perf: five simulation suites are one twenty-minute test each

[MarcusKainth]

What kind

Cost grows where the model says it should not

Which benchmark

scripts/test-group.sh (the CI simulation groups)

The numbers

Per-test wall time on CI, median of five main runs on the seven-group
matrix, parsed from the job logs. Runs: 99135ec, 80b75eb, 89e73f9,
bb31393, c266c90, all ubuntu-latest at TEST_THREADS=4.

suite                     test                                        median    min    max
sim_missile_live          the_missile_check_draws_only_where_the_...   20.6m  12.0m  22.4m
sim_fall_live             a_thing_falls_and_clips_the_way_the_eng...   19.8m  11.6m  22.3m
sim_hearing_live          a_look_wakes_on_what_its_sector_heard        18.4m  13.4m  20.9m
sim_punch_live            a_punch_lands_only_where_the_engine_lan...   17.6m  10.1m  18.0m
sim_player_damage_live    a_claw_reaches_the_player_through_its_o...   16.5m  11.6m  18.0m

Each is one test. Its suite is that test plus a trivial one, except
sim_player_damage_live, which carries six.

All simulation work is 443.9 test-minutes. Over six groups at four threads
a perfect pack would be 18.5 minutes, but the floor is 20.6, set by
sim_missile_live's single test, because a test cannot be split across
threads.

The machine, and how quiet it was

GitHub's ubuntu-latest runners, one job per group, nothing else on the
runner. Per-test time across the five runs has a median coefficient of
variation of 17% and a maximum of 35%, which is the noise any measurement
here carries.

What you think is happening

A test that opens a session pays the tic statement's analysis once, about
three minutes on a standard runner, and then runs its tics. These five run
long because each one drives many tics through a single session rather than
because any one tic is slow.

The consequence for CI is structural rather than a matter of packing.
#547 packs the suites over six groups and every group lands between 19.8
and 20.6 minutes, which is where the longest single test puts the floor. No
assignment of suites to groups can go below it, and adding a seventh group
would not either: the total is already below the floor at six.

What would move it is splitting these tests so no single one is 20 minutes.
Two shapes worth weighing, neither obviously right:

• Split by tic range, so a suite's tics run as several tests against the
  same seeded state. Each new test pays the analysis again, about three
  minutes, so splitting one 20-minute test in two gives roughly 11.5 and
  11.5 rather than 10 and 10, and the floor falls to about 12 minutes.
• Split by case rather than by tic, where a test covers several independent
  seeded arms that could each be their own test. This costs the same
  analysis per test and is only worth it where the arms are genuinely
  independent.

Either way the gain is bounded by the analysis cost, so the question is how
many pieces are worth paying it for. Worth measuring before anyone commits
to a shape: what the analysis actually costs on a runner today, per
session, against what each of these tests spends on its tics.

This is not a regression. The tests have been this long since they were
written; the cost only became visible when the groups were packed on
measured times rather than on counts.

[MarcusKainth]

Measured: the analysis is the cost, not the tics

Taken on a quiet machine (lock held, no other container) against
clickhouse/clickhouse-server:26.8.2.7 on main at 38a7d59, one suite at a
time, read off system.query_log's own ProfileEvents rather than the wall
clock. QueryAnalysisMicroseconds is the analysis; UserTimeMicroseconds
minus it is execution. Duration is not used: on a server with a streamed
statement most of it is waiting.

suite                       stage1   analysis   per     execution
                           queries        (s)  analysis       (s)
sim_hearing                      5      201.8    40.4       15.3
sim_player_damage_killing        5      200.5    40.1       14.3
sim_fall                         4      174.9    43.7       13.4
sim_missile                      4      170.6    42.6       19.4
sim_player_damage_living         4      161.5    40.4       12.1
sim_punch                        4      156.0    39.0        8.6

Stage2 is negligible beside it: 8 to 10 queries for 3.6 to 4.6 s of
analysis in every suite, about 0.45 s each.

Three things follow.

A stage1 analysis costs 40.9 s and does not vary with what the statement
runs. Six suites, 39.0 to 43.7 s, spread under 12%. It is the same
statement every time.

A tic costs about 0.13 s. sim_missile_live executes 127 tics in 19.4 s.
An analysis is roughly 300 tics' worth of work.

A suite's cost is its statement count. The number of stage1 analyses is
1 + sum over arms of that arm's tics, exactly:

sim_missile               1 walk + 3 arms x 1 tic  = 4
sim_fall                  1 walk + 3 arms x 1 tic  = 4
sim_hearing               1 walk + 4 arms x 1 tic  = 5
sim_player_damage_living  1 walk + 3 arms x 1 tic  = 4
sim_player_damage_killing 1 walk + 2 arms x 2 tics = 5
sim_punch                 no walk, 4 statements    = 4

So wall time is about 40.9 x statements + 0.13 x tics, and the second term
never reaches a minute.

What this rules out

Splitting these tests makes them slower. Each new test repeats the walk
and pays another analysis for it. Splitting sim_missile_live's three arms
into three tests gives 3 walks and 6 analyses where there are 1 and 4.

Deleting the walk is worth seconds, not minutes. It was the fix I
expected from reading the sources. sim_missile_live's 127-tic walk is
about 16 s of the 19.4 s of execution, against 170.6 s of analysis. Worth
doing where it is free, not as the lever.

What the numbers point at

One statement pair driving every pre-seeded arm, instead of one pair per
arm per tic.

run_statement(db, rows) already takes a slice and drives it through
numbers() so each row is its own block, and a row reads the tic before it
through joinGet. The arms are seeded before any statement runs and sit at
tics far enough apart not to read each other, so one pair can carry all of
them. Within an arm, consecutive tics are already in dependency order.

Each suite then pays 2 analyses, the walk and the arms, or 1 where the walk
can be replaced by a seed from the probe's own row at that tic:

suite                       now   one pair   and probe-seeded
sim_missile                   4          2                  1
sim_fall                      4          2                  1
sim_hearing                   5          2                  1
sim_player_damage_living      4          2                  1
sim_player_damage_killing     5          2                  1
sim_punch                     4          1                  1

At 40.9 s an analysis and the CI runners about 8x slower than this machine,
a 1-analysis suite is roughly the 340 s sim_thinker_order_live costs
today, which is the one suite already built this way: one tic, seeded from
a committed probe row, no walk.

sim_player_damage_live is the largest single win. Its claw_arms runs
run_statement per arm per tic inside a nested loop, and the suite calls it
twice across six tests.

The constraint

What each arm asserts has to stay byte-identical. Nothing above changes a
seed, an override or an assertion: the arms are already independent and
already seeded ahead of the run. Only the number of statements carrying them
changes. Any suite whose walk is itself the assertion keeps its walk;
sim_parity_live is that case and is not in this set.

Not yet measured

sim_player_damage_live's other four tests were still running when this was
taken, so its two rows here are the two claw_arms calls that had finished.
The per-analysis cost is steady enough across six independent readings that
I would expect them to add statements at the same 40.9 s, but that is an
expectation rather than a measurement.