Skip to content

ci: run the driver's native suites as their own group - #591

Merged
MarcusKainth merged 2 commits into
mainfrom
ci/driver-native-group
Sep 25, 2026
Merged

MarcusKainth merged 2 commits into
mainfrom
ci/driver-native-group

Conversation

@MarcusKainth

@MarcusKainth MarcusKainth commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

What this changes, and why

native-rest is the longest test job. Over the six main runs 36126756550 to 36132815983, its suites step took 275 to 391 s, while the slowest simulation group took 188 to 261 s. It ran two nextest invocations one after the other, each at --test-threads 2: the native crate's non-simulation suites, then the driver's native_* suites.

The two threads were a packing choice, made when native-rest absorbed driver-native (b01ae55). Two cases matter for moving to more threads:

  • native_connections_live counts the server's connections, so it can't run next to another test. It already runs in emulator at one thread.
  • The other suites read only their own database (session_live's system.query_log count filters on its database) or their own query id (native_session_live's KILL QUERY WHERE query_id = ...).

This PR makes two changes:

  • The driver's native_* suites become the driver-native group. It runs from the workspace archive at TEST_THREADS, like a simulation group.
  • The native crate's non-simulation suites join native-sim-e, which already downloads the native archive. Its filter becomes package(clickdoom-native) and not (<lettered groups>), so it catches every native suite the other groups don't name.

native_stream_live goes to emulator, not driver-native (second commit, 6a930eb). Its pacing test asserts a median send-to-visible latency under 5 ms (P50_LIMIT). On this PR's first CI run (36137795545) it ran beside three resident sessions in driver-native and measured p50 5.61 ms (p99 21.78 ms, max 29.24 ms), then failed. The limit stays. emulator runs one test at a time, and the connection suite runs there for the same reason.

No other suite in driver-native or the simulation groups asserts a latency or a duration:

  • native_session_live measures frame waits and prints them without asserting on them.
  • sim_cost_live asserts only that a statement takes at least as long as its own analysis.
  • The remaining time limits are harness timeouts of 10 s or more (FRAME_TIMEOUT, VISIBLE_TIMEOUT, FIRST_ROW_TIMEOUT, TIC_TIMEOUT).

The test job count stays the same, and ci-passed's needs doesn't change. group_coverage.rs finds the catch-all by its new filter text.

Evidence

Per-test times come from the nextest PASS lines of the six main runs. Each group is modelled as longest-test-first scheduling over four slots:

native-rest (now), by test, median / worst over 6 runs:
   150.8 max 161.7 native_demo_live a_sim_run_draws_frames_from_the_tics_it_commits
   109.0 max 118.7 native_diff_live a_differential_run_reports_the_first_field_that_differs
    93.2 max 100.2 render_live a_rendered_frame_draws_what_the_engine_drew
    92.3 max  95.3 native_diff_live a_tic_that_refuses_stops_before_the_field_comparison
    73.5 max  77.7 session_live arms_in_one_session_leave_the_rows_each_leaves_alone
    29.9 max  31.1 native_play_live a_key_the_driver_streams_reaches_the_tic_command

model at 4 threads        median   worst
native-sim-e now            171 s   176 s
native-sim-e + native       213 s   230 s
driver-native               151 s   162 s

The same model against each group's observed suites step in those runs (6 runs x 8 groups) gives observed/model between 0.96 and 1.17, mostly 1.05 to 1.10. nextest's own startup accounts for the steady part of that gap.

$ CARGO_PROFILE_TEST_DEBUG=0 scripts/test-group.sh --check; echo "check exit=$?"
emulator:      467 tests
native-sim-a:       10 tests
native-sim-b:       12 tests
native-sim-c:       12 tests
native-sim-d:       12 tests
native-sim-e:      411 tests
native-sim-f:       21 tests
driver-native:       15 tests
every test:      960; selected:      960
check exit=0
$ cargo test --locked -p clickdoom-native --test group_coverage
test result: ok. 2 passed; 0 failed
$ shellcheck scripts/test-group.sh; make actionlint zizmor; cargo fmt --check
all exit 0

native-sim-e's 411 tests are its 28 earlier ones plus the 383 native tests native-rest ran. Of the 19 driver tests native-rest ran, 15 are in driver-native and native_stream_live's 4 are in emulator.

What this PR's CI run should show

  • test (driver-native): the first run's suites step took 217.5 s with the stream suite in it, which is slower than the model's 151 to 162 s at four sessions at once. Expect about the same without that suite (about 3 s of tests), well under native-rest's 275 to 391 s.
  • test (emulator): 3 to 6 s longer than before, with native_stream_live passing alone.
  • test (native-sim-e): about 230 to 250 s, up from 136 to 188 s.
  • The other groups are unchanged: emulator 210 to 246 s, native-sim-f 188 to 261 s.
  • test-groups: 960 of 960 selected, emulator 467 tests, driver-native 15 tests.

Invariants

None. Test grouping only.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates: not run. The checks above ran locally
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

native-rest was the longest test job: its suites took 275 to 391 s over
the six main runs 36126756550 to 36132815983, against 188 to 261 s for
the slowest simulation group. It ran the native crate's non-simulation
suites and the driver's native_* suites as two nextest invocations one
after the other, each at two threads. The two threads were a packing
choice (b01ae55). The one suite that can't share the server,
native_connections_live, already runs in emulator at one thread, and the
other native-rest suites read only their own database and query ids.

The driver's native_* suites become the driver-native group, run from
the workspace archive at TEST_THREADS like a simulation group. The
native crate's non-simulation suites join native-sim-e, whose filter now
takes every native crate suite the lettered groups don't name. They run
from the native archive, which that group already downloads.

Modelled from each test's worst time over those six runs, longest test
first over four slots: driver-native 162 s (longest test
native_demo_live's sim run, 162 s), native-sim-e 230 s. The same model
reproduces the observed suites step of every group in those runs within
4 to 17%. The test job count stays the same.
@github-actions github-actions Bot added area: ci Workflows, the Makefile, and the scripts they run area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader labels Sep 25, 2026
native_stream_live's pacing test asserts that the median send-to-visible
latency over 100 tics stays under 5 ms (P50_LIMIT). In driver-native it
ran beside three resident sessions on a four-core runner, and on run
36137795545 it measured a p50 of 5.61 ms (p99 21.78 ms) and failed. It
passed in native-rest, where it had at most one neighbour.

The limit stays. The suite moves to emulator, which runs one test at a
time for the same reason the connection suite runs there: its check is
about the server and the runner, so a neighbour changes what it
measures.

No other suite in driver-native or the simulation groups asserts a
latency or a duration. native_session_live measures frame waits and
prints them without asserting on them, sim_cost_live asserts only that a
statement takes at least as long as its own analysis, and the
remaining time limits are harness timeouts of 10 s or more.
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 13:11
@MarcusKainth
MarcusKainth merged commit 27fb257 into main Sep 25, 2026
22 checks passed
@MarcusKainth
MarcusKainth deleted the ci/driver-native-group branch September 25, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant