Skip to content

ci: pack the native suites into four groups - #592

Merged
MarcusKainth merged 1 commit into
mainfrom
ci/repack-native-groups
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
ci/repack-native-groups

Conversation

@MarcusKainth

@MarcusKainth MarcusKainth commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #591 (ci/driver-native-group); the diff against that branch is this one commit.

What this changes, and why

The six simulation groups were sized when a single test took 14 to 21 minutes. With every arm of a suite run in one session and the analysis cut, the slowest test now takes 227 s at its worst. The six groups finish their suites in 78 to 261 s each over the main runs 36126756550 to 36132815983, and emulator takes 210 to 246 s. Each group is a runner, and the runners queue against the account's concurrent-job limit.

The native crate's suites are repacked into four groups:

  • The packing uses each test's worst time across those six runs.
  • Each group is modelled as longest test first over four slots.
  • Suites are placed greedily, then moved and swapped until the worst group stops falling.
  • The crate's non-simulation suites stay in the catch-all group, native-sim-d, because group_coverage.rs only accepts simulation suites in the lettered lists.

The timeouts are re-derived from the same runs:

  • nextest cutoff: 2,400 s drops to 720 s (period = "120s", terminate-after = 6). The slowest test's worst is 227 s. The cutoff sits above FIRST_TIC_TIMEOUT (600 s) plus a load, so a first tic that never comes fails with the session's own message.
  • timeout-minutes for test and test-sim: 45 drops to 15. That is about twice the slowest test job seen (7.1 min, native-rest in 36131955884), and it stays above nextest's 12-minute cutoff.

The test jobs go from eight to six: emulator, driver-native and four simulation groups. ci-passed needs no change because the job names stay the same.

Evidence

model at 4 threads    tests   median   worst   sum of worst
native-sim-a            28     223 s   253 s     1,011 s
native-sim-b            26     219 s   251 s     1,000 s
native-sim-c            25     224 s   252 s     1,000 s
native-sim-d (+native) 399     232 s   253 s     1,013 s

alternatives, worst-time model: 3 groups 337 s, 4 groups 253 s, 5 groups 227 s (bound by sim_plat_live's longest test)

The same model against each group's observed suites step in the six runs (48 group-runs) gives observed/model between 0.96 and 1.17, mostly 1.05 to 1.10.

$ CARGO_PROFILE_TEST_DEBUG=0 scripts/test-group.sh --check; echo "check exit=$?"
emulator:      467 tests
native-sim-a:       28 tests
native-sim-b:       26 tests
native-sim-c:       25 tests
native-sim-d:      399 tests
driver-native:       15 tests
every test:      960; selected:      960
check exit=0
$ cargo test --locked -p clickdoom-native --test group_coverage
test result: ok. 2 passed; 0 failed
$ shellcheck scripts/test-group.sh; make actionlint zizmor; cargo nextest list -p clickdoom-native --test group_coverage
all exit 0 (the last parses the edited .config/nextest.toml)

What this PR's CI run should show

  • Simulation groups: native-sim-a to native-sim-d each run their suites step in about 235 to 280 s, which is the model times 1.05 to 1.10 plus spread. Before, the six groups took 78 to 261 s.
  • driver-native: about 215 s, as ci: run the driver's native suites as their own group #591's first run measured.
  • emulator: 210 to 250 s, unchanged.
  • Critical path: the slowest simulation group, about 20 to 30 s longer than emulator, with two fewer runners.
  • test-groups: 960 of 960, with the counts above.

If the simulation groups come in past about 280 s, five groups (modelled at 227 s) is the next step.

Invariants

None. Test grouping and timeouts only.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates: not run. The checks above ran locally
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added the area: ci Workflows, the Makefile, and the scripts they run label Sep 25, 2026
@MarcusKainth
MarcusKainth force-pushed the ci/repack-native-groups branch from 2b2cfd5 to 51489b8 Compare September 25, 2026 13:04
Base automatically changed from ci/driver-native-group to main September 25, 2026 13:11
The simulation groups were packed when a single test took 14 to 21
minutes. Running every arm of a suite in one session and cutting the tic
statement's analysis brought the slowest test down to 227 s, and six
simulation groups now finish their suites in 78 to 261 s each on the six
main runs 36126756550 to 36132815983, while emulator takes 210 to 246 s.
Each group is a runner, and the runners queue against the account's
concurrent-job limit.

The native crate's suites are repacked into four groups from each test's
worst time across those six runs. Each group is modelled as longest test
first over four slots, and suites are placed and swapped until the worst
group stops falling. The crate's non-simulation suites stay in the
catch-all group, native-sim-d, because the lettered lists name only
simulation suites. Worst-time model per group: a 253 s, b 251 s,
c 252 s, d 253 s (medians: 223, 219, 224, 232 s). The same model
reproduced the observed suites step of every group in those runs within
4 to 17%. Five groups would model at 227 s, the length of sim_plat_live's
longest test, for one more runner.

The timeouts are re-derived from the same runs:
- nextest's cutoff drops from 2,400 s to 720 s (120 s x 6). The slowest
  test's worst was 227 s, and the cutoff sits above FIRST_TIC_TIMEOUT's
  600 s plus a load, so a first tic that never comes fails with the
  session's own message.
- The test jobs' timeout-minutes drops from 45 to 15: about twice the
  slowest test job seen (7.1 minutes), and above nextest's 12-minute
  cutoff.
@MarcusKainth
MarcusKainth force-pushed the ci/repack-native-groups branch from 51489b8 to fa7b23a Compare September 25, 2026 13:11
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 13:25
@MarcusKainth
MarcusKainth merged commit 0aed0a9 into main Sep 25, 2026
20 checks passed
@MarcusKainth
MarcusKainth deleted the ci/repack-native-groups branch September 25, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant