ci: pack the native suites into four groups - #592
Merged
Merged
Conversation
MarcusKainth
force-pushed
the
ci/repack-native-groups
branch
from
September 25, 2026 13:04
2b2cfd5 to
51489b8
Compare
The simulation groups were packed when a single test took 14 to 21 minutes. Running every arm of a suite in one session and cutting the tic statement's analysis brought the slowest test down to 227 s, and six simulation groups now finish their suites in 78 to 261 s each on the six main runs 36126756550 to 36132815983, while emulator takes 210 to 246 s. Each group is a runner, and the runners queue against the account's concurrent-job limit. The native crate's suites are repacked into four groups from each test's worst time across those six runs. Each group is modelled as longest test first over four slots, and suites are placed and swapped until the worst group stops falling. The crate's non-simulation suites stay in the catch-all group, native-sim-d, because the lettered lists name only simulation suites. Worst-time model per group: a 253 s, b 251 s, c 252 s, d 253 s (medians: 223, 219, 224, 232 s). The same model reproduced the observed suites step of every group in those runs within 4 to 17%. Five groups would model at 227 s, the length of sim_plat_live's longest test, for one more runner. The timeouts are re-derived from the same runs: - nextest's cutoff drops from 2,400 s to 720 s (120 s x 6). The slowest test's worst was 227 s, and the cutoff sits above FIRST_TIC_TIMEOUT's 600 s plus a load, so a first tic that never comes fails with the session's own message. - The test jobs' timeout-minutes drops from 45 to 15: about twice the slowest test job seen (7.1 minutes), and above nextest's 12-minute cutoff.
MarcusKainth
force-pushed
the
ci/repack-native-groups
branch
from
September 25, 2026 13:11
51489b8 to
fa7b23a
Compare
MarcusKainth
marked this pull request as ready for review
September 25, 2026 13:25
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #591 (
ci/driver-native-group); the diff against that branch is this one commit.What this changes, and why
The six simulation groups were sized when a single test took 14 to 21 minutes. With every arm of a suite run in one session and the analysis cut, the slowest test now takes 227 s at its worst. The six groups finish their suites in 78 to 261 s each over the main runs 36126756550 to 36132815983, and
emulatortakes 210 to 246 s. Each group is a runner, and the runners queue against the account's concurrent-job limit.The native crate's suites are repacked into four groups:
native-sim-d, becausegroup_coverage.rsonly accepts simulation suites in the lettered lists.The timeouts are re-derived from the same runs:
period = "120s",terminate-after = 6). The slowest test's worst is 227 s. The cutoff sits aboveFIRST_TIC_TIMEOUT(600 s) plus a load, so a first tic that never comes fails with the session's own message.testandtest-sim: 45 drops to 15. That is about twice the slowest test job seen (7.1 min,native-restin 36131955884), and it stays above nextest's 12-minute cutoff.The test jobs go from eight to six:
emulator,driver-nativeand four simulation groups.ci-passedneeds no change because the job names stay the same.Evidence
The same model against each group's observed suites step in the six runs (48 group-runs) gives observed/model between 0.96 and 1.17, mostly 1.05 to 1.10.
What this PR's CI run should show
native-sim-atonative-sim-deach run their suites step in about 235 to 280 s, which is the model times 1.05 to 1.10 plus spread. Before, the six groups took 78 to 261 s.driver-native: about 215 s, as ci: run the driver's native suites as their own group #591's first run measured.emulator: 210 to 250 s, unchanged.emulator, with two fewer runners.test-groups: 960 of 960, with the counts above.If the simulation groups come in past about 280 s, five groups (modelled at 227 s) is the next step.
Invariants
None. Test grouping and timeouts only.
Spec impact
Checks
make gates: not run. The checks above ran locallyWritten mostly by Claude Opus 5.5.