Repository navigation
ci: check that every test runs in exactly one group - #587
Merged
Merged
Conversation
2 of 3 tasks
MarcusKainth
force-pushed
the
ci/split-test-archives
branch
from
September 25, 2026 11:21
0ac7839 to
712e4eb
Compare
A group's filter that stops matching some tests leaves them running nowhere, and CI stays green. group_coverage.rs caught that for the simulation suites only, by reading the script's lettered filters. Every other package's tests, and the split of the driver's native_* suites between native-rest and emulator, had no check. scripts/test-group.sh gains two modes. `--list <group>` prints the tests the group selects, through `cargo nextest list` with the same archives and filters the run uses. `--check` lists every test in the archives (or the workspace, without NEXTEST_ARCHIVE_DIR) and every group, and fails when a test is selected by no group or by two, or when a group selects nothing. A test a group reaches from two archives counts once for that group: four refemu tests in probe_fixture and reference_trace run in both the debug workspace archive and the release ROM suites archive, both in emulator. The new test-groups job runs --check against the three archives, and ci-passed needs it. group_coverage.rs also checks that ci.yml's test matrix, the script's case arms and its new `groups=(...)` line name the same groups, and it reads group names of any length rather than one letter.
MarcusKainth
force-pushed
the
ci/test-group-coverage
branch
from
September 25, 2026 11:45
fed239b to
bd4c759
Compare
MarcusKainth
marked this pull request as ready for review
September 25, 2026 11:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #586 (
ci/split-test-archives); the diff against that branch is this one commit.What this changes, and why
If a group's filter stops matching some tests, those tests run nowhere and CI stays green.
group_coverage.rscaught this for the simulation suites only, by reading the script's lettered filters. Nothing checked any other package, or how the driver'snative_*suites split betweennative-restandemulator, which the archive split makes easier to get wrong.scripts/test-group.sh --list <group>prints the tests a group selects, throughcargo nextest listwith the same archives and filters the run uses.scripts/test-group.sh --checklists every test in the archives (or in the workspace, withoutNEXTEST_ARCHIVE_DIR) and every group. It fails when a test is selected by no group or by two, or when a group selects nothing.test-groupsjob runs--checkagainst the three archives. It lists tests and runs none, so it needs no ClickHouse.ci-passedneeds it.group_coverage.rsalso checks that ci.yml's test matrix, the script's case arms and its newgroups=(...)line name the same groups. It reads group names of any length, where it used to read one letter.One finding: four refemu tests (three in
probe_fixture, one inreference_trace) run twice insideemulator, once from the debug workspace archive and once from the release ROM suites archive.--checkcounts a test once per group, so this passes. Changing which of the two runs them is a separate decision, and this PR leaves it alone.Evidence
On local archives built the way CI builds them:
Without archives (
CARGO_PROFILE_TEST_DEBUG=0 scripts/test-group.sh --check) it exits 0 with 960 of 960. The extra test is the newgroup_coveragetest, which the local archives were built without.Each guard shown failing on a mutation, restored with
git checkout HEAD -- <file>after each one:native-rest's driver filter stops matching
native_stream_live:sim_aim_liveadded tosim_aas well assim_f:native-sim-b's filter narrowed with
and test(=no_such_test):A filter that names a binary that doesn't exist (
and binary(no_such_suite)) stops earlier: nextest refuses the filterset (operator didn't match any binary names) and--checkexits 94.native-restdropped from ci.yml's matrix:The
test-groupsjob on this PR's CI run 36124924215 took 26 s: 4 s to download the three archives and 14 s for the check.Every group passed and
ci-passed, which needstest-groups, succeeded.Invariants
None. CI and the test-group script only.
Spec impact
Checks
make gates: not run locally. CI's lint job ranmake lintand every test group passedWritten mostly by Claude Opus 5.5.