Skip to content

ci: check that every test runs in exactly one group - #587

Merged
MarcusKainth merged 1 commit into
mainfrom
ci/test-group-coverage
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
ci/test-group-coverage

Conversation

@MarcusKainth

@MarcusKainth MarcusKainth commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #586 (ci/split-test-archives); the diff against that branch is this one commit.

What this changes, and why

If a group's filter stops matching some tests, those tests run nowhere and CI stays green. group_coverage.rs caught this for the simulation suites only, by reading the script's lettered filters. Nothing checked any other package, or how the driver's native_* suites split between native-rest and emulator, which the archive split makes easier to get wrong.

  • scripts/test-group.sh --list <group> prints the tests a group selects, through cargo nextest list with the same archives and filters the run uses.
  • scripts/test-group.sh --check lists every test in the archives (or in the workspace, without NEXTEST_ARCHIVE_DIR) and every group. It fails when a test is selected by no group or by two, or when a group selects nothing.
  • A new test-groups job runs --check against the three archives. It lists tests and runs none, so it needs no ClickHouse. ci-passed needs it.
  • group_coverage.rs also checks that ci.yml's test matrix, the script's case arms and its new groups=(...) line name the same groups. It reads group names of any length, where it used to read one letter.

One finding: four refemu tests (three in probe_fixture, one in reference_trace) run twice inside emulator, once from the debug workspace archive and once from the release ROM suites archive. --check counts a test once per group, so this passes. Changing which of the two runs them is a separate decision, and this PR leaves it alone.

Evidence

On local archives built the way CI builds them:

$ NEXTEST_ARCHIVE_DIR=target/nextest scripts/test-group.sh --check; echo "exit=$?"
emulator:      463 tests
native-sim-a:       10 tests
native-sim-b:       12 tests
native-sim-c:       12 tests
native-sim-d:       12 tests
native-sim-e:       28 tests
native-sim-f:       21 tests
native-rest:      401 tests
every test:      959; selected:      959
exit=0

Without archives (CARGO_PROFILE_TEST_DEBUG=0 scripts/test-group.sh --check) it exits 0 with 960 of 960. The extra test is the new group_coverage test, which the local archives were built without.

Each guard shown failing on a mutation, restored with git checkout HEAD -- <file> after each one:

native-rest's driver filter stops matching native_stream_live:

check exit=1
native-rest:      397 tests
every test:      959; selected:      955
clickdoom-driver::native_stream_live a_statement_that_fails_on_a_row_stops_writing_and_says_why
clickdoom-driver::native_stream_live a_statement_the_server_cannot_parse_reports_its_message_on_close
clickdoom-driver::native_stream_live a_statement_too_large_for_a_url_parameter_opens
clickdoom-driver::native_stream_live one_statement_takes_a_hundred_tics_and_chains_them
::error::the tests above are selected by no group

sim_aim_live added to sim_a as well as sim_f:

check exit=1
clickdoom-native::sim_aim_live support::seed::tests::a_column_overridden_twice_panics
clickdoom-native::sim_aim_live the_aim_finds_what_the_engine_finds
::error::the tests above are selected by more than one group
group_coverage exit=101
named by more than one group: ["sim_aim_live in [\"a\", \"f\"]"]

native-sim-b's filter narrowed with and test(=no_such_test):

check exit=1
native-sim-b:        0 tests
::error::native-sim-b selects no tests
::error::the tests above are selected by no group

A filter that names a binary that doesn't exist (and binary(no_such_suite)) stops earlier: nextest refuses the filterset (operator didn't match any binary names) and --check exits 94.

native-rest dropped from ci.yml's matrix:

group_coverage exit=101
assertion `left == right` failed: ci.yml's test matrix and the script's groups line name different groups
  left: ["emulator", "native-sim-a", "native-sim-b", "native-sim-c", "native-sim-d", "native-sim-e", "native-sim-f"]
 right: ["emulator", "native-rest", "native-sim-a", "native-sim-b", "native-sim-c", "native-sim-d", "native-sim-e", "native-sim-f"]
$ make actionlint zizmor shellcheck; echo "exit=$?"
exit=0
$ cargo test --locked -p clickdoom-native --test group_coverage
test result: ok. 2 passed; 0 failed

The test-groups job on this PR's CI run 36124924215 took 26 s: 4 s to download the three archives and 14 s for the check.

Found 5 artifact(s)
- workspace-tests (ID: 10859878237, Size: 70560360, ...)
- native-tests (ID: 10859621990, Size: 101214217, ...)
- rom-suites (ID: 10859607711, Size: 5064907, ...)
emulator: 463 tests
native-sim-a: 10 tests
native-sim-b: 12 tests
native-sim-c: 12 tests
native-sim-d: 12 tests
native-sim-e: 28 tests
native-sim-f: 21 tests
native-rest: 402 tests
every test: 960; selected: 960

Every group passed and ci-passed, which needs test-groups, succeeded.

Invariants

None. CI and the test-group script only.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates: not run locally. CI's lint job ran make lint and every test group passed
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader labels Sep 25, 2026
Base automatically changed from ci/split-test-archives to main September 25, 2026 11:31
A group's filter that stops matching some tests leaves them running
nowhere, and CI stays green. group_coverage.rs caught that for the
simulation suites only, by reading the script's lettered filters. Every
other package's tests, and the split of the driver's native_* suites
between native-rest and emulator, had no check.

scripts/test-group.sh gains two modes. `--list <group>` prints the tests
the group selects, through `cargo nextest list` with the same archives
and filters the run uses. `--check` lists every test in the archives
(or the workspace, without NEXTEST_ARCHIVE_DIR) and every group, and
fails when a test is selected by no group or by two, or when a group
selects nothing. A test a group reaches from two archives counts once
for that group: four refemu tests in probe_fixture and reference_trace
run in both the debug workspace archive and the release ROM suites
archive, both in emulator. The new test-groups job runs --check against
the three archives, and ci-passed needs it.

group_coverage.rs also checks that ci.yml's test matrix, the script's
case arms and its new `groups=(...)` line name the same groups, and it
reads group names of any length rather than one letter.
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 11:54
@MarcusKainth
MarcusKainth merged commit ea56c53 into main Sep 25, 2026
22 checks passed
@MarcusKainth
MarcusKainth deleted the ci/test-group-coverage branch September 25, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant