Skip to content

ci: split the test archive by crate - #586

Merged
MarcusKainth merged 1 commit into
mainfrom
ci/split-test-archives
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
ci/split-test-archives

Conversation

@MarcusKainth

@MarcusKainth MarcusKainth commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #584 (ci/archive-no-debuginfo); the diff against that branch is this one commit.

What this changes, and why

Every test job downloaded one archive holding every crate's test binaries, and build-tests built that archive and the ROM suites archive one after the other. A simulation group needs only the native crate's suites.

  • build-tests is a matrix of two archives built side by side: native.tar.zst (the native crate's suites) and workspace.tar.zst (every other crate's). Each leg has its own rust-cache key.
  • The ROM suites archive moves to build-refemu, which already builds the reference emulator in release and isn't on the path to the test jobs.
  • Each group downloads only what scripts/test-group.sh runs it from: the simulation groups get the native archive, emulator gets the workspace and ROM suites archives, and native-rest gets both. It runs the native crate's other suites and the driver's native_* suites as two nextest invocations, one per archive, still at two threads each. native_connections_live stays in emulator at --test-threads 1.
  • A group that lacks an archive it needs fails on the missing file.

ci-passed still needs build-tests, which now reports the result of both legs together.

Evidence

Each group selects the same tests

Every group's filter over the whole workspace before the split, against the split filters over the two archives after (nextest list --message-format oneline, sorted, compared with cmp), with local archives built the way CI builds them:

emulator       old  457 new  457 same
native-sim-a   old   10 new   10 same
native-sim-b   old   12 new   12 same
native-sim-c   old   12 new   12 same
native-sim-d   old   12 new   12 same
native-sim-e   old   28 new   28 same
native-sim-f   old   21 new   21 same
native-rest    old  401 new  401 same
every test in the workspace:      953; in some group after the split:      953; duplicates:        0
exit=0

This PR's CI run 36124399424

Each build leg has a new rust-cache key, so this run compiled the dependencies too. The warm figures for the same split come from the measurement run 36122853256, with the native crate at opt-level 0 (#585): 48 s for the native archive and 94 s for the rest.

#584 alone (run 36122982586, attempt 2, warm) this PR (cold cache)
build-tests 1 job, 342 s: 316 s archive, 6 s upload 2 jobs side by side: native 229 s (199 s archive), workspace 275 s (248 s archive)
ROM suites archive in build-tests build-refemu, 36 s. The job takes 67 s and gates only emulator
artifacts test-archives 191,650,427 B native-tests 101,225,013 B, workspace-tests 70,556,324 B, rom-suites 5,064,897 B
download per group 3 to 7 s, both archives 3 to 6 s, only the group's own archives

The nextest summaries show every group running what it ran before:

test (emulator)       Summary [ 196.129s] 457 tests run: 457 passed, 0 skipped
                      Summary [  29.100s] 10 tests run: 10 passed, 0 skipped
test (native-rest)    Summary [  69.286s] 382 tests run: 382 passed, 0 skipped
                      Summary [ 143.356s] 19 tests run: 19 passed, 0 skipped
test (native-sim-a)   Summary [  75.511s] 10 tests run: 10 passed, 0 skipped
test (native-sim-b)   Summary [ 160.567s] 12 tests run: 12 passed, 0 skipped
test (native-sim-c)   Summary [ 107.962s] 12 tests run: 12 passed, 0 skipped
test (native-sim-d)   Summary [ 145.380s] 12 tests run: 12 passed, 0 skipped
test (native-sim-e)   Summary [ 163.015s] 28 tests run: 28 passed, 0 skipped
test (native-sim-f)   Summary [ 224.610s] 21 tests run: 21 passed, 0 skipped

The whole run took 9 min 15 s from the first job's start (10:31:02) to ci-passed (10:40:17). native-rest ran in 220 s as two invocations, against 356 s in #584's attempt 2 and 276 s in #584's attempt 1.

$ make actionlint zizmor shellcheck; echo "exit=$?"
No findings to report. Good job! (9 suppressed)
exit=0

Invariants

None. CI layout only; the same tests run, from different archives.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates: not run locally. CI's lint job ran make lint and every test group passed
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added the area: ci Workflows, the Makefile, and the scripts they run label Sep 25, 2026
Base automatically changed from ci/archive-no-debuginfo to main September 25, 2026 11:21
build-tests made one archive holding every crate's test binaries and one
holding the reference emulator's release ROM suites, one after the
other, and every test job downloaded both. A simulation group needs only
the native crate's suites.

build-tests is now a matrix of two archives built side by side:
native.tar.zst with the native crate's suites and workspace.tar.zst with
every other crate's. The ROM suites archive moves to build-refemu, which
already builds the reference emulator in release and is off the path to
the test jobs. Each group downloads only what scripts/test-group.sh runs
it from: the simulation groups the native archive, the emulator group
the workspace and ROM suites archives, and native-rest both, since it
runs the native crate's other suites and the driver's native_* suites.
native-rest runs them as two nextest invocations, one per archive, at the
same two threads.

Measured on the throwaway run 36122853256 with a warm cache, no debug
info and the native crate's suites at opt-level 0, the native archive
built in 48 s (119 MB) and the other crates' in 94 s (72 MB), against
2 m 16 s (206 MB) for the single archive under the same settings.

Each group selects the same tests as before: listing every group's
filter over the workspace before the split and over the two archives
after gives the same test lists for all eight groups, 953 tests in all,
none in two groups.
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 11:31
@MarcusKainth
MarcusKainth merged commit 885e786 into main Sep 25, 2026
21 checks passed
@MarcusKainth
MarcusKainth deleted the ci/split-test-archives branch September 25, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant