ci: measure the combined CI changes (throwaway) - #590
Closed
MarcusKainth wants to merge 6 commits into
Closed
MarcusKainth wants to merge 6 commits into
MarcusKainth wants to merge 6 commits into
Conversation
The workspace archive was 2.63 GB (2,631,156,768 bytes), almost all of it DWARF: a driver test binary is about 465 MB with full debug info and about 9 MB without. Uploading it took 78 to 83 s and a test job's download took 33 to 363 s. With CARGO_PROFILE_TEST_DEBUG=0 the same archive is 187 MB. Line tables only (line-tables-only) would keep 860 MB. A panic's message and its file and line come from the panic location, which does not depend on debug info, so a failing test reports the same thing it reports now. The variable is set on the job so rust-cache includes it in its key; the dependencies it restores were built under the same setting.
The test profile's opt-level 2 is there for the emulator's fixture and trace suites, which run millions of instructions in Rust. The native crate builds SQL text and its suites spend their time waiting on the server, so it gets opt-level 0 through a package override. Every other crate keeps opt-level 2, the reference emulator included. On a standard runner the native crate's 57 test binaries took 876 CPU seconds to compile at opt-level 2 and 200 at opt-level 0 (cargo --timings, CI run 36121287631). Its non-live suites ran in 0.87 s at opt-level 2 and 1.98 s at opt-level 0 locally. opt-level does not change what safe Rust computes. The generated text is byte-identical anyway: xxh64 of both resident statements, run_statement, demo_statement, the load, level and render plans and the parity queries match at opt-level 2, 1 and 0.
build-tests made one archive holding every crate's test binaries and one holding the reference emulator's release ROM suites, one after the other, and every test job downloaded both. A simulation group needs only the native crate's suites. build-tests is now a matrix of two archives built side by side: native.tar.zst with the native crate's suites and workspace.tar.zst with every other crate's. The ROM suites archive moves to build-refemu, which already builds the reference emulator in release and is off the path to the test jobs. Each group downloads only what scripts/test-group.sh runs it from: the simulation groups the native archive, the emulator group the workspace and ROM suites archives, and native-rest both, since it runs the native crate's other suites and the driver's native_* suites. native-rest runs them as two nextest invocations, one per archive, at the same two threads. Measured on the throwaway run 36122853256 with a warm cache, no debug info and the native crate's suites at opt-level 0, the native archive built in 48 s (119 MB) and the other crates' in 94 s (72 MB), against 2 m 16 s (206 MB) for the single archive under the same settings. Each group selects the same tests as before: listing every group's filter over the workspace before the split and over the two archives after gives the same test lists for all eight groups, 953 tests in all, none in two groups.
A group's filter that stops matching some tests leaves them running nowhere, and CI stays green. group_coverage.rs caught that for the simulation suites only, by reading the script's lettered filters. Every other package's tests, and the split of the driver's native_* suites between native-rest and emulator, had no check. scripts/test-group.sh gains two modes. `--list <group>` prints the tests the group selects, through `cargo nextest list` with the same archives and filters the run uses. `--check` lists every test in the archives (or the workspace, without NEXTEST_ARCHIVE_DIR) and every group, and fails when a test is selected by no group or by two, or when a group selects nothing. A test a group reaches from two archives counts once for that group: four refemu tests in probe_fixture and reference_trace run in both the debug workspace archive and the release ROM suites archive, both in emulator. The new test-groups job runs --check against the three archives, and ci-passed needs it. group_coverage.rs also checks that ci.yml's test matrix, the script's case arms and its new `groups=(...)` line name the same groups, and it reads group names of any length rather than one letter.
Every test job needed every build job: both test archives, the ROM and the reference emulator. A simulation group reads only the native crate's archive, so it waited on the slowest of the others for nothing. The native archive gets its own job, build-native-tests, and build-tests builds the other crates' archive. The simulation groups move to their own matrix job, test-sim, which needs only build-native-tests and downloads only its archive. `test` keeps emulator and native-rest and every build job. ci-passed needs the two new jobs. group_coverage.rs reads the groups of every test matrix in ci.yml, so the check that CI runs every group the script defines covers both jobs.
github-actions
Bot
added
area: ci
Owner
Author
|
Measured: run 36126756550, attempt 1 cold and attempt 2 warm. The numbers are in #588. |
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway: #588 (top of the #584 stack) merged with #585, to measure the combined build, download and suite times. Not for merging; closed once measured.
Written mostly by Claude Opus 5.5.