Repository navigation
ci: run unit and end-to-end tests in parallel - #1965
Conversation
The Main tests job ran the unit and end-to-end suites one after the other in a single pytest process, so the job grew from under a minute in August to about 25 minutes as the suite reached about 5,500 tests. No single test dominates: the slowest 40 account for under a fifth of the serial run. The make test targets now spread tests across one pytest-xdist worker per CPU, with PYTEST_WORKERS=0 restoring a single process, and the Main matrix runs the unit and end-to-end suites as separate jobs.
The Bub harness lock records the root project's dev group, so the new pytest-xdist requirement left it stale and harness-sync rejected it.
|
The The job stops at This PR does not touch the Skill or the pin. Because a changed pin needs a new model run, I've left it to whoever owns the ZCode guidance evaluation. |
Which issue or RFC does this PR close?
No linked issue. This PR addresses Main CI duration, which has grown steadily over the last two months.
Rationale for this change
The
tests (3.x)jobs in the Main workflow are the critical path of every PR. Each one runsmake unit-testand thenmake e2e-testin a single pytest process. Sampledtests (3.14)step durations on master pushes (minutes, unit / e2e):The suite now collects about 5,500 tests (837 under
tests/e2e). A local--durations=40profile shows no single hotspot: the slowest 40 tests total about 125 s of a 643 s serial run, and the slowest single test takes 12 s. Fixing individual tests would recover little, so this PR changes how the suite runs instead.What changes are included in this PR?
pytest-xdistto thedevdependency group.make test,make unit-test, andmake e2e-testpass-n $(PYTEST_WORKERS), defaulting toauto(one worker per CPU).PYTEST_WORKERS=0runs a single process, for example when debugging withpdb.testsmatrix gains asuite: [unit, e2e]axis, so the two suites run as separate parallel jobs. The DSH configuration runtime setup runs only in the unit jobs, which are the only ones that pass--require-dsh-config-runtime.CONTRIBUTING.mddocuments the parallel default and the opt-out.Job names change from
tests (3.14)totests (3.14, unit)andtests (3.14, e2e). The master ruleset has no required status checks, so the rename does not block merging.The
E2E harnessworkflow invokes pytest directly and is unchanged.Are there any user-facing changes?
No product changes. Contributors running the make test targets get parallel execution by default.
How was this change tested?
Locally on macOS (18 CPUs), Python 3.13:
pytest --doctest-modules(serial, before)-n 4-n 4make unit-test(-n auto)make e2e-test(-n auto)make checkpasses, and-n 0runs a single process.Not validated locally: tests skipped here because the DSH configuration runtime and other CI-only prerequisites were absent. This PR's own CI run is the first parallel run of those tests on the 4-CPU Linux runners.
AI usage statement
Claude Code (Claude Opus 5.5) was used to profile the CI and test durations, implement the change, and draft this description. I reviewed the changes and the results.