ci: split surface unit tests into server and web jobs#625
Merged
Conversation
The combined "Surface unit tests" job ran the Postgres-backed server suite (the ~200s+ long pole) alongside the jsdom suites (web, mobile, …) on one 4-vCPU runner. Server's 3 workers competed for CPU with mobile's + web's, and the resulting starvation inflated server's wall time enough to trip the 6-minute step timeout on an otherwise-green suite (#621: 909/909 passed, killed at 344s; a clean rerun did the same tests in 205s). Measured the mechanism locally (10-core box, so contention is milder than CI): serializing packages made total wall WORSE (+40%) by de-overlapping the IO-bound server from the CPU-bound jsdom suites — so `--workspace-concurrency=1` is the wrong lever. But isolating the jsdom suites cut their test time sharply (web 37s→16s, mobile 20s→9s), confirming CPU starvation among them. So: put the server suite on its own runner (keeps Postgres, 10-min step budget) and the DB-free jsdom suites on a second runner in parallel. Also cap mobile's worker pool (it was the one heavy jsdom suite left uncapped) to match web. This is being pushed to measure the two jobs' real 4-vCPU times before it's proposed as the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
The combined Surface unit tests job runs the Postgres-backed server suite (the ~200s+ long pole) alongside the jsdom suites (web, mobile, shared, desktop, centaur-client) on one 4-vCPU runner. Server's 3 vitest workers compete for CPU with mobile's + web's, and the resulting starvation inflates server's wall time enough to trip the 6-minute step timeout on an otherwise-green suite — exactly what happened on #621 (909/909 tests passed, killed at 344s; a clean rerun ran the same tests in 205s).
This splits the server suite (keeps Postgres, own runner, 10-min step budget) from the DB-free jsdom suites (second runner, in parallel), and caps mobile's worker pool — it was the one heavy jsdom suite left uncapped, so it oversubscribed the shared runner. Verified only the server package touches Postgres, so the split is safe.
Measured, not assumed
Local A/B (10-core box, so contention is milder than CI — a lower bound) first ruled out the obvious lever:
--workspace-concurrency=1made total wall worse (195s → 272s, +40%) by de-overlapping the IO-bound server suite from the CPU-bound jsdom suites. But isolating the jsdom suites cut their test-time sharply (web 37s→16s, mobile 20s→9s), pinning the contention on CPU starvation.Then measured on real 4-vCPU CI runners (this PR, 3 samples of the server job):
The point isn't just that it's faster — it's that server's variance collapsed from a 68% swing (205↔344s) to 1.6% (126.5↔128.5s) once it stops fighting the jsdom suites for cores. That swing was the flakiness. The web job (all jsdom suites, concurrent, mobile now capped) runs in ~75s of vitest wall on its own runner.