Skip to content

CI: plan e2e shards by measured duration on Linux and macOS - #8

Merged
johnpippett merged 4 commits into
mainfrom
ci/shard-balance
Nov 3, 2026
Merged

johnpippett merged 4 commits into
mainfrom
ci/shard-balance

Conversation

@johnpippett

@johnpippett johnpippett commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

What this changes

The e2e suite now runs in six planned shards on Linux and on macOS. The Linux unit suite runs in two shards. macOS runs its unit suite and its e2e shards as separate jobs.

Why

The Linux e2e shards split files by count. Vitest sorts files by hash and slices them into contiguous groups, so the load was random. In run 41559401063 the four shards took 145 s, 101 s, 69 s and 58 s. One file, packages/core/test/swipe-approval.e2e.test.ts, takes about 63 s. It is one sixth of the suite, so the shard that holds it cannot finish sooner than 63 s.

macOS had the same problem and a bigger one. One job ran the static checks, the unit suite and the whole e2e suite: about 11m45s. Its e2e files take 466.6 s, against 377.7 s on Linux.

How the plan works

  • scripts/e2e-durations.json records measured seconds per e2e file for linux and darwin.
  • scripts/e2e-shard.ts <index> <count> assigns files longest-first to the lightest shard and prints the files for one shard. It selects the table for the platform it runs on.
  • The plan puts six Linux shards between 62.8 s and 63.3 s, and four macOS shards between 116.7 s and 116.8 s. Four, not six, because five macOS jobs run at once: the unit job takes one slot.
  • scripts/lib/e2e-shards.test.ts checks coverage, determinism, load balance, stale entries and filter collisions for both tables. A new e2e file without a measurement is still planned.

Unit shards

The old static-and-unit job is now the static job plus a two-shard unit matrix, with pnpm test:unit --maxWorkers="$JOB_CPUS" --shard=N/2. macOS gets a separate unit job. The static checks are platform-independent, so macOS does not repeat them.

Test plan

  • pnpm check:static: all five checks pass.
  • Unit shards with shared workers: 45 s and 57 s. 1877 tests passed and 3 were skipped.
  • Planned e2e shard 6/6 on Linux: 10 files, 66 tests passed, 4 skipped, 71 s wall clock.
  • Shard plans: Linux 63.3 s max, macOS 116.8 s max.
  • The macOS run of this pull request is the end-to-end check for that platform.

Not changed

Chromium is installed in each e2e shard on both platforms and is not cached.

The four count-balanced e2e shards were uneven (145/101/69/58 s) because Vitest shards by hashed file order and one file is about 63 s of the suite. A committed duration table plus longest-first assignment puts six shards within a second of each other. Also split static from unit and run the unit suite in two shards.
macOS ran the static checks, the unit suite and the whole e2e suite in one job: about 11m45s. The static checks are platform-independent and already run in ci.yml. Its e2e files take 466.6s, against 377.7s on Linux, so e2e-durations.json now holds a linux and a darwin table and e2e-shard.ts selects by platform. Six macOS shards plan to 78.5s at most. Also replace mapfile, which macOS bash 3.2 lacks.
@johnpippett johnpippett changed the title CI: plan e2e shards by measured duration and shard the unit suite CI: plan e2e shards by measured duration on Linux and macOS Nov 3, 2026

Copy link
Copy Markdown
Owner Author

Review: PR #8 at 833d1dc (base 1a1c5fe)

Verdict: good to merge on the owner's OK. The planner does what the PR says. The six planned shards hold every e2e file exactly once and match the list that Vitest itself resolves for the e2e project (52 files, no gaps, no duplicates). The planned loads are 62.8 s to 63.3 s. The two unit shards hold all 161 unit files. All local checks and all CI jobs pass. No security invariant is weakened. Two low findings are below. Neither blocks the merge.

Contract changes needing the owner's sign-off: this PR changes protected files. These are .github/workflows/ci.yml (the jobs are now static, unit x2 and e2e x6, and check now needs all three), AGENTS.md (CI description only, lines 173-176), and new files under scripts/ (e2e-shard.ts, e2e-durations.json, lib/e2e-shards.ts, lib/e2e-shards.test.ts). No SPEC.md, features.json, CLAUDE.md, tests/acceptance, .nightshift/, .claude/, vitest config, package.json script or inv* test changed.

Security: this PR adds no new action, permission or trigger. There is still no pull_request_target, and permissions: at ci.yml:15 did not change. The new unit job sets persist-credentials: false the same as the other jobs. The only ${{ }} value in the new run: steps is the static matrix integer. The required check job still fails when any job fails or is skipped (ci.yml check step). The planner fails closed: a bad index exits 1, and under the runner's bash -e, plan=$(...) stops the step. An empty plan also exits 1, and a planned path that matches no file makes Vitest exit 1 ("No test files found"). The runner-guard test (packages/cli/test/workflow-runner-guard.test.ts) still passes. No personal info is in the diff. The commit uses the noreply address.

Findings (by severity)

  1. Low: the e2e file pattern is in two places, and drift between them is not detected. scripts/lib/e2e-shards.ts:3-4 copies the e2e project include and exclude from vitest.config.ts:44-45. The guard test compares the plan against e2eFiles(), so it compares the planner against itself. Hands-on: I added a demo/test/*.e2e.test.ts include to the vitest e2e project and added a matching file. vitest list --project e2e showed the file, but no CI shard planned it, and all 8 guard tests stayed green. In CI, that file would never run. Suggestion: get both from one exported constant, or make the guard test compare against Vitest's resolved e2e file list.
  2. Low (docs): a new e2e file without a measured time makes CI fail. The docs say that it is still planned. docs/IMPLEMENTATION_NOTES.md:2867 and the PR body say a new file "is still planned, at the mean; refresh the table when it drifts". This is true for the planner. But the test has a measured time for every e2e file (scripts/lib/e2e-shards.test.ts:30) fails the unit job. Hands-on: a copied e2e file was planned into one shard, and that test failed. The strict behavior is the safer one, so I suggest changing only the docs: "add the new file to scripts/e2e-durations.json in the same PR". Also, the docs say a missing file "takes the mean", but e2e-durations.json:57 sets defaultSeconds: 7.41, and e2e-shards.ts:27 uses that first (the measured mean is 7.26 s). This difference is trivial.

Test results (PR head, this computer)

Check Result
pnpm install --frozen-lockfile pass
typecheck / lint / check:escapes / check:comments / check:bidi all pass
test 211 files pass, 2 skipped; 2,147 tests pass, 25 skipped (8 more than before: the new guard tests)
test:e2e (unsharded) 51 files pass, 1 skipped; 279 tests pass, 13 skipped
test:isolation 7 files skipped (no rootless podman or docker here)
CI shard 1/6, run with the exact CI command (mapfile plus vitest run --project e2e) 1 file, 16 tests pass, 64.8 s

On GitHub CI, all 11 jobs pass, and macOS passes too. The unit shards hold 81 + 80 = 161 files and 957 + 923 = 1,880 tests. The e2e shards hold 1 + 9 + 10 + 11 + 11 + 10 = 52 files and 292 tests. These counts match the unsharded totals. The e2e jobs took 1m37s to 2m7s each, including setup and the Chromium install.

Hands-on checks (all edits reverted, working tree clean)

Change Result
Planner skips the last file caught (1 test fails)
Planner always picks shard 1 caught (2 tests fail)
Glob drops packages/core caught (2 tests fail)
New e2e file with no duration planned once; the guard test fails (see finding 2)
Vitest e2e include widened, planner glob unchanged not caught (see finding 1)
e2e-shard.ts 7 6, 0 6, "1;id" 6 each one exits 1 with a clear message

Not covered

  • The base was not run again on this computer. The base test count comes from the previous review (2,139). It agrees with 2,147 minus the 8 new tests.
  • vitest list does not apply --shard, so I checked the unit shard split from the CI logs, not locally.
  • The isolation suite was not run (no rootless podman or docker here).
  • The actions are still pinned by tag, not by SHA. This existed before this PR, so it is not a regression.

@claude The review is done. Do not merge without the owner's OK.

Run 41568408211 started five macOS jobs at once; the sixth and seventh waited about 90 s for a slot. The unit job takes one slot, so four e2e shards keep every job running. Four macOS shards plan to 116.8 s at most.
@johnpippett

Copy link
Copy Markdown
Owner Author

Second review: PR #8 at e3417b1 (new since the first review: 002eaca, e3417b1)

Verdict: good to merge on the owner's OK. No high or medium findings.

What changed: macos.yml used to run one serial job (static + unit + e2e, about 11m45s). It now runs a unit job, four planned e2e shards and a check aggregate. The static checks are dropped there because ci.yml already runs them. e2e-durations.json now holds a linux table and a darwin table, and planFor() picks the table from process.platform. The guard tests now run against both tables. In both workflows, mapfile is replaced with a while read loop, because macOS ships bash 3.2. The AGENTS.md and IMPLEMENTATION_NOTES.md text matches the code.

Security: no regression. The triggers are still pull_request/push/workflow_dispatch, with no pull_request_target. permissions: contents: read is unchanged. Both new checkout steps set persist-credentials: false. No secrets are used. The only ${{ }} values in run: are the static matrix integer, and the check job passes needs.*.result through env:, not inline. All runners are GitHub-hosted macos-15 (image macos-15-arm64), and the runner-guard test still passes. The setup-node pnpm cache is the same pattern as before: PR-scoped caches can't write to main's cache, and installs use --frozen-lockfile. The actions are still pinned by tag, not SHA (this predates the PR; release.yml is SHA-pinned). The new diff contains no personal info, and both commits use the noreply address. The planner still fails closed: if the darwin table is removed, shardFiles throws, so the step exits 1.

Findings

  1. Low, still open (first-review Docs: match what ships today #1): scripts/lib/e2e-shards.ts:3-4 still copies the vitest e2e include/exclude, and nothing checks the two for drift. This now affects both platforms.
  2. Low (docs), still open (first-review README: the current mixed-case logo #2): docs/IMPLEMENTATION_NOTES.md:2867 still says that a new file without a duration "is still planned, at the mean". The guard test still fails CI in that case, and it now needs an entry in both tables. Suggested wording: "add the file to both linux and darwin in scripts/e2e-durations.json in the same PR; use defaultSeconds for darwin until a macOS run measures it."
  3. Low, not addressed (PR Tests: make browser-shell independent of the host platform #7 note): this PR changes no tests under packages/. The macOS runner does now run browser-shell.test.ts, but that test fakes uname as Linux x86_64 (browser-shell.test.ts:52). So browser_is_current is still never checked against the chrome-mac-arm64 layout. The test covers the cft_platform/cft_inner name mapping for mac-arm64, but nothing more.
  4. Nit: the macOS check aggregate uses a macos-15 slot for a 4 s shell loop. ubuntu-latest would do the same job.

Evidence

Check Result
Live CI at e3417b1 CI run 41569000538 and macOS run 41569000640: all 16 jobs pass (Linux 10, macOS 6). macOS wall time was about 3 min (02:54 to 02:57 UTC), against about 11m45s before.
macOS shards 10/13/14/15 files = 52, and 73+66+66+87 = 292 tests, the same as the Linux total. The macOS unit job ran 161 files and 1,886 tests (1,865 passed, 21 skipped).
Plan check (local) darwin x4 planned 116.5–116.8 s against a 116.7 s mean, with 10/13/14/15 files. This is the same split CI ran. linux x6 is unchanged at 62.8–63.3 s. Both tables have the same 52 keys as the e2e glob.
Local (Linux x86_64) install, typecheck, lint, check:escapes, check:comments and check:bidi all pass. test: 211 files, 2,153 passed, 25 skipped (+6, the new per-platform guard tests). test:e2e: 51 files, 279 passed, 13 skipped. test:isolation: skipped (no rootless podman).
Not run The macOS jobs can't run here. The macOS evidence above comes from the CI logs only.

@claude Second review done. None of the findings blocks the merge. Good to merge on the owner's OK. Do not merge without it.

The planner copied the e2e include and exclude from vitest.config.ts, so the guard test compared the planner against itself. Widening the Vitest include left an e2e file unplanned and the tests still passed. Both now import scripts/lib/e2e-files.ts. Also drop defaultSeconds, which disagreed with the measured mean, and correct the notes: a new e2e file without a duration fails the guard test on purpose.
@johnpippett

Copy link
Copy Markdown
Owner Author

Both findings are addressed in 04604d4.

Finding 1, the e2e pattern in two places. The include and exclude now live in scripts/lib/e2e-files.ts. vitest.config.ts and scripts/lib/e2e-shards.ts both import them, so there is one definition and the plan cannot drift from what Vitest runs. Verified on the PR head: vitest list --project e2e --filesOnly resolves 52 files, and the planner resolves the same 52 from the shared constant. Your hands-on case, widening the Vitest include while the planner glob stays put, is no longer expressible: widening means editing the one constant, which moves both.

Finding 2, the docs promised something the test forbids. The notes now say a new e2e file without a duration fails the guard test on purpose, and that the file goes into scripts/e2e-durations.json in the same pull request. The failure message names the file and the fix (add the file to scripts/e2e-durations.json) instead of an anonymous empty-array diff. The PR body says the same.

I also dropped defaultSeconds, which you were right to flag. It read 7.41 while the measured mean is 7.26. A file with no entry now falls back to the mean computed from the table, so there is no second number to keep in step. Nothing reads it, because the guard test already requires an entry for every file.

One correction to your first finding's wording: the duplicate pattern was copied from vitest.config.ts into the planner, so the guard test compared the planner against its own glob. It was not comparing the planner against the Vitest config. The effect you described is the same, and it is now fixed.

Re-run results on 04604d4: pnpm check:static passes all five checks, and the guard test passes 14 tests against both tables.

@johnpippett
johnpippett merged commit 34f2b4d into main Nov 3, 2026
16 checks passed
@johnpippett
johnpippett deleted the ci/shard-balance branch November 3, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant