Repository navigation
CI: plan e2e shards by measured duration on Linux and macOS - #8
Conversation
The four count-balanced e2e shards were uneven (145/101/69/58 s) because Vitest shards by hashed file order and one file is about 63 s of the suite. A committed duration table plus longest-first assignment puts six shards within a second of each other. Also split static from unit and run the unit suite in two shards.
macOS ran the static checks, the unit suite and the whole e2e suite in one job: about 11m45s. The static checks are platform-independent and already run in ci.yml. Its e2e files take 466.6s, against 377.7s on Linux, so e2e-durations.json now holds a linux and a darwin table and e2e-shard.ts selects by platform. Six macOS shards plan to 78.5s at most. Also replace mapfile, which macOS bash 3.2 lacks.
Review: PR #8 at
|
| Check | Result |
|---|---|
pnpm install --frozen-lockfile |
pass |
| typecheck / lint / check:escapes / check:comments / check:bidi | all pass |
test |
211 files pass, 2 skipped; 2,147 tests pass, 25 skipped (8 more than before: the new guard tests) |
test:e2e (unsharded) |
51 files pass, 1 skipped; 279 tests pass, 13 skipped |
test:isolation |
7 files skipped (no rootless podman or docker here) |
CI shard 1/6, run with the exact CI command (mapfile plus vitest run --project e2e) |
1 file, 16 tests pass, 64.8 s |
On GitHub CI, all 11 jobs pass, and macOS passes too. The unit shards hold 81 + 80 = 161 files and 957 + 923 = 1,880 tests. The e2e shards hold 1 + 9 + 10 + 11 + 11 + 10 = 52 files and 292 tests. These counts match the unsharded totals. The e2e jobs took 1m37s to 2m7s each, including setup and the Chromium install.
Hands-on checks (all edits reverted, working tree clean)
| Change | Result |
|---|---|
| Planner skips the last file | caught (1 test fails) |
| Planner always picks shard 1 | caught (2 tests fail) |
Glob drops packages/core |
caught (2 tests fail) |
| New e2e file with no duration | planned once; the guard test fails (see finding 2) |
Vitest e2e include widened, planner glob unchanged |
not caught (see finding 1) |
e2e-shard.ts 7 6, 0 6, "1;id" 6 |
each one exits 1 with a clear message |
Not covered
- The base was not run again on this computer. The base test count comes from the previous review (2,139). It agrees with 2,147 minus the 8 new tests.
vitest listdoes not apply--shard, so I checked the unit shard split from the CI logs, not locally.- The isolation suite was not run (no rootless podman or docker here).
- The actions are still pinned by tag, not by SHA. This existed before this PR, so it is not a regression.
@claude The review is done. Do not merge without the owner's OK.
Run 41568408211 started five macOS jobs at once; the sixth and seventh waited about 90 s for a slot. The unit job takes one slot, so four e2e shards keep every job running. Four macOS shards plan to 116.8 s at most.
Second review: PR #8 at
|
| Check | Result |
|---|---|
Live CI at e3417b1 |
CI run 41569000538 and macOS run 41569000640: all 16 jobs pass (Linux 10, macOS 6). macOS wall time was about 3 min (02:54 to 02:57 UTC), against about 11m45s before. |
| macOS shards | 10/13/14/15 files = 52, and 73+66+66+87 = 292 tests, the same as the Linux total. The macOS unit job ran 161 files and 1,886 tests (1,865 passed, 21 skipped). |
| Plan check (local) | darwin x4 planned 116.5–116.8 s against a 116.7 s mean, with 10/13/14/15 files. This is the same split CI ran. linux x6 is unchanged at 62.8–63.3 s. Both tables have the same 52 keys as the e2e glob. |
| Local (Linux x86_64) | install, typecheck, lint, check:escapes, check:comments and check:bidi all pass. test: 211 files, 2,153 passed, 25 skipped (+6, the new per-platform guard tests). test:e2e: 51 files, 279 passed, 13 skipped. test:isolation: skipped (no rootless podman). |
| Not run | The macOS jobs can't run here. The macOS evidence above comes from the CI logs only. |
@claude Second review done. None of the findings blocks the merge. Good to merge on the owner's OK. Do not merge without it.
The planner copied the e2e include and exclude from vitest.config.ts, so the guard test compared the planner against itself. Widening the Vitest include left an e2e file unplanned and the tests still passed. Both now import scripts/lib/e2e-files.ts. Also drop defaultSeconds, which disagreed with the measured mean, and correct the notes: a new e2e file without a duration fails the guard test on purpose.
|
Both findings are addressed in Finding 1, the e2e pattern in two places. The include and exclude now live in Finding 2, the docs promised something the test forbids. The notes now say a new e2e file without a duration fails the guard test on purpose, and that the file goes into I also dropped One correction to your first finding's wording: the duplicate pattern was copied from Re-run results on |
What this changes
The e2e suite now runs in six planned shards on Linux and on macOS. The Linux unit suite runs in two shards. macOS runs its unit suite and its e2e shards as separate jobs.
Why
The Linux e2e shards split files by count. Vitest sorts files by hash and slices them into contiguous groups, so the load was random. In run 41559401063 the four shards took 145 s, 101 s, 69 s and 58 s. One file,
packages/core/test/swipe-approval.e2e.test.ts, takes about 63 s. It is one sixth of the suite, so the shard that holds it cannot finish sooner than 63 s.macOS had the same problem and a bigger one. One job ran the static checks, the unit suite and the whole e2e suite: about 11m45s. Its e2e files take 466.6 s, against 377.7 s on Linux.
How the plan works
scripts/e2e-durations.jsonrecords measured seconds per e2e file forlinuxanddarwin.scripts/e2e-shard.ts <index> <count>assigns files longest-first to the lightest shard and prints the files for one shard. It selects the table for the platform it runs on.scripts/lib/e2e-shards.test.tschecks coverage, determinism, load balance, stale entries and filter collisions for both tables. A new e2e file without a measurement is still planned.Unit shards
The old
static-and-unitjob is now thestaticjob plus a two-shardunitmatrix, withpnpm test:unit --maxWorkers="$JOB_CPUS" --shard=N/2. macOS gets a separateunitjob. The static checks are platform-independent, so macOS does not repeat them.Test plan
pnpm check:static: all five checks pass.Not changed
Chromium is installed in each e2e shard on both platforms and is not cached.