Repository navigation
ci: measure and shard Playwright E2E suites (#434) - #436
Merged
Merged
Conversation
# Conflicts: # .github/workflows/deploy.yml
…sue-434-e2e-shards
…sue-434-e2e-shards
dborup
marked this pull request as ready for review
October 13, 2026 20:26
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #434
Summary
Split the existing 155 Playwright suites into three independently starting shards using measured longest-first (LPT) assignment. Each shard retains baseline-relative execution order and prepares its own unchanged fixture/server on localhost ports 13581, 13582 and 13583.
The checked-in manifest includes every assignment, command environment, three measured timing samples and provenance. The permanent three-repeat slide-over gate is bundled with its regular suite and runs exactly once on shard 3. Core frontend coverage belongs to shard 2.
The required check remains
🏗️ Build & Publish Docker Image. It explicitly requires Go, all three shards, the verified shard report and image validation to report success. Failure, skipped, cancelled and empty results fail the gate; a cancelled workflow does not turn the required gate green.Draft only; auto-merge off. Latest head
fa84117fis fully green in automatic PR run 38452595221, attempt 1, with seven successful jobs and three skipped. It includes the required ordinary master merges for issue #426 hardening and #427. Five completed successful post-change PR executions are reported separately below; failed historical executions remain disclosed, not hidden or claimed fixed. Nothing is marked ready, merged or deployed.Measurement first
Downloaded full logs of three recent green PR runs: 38404492174, 38392091612 and 38391488315. None contained runtime
=== E2E SUITE: ... ===markers; the green #383 log was also unmarked. Echoed shell commands are not runtime evidence, so no per-suite values were invented from those logs.A marker-only baseline was therefore committed first. Removing its paired START/END echoes reproduced the then-current workflow byte-for-byte. Three full, fresh green attempts of the same PR run/head were collected before choosing assignments: run 38410864651, attempts 1–3, head
529e008c941f86c1901e21a62da85173e739200d. These are three distinct executions/job IDs, not three distinct GitHub run IDs.Runner minutes are the sum of actual non-skipped job start-to-completion durations, not billed/rounded minutes. Execution span excludes queue and idle wait between reruns. Measurements are wall time, so hosted-runner variability is visible.
Measured three-way assignment
Greedy assignment sorts by descending mean suite duration, with filename then shard ID as deterministic ties. The three-repeat gate's measured mean (45.494 s) is added to its base suite's assignment weight. Execution order inside each shard is retained from the old workflow.
These predictions exclude duplicated checkout/build/install/fixture setup, report and upload overhead. Three shards are retained as requested; assignment loads differ by under 0.2 seconds. Go may now dominate total pipeline elapsed time, so E2E improvement must not be presented as equal whole-pipeline improvement.
Full measured suite timing table: seconds in each baseline execution
All 155 regular suite durations use exact paired START/END log timestamps. The permanent three-repeat gate has one aggregate START/END wrapper; individual repeat durations are not claimed. The repeat weight is bundled with its regular suite for balancing. Predictions exclude shard setup, fixture preparation, teardown/uploads, and coverage-report overhead.
Fixture, coverage and fail-closed behavior
Verification before push
-race: PASS.sh test-all.sh: 243 files passed, 0 failed. Later master-merge validation is reported separately below.git diff --check: PASS. Actionlint's YAML/Actions check has only the existing custom self-hosted label diagnostic; full ShellCheck output retains pre-existing style diagnostics.Red mutants / green controls
Scratch-only mutation runs, never pushed:
test-filter-ux-e2e.jsassignment: validator and registry exit 1 withunassigned E2E suite.Completed post-change PR CI measurements
Run 38425252311, attempt 1 on
4ef231a7, ended failure. Go Build & Test failed in Build and test Go server (with coverage):TestPollerBroadcastsNewData,websocket_test.go:241,expected data.packet.timestamp to exist. The poller, database query and test are unchanged by this PR. There was no race-detector warning.All three shards and their report passed, with 9m15s E2E/report wall span (shard jobs 8m48s / 8m08s / 8m55s, report 17s). All 158 suite/repeat child executions were verified from the artifacts with exit 0 and the same manifest. Image-check's local Docker build passed. The required gate correctly failed with
GO_RESULT=failure, while every shard/report/image result was success; GHCR, release, deploy and badge publishing did not run.The failed attempt consumed 47m39s actual runner time. This is an incomplete failed-pipeline measurement, not a green before/after cost comparison, and will remain separately disclosed. No Go assertion, timing, retry or skip is changed in this PR.
Attempt 2 of the same pull-request run on
4ef231a7was a complete SUCCESS, with fresh Go, three shard, report, image and required-gate job IDs. All required jobs passed; release, deployment and publication stayed skipped. The three shard artifacts contain all 158 expected suite/repeat child runs with exit 0 and matching manifest/checkout context.The observed E2E/report span is 58.9% shorter, while actual all-job runner time is 6.36% higher. The after sample is n=1, so it has no repeatability/variance estimate. Whole execution is now mostly Go-bound. Runner minutes sum actual non-skipped job durations, not billed rounded minutes; queue and rerun idle wait are excluded. Baseline
529e008cand after head4ef231a7also differ by incoming #422 Go configuration/test changes; this is not a source-identical A/B comparison. Go changes and hosted-runner variability are not attributed to sharding. The three baseline executions are attempts of one baseline run ID, and the successful after execution is attempt 2 of a different run ID. At the latest merge head, the manifest's weights remain historical baseline estimates; #435 adds core browser checks without inventing new timing samples for them.Attempt 3 used the same unchanged head but shard 3 failed in
test-issue-315-estimated-positions-e2e.js,browserChecks:173:16(caller277:7):page.clicktimed out after 10,000 ms waiting for the synthetic missing-GPS node rowtr[data-key="3152000000000000000000000000000000000000000000000000000000000000"]. Go, both other shards and image validation passed; the report and required check correctly failed closed. This is the suite body, not fixture preparation or Leaflet download: all five pinned Leaflet checks passed immediately beforehand. The suite starts its own synthetic DB/server/browser context rather than sharing the shard fixture. Its source, fixture and relevant node UI are byte-identical to the master included in that head. The failed execution's E2E/report span was 10m38s, and actual all-job runtime 51m44s; these are failed-execution diagnostic numbers, not a green comparison. All publishing/deployment jobs stayed skipped.The second complete green measurement at that unchanged head was not obtained. There is no fourth retry of it. Master subsequently advanced to
8665b0c7(#435), requiring the ordinary merge and fresh validation described below. Fresh pull-request run 38436317680, attempt 1, completed SUCCESS on merge head045cf701, tree277f3372: Go, all three shards, receipt/coverage report, image validation and the stable required gate passed. All 158 expected child suite/repeat executions had exit 0 and the same manifest/context. The actual GitHub checkout103a5705has exactly that tested tree and parents master8665b0c7plus head045cf701. GHCR login/push, release, deployment and badge publishing stayed skipped.4ef231a7045cf701514392a71ec70c20fa84117fCompared separately with the three-execution baseline mean (26m33.7s E2E, 50.64 actual runner minutes), E2E/report time is 58.9%, 57.4%, 59.0%, 58.7% and 56.8% shorter in these observations. Actual runner time was 6.36% and 11.07% higher in the first two, 0.42% lower at
514392a7, 10.97% higher at1ec70c20, and 11.10% higher atfa84117f; no causal cost-saving claim is made. Each post head has n=1, and #435 adds core E2E checks, so they are not pooled into an identical-input after mean or variance estimate. Baseline-to-after also includes #422 Go config/test drift; later heads add #431 MQTT/pprof operations, #430 API cache bounds, #426 hardening, #433 WS history, #424 backup, #440/Kpa-clawbot#1692 fixture-window changes and #427 frontend/#165 tests, which may affect runtime. This is not a source-identical A/B experiment, and observed all-job cost or Go-time changes are not attributed solely to sharding. Manifest weights remain historical baseline estimates. Failed attempts 1 and 3 stay separately disclosed and excluded; green results do not fix their independent defects.During the final read-only check, master advanced again to
dd712bc9(#431 and #430). It is included by ordinary merge514392a7; fresh PR run 38441232737, attempt 1, completed SUCCESS on that exact head. Go, all three shards, the receipt/coverage report, local Docker-image validation and🏗️ Build & Publish Docker Imageall passed. GHCR login/push, release, staging-deploy and badge publishing were explicitly skipped at job/step level. All 158 ordered suite/repeat child runs had exit 0, no signal and matching manifest hash. Their common GitHub checkoutf76c66f0was independently checked through the GitHub Git API: tree0ff34246, parents masterdd712bc9and PR head514392a7, identical to the local tested tree.Latest E2E/report wall span includes the full shard jobs and report: 10m53s. Actual all-job runner time is 50m26s (50.43 min), not billed rounded minutes. Comparable whole execution, earliest actual job start through last non-skipped job completion, is 18m50s; GitHub's API run-level start/completion span is 18m54s, including three seconds of bootstrap and one second of completion bookkeeping. Queue and rerun idle time are excluded throughout. The three baseline executions are three attempts of one run ID, while the three completed successful sharded executions above are separately identified run IDs/heads.
Post-push local browser verification also passed: Kpa-clawbot#1122 18/18 on each of the three independent local fixture servers, three extra slide-over runs 27/27 each on the owner port, and #406's self-contained Chromium smoke 21/21. All owned processes exited and ports were freed.
Known independent Go-test gap
A separate read-only diagnosis confirmed identical
websocket.go,websocket_test.goanddb.goon the historical tested masterd34b228aand measured head4ef231a7. An initialization-barrier probe reproduced the missing fallbacktimestamp3/3 on both; the representative production packet-store path passed 3/3 on both. The original test passed 25/25 on both by initializing its cursor after the INSERT and then accepting no broadcast. Those passes were vacuous for this assertion. Later incoming WebSocket changes are not covered by this historical source-identity claim.Production
main.gosetspoller.store; the legacy nil-store database fallback instead returnsfirst_seenwithouttimestamp. This is an existing fallback-contract/test-ordering defect, not evidence that #434 introduced it. It is not fixed by a green rerun and needs a separate follow-up; no Go production/test behavior is weakened or changed here. This draft is not a merge authorization.Independent existing navigation/theme vulnerability
A separate reviewer reproduced the exact #315 missing-row timeout on both unchanged master
d34b228aand measured candidate4ef231a7by forcingtheme-refreshto run between list-navigation hash assignment and the router handling it. The node page reselects the old node, rewrites the hash to its full detail and leaves no list row to click. Captured API responses still contained all four synthetic nodes; captured DOM/URL showed full detail instead of the list. An unforced local execution passed all three #315 policy modes. Original suite, fixture and node-UI hashes were unchanged, and owned probe processes exited.This proves a pre-existing navigation/theme ordering vulnerability, not the actual cause of CI attempt 3: that artifact lacks failure DOM/trace, so a matching forced-order reproduction is not causal proof. No concrete sharding regression was established. The failure remains visible, and this PR does not skip the suite, relax its assertion, raise its timeout or fix unrelated production code. A separate focused fix should prevent cosmetic theme refresh from overwriting pending navigation and add a deterministic regression test.
Scope and master drift
Master
d34b228a88bc3b17e6602d3b9d34ad8c453f0a2awas included through ordinary merge commits for the measured head4ef231a7. The incoming #423 E2E and #422 config changes are preserved. During measurement, master advanced to8665b0c7603cd4a54a0fceadfd4d44c1a3a5ad6d(#435 packet-count help); it is now included and pushed in ordinary merge commit045cf701430b128c0b5ef118d87f0b4e5f2d065d, tree277f337264178877e4274c9488ed053c6bc84ec3. All seven incoming files are inherited byte-identically from master without conflict; all 11 reviewed sharding feature files are unchanged. Fresh local checks passed: all 243 registered JS test files, server package (41.819 s), focused fork guards under-race(1.582 s), syntax/diff checks and exact 9/1 literal fork guards. No rebase or force-push. All feature and merge commits usedborup <kontakt@meshview.dk>.The exact new merge tree also passed bounded local Chromium validation with the unchanged CI fixture preparation on its own port: core
test-e2e-playwright.jsexit 0, 151 checks passed and three existing skips (106.7 s), including every incoming packet-count-help check; Kpa-clawbot#1281 exit 0, 6/6 (5.6 s). The initial archive lacked the runtime Git metadata available in CI's checkout, so its Version-card assertion failed; supplying only the exact exported commit via scratch.git-commitresolved that confirmed harness mismatch without changing source, tests or fixtures. Owned servers/browser processes exited and port 13582 was freed. This local macOS pass is supplementary, not a substitute for the fresh Linux PR run.No staging/production contact, ready marking, PR merge, issue closure or upstream writes. This PR remains draft for a separate merge decision.
Historical master merge
514392a7Final preflight found master
dd712bc92ad916b489be3a9002cc0fefc328f43f, incorporating #431 MQTT/pprof settings and #430 bounded analytics caches. Ordinary merge commit514392a751c07f78c80c911546a20b6f94de9c87has parents045cf701anddd712bc9and tree0ff34246702dc50152813b2cb3019da22abf2ee7. All 35 incoming files match master except the two expected feature overlaps:deploy.ymlandtest-all.shpreserve both sides' additions. The newinternal/pprofbindtest line and broker-check unit registration are retained. Diff against this master remains exactly the 11 intended sharding files; the other nine feature blobs are unchanged. Independent immutable-tree audit passed with no findings, unchanged fixtures/aliases/ports/gate and exact 9/1 fork guards.Fresh local validation at this historical merge: server 43.763 s, ingestor 110.729 s, new pprof module under
-race1.301 s, and all 244 JS unit-test files passed. Server/ingestor/migrate binaries were rebuilt from the exact new tree rather than reusing old production code. The core browser suite passed 151 checks with three existing skips (108.6 s); #315 passed default/enabled/disabled real API and browser policy modes (6.8 s), with the same pinned Leaflet files. Owned processes exited and localhost 13582 was freed. Linux PR CI on514392a7completed successfully as detailed above; this is not evidence of CI success on the later1ec70c20merge.Integration with issue #426 hardening
Master advanced to
a20c611eafe0711894e2164d5f2a0ad34d450edeafter that green execution, including #438 (issue #426 CI hardening), #433 (WebSocket-client history), #424 (backup locking) and #440 (the existing Kpa-clawbot#1692 E2E fixture-window fix). Normal merge1ec70c20fe52f98e3829d1a1e51e49390e55163dhas parents514392a7anda20c611e, tree87544b279349fa6354fcf447a44bb55b0e62382f. No rebase, amend or force push.The workflow retains #426's local image job
contents: readpermission and 20-minute timeout. The only conflict was intest-ci-parallel-gates.js; resolution retains both the real six-prerequisite gate cases and the new every-job publishing-surface/permissions/timeout checks. Incoming Go hardening tests and the new Perf WS-client unit registration are also retained. Own diff against that master remains exactly the 11 sharding files. Fresh immutable-tree audit, mutation/control checks, local Go/JS suites and rebuilt local browser validation passed. Automatic PR run 38446859914, attempt 1, completed SUCCESS on exact head1ec70c20. These incoming server/test changes mean its timings cannot be pooled with earlier heads as identical-input samples.Local verification on
1ec70c20: server 45.010 s, ingestor 112.576 s, pprof binding module under-race1.217 s, and both server/ingestorgo vetpassed. The full JS runner reported 244/245 files green inside the sandbox: the newly inherited backup suite's dead-PID checks failed because the sandbox deniedps -p. That restriction was reproduced directly. The same unmodified suite then passed with normal local process access (95 passed, 0 failed, 50 clearly reported unavailable-host-tool skips), using only mock Docker and disposable files. No source, assertion, skip or retry rule was changed to obtain that result. All 245 registered JS files are therefore covered by the full run plus this explicit environment-corrected backup check, not by a falsely claimed all-green original sandbox run.Independent audit of exact
1ec70c20passed: all 17 incoming paths are inherited correctly, with four intentional workflow/test overlaps retaining both sides. Other workflows match master, aliases resolve equally, manifest covers all 155 suites, stable check and six prerequisites remain intact, and fork guards are exactly 9/1. All 25 actual Bash gate cases passed. Ignoring shard 2/3 or accepting skipped results made the gate red; removing an assignment or adding a new unassigned suite made both validator and registry red. Incoming JavaScript and Go publishing/timeout/permissions mutants also turned red. Restored controls passed; 11 focused Go workflow guards passed under-race(1.571 s).The three native binaries were rebuilt from this exact tree. Its full CI fixture recipe and instrumented local Chromium verification passed: core 151 checks passed / three existing skips (110.9 s), updated Kpa-clawbot#1692 2/2 (1.6 s), Perf WS-client unit 17/17, and unmodified #315 default/enabled/disabled real API/browser checks (6.6 s). Only localhost port 13584 was used; owned browser/server processes exited and the port is free. This supplements the subsequently successful Linux CI run on
1ec70c20.Run
38446859914had exactly seven successful jobs and three skipped jobs. Its 158 ordered child runs all exited 0 with no signal and the unchanged manifest. The actual checkout6a1ffcb224b58028951dc00a4c66f096698babb6has tree87544b27and parents mastera20c611eplus head1ec70c20, verified through GitHub's Git API. E2E/report wall span was 10m58s, actual all-job runner time 56m12s (56.20 min), and comparable actual-job execution span 24m44s. API-level run span was 24m48s, including bootstrap/completion bookkeeping. Go took 24m34s and was the critical path; report took 19s. GHCR login/push, Buildx, QEMU, Docker metadata, release, staging deployment and badge publishing were explicitly skipped. A separate read-only audit reproduced all timestamps, totals and percentages from the raw metadata.Subsequent #427 master integration
The final ref check then found master
c04861bf7f38c78362a854db9b0eabe31197bbb1, incorporating #427's direct-hop/receiver-hint resolution. Required ordinary mergefa84117fb00611c2908fe351f38712e6a1fdb08ehas parents1ec70c20andc04861bf, treedc70a36c1773856c9fff06afa0d1838e4807ca27. Incoming scope is six frontend/test files, including the existing #165 grouped-hop E2E; there are no new suite files or workflow/Go changes. All 11 sharding feature blobs are unchanged from the reviewed, green1ec70c20; assignment weights remain explicitly historical. Fresh local combination checks passed. Before the non-force push, local/remote head, clean worktree, and master were freshly locked and verified. Automatic PR run38452595221, attempt 1, subsequently completed SUCCESS on exactfa84117f, not merely a historical head. Post-CI refs still match masterc04861bfand feature headfa84117f.Local checks at
fa84117f: all 245 registered JS files passed, server 43.677 s, ingestor 109.478 s. Independent immutable audit passed: all six incoming blobs match master, all 11 sharding blobs match1ec70c20, own scope remains 11 files, guards remain 9/1, manifest covers 155 suites. Fresh gate 25 real Bash cases, shard behavior 9/9, registry 10/10 and 11 focused Go guards under-racepassed. Previous red mutants remain applicable because every involved workflow/test/manifest/helper blob is byte-identical, and restored controls were freshly executed.Fresh native builds and one-pass CI fixture setup also passed in a new immutable archive. Instrumented Chromium ran each suite once: core 151/154 with three existing skips (110.7 s), changed #165 grouped-hop 12/12 (6.2 s), #123 map lifecycle 16/16 (24.2 s), and #315 all three policy/API/browser modes (5.9 s). Actual health commit was
fa84117f; port 13585 and all owned processes were clean after exit. Two earlier bootstrap-only attempts executed no tests: sandbox bind/ps restrictions, then duplicate seeding from a reused fixture. Both were stopped and preserved separately; final validation used an entirely fresh source archive/database/public assets and normal localhost/process permission. No suite was retried or weakened.Completed latest-head CI and acceptance evidence
PR run 38452595221, attempt 1, ended success on head
fa84117fb00611c2908fe351f38712e6a1fdb08e: Go, all three shards, verified receipt/coverage report, local image validation and the unchanged required check🏗️ Build & Publish Docker Imagepassed. The successful Go/JS unit steps actually ran; the local image build passed. Release, staging deployment and badge publishing stayed skipped, as did Buildx, QEMU, GHCR login, Docker metadata and GHCR push. No staging/production host was contacted.All three artifacts independently verify the 158 exact ordered child executions (155 suites + three extra slide-over repeats), all code 0/no signal, same manifest hash
af8c01cc…. Common checkout4e5a75478a8e2fc845230526339d7cc566649958has treedc70a36c1773856c9fff06afa0d1838e4807ca27and parents masterc04861bfplus PR headfa84117f, verified through GitHub's Git API. The offline measurement validator additionally checks each receipt lies inside its owning job's timing bounds, rejects incomplete/failed/stale/empty proofs, and retains separate failed historical diagnostics.The measured suite table, checked-in manifest, five separately labeled successful PR executions, exact-head green CI and real red-mutant/restored-green gate/unassigned-suite evidence are now complete. Draft and auto-merge-off status are preserved; issue #434 remains open. No ready marking, PR merge, issue closure, force-push, settings change, squad edit, deployment or upstream write was performed.