You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: full Lighthouse CI Core Web Vitals budget (deferred from #95)Β #100
#95 asked for a CI performance gate. We shipped the deterministic fallback it sanctioned β a brotli client bundle-size budget (bundle-budget.json + scripts/check-bundle-budget.mjs, wired into the Build job). That catches byte-weight regressions without flake, but it does not measure real Core Web Vitals (LCP, CLS, TBT, FCP) or runtime rendering cost.
Proposed direction
Once the E2E environment is stable enough to host a server reliably in CI, add @lhci/cli:
An lhci:autorun step after build, starting the Node adapter server with DEV_SKIP_AUTH=true, scanning the login + program pages, then stopping.
Run it as a non-required lane first (warn-only / assert.assertions informational) to gauge flakiness against the Node adapter (port binding + cold-start jitter is why it was deferred in perf: Lighthouse CI performance budgetΒ #95) before promoting it into CI - Required Checks.
Acceptance criteria
Lighthouse CI runs against the built app in CI.
Budgets documented in lighthouserc.json.
Stable (no flaky failures over ~10 runs) before it becomes a required check.
Size: M
Follow-up to #95. The bundle-size budget remains the deterministic baseline gate.
Phase 1 (the warn-only lane) is now on main β @lhci/cli + lighthouserc.json (LCP/CLS/TBT/FCP + Performance β₯ 0.85, all warn) + scripts/lighthouse.mjs + a dedicated non-requiredlighthouse job in ci.yaml that builds the app and runs npm run lhci with DEV_SKIP_AUTH=true, deliberately kept out of CI - Required Checks.
Acceptance status:
Lighthouse CI runs against the built app in CI
Budgets documented in lighthouserc.json
Stable (no flaky failures over ~10 runs) before it becomes a required check
Keeping this open to track the remaining step: gauge the lane's red/green over ~10 runs against the Node-adapter server (port binding + cold-start jitter was the original deferral reason), then decide whether to promote the budgets into CI - Required Checks.
(Docs were drifting too β AGENTS.md still said the full Lighthouse run was "deferred"; synced in #121.)
Stability checkpoint β the third acceptance criterion is empirically not met yet, and I found why.
Over the last 16 β CI runs the Lighthouse CI (non-required) lane failed 3Γ (kit-2.66.0, adapter-node-5.5.5, plus a transient) β ~80% green, short of the "no flaky failures over ~10 runs" bar.
Root cause (from the failing run's log): the @sveltejs/adapter-node server exits early with code 13 β Detected unsettled top-level await β¦ await server.init({β¦}) β on a cold start, and scripts/lighthouse.mjs kept polling the dead port until the 30s waitForServer timeout. It's a race (the same commit passed one run, failed the next), i.e. exactly the port-binding/cold-start jitter that deferred this to warn-only in #95.
Draft #148 hardens the lane (abort the poll on early exit + retry the launch up to 3Γ), which should turn the jitter into a stable green signal. Keeping this issue open per its plan: once the lane runs ~10Γ clean post-#148, we can decide on promoting the budgets into CI - Required Checks.
Correction + stronger fix in #148. My earlier read (cold-start race, fix by retry) was incomplete: pushing the retry revealed the adapter-node server.init code-13 exit is deterministic on the current build when started DB-free β all 3 retries hit it. So the real fix is to stop using node build/index.js for this lane and serve the build via vite preview, exactly as the e2e lane already does reliably under DEV_SKIP_AUTH. Verified locally: /login 200, / (+session cookie) 200, / (no cookie) 302. The early-exit-aware retry is kept for residual bind jitter.
Separately worth your eye (not blocking, prod unaffected since DATABASE_URL is set there): the adapter-node server exits with code 13 (unsettled top-level await in await server.init(...)) when booted without a DB β i.e. node build/index.js won't come up DB-free. Harmless for this perf lane now that it uses preview, but I can file a separate issue to dig into the server.init top-level-await if you'd like.
Promotion is now gated on the budgets actually passing: the warn-mode assertions currently fail on / (perf 0.66β0.77 vs β₯0.85, FCP 2.1β4.2 s vs β€1.8 s) and are borderline on /login (LCP 2573 ms vs β€2500 ms). Root causes + fix plan tracked in #176; once assertions pass consistently, the final flip (warnβerror + required aggregate) closes this issue.
Post-#179/#180 Lighthouse read (CI run 29177777760, at the #180 head with responsive gallery images + self-hosted fonts, 2026-07-12 03:11Z):
Now passing (previously failing): performance score (was 0.66β0.77 vs β₯0.85 β no longer warned), TBT, CLS.
Still over budget: LCP on / β 3391 ms median (β€2500; all samples 3391/4141/4137) and LCP on /login β 2865 ms (β€2500; samples 3011/2865/2868). FCP on /login borderline at 1897 ms (β€1800).
So the #176 lever chain (gallery lazy-window β hero preload β responsive images β self-hosted fonts) recovered the perf score but the LCP element itself still renders too late under mobile throttling on both pages. Graduation (warnβerror) stays gated until LCP passes over multi-run evidence (single hosted-runner samples stay noisy β 3 in-run samples here are consistent, but we hold the ~multi-run bar from #148).
Next lever: profile what the LCP element is on each page post-#180 (hero image on /, likely the form card or heading on /login) and whether render-delay (not resource-load) now dominates β if so the remaining cost is in the critical JS/CSS path, not assets. Tracking continues on #176.
AC1 and AC2 have quietly been done for a while β the Lighthouse CI (non-required) job runs npm run lhci against the built app, and lighthouserc.json carries all five budgets (LCP 2500 / CLS 0.1 / TBT 200 / FCP 1800 / perf 0.85). What had never actually been taken is AC3's stability reading, so this issue sat at the head of the backlog with no one measuring the thing it asks for.
The reading, over the last 40 β CI runs:
Outcome
Count
Lighthouse job success
38
Cancelled (superseded push β not a lane signal)
1
Failure
1
So: 1 flaky failure in 39 usable runs (~2.6%), not zero. AC3 is not yet met.
The single failure has one root cause, not a budget miss (budgets are warn-level and cannot fail the job). In run 29641332997 Chrome launched, then every gatherer hit PROTOCOL_TIMEOUT, the page never painted (NO_FCP), and the browser died with Target closed β classic runner starvation.
Why it reddened the lane: the lane retried the server launch but treated a failed lhci collect as terminal. --numberOfRuns=3 does not cover it β LHCI aborts the whole collect on its first failed run.
PR #204 makes a failed collect retry (a deterministic failure still fails the lane β verified with a negative control).
Next step to close this issue: after #204 merges, observe ~10+ further runs. Clean β AC3 met, this closes, and graduation to required becomes purely #176's budget work. Any further failure β re-diagnose here.
#204 merged (dec84fcf). The lane now retries a failed lhci collect, and CI on the PR itself was green with the Lighthouse CI (non-required) job passing.
This issue stays open pending the AC3 re-measurement β shipping the fix is not proof it worked. The prior reading was 1 flake in 39 runs (~2.6%); the target is a clean window.
Measurement date: 2026-08-02. On that date, re-run the same reading over the β CI runs since dec84fcf:
Any failure β pull that run's Lighthouse job log and check whether it is the same Chrome/CDP class (PROTOCOL_TIMEOUT / NO_FCP / Target closed) surviving the retry, or a new mode; re-diagnose here.
<10 runs by then β extend the window rather than concluding from a thin sample.
π€ Generated by the Agentic Engineer\n\n## Measurement decision β stability criterion met\n\nI reproduced the post-#204 reading against the live Actions API rather than inferring from aggregate CI:\n\n- Fix checkpoint: #204 merged as dec84fc on 2026-07-19.\n- Named measurement window: 2026-08-02 through 2026-08-22.\n- Window size: 59 complete CI runs.\n- Lighthouse CI result: 59 success, 0 failure.\n- The two aggregate-red workflows in that window failed in E2E Tests; their Lighthouse CI jobs both completed successfully.\n- Broader post-fix control: 99 CI runs. 93 Lighthouse jobs succeeded. The six job-level failures all stopped before the Lighthouse collector/server path: two deterministic npm dependency-resolution failures and four sub-five-second job-start failures with no job log. None reproduced PROTOCOL_TIMEOUT, NO_FCP, Target closed, server-bind jitter, or a failed lhci collect after the retry.\n\nThat exceeds the requested clean window of roughly 10 runs and answers the experiment rather than merely observing another single pass.\n\nAcceptance status:\n\n- [x] Lighthouse CI runs against the built app in CI.\n- [x] Budgets are documented in lighthouserc.json.\n- [x] Stable over the named multi-run observation window.\n\nDecision: AC3 is met, so this experiment is complete and this issue can close. Graduation from warn-only to required remains separate budget work on #176; this stability result does not claim that the LCP/FCP budgets themselves already pass.
Activity
perf: add non-required Lighthouse CI Core Web Vitals lane (#102)
devantler commentedon Jun 19, 2026
Phase 1 (the warn-only lane) is now on
mainβ@lhci/cli+lighthouserc.json(LCP/CLS/TBT/FCP + Performance β₯ 0.85, allwarn) +scripts/lighthouse.mjs+ a dedicated non-requiredlighthousejob inci.yamlthat builds the app and runsnpm run lhciwithDEV_SKIP_AUTH=true, deliberately kept out ofCI - Required Checks.Acceptance status:
lighthouserc.jsonKeeping this open to track the remaining step: gauge the lane's red/green over ~10 runs against the Node-adapter server (port binding + cold-start jitter was the original deferral reason), then decide whether to promote the budgets into
CI - Required Checks.(Docs were drifting too β AGENTS.md still said the full Lighthouse run was "deferred"; synced in #121.)
devantler commentedon Jun 28, 2026
Stability checkpoint β the third acceptance criterion is empirically not met yet, and I found why.
Over the last 16
β CIruns theLighthouse CI (non-required)lane failed 3Γ (kit-2.66.0,adapter-node-5.5.5, plus a transient) β ~80% green, short of the "no flaky failures over ~10 runs" bar.Root cause (from the failing run's log): the
@sveltejs/adapter-nodeserver exits early with code 13 βDetected unsettled top-level await β¦ await server.init({β¦})β on a cold start, andscripts/lighthouse.mjskept polling the dead port until the 30swaitForServertimeout. It's a race (the same commit passed one run, failed the next), i.e. exactly the port-binding/cold-start jitter that deferred this to warn-only in #95.Draft #148 hardens the lane (abort the poll on early exit + retry the launch up to 3Γ), which should turn the jitter into a stable green signal. Keeping this issue open per its plan: once the lane runs ~10Γ clean post-#148, we can decide on promoting the budgets into
CI - Required Checks.ci: serve the Lighthouse lane via vite preview (e2e parity)
devantler commentedon Jun 28, 2026
Correction + stronger fix in #148. My earlier read (cold-start race, fix by retry) was incomplete: pushing the retry revealed the adapter-node
server.initcode-13 exit is deterministic on the current build when started DB-free β all 3 retries hit it. So the real fix is to stop usingnode build/index.jsfor this lane and serve the build viavite preview, exactly as the e2e lane already does reliably underDEV_SKIP_AUTH. Verified locally:/login200,/(+session cookie) 200,/(no cookie) 302. The early-exit-aware retry is kept for residual bind jitter.Separately worth your eye (not blocking, prod unaffected since
DATABASE_URLis set there): the adapter-node server exits with code 13 (unsettled top-level awaitinawait server.init(...)) when booted without a DB β i.e.node build/index.jswon't come up DB-free. Harmless for this perf lane now that it uses preview, but I can file a separate issue to dig into theserver.inittop-level-await if you'd like.devantler commentedon Jul 11, 2026
Status triage (2026-07-11):
Lighthouse CI (non-required)job, served via the built adapter-node app,DEV_SKIP_AUTH=true, scanning/login+/).lighthouserc.jsonβ all five assertions from this issue are in place (warn level).Promotion is now gated on the budgets actually passing: the warn-mode assertions currently fail on
/(perf 0.66β0.77 vs β₯0.85, FCP 2.1β4.2 s vs β€1.8 s) and are borderline on/login(LCP 2573 ms vs β€2500 ms). Root causes + fix plan tracked in #176; once assertions pass consistently, the final flip (warnβerror + required aggregate) closes this issue.devantler commentedon Jul 12, 2026
Post-#179/#180 Lighthouse read (CI run 29177777760, at the #180 head with responsive gallery images + self-hosted fonts, 2026-07-12 03:11Z):
/β 3391 ms median (β€2500; all samples 3391/4141/4137) and LCP on/loginβ 2865 ms (β€2500; samples 3011/2865/2868). FCP on/loginborderline at 1897 ms (β€1800).So the #176 lever chain (gallery lazy-window β hero preload β responsive images β self-hosted fonts) recovered the perf score but the LCP element itself still renders too late under mobile throttling on both pages. Graduation (warnβerror) stays gated until LCP passes over multi-run evidence (single hosted-runner samples stay noisy β 3 in-run samples here are consistent, but we hold the ~multi-run bar from #148).
Next lever: profile what the LCP element is on each page post-#180 (hero image on
/, likely the form card or heading on/login) and whether render-delay (not resource-load) now dominates β if so the remaining cost is in the critical JS/CSS path, not assets. Tracking continues on #176.devantler commentedon Jul 19, 2026
AC3 measured for the first time
AC1 and AC2 have quietly been done for a while β the
Lighthouse CI (non-required)job runsnpm run lhciagainst the built app, andlighthouserc.jsoncarries all five budgets (LCP 2500 / CLS 0.1 / TBT 200 / FCP 1800 / perf 0.85). What had never actually been taken is AC3's stability reading, so this issue sat at the head of the backlog with no one measuring the thing it asks for.The reading, over the last 40
β CIruns:So: 1 flaky failure in 39 usable runs (~2.6%), not zero. AC3 is not yet met.
The single failure has one root cause, not a budget miss (budgets are
warn-level and cannot fail the job). In run29641332997Chrome launched, then every gatherer hitPROTOCOL_TIMEOUT, the page never painted (NO_FCP), and the browser died withTarget closedβ classic runner starvation.Why it reddened the lane: the lane retried the server launch but treated a failed
lhci collectas terminal.--numberOfRuns=3does not cover it β LHCI aborts the whole collect on its first failed run.PR #204 makes a failed collect retry (a deterministic failure still fails the lane β verified with a negative control).
Next step to close this issue: after #204 merges, observe ~10+ further runs. Clean β AC3 met, this closes, and graduation to required becomes purely #176's budget work. Any further failure β re-diagnose here.
devantler commentedon Jul 19, 2026
#204 merged (
dec84fcf). The lane now retries a failedlhci collect, and CI on the PR itself was green with theLighthouse CI (non-required)job passing.This issue stays open pending the AC3 re-measurement β shipping the fix is not proof it worked. The prior reading was 1 flake in 39 runs (~2.6%); the target is a clean window.
Measurement date: 2026-08-02. On that date, re-run the same reading over the
β CIruns sincedec84fcf:PROTOCOL_TIMEOUT/NO_FCP/Target closed) surviving the retry, or a new mode; re-diagnose here.chore: agent-claim #100 nonce=034d673a23126b4f42b1b3595882c47c
chore: agent-claim #100 nonce=802548c1399eeda842b0511b99fe44b2
devantler commentedon Aug 23, 2026