Skip to content

perf: full Lighthouse CI Core Web Vitals budget (deferred from #95)Β #100

Description

@devantler

πŸ€– Generated by the Daily AI Assistant

Problem

#95 asked for a CI performance gate. We shipped the deterministic fallback it sanctioned β€” a brotli client bundle-size budget (bundle-budget.json + scripts/check-bundle-budget.mjs, wired into the Build job). That catches byte-weight regressions without flake, but it does not measure real Core Web Vitals (LCP, CLS, TBT, FCP) or runtime rendering cost.

Proposed direction

Once the E2E environment is stable enough to host a server reliably in CI, add @lhci/cli:

  1. lighthouserc.json with budgets: LCP ≀ 2.5s, CLS ≀ 0.1, TBT ≀ 200ms, FCP ≀ 1.8s, Performance β‰₯ 85.
  2. An lhci:autorun step after build, starting the Node adapter server with DEV_SKIP_AUTH=true, scanning the login + program pages, then stopping.
  3. Run it as a non-required lane first (warn-only / assert.assertions informational) to gauge flakiness against the Node adapter (port binding + cold-start jitter is why it was deferred in perf: Lighthouse CI performance budgetΒ #95) before promoting it into CI - Required Checks.

Acceptance criteria

  • Lighthouse CI runs against the built app in CI.
  • Budgets documented in lighthouserc.json.
  • Stable (no flaky failures over ~10 runs) before it becomes a required check.

Size: M

Follow-up to #95. The bundle-size budget remains the deterministic baseline gate.

Activity

  1. devantler commented on Jun 19, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Assistant

    Phase 1 (the warn-only lane) is now on main β€” @lhci/cli + lighthouserc.json (LCP/CLS/TBT/FCP + Performance β‰₯ 0.85, all warn) + scripts/lighthouse.mjs + a dedicated non-required lighthouse job in ci.yaml that builds the app and runs npm run lhci with DEV_SKIP_AUTH=true, deliberately kept out of CI - Required Checks.

    Acceptance status:

    • Lighthouse CI runs against the built app in CI
    • Budgets documented in lighthouserc.json
    • Stable (no flaky failures over ~10 runs) before it becomes a required check

    Keeping this open to track the remaining step: gauge the lane's red/green over ~10 runs against the Node-adapter server (port binding + cold-start jitter was the original deferral reason), then decide whether to promote the budgets into CI - Required Checks.

    (Docs were drifting too β€” AGENTS.md still said the full Lighthouse run was "deferred"; synced in #121.)

  2. devantler commented on Jun 28, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Assistant

    Stability checkpoint β€” the third acceptance criterion is empirically not met yet, and I found why.

    Over the last 16 βœ… CI runs the Lighthouse CI (non-required) lane failed 3Γ— (kit-2.66.0, adapter-node-5.5.5, plus a transient) β†’ ~80% green, short of the "no flaky failures over ~10 runs" bar.

    Root cause (from the failing run's log): the @sveltejs/adapter-node server exits early with code 13 β€” Detected unsettled top-level await … await server.init({…}) β€” on a cold start, and scripts/lighthouse.mjs kept polling the dead port until the 30s waitForServer timeout. It's a race (the same commit passed one run, failed the next), i.e. exactly the port-binding/cold-start jitter that deferred this to warn-only in #95.

    Draft #148 hardens the lane (abort the poll on early exit + retry the launch up to 3Γ—), which should turn the jitter into a stable green signal. Keeping this issue open per its plan: once the lane runs ~10Γ— clean post-#148, we can decide on promoting the budgets into CI - Required Checks.

  3. devantler commented on Jun 28, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Assistant

    Correction + stronger fix in #148. My earlier read (cold-start race, fix by retry) was incomplete: pushing the retry revealed the adapter-node server.init code-13 exit is deterministic on the current build when started DB-free β€” all 3 retries hit it. So the real fix is to stop using node build/index.js for this lane and serve the build via vite preview, exactly as the e2e lane already does reliably under DEV_SKIP_AUTH. Verified locally: /login 200, / (+session cookie) 200, / (no cookie) 302. The early-exit-aware retry is kept for residual bind jitter.

    Separately worth your eye (not blocking, prod unaffected since DATABASE_URL is set there): the adapter-node server exits with code 13 (unsettled top-level await in await server.init(...)) when booted without a DB β€” i.e. node build/index.js won't come up DB-free. Harmless for this perf lane now that it uses preview, but I can file a separate issue to dig into the server.init top-level-await if you'd like.

  4. devantler commented on Jul 11, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Assistant

    Status triage (2026-07-11):

    Promotion is now gated on the budgets actually passing: the warn-mode assertions currently fail on / (perf 0.66–0.77 vs β‰₯0.85, FCP 2.1–4.2 s vs ≀1.8 s) and are borderline on /login (LCP 2573 ms vs ≀2500 ms). Root causes + fix plan tracked in #176; once assertions pass consistently, the final flip (warnβ†’error + required aggregate) closes this issue.

  5. devantler commented on Jul 12, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Engineer

    Post-#179/#180 Lighthouse read (CI run 29177777760, at the #180 head with responsive gallery images + self-hosted fonts, 2026-07-12 03:11Z):

    • Now passing (previously failing): performance score (was 0.66–0.77 vs β‰₯0.85 β€” no longer warned), TBT, CLS.
    • Still over budget: LCP on / β€” 3391 ms median (≀2500; all samples 3391/4141/4137) and LCP on /login β€” 2865 ms (≀2500; samples 3011/2865/2868). FCP on /login borderline at 1897 ms (≀1800).

    So the #176 lever chain (gallery lazy-window → hero preload → responsive images → self-hosted fonts) recovered the perf score but the LCP element itself still renders too late under mobile throttling on both pages. Graduation (warn→error) stays gated until LCP passes over multi-run evidence (single hosted-runner samples stay noisy — 3 in-run samples here are consistent, but we hold the ~multi-run bar from #148).

    Next lever: profile what the LCP element is on each page post-#180 (hero image on /, likely the form card or heading on /login) and whether render-delay (not resource-load) now dominates β€” if so the remaining cost is in the critical JS/CSS path, not assets. Tracking continues on #176.

  6. self-assigned this
    on Jul 19, 2026
  7. devantler commented on Jul 19, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Engineer

    AC3 measured for the first time

    AC1 and AC2 have quietly been done for a while β€” the Lighthouse CI (non-required) job runs npm run lhci against the built app, and lighthouserc.json carries all five budgets (LCP 2500 / CLS 0.1 / TBT 200 / FCP 1800 / perf 0.85). What had never actually been taken is AC3's stability reading, so this issue sat at the head of the backlog with no one measuring the thing it asks for.

    The reading, over the last 40 βœ… CI runs:

    Outcome Count
    Lighthouse job success 38
    Cancelled (superseded push β€” not a lane signal) 1
    Failure 1

    So: 1 flaky failure in 39 usable runs (~2.6%), not zero. AC3 is not yet met.

    The single failure has one root cause, not a budget miss (budgets are warn-level and cannot fail the job). In run 29641332997 Chrome launched, then every gatherer hit PROTOCOL_TIMEOUT, the page never painted (NO_FCP), and the browser died with Target closed β€” classic runner starvation.

    Why it reddened the lane: the lane retried the server launch but treated a failed lhci collect as terminal. --numberOfRuns=3 does not cover it β€” LHCI aborts the whole collect on its first failed run.

    PR #204 makes a failed collect retry (a deterministic failure still fails the lane β€” verified with a negative control).

    Next step to close this issue: after #204 merges, observe ~10+ further runs. Clean β†’ AC3 met, this closes, and graduation to required becomes purely #176's budget work. Any further failure β†’ re-diagnose here.

  8. devantler commented on Jul 19, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Daily AI Engineer

    #204 merged (dec84fcf). The lane now retries a failed lhci collect, and CI on the PR itself was green with the Lighthouse CI (non-required) job passing.

    This issue stays open pending the AC3 re-measurement β€” shipping the fix is not proof it worked. The prior reading was 1 flake in 39 runs (~2.6%); the target is a clean window.

    Measurement date: 2026-08-02. On that date, re-run the same reading over the βœ… CI runs since dec84fcf:

    • β‰₯10 runs, zero Lighthouse failures β†’ AC3 met, close this issue, and graduation to required becomes purely perf: pass the Core Web Vitals budgets so the Lighthouse lane can graduate to requiredΒ #176's budget work.
    • Any failure β†’ pull that run's Lighthouse job log and check whether it is the same Chrome/CDP class (PROTOCOL_TIMEOUT / NO_FCP / Target closed) surviving the retry, or a new mode; re-diagnose here.
    • <10 runs by then β†’ extend the window rather than concluding from a thin sample.
  9. moved this to πŸ“₯ Backlog in 🌊 Project Boardon Jul 25, 2026
  10. added 2 commits that reference this issue on Aug 23, 2026
  11. devantler commented on Aug 23, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer\n\n## Measurement decision β€” stability criterion met\n\nI reproduced the post-#204 reading against the live Actions API rather than inferring from aggregate CI:\n\n- Fix checkpoint: #204 merged as dec84fc on 2026-07-19.\n- Named measurement window: 2026-08-02 through 2026-08-22.\n- Window size: 59 complete CI runs.\n- Lighthouse CI result: 59 success, 0 failure.\n- The two aggregate-red workflows in that window failed in E2E Tests; their Lighthouse CI jobs both completed successfully.\n- Broader post-fix control: 99 CI runs. 93 Lighthouse jobs succeeded. The six job-level failures all stopped before the Lighthouse collector/server path: two deterministic npm dependency-resolution failures and four sub-five-second job-start failures with no job log. None reproduced PROTOCOL_TIMEOUT, NO_FCP, Target closed, server-bind jitter, or a failed lhci collect after the retry.\n\nThat exceeds the requested clean window of roughly 10 runs and answers the experiment rather than merely observing another single pass.\n\nAcceptance status:\n\n- [x] Lighthouse CI runs against the built app in CI.\n- [x] Budgets are documented in lighthouserc.json.\n- [x] Stable over the named multi-run observation window.\n\nDecision: AC3 is met, so this experiment is complete and this issue can close. Graduation from warn-only to required remains separate budget work on #176; this stability result does not claim that the LCP/FCP budgets themselves already pass.

  12. moved this from πŸ“₯ Backlog to βœ… Done in 🌊 Project Boardon Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Performance.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions