Skip to content

ci: shard Playwright E2E by measured suite timings (follow-up to #383) #434

Closed

Description

@dborup

Problem

After #383 parallelised Go tests, E2E and the image check, the Playwright E2E job is now the critical path: about 25 min of the 25–28 min PR runs. With several PRs in flight, jobs also queue behind each other, so total wall time often reaches 40–50 min.

#383 deliberately deferred E2E sharding until we had measured timings ("sharding should follow measured timings from a live run rather than copying upstream's assignments").

Proposal

Split the E2E job into N shards (start with 3) that run in parallel, based on measured per-suite durations from recent green runs:

  • Extract per-suite wall time from the last several green 🎭 Playwright E2E Tests logs (the === E2E SUITE: … === markers), and put the table in the PR.
  • Assign suites to shards to balance total time (greedy longest-first). Keep the assignment in a checked-in manifest, so it is reviewable and updatable.
  • Each shard prepares the same fixture as today (freshen, seed SQL, corescope-migrate, seeds) and runs its own local server on its own port.
  • Fail-closed: the existing 🏗️ Build & Publish Docker Image gate must require every shard to succeed. Skipped, cancelled or empty counts as failure, as ci: parallelize independent checks with fail-closed gate #383 established.
  • A guard test fails CI if a new E2E suite file is not assigned to any shard, so suites cannot silently drop out.
  • Keep the fork guards (9 in deploy.yml, 1 in release-fast-path.yml). No publish or deploy from the fork.

Acceptance

  • The per-suite timing table and the shard assignment are in the PR.
  • A PR-triggered run shows every shard green, and the measured E2E wall time is down versus the current ~25 min. Report before/after for at least two runs.
  • The unassigned-suite guard and the fail-closed gate are tested (mutants shown red).
  • Consider the runner-minute cost: shards repeat the setup. Report total runner minutes before and after.

Activity

  1. dborup commented on Oct 13, 2026

    @dborup
    OwnerAuthor

    Fixed by #436 (merged as 822cd34c1f437920e6ed13508837fb8c5d92ff71). Remaining nits from the review on #436: job-level env now reaches every suite, e2e-report has no timeout-minutes, 1-day artifact retention limits late re-runs, and the manifest could be one suite per line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions