CI: runner hangs in install/build steps; e2e workflow has no step timeouts so hung shards sit for hours

[tomsmith8]

What happened

On 2026-08-19 from ~15:11 UTC, GitHub-hosted runners (ubuntu-latest) began hanging in network-heavy setup steps across every branch, including master — no correlation with any diff. githubstatus.com showed no incident (typical for short-lived runner/registry degradations).

Two workflows, two very different failure shapes from the same cause:

Workflow	Step that hangs	Behavior
Deploy-Preview (deployPR.yml)	Install Dependencies	Fails loudly at exactly 10:00 — the step is wrapped in nick-fields/retry@v3 with timeout_minutes: 10, so the hung install is killed (Error: kill EPERM) and retried
Playwright e2e (e2e.yml)	Build app and setup database	Sits silently — no timeout-minutes on any step, so a hung shard runs until the 6-hour job default and the PR just looks stuck

Evidence

• Deploy-Preview failing repo-wide, then recovering on retry: https://github.com/stakwork/hive/actions/runs/32269577633 (killed at exactly 10:00 in Install Dependencies, twice, then passed)
• Playwright shard 3/3 hung twice on the same PR while shards 1/3 and 2/3 finished in ~6 min: https://github.com/stakwork/hive/actions/runs/32269577461 — first hang sat >1h, second sat 40 min in "Build app and setup database"; both cancelled and re-run manually
• Same-day failures on unrelated branches (feature/...-immutable-docx-editor-state-machine, feature/...-scoped-node-resolution-api) and master at 15:11 UTC

Impact

A hung e2e shard blocks auto-merge indefinitely with no red X — it is indistinguishable from a slow run without inspecting job steps. PR #5042 lost several hours to this today.

Proposed fix

Mirror the Deploy-Preview hardening in e2e.yml:

1. Add timeout-minutes to the install/build/test steps (healthy shards complete the whole job in ~6 min; 15 would be generous)
2. Optionally wrap the install/build steps in the same nick-fields/retry@v3 used by deployPR.yml so transient runner sickness self-heals
3. Consider a job-level timeout-minutes: 30 as a backstop so no shard can ever occupy a runner for 6 hours

No product code involved — workflow YAML only.

[tomsmith8]

Root cause found for the e2e hangs — it's more specific than general runner sickness.

Cancelled the hung shard and pulled its log. The cleanup phase shows the process that was actually stuck inside "Build app and setup database":

Terminate orphan process: pid (2879) (npm exec playwright install-deps chromium)

playwright install-deps is an apt-get call for Chromium's system libraries. It runs on every job and is not covered by the Playwright browser cache (that cache only skips playwright install, hence "Install Playwright Browsers: skipped" while install-deps still runs). When the runner's route to the Ubuntu package mirrors degrades — as it did repo-wide today — apt hangs indefinitely, and with no step timeout the shard sits until the 6-hour job default.

The recurring FATAL: role "root" does not exist lines in the postgres service container are health-check noise present in passing runs too — red herring.

Sharpened recommendations:

1. timeout-minutes on the build/setup step (as above) — turns a 6-hour silent hang into a 10-minute retryable failure
2. Avoid the uncached apt call per shard: either bake deps via npx playwright install --with-deps chromium under the existing cache key discipline, run e2e in the mcr.microsoft.com/playwright container image (deps preinstalled, no apt at job time), or cache the apt archives
3. The retry wrapper from deployPR.yml composes with either