Skip to content

fix(ci): retry runs that failed before executing any of our code - #280

Merged
github-actions[bot] merged 2 commits into
mainfrom
ci/retry-infra-failures
Aug 7, 2026
Merged

fix(ci): retry runs that failed before executing any of our code#280
github-actions[bot] merged 2 commits into
mainfrom
ci/retry-infra-failures

Conversation

@catomean

@catomean catomean commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

The gap

The sweep retries CANCELLED checks but deliberately leaves genuine failures alone. A GitHub Actions incident lands exactly in the gap between those two: the run reports conclusion=failure, so it is treated as a real verdict — but it never executed a line of this repo's code.

The consequence is not a slow queue, it is a permanent stall. During an incident every open PR is stranded, and since nobody is in the merge loop, nobody notices.

This is live right now. Actions has been in a major outage since 15:22Z (webhooks throttled to ~15%). Three jobs on #278 died with:

Failed to resolve action download info. Error: Service Unavailable

That PR cannot go green again without a human running gh run rerun --failed — which is precisely the "someone must be watching" assumption this whole auto-merge setup exists to remove.

The discriminator

A job whose only failed step is Set up job never got as far as running our code — GitHub could not resolve an action, provision the runner, or start the container. It produced no verdict about the code. A real failure names a real step.

Verified against live runs in this repo:

Run Failed steps Verdict
#278, outage [Set up job] infra → retry
#228, TypeScript 7 [Verify (lint + umlauts + typecheck + build)] real → leave alone
#265, ESLint 10 [Verify (lint + umlauts + typecheck + build)] real → leave alone
nonexistent run id no data refuses to guess

Safety

  • Conservative by construction: retries only when every failed job in the run failed at set-up. One real step failure anywhere and the PR is left alone.
  • Refuses to guess: if the jobs API returns nothing or is unreachable, it returns "not infra". Guessing the other way would re-run real failures forever.
  • Capped at MAX_RUN_ATTEMPTS (default 3). An incident can last hours; an uncapped retry would re-run the same doomed run every 10 minutes indefinitely and bury the real signal.
  • Needs no new permissions — actions: write is already granted for the existing cancelled-run retry.

Note on fleet drift

My notes said this script was byte-identical across the fleet. That is no longer true — orangecat 190 lines, fleetcrown 232, evig 208 (evig has #275's oldest-first fix). This change is evig-only and worth porting to the other two, since the failure mode is not evig-specific.

🤖 Generated with Claude Code

The sweep retries CANCELLED checks but deliberately leaves genuine failures
alone. A GitHub Actions incident lands in the gap between those two: the run
fails with conclusion=failure, so it is treated as a real verdict — but it
never executed a line of this repo's code.

Consequence: during an incident EVERY open PR is stranded permanently, and
nobody is in the merge loop to notice. Today (2026-08-06, Actions major
outage, webhooks throttled to ~15%) three jobs on PR #278 died with "Failed
to resolve action download info. Error: Service Unavailable" and that PR
cannot become green again without a human running `gh run rerun --failed`.

The discriminator is precise: a job whose ONLY failed step is "Set up job"
never got as far as running our code, so it produced no verdict about it.
A real failure names a real step.

Verified against live runs:
  PR #278 outage run  -> [Set up job]                                -> infra
  PR #228 (TS 7)      -> [Verify (lint + umlauts + typecheck + build)] -> real
  PR #265 (ESLint 10) -> [Verify (lint + umlauts + typecheck + build)] -> real
  nonexistent run     -> no data -> refuses to guess

Conservative by construction: retries only when EVERY failed job in the run
failed at set-up. One real step failure anywhere and the PR is left alone.

Capped at MAX_RUN_ATTEMPTS (default 3) — an incident can last hours, and an
uncapped retry would re-run the same doomed run every sweep forever.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@catomean

catomean commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Re-firing CI: this PR's pull_request webhook was dropped during the GitHub Actions incident of 2026-08-06 (webhooks throttled to ~15%; GitHub's closing update states the missed events "cannot be replayed automatically"). No workflow run ever existed for head 875e7609, so the auto-merge sweep skipped it indefinitely with no checks reported yet. Closing and reopening replays the event.

@catomean catomean closed this Aug 7, 2026
@catomean catomean reopened this Aug 7, 2026
The sweep already knew that a cancelled/infra-failed run is noise rather than
a judgement — but only for PR checks. The base branch kept the original
"refuse and exit" behaviour, which deadlocks: the only thing that produces a
new CI run on main is a merge, and merges are exactly what the guard blocks.

Observed here on 2026-08-07. An Actions incident left main's run `failure`
with no failed job at all (Migration Drift cancelled, everything else green).
Eleven PRs sat for ~14h while every sweep exited 0 and looked healthy.

So: extract the "is this a verdict about the code?" test and apply it to the
base run too. A genuine failure still blocks — that IS a verdict. Retries are
capped by the run's own attempt counter, which is why this re-runs rather than
dispatching fresh (a new dispatch resets to attempt 1 and could churn forever).

Also drops `--failed` from the PR-side retry. A partial re-run flips SKIPPED
jobs to CANCELLED, so the PR ends up non-green for a brand new reason and needs
yet another retry — seen on #278 today.

Tested by running the real script against a fake `gh` on PATH, so this covers
shipped control flow rather than a re-description of it. Mutation-checked:
disabling the self-heal turns 3 of the 6 tests red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit ca6f807 into main Aug 7, 2026
8 checks passed
@github-actions
github-actions Bot deleted the ci/retry-infra-failures branch August 7, 2026 08:19
github-actions Bot pushed a commit that referenced this pull request Aug 7, 2026
#283)

Three consecutive sweeps read #278 and #282 as MERGEABLE/CLEAN, announced the
merge, and were refused:

    X Pull request #278 is not mergeable:
      the base branch policy prohibits the merge.

There is no such policy. main has branch protection with every option
disabled, no rulesets, no CODEOWNERS, no required checks and no required
reviews. Throughout the same window the bot merged #225, #226, #279 and #280
without trouble — including #225/#226, which modify .github/workflows/, so
this is not the token's workflows scope either.

The refusal is gh's client-side precheck reading the mergeStateStatus that
GITHUB_TOKEN sees, which is not the one a PAT sees — nothing this script can
inspect its way around. So log both what we saw and delegate: native
auto-merge is GitHub performing the merge itself, and does not go through that
precheck. It is the escape hatch gh names in its own error message.

Still one car per sweep — the fallback is followed by the same `break`, so at
most one PR per sweep is handed over, and the green-base guard is unchanged.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant