fix(ci): retry runs that failed before executing any of our code - #280
Merged
Conversation
The sweep retries CANCELLED checks but deliberately leaves genuine failures alone. A GitHub Actions incident lands in the gap between those two: the run fails with conclusion=failure, so it is treated as a real verdict — but it never executed a line of this repo's code. Consequence: during an incident EVERY open PR is stranded permanently, and nobody is in the merge loop to notice. Today (2026-08-06, Actions major outage, webhooks throttled to ~15%) three jobs on PR #278 died with "Failed to resolve action download info. Error: Service Unavailable" and that PR cannot become green again without a human running `gh run rerun --failed`. The discriminator is precise: a job whose ONLY failed step is "Set up job" never got as far as running our code, so it produced no verdict about it. A real failure names a real step. Verified against live runs: PR #278 outage run -> [Set up job] -> infra PR #228 (TS 7) -> [Verify (lint + umlauts + typecheck + build)] -> real PR #265 (ESLint 10) -> [Verify (lint + umlauts + typecheck + build)] -> real nonexistent run -> no data -> refuses to guess Conservative by construction: retries only when EVERY failed job in the run failed at set-up. One real step failure anywhere and the PR is left alone. Capped at MAX_RUN_ATTEMPTS (default 3) — an incident can last hours, and an uncapped retry would re-run the same doomed run every sweep forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Re-firing CI: this PR's |
The sweep already knew that a cancelled/infra-failed run is noise rather than a judgement — but only for PR checks. The base branch kept the original "refuse and exit" behaviour, which deadlocks: the only thing that produces a new CI run on main is a merge, and merges are exactly what the guard blocks. Observed here on 2026-08-07. An Actions incident left main's run `failure` with no failed job at all (Migration Drift cancelled, everything else green). Eleven PRs sat for ~14h while every sweep exited 0 and looked healthy. So: extract the "is this a verdict about the code?" test and apply it to the base run too. A genuine failure still blocks — that IS a verdict. Retries are capped by the run's own attempt counter, which is why this re-runs rather than dispatching fresh (a new dispatch resets to attempt 1 and could churn forever). Also drops `--failed` from the PR-side retry. A partial re-run flips SKIPPED jobs to CANCELLED, so the PR ends up non-green for a brand new reason and needs yet another retry — seen on #278 today. Tested by running the real script against a fake `gh` on PATH, so this covers shipped control flow rather than a re-description of it. Mutation-checked: disabling the self-heal turns 3 of the 6 tests red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 7, 2026
#283) Three consecutive sweeps read #278 and #282 as MERGEABLE/CLEAN, announced the merge, and were refused: X Pull request #278 is not mergeable: the base branch policy prohibits the merge. There is no such policy. main has branch protection with every option disabled, no rulesets, no CODEOWNERS, no required checks and no required reviews. Throughout the same window the bot merged #225, #226, #279 and #280 without trouble — including #225/#226, which modify .github/workflows/, so this is not the token's workflows scope either. The refusal is gh's client-side precheck reading the mergeStateStatus that GITHUB_TOKEN sees, which is not the one a PAT sees — nothing this script can inspect its way around. So log both what we saw and delegate: native auto-merge is GitHub performing the merge itself, and does not go through that precheck. It is the escape hatch gh names in its own error message. Still one car per sweep — the fallback is followed by the same `break`, so at most one PR per sweep is handed over, and the green-base guard is unchanged. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
The sweep retries
CANCELLEDchecks but deliberately leaves genuine failures alone. A GitHub Actions incident lands exactly in the gap between those two: the run reportsconclusion=failure, so it is treated as a real verdict — but it never executed a line of this repo's code.The consequence is not a slow queue, it is a permanent stall. During an incident every open PR is stranded, and since nobody is in the merge loop, nobody notices.
This is live right now. Actions has been in a major outage since 15:22Z (webhooks throttled to ~15%). Three jobs on #278 died with:
That PR cannot go green again without a human running
gh run rerun --failed— which is precisely the "someone must be watching" assumption this whole auto-merge setup exists to remove.The discriminator
A job whose only failed step is
Set up jobnever got as far as running our code — GitHub could not resolve an action, provision the runner, or start the container. It produced no verdict about the code. A real failure names a real step.Verified against live runs in this repo:
[Set up job][Verify (lint + umlauts + typecheck + build)][Verify (lint + umlauts + typecheck + build)]Safety
MAX_RUN_ATTEMPTS(default 3). An incident can last hours; an uncapped retry would re-run the same doomed run every 10 minutes indefinitely and bury the real signal.actions: writeis already granted for the existing cancelled-run retry.Note on fleet drift
My notes said this script was byte-identical across the fleet. That is no longer true — orangecat 190 lines, fleetcrown 232, evig 208 (evig has #275's oldest-first fix). This change is evig-only and worth porting to the other two, since the failure mode is not evig-specific.
🤖 Generated with Claude Code