Repository navigation
Description
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
Evidence
- Every platform change reaches
mainthrough the merge queue, and each queue candidate deploys to production before it merges. A candidate whose deploy fails is evicted. - In the 30 days to 2026-10-17,
maintook 689 commits. Leaving out dependency updates, the two largest commit scopes werefix(ci)(48) andfix(deploy)(22): about one delivery-pipeline repair a day. - Open bugs where a deploy fails, stalls or is judged wrongly for reasons outside the change being deployed:
- Production deploys fail intermittently at the data product rollout check and evict unrelated pull requests #4545 — deploys fail intermittently at the data product rollout check and evict unrelated pull requests
- fix(deploy): an unrelated Coroot fault must not refuse every production deploy #4659 — an unrelated Coroot fault refuses every production deploy
- Avoid redeploying production after an evicted candidate was refused before deployment #4937 — production is redeployed after an evicted candidate was refused before it deployed
- Deploy GHCR step can time out on the Flux policy pause without saying which condition never held #4873 — the GHCR step times out on the Flux policy pause without naming the condition that never held
- Talos node upgrades can exceed the 10-minute drain timeout, evicting the bump from the merge queue #3388 — Talos node upgrades can exceed the 10-minute drain timeout, which evicts the bump
- The deploy bridge loses its own sync lease mid-run, and each failure fences a node that fails the next deploy #3071 — the deploy bridge loses its own sync lease mid-run
- A deploy passes its Flux step when Flux could not delete a removed resource #4734 — a deploy passes its Flux step when Flux could not delete a removed resource (the opposite failure: a broken deploy judged healthy)
- Complete the Platform release left after a rejected notes push #4839 — a Platform release is left incomplete after a rejected notes push
- No current epic owns this class: the earlier delivery-correctness theme on Roadmap: a resilient, current, well-abstracted GitOps platform #2043 has closed all its children.
Audience and problem
Everyone who changes the platform (the maintainer and both agent lanes) and every tenant waiting on a platform change. When a deploy gate fails for a reason the candidate did not cause, the candidate is evicted, the next candidate inherits the same fault, and a working change can take hours to land. When a gate passes something broken, production drifts from main unnoticed.
Hypothesis
If every deploy gate judges only the effect of the candidate it is deploying, and names the condition that failed, then evictions caused by something outside the evicted pull request become rare, and a gate never reports success for a step it could not complete.
Success signal
- Signal: the share of
failed_checksmerge-queue evictions whose root cause lies outside the evicted pull request. - Baseline: not measured yet. The first child classifies the evictions from the last 30 days, so the baseline is a number, not an estimate.
- Target: under 10% over a 30-day window after the fixes, with every eviction's cause named by the failing step's own output.
- Guardrail: no gate is removed or made non-blocking, and a candidate that does break production is still evicted and healed. Fixes narrow what a gate looks at; they never stop it looking.
Smallest useful change
Measure first: classify the last 30 days of failed_checks evictions by root cause (candidate's own change, unrelated platform fault, infrastructure transient, timeout) and rank the unrelated causes by how many evictions each caused. Then fix the top cause.
Acceptance criteria
- An eviction-cause baseline for the last 30 days is recorded on this issue.
- The children above are fixed or closed with a reason, highest-ranked cause first.
- A 30-day re-measurement meets the target without breaching the guardrail.
Size: large — about 8 child issues, each small to medium. Decompose further as the baseline names new causes.
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- StatusShow more project fields📥 Backlog
Activity
10 remaining items