Skip to content

Production deploys fail only for the change they deploy #4979

Open
20 of 31 issues completed
Open

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

Audience and problem

Everyone who changes the platform (the maintainer and both agent lanes) and every tenant waiting on a platform change. When a deploy gate fails for a reason the candidate did not cause, the candidate is evicted, the next candidate inherits the same fault, and a working change can take hours to land. When a gate passes something broken, production drifts from main unnoticed.

Hypothesis

If every deploy gate judges only the effect of the candidate it is deploying, and names the condition that failed, then evictions caused by something outside the evicted pull request become rare, and a gate never reports success for a step it could not complete.

Success signal

  • Signal: the share of failed_checks merge-queue evictions whose root cause lies outside the evicted pull request.
  • Baseline: not measured yet. The first child classifies the evictions from the last 30 days, so the baseline is a number, not an estimate.
  • Target: under 10% over a 30-day window after the fixes, with every eviction's cause named by the failing step's own output.
  • Guardrail: no gate is removed or made non-blocking, and a candidate that does break production is still evicted and healed. Fixes narrow what a gate looks at; they never stop it looking.

Smallest useful change

Measure first: classify the last 30 days of failed_checks evictions by root cause (candidate's own change, unrelated platform fault, infrastructure transient, timeout) and rank the unrelated causes by how many evictions each caused. Then fix the top cause.

Acceptance criteria

  • An eviction-cause baseline for the last 30 days is recorded on this issue.
  • The children above are fixed or closed with a reason, highest-ranked cause first.
  • A 30-day re-measurement meets the target without breaching the guardrail.

Size: large — about 8 child issues, each small to medium. Decompose further as the baseline names new causes.

Activity

  1. added theissue type on Oct 17, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Oct 17, 2026
  3. 10 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    No fields configured for Epic.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions