Skip to content

Production delivery stalls for hours when the organisation's runners are full #6203

Open
0 of 7 issues completed

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

Measured on 13 November 2026 from 930 merge-queue CI runs and 800 queued pull requests since 17 October, and every workflow job in nine organisation repositories since 12 November.

  • The queue itself is fast when runners are free. Since ci: preserve production deploy order while merge groups validate #6036 a queued change validates while the change ahead of it deploys. On 12 November the queue merged one change every 13.5 minutes while it was backlogged, and a change waited 25 minutes (median) from queueing to merging, across 45 merges.
  • On 13 November the same queue took 38.8 minutes per change, and a change waited 286 minutes (median). Nothing in the queue or the deploy changed. The deploy itself still took about 12.5 minutes.
  • The difference is jobs waiting for a runner. Between two production deploys three jobs each need one: the final required check of the change that just deployed, a post-merge observer that holds the production lock while it waits (0.6 minutes of work), and the deploy job of the next change. Each normally starts within seconds. On 13 November the required check waited 8.1 minutes and the next deploy 13.7 (medians), 9.2 of them behind the observer's lock, with production idle.
  • The organisation's runner allowance was full. The plan allows 60 jobs at once. From 12 November 17:00 UTC to 13 November 10:40 UTC the organisation averaged 57.7 running jobs and sat at 55 or more for three quarters of the time. Pull-request checks used most of it: ksail 54% of job time (its system tests alone 23%), platform 32%.
  • This has happened before. ksail#7579 and A burst of pull requests here starves the organization's runners and delayed a production deploy by an hour .github#475 recorded the same stall on 5 and 6 October, and the runner waits show up again on 3, 4 and 10 November.
  • Four in ten production deploys change nothing in production. 244 of 633 deploys in the window published manifests identical to main: the change touched only scripts, tests, CI files or documentation. Each still took the production lane for about 11 minutes, or about 40 minutes when runners were scarce.
  • Half of the evictions were not the evicted change's fault. 49 of 98 evicted changes later merged without any new commit. Failed deploys and the recovery that follows them used 18% of the production lane's time.

The full numbers and method are in the decision record (linked below once merged).

Audience and problem

The maintainer, both agent lanes, and every tenant waiting on a platform change. Production changes go through one lane, one at a time. When pull-request checks anywhere in the organisation fill the runner allowance, that lane spends most of its time waiting for a runner, the queue grows faster than it drains, and a reviewed, green change takes six to twelve hours to reach production.

Hypothesis

If changes that alter no production input stop taking the production lane, and the organisation's pull-request checks stop running at the runner limit, then the queue keeps up with a busy day even when runners are scarce, without any change merging with less proof than today.

Success signal

  • Signal: median time from queueing to merging on a day with 40 or more merges, and the time from one production deploy ending to the next starting.
  • Baseline: 286 minutes on 13 November (runners full) and 25 minutes on 12 November (runners free); 29.5 minutes from one deploy ending to the next starting on 13 November, against 1.6 on 12 November, when runners were free.
  • Target: under 45 minutes median from queueing to merging on any day with up to 60 merges, including days when pull-request checks are busy; under 5 minutes at the 90th percentile from one deploy ending to the next starting.
  • Measure on: the first two days with 40 or more merges after the first child lands.
  • Guardrail: every change that alters a production input is still deployed to production and verified before it merges. No required check is removed, skipped or made non-blocking, and no timeout is raised to hide a wait.

Plan

In order, each taken only as far as the numbers require:

  1. Do not deploy when no production input changed (Avoid production redeployment when only CI validation inputs change #5077). The first delivery removes about one deploy in eight (74 of 629 replayed); the ceiling is about one in three.
  2. Do less runner work per pull request where it changes no evidence: the duplicate Go validation here, runs cancelled by the next push, and the two repositories that fill the allowance (ksail#7579, A burst of pull requests here starves the organization's runners and delayed a production deploy by an hour .github#475).
  3. Shorten the wait between two deploys (The next production deploy waits behind a post-merge observer and two fresh runners #6205, Queued changes wait up to 15 minutes after validation for earlier production deploys #6139), starting with the observer that holds the production lock while it waits for a runner.
  4. Keep it measured, so the next stall is seen as a number and not found by hand.

False evictions stay with #4979.

Not chosen

  • One deploy for several changes. GitHub's queue runs one CI run per change, so a shared deploy needs coordination between runs and loses the answer to "which change broke production". Revisit only if the lane is busy more than 70% of a week after step 1.
  • Building more queued changes at once. Validation already finishes inside the deploy of the change ahead.
  • Proving changes on a separate cluster instead of production. There is no production-like cluster, and it would be weaker proof.
  • Capping how many runners ksail system tests may take. Simulated on the measured window: the allowance stays full, because the same work still has to run.
  • Self-hosted runners for the deploy. The organisation pool is not active, and a deploy that restarts cluster nodes should not run on that cluster.

Size

Large. Step 1 is one careful change here plus a production proof; the rest are small and independent.

Activity

  1. added theissue type on Nov 13, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 13, 2026
  3. updated Priority High and Effort High on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    High

    Effort

    High

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions