Repository navigation
Description
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
Evidence
Measured on 13 November 2026 from 930 merge-queue CI runs and 800 queued pull requests since 17 October, and every workflow job in nine organisation repositories since 12 November.
- The queue itself is fast when runners are free. Since ci: preserve production deploy order while merge groups validate #6036 a queued change validates while the change ahead of it deploys. On 12 November the queue merged one change every 13.5 minutes while it was backlogged, and a change waited 25 minutes (median) from queueing to merging, across 45 merges.
- On 13 November the same queue took 38.8 minutes per change, and a change waited 286 minutes (median). Nothing in the queue or the deploy changed. The deploy itself still took about 12.5 minutes.
- The difference is jobs waiting for a runner. Between two production deploys three jobs each need one: the final required check of the change that just deployed, a post-merge observer that holds the production lock while it waits (0.6 minutes of work), and the deploy job of the next change. Each normally starts within seconds. On 13 November the required check waited 8.1 minutes and the next deploy 13.7 (medians), 9.2 of them behind the observer's lock, with production idle.
- The organisation's runner allowance was full. The plan allows 60 jobs at once. From 12 November 17:00 UTC to 13 November 10:40 UTC the organisation averaged 57.7 running jobs and sat at 55 or more for three quarters of the time. Pull-request checks used most of it:
ksail54% of job time (its system tests alone 23%),platform32%. - This has happened before. ksail#7579 and A burst of pull requests here starves the organization's runners and delayed a production deploy by an hour .github#475 recorded the same stall on 5 and 6 October, and the runner waits show up again on 3, 4 and 10 November.
- Four in ten production deploys change nothing in production. 244 of 633 deploys in the window published manifests identical to
main: the change touched only scripts, tests, CI files or documentation. Each still took the production lane for about 11 minutes, or about 40 minutes when runners were scarce. - Half of the evictions were not the evicted change's fault. 49 of 98 evicted changes later merged without any new commit. Failed deploys and the recovery that follows them used 18% of the production lane's time.
The full numbers and method are in the decision record (linked below once merged).
Audience and problem
The maintainer, both agent lanes, and every tenant waiting on a platform change. Production changes go through one lane, one at a time. When pull-request checks anywhere in the organisation fill the runner allowance, that lane spends most of its time waiting for a runner, the queue grows faster than it drains, and a reviewed, green change takes six to twelve hours to reach production.
Hypothesis
If changes that alter no production input stop taking the production lane, and the organisation's pull-request checks stop running at the runner limit, then the queue keeps up with a busy day even when runners are scarce, without any change merging with less proof than today.
Success signal
- Signal: median time from queueing to merging on a day with 40 or more merges, and the time from one production deploy ending to the next starting.
- Baseline: 286 minutes on 13 November (runners full) and 25 minutes on 12 November (runners free); 29.5 minutes from one deploy ending to the next starting on 13 November, against 1.6 on 12 November, when runners were free.
- Target: under 45 minutes median from queueing to merging on any day with up to 60 merges, including days when pull-request checks are busy; under 5 minutes at the 90th percentile from one deploy ending to the next starting.
- Measure on: the first two days with 40 or more merges after the first child lands.
- Guardrail: every change that alters a production input is still deployed to production and verified before it merges. No required check is removed, skipped or made non-blocking, and no timeout is raised to hide a wait.
Plan
In order, each taken only as far as the numbers require:
- Do not deploy when no production input changed (Avoid production redeployment when only CI validation inputs change #5077). The first delivery removes about one deploy in eight (74 of 629 replayed); the ceiling is about one in three.
- Do less runner work per pull request where it changes no evidence: the duplicate Go validation here, runs cancelled by the next push, and the two repositories that fill the allowance (ksail#7579, A burst of pull requests here starves the organization's runners and delayed a production deploy by an hour .github#475).
- Shorten the wait between two deploys (The next production deploy waits behind a post-merge observer and two fresh runners #6205, Queued changes wait up to 15 minutes after validation for earlier production deploys #6139), starting with the observer that holds the production lock while it waits for a runner.
- Keep it measured, so the next stall is seen as a number and not found by hand.
False evictions stay with #4979.
Not chosen
- One deploy for several changes. GitHub's queue runs one CI run per change, so a shared deploy needs coordination between runs and loses the answer to "which change broke production". Revisit only if the lane is busy more than 70% of a week after step 1.
- Building more queued changes at once. Validation already finishes inside the deploy of the change ahead.
- Proving changes on a separate cluster instead of production. There is no production-like cluster, and it would be weaker proof.
- Capping how many runners
ksailsystem tests may take. Simulated on the measured window: the allowance stays full, because the same work still has to run. - Self-hosted runners for the deploy. The organisation pool is not active, and a deploy that restarts cluster nodes should not run on that cluster.
Size
Large. Step 1 is one careful change here plus a production proof; the rest are small and independent.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields📥 Backlog
Activity
docs(adr): separate lock time from runner time and correct the forecast
docs(adr): count every lock holder in the deploy wait split
docs(adr): correct the figures an independent review could not reproduce
docs(adr): correct seven figures and two claims found in review
docs(adr): require every deploy path to run from the list before any …
docs(adr): say why seven pairs and nine deploys differ, and which del…