Repository navigation
Description
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
Evidence
- Production deploys run one at a time. Between two of them, three jobs each need a fresh runner, one after the other:
- the final required check of the change that just deployed;
Observe Production Convergenceinvalidate-main.yaml, which starts on every push tomain, takes the production lock and holds it while it waits for a runner. Its own work takes 0.6 minutes;- the deploy job of the next change, which waits for that lock and then for a runner of its own.
- When runners are free the hand-off takes 1.5 minutes (median, 28 hand-offs from 5 to 12 November) and a deploy-to-deploy cycle takes 14.3 minutes. On 12 November, 17 of 38 deploys still queued behind the observer, for about 0.9 minutes each.
- On 13 November the hand-off took 29.5 minutes and the cycle 39.8 minutes. The deploy itself was unchanged.
| 13 November, medians | Minutes |
|---|---|
| Required check, wait for a runner | 8.1 |
| Observer, wait for a runner while holding the lock | 10.6 |
| Next deploy job, total wait | 13.7 |
| of which behind the observer's lock | 9.2 |
| of which for its own runner | 6.7 |
13 of that day's 14 deploys were queued behind the observer. The prod environment has no reviewers and no wait timer, so none of this is an approval wait.
Problem
The one part of delivery that cannot run in parallel pays for a busy runner pool three times per change, and one of the three is a 0.6-minute read that holds the production lock while it waits.
Options to evaluate
Each must keep today's order and proof: a change deploys only when every change ahead of it is on main, and merges only after its own deploy is verified.
- Take the observer out of the hand-off. Do this first: it changes no merge proof. The observer exists to catch production falling behind
mainwhen nothing else will redeploy it (A merge-group heal can leave prod behind a merge that lands just after it #2838, Redeploy main after a merge when prod has not converged on it #3848). While another change is queued, that change's own deploy or recovery brings production up to date, so the observation only matters once the queue is empty. Candidates: observe only when the queue is empty at the push, with a way to still observe if the remaining queued changes leave without merging; or keep observing on every push but run the observer's own self-tests outside the lock. Note that making the observer start later does not help under sustained load: each merge still adds one observer that holds the lock while it waits for a runner. - Overlap the two remaining waits. Let the next change start looking for its deploy runner when the previous deploy succeeds, and still check under the production lock that it is first on
mainbefore it touches production. Saves the shorter of the two waits. - Take the final check off the shared pool. It runs no repository code and needs no secrets, so it is the natural first job for a dedicated runner once the organisation pool is active (Establish a zero-runner baseline before reopening ARC #4749).
- Start the final check's runner during the deploy. Removes its wait, but the check would read job results from the API and not from the workflow itself. Evaluate that trade before choosing it.
Acceptance criteria
- Option 1 is delivered, or refuted with a reason: on a busy day no deploy job waits behind the observer, and production falling behind
mainwith an empty queue is still detected and redeployed. - For options 2 to 4, choose one and record why, with the measured saving it is expected to give.
- Tests show a change still cannot reach production out of order, and a failed or cancelled deploy is still restored.
- Before and after: hand-off time between consecutive deploys over at least ten backlogged hand-offs, on a day when runners are busy.
Size
Option 1 is small and touches only the post-merge workflow. Options 2 to 4 are medium and touch the most carefully guarded part of the queue workflow, so they come after #5077.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- StatusShow more project fields🏃🏻♂️ In Progress
Activity
[-]The next production deploy waits for two fresh runners after the previous one finishes[/-][+]The next production deploy waits behind a post-merge observer and two fresh runners[/+]fix(validate-main): bound the observer's wait by its own hour
fix(validate-main): let no run hand its observation to another
fix(validate-main): hold back only the push of a change that deployed
devantler commentedon Nov 16, 2026
Post-merge check of #6212: deploys mostly no longer wait behind the production check
#6212 asked for this check after merge: on a busy day, confirm that deploys no longer queue behind the production check, and that the check still runs once the queue is empty. I read the job timings of every merge-queue
🚀 Deploy to Prodjob and everyvalidate-main.yamlrun from 11 November until now, using the GitHub API only.🚦 Wait for an idle merge queuefinished successfully 129 times out of 129. None of those waits hit the 75-minute limit, and the median wait was 10.8 minutes. The production check then ran 54 times and was skipped 76 times. A skip happens only when a later deploy has already carried that push to production.Result: option 1 delivered what it set out to do. This issue stays open for the options that are not yet built: overlapping the two remaining waits (2), and taking the final check off the shared runner pool (3 and 4, which wait on #4749).