Skip to content

The next production deploy waits behind a post-merge observer and two fresh runners #6205

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

  • Production deploys run one at a time. Between two of them, three jobs each need a fresh runner, one after the other:
    1. the final required check of the change that just deployed;
    2. Observe Production Convergence in validate-main.yaml, which starts on every push to main, takes the production lock and holds it while it waits for a runner. Its own work takes 0.6 minutes;
    3. the deploy job of the next change, which waits for that lock and then for a runner of its own.
  • When runners are free the hand-off takes 1.5 minutes (median, 28 hand-offs from 5 to 12 November) and a deploy-to-deploy cycle takes 14.3 minutes. On 12 November, 17 of 38 deploys still queued behind the observer, for about 0.9 minutes each.
  • On 13 November the hand-off took 29.5 minutes and the cycle 39.8 minutes. The deploy itself was unchanged.
13 November, medians Minutes
Required check, wait for a runner 8.1
Observer, wait for a runner while holding the lock 10.6
Next deploy job, total wait 13.7
of which behind the observer's lock 9.2
of which for its own runner 6.7

13 of that day's 14 deploys were queued behind the observer. The prod environment has no reviewers and no wait timer, so none of this is an approval wait.

Problem

The one part of delivery that cannot run in parallel pays for a busy runner pool three times per change, and one of the three is a 0.6-minute read that holds the production lock while it waits.

Options to evaluate

Each must keep today's order and proof: a change deploys only when every change ahead of it is on main, and merges only after its own deploy is verified.

  1. Take the observer out of the hand-off. Do this first: it changes no merge proof. The observer exists to catch production falling behind main when nothing else will redeploy it (A merge-group heal can leave prod behind a merge that lands just after it #2838, Redeploy main after a merge when prod has not converged on it #3848). While another change is queued, that change's own deploy or recovery brings production up to date, so the observation only matters once the queue is empty. Candidates: observe only when the queue is empty at the push, with a way to still observe if the remaining queued changes leave without merging; or keep observing on every push but run the observer's own self-tests outside the lock. Note that making the observer start later does not help under sustained load: each merge still adds one observer that holds the lock while it waits for a runner.
  2. Overlap the two remaining waits. Let the next change start looking for its deploy runner when the previous deploy succeeds, and still check under the production lock that it is first on main before it touches production. Saves the shorter of the two waits.
  3. Take the final check off the shared pool. It runs no repository code and needs no secrets, so it is the natural first job for a dedicated runner once the organisation pool is active (Establish a zero-runner baseline before reopening ARC #4749).
  4. Start the final check's runner during the deploy. Removes its wait, but the check would read job results from the API and not from the workflow itself. Evaluate that trade before choosing it.

Acceptance criteria

  • Option 1 is delivered, or refuted with a reason: on a busy day no deploy job waits behind the observer, and production falling behind main with an empty queue is still detected and redeployed.
  • For options 2 to 4, choose one and record why, with the measured saving it is expected to give.
  • Tests show a change still cannot reach production out of order, and a failed or cancelled deploy is still restored.
  • Before and after: hand-off time between consecutive deploys over at least ten backlogged hand-offs, on a day when runners are busy.

Size

Option 1 is small and touches only the post-merge workflow. Options 2 to 4 are medium and touch the most carefully guarded part of the queue workflow, so they come after #5077.

Part of #6203. Related: #6139, #3848.

Activity

  1. moved this to 📥 Backlog in 🌊 Project Boardon Nov 13, 2026
  2. updated Priority Medium and Effort Medium on Nov 13, 2026
  3. changed the title [-]The next production deploy waits for two fresh runners after the previous one finishes[/-] [+]The next production deploy waits behind a post-merge observer and two fresh runners[/+] on Nov 13, 2026
  4. moved this from 📥 Backlog to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 13, 2026
  5. self-assigned this
    on Nov 13, 2026
  6. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Post-merge check of #6212: deploys mostly no longer wait behind the production check

    #6212 asked for this check after merge: on a busy day, confirm that deploys no longer queue behind the production check, and that the check still runs once the queue is empty. I read the job timings of every merge-queue 🚀 Deploy to Prod job and every validate-main.yaml run from 11 November until now, using the GitHub API only.

    Before #6212 (11 Nov – 13 Nov 17:01Z) After #6212 (13 Nov 17:31Z – 16 Nov 04:00Z)
    Deploys 117 119
    Deploy wait to start, median 0.6 min 0.1 min
    Deploy wait to start, 90th percentile 13.0 min 6.8 min
    Deploys whose wait overlapped a running production check 66 (56%) 12 (10%)
    • Busiest day: 15 November. It had 59 deploys, and 6 of them overlapped a check. Its 90th-percentile wait was 28.2 minutes, which comes from waiting for runners. The check did not cause it.
    • The check still runs. After the merge, 🚦 Wait for an idle merge queue finished successfully 129 times out of 129. None of those waits hit the 75-minute limit, and the median wait was 10.8 minutes. The production check then ran 54 times and was skipped 76 times. A skip happens only when a later deploy has already carried that push to production.
    • How to read the overlap row. It counts deploys that were waiting while a production check job was running. It does not prove that the check held the lock in each case. By design, a push that bypassed the queue or a merge that did not deploy still checks straight away, and that accounts for the remaining overlaps.

    Result: option 1 delivered what it set out to do. This issue stays open for the options that are not yet built: overlapping the two remaining waits (2), and taking the final check off the shared runner pool (3 and 4, which wait on #4749).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

Priority

Medium

Effort

Medium

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions