Skip to content

Queued changes wait up to 15 minutes after validation for earlier production deploys #6139

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

On 12 November, 7 of 16 successful merge-queue CI runs that deployed to production had to wait for the entries ahead of them to deploy first. The wait is the 🧾 Wait until the candidate is first on current main step. It starts about 20 seconds into the run and runs alongside validation, so it only delays the deploy by the part that outlasts validation:

Run Wait for the deploy turn Validation finished at Delay added to the deploy
43233842713 409 s 422 s 9 s
43246082229 535 s 524 s 33 s
43208254811 489 s 417 s 92 s
43203738969 677 s 460 s 247 s
43188519865 755 s 454 s 360 s
43205453155 792 s 418 s 396 s
43200234933 1,351 s 557 s 876 s

The other 9 runs waited 17–24 s, and their deploy started as soon as validation finished. Each production deploy job took 630–1,144 s (637–825 s in the afternoon runs).

Problem

Production deploys run one at a time, and every merged change gets its own deploy. In a busy queue, a change that has finished its own validation can sit for up to one deploy per entry ahead of it: up to 15 minutes in the runs above. #5495 made validation and the deploy itself faster, so this queueing is now the largest variable part of the time from queueing a change to having it live.

Hypothesis and success signal

Fewer deploys per queued change should cut the delay without weakening any check. There are two independent levers:

  1. Skip the production deploy when a change touches only CI validation inputs. Avoid production redeployment when only CI validation inputs change #5077 already tracks this.
  2. Let one deploy cover several queued changes, for example by having the merge queue build more than one entry per group. The per-change checks then need to hold for a group: the deployed-change-still-queued check (The deployed-PR-still-queued check gives up after an hour of reads #5014), Heal Prod's restore of main, and blame for a failed deploy (Production deploys fail only for the change they deploy #4979).

Success: across at least ten comparable busy-period runs, the delay between an entry finishing validation and starting its deploy drops well below the current 92–876 s for entries that were not first in the queue. Every required check still runs, and a failed deploy still names the change that caused it.

Acceptance criteria

  • Decide which lever to try first and record why, including what happens to the per-change checks under that lever.
  • Ship it behind the existing required checks, with no check removed or skipped for a change that touches deployed manifests.
  • Measure the delay on comparable busy-period runs and report it next to the whole-run time, the same way as Reduce CI critical-path time while preserving validation #5495.
  • Confirm main and production stay healthy after the change.

Part of #2043. Split out of #5495, whose validation and deploy-duration goals are met.

Activity

  1. coderabbitai commented on Nov 12, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #6036 - ci: preserve production deploy order while merge groups validate [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. moved this to 📥 Backlog in 🌊 Project Boardon Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Performance.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions