Skip to content

perf(ci): pace GITHUB_TOKEN spend so busy hours stop tripping the kill switch #17460

Open

Description

@kyle-sexton

The repository's GITHUB_TOKEN pool (5,000 calls/h) runs out in every busy hour, which sets the CI kill switch: 15 trips on 2026-11-18 and 9 on 2026-11-19, each 0-14 minutes before the hourly reset. CI spent 103,065 calls in 24 h (4,294/h average). Quiet hours (29-43 runs/h) did not trip.

Cause

  • The brake drives its own cycle: on BRAKE-CLEAR the hub dispatches pr-merge and work-unblock at once (token-headroom.sh:124), and lanes check the budget only when they start, so jobs already running spend it to 0%. 15 of the last 26 trips were refusals (calls already rejected).
  • Top consumers (calls/h, from limits/gh-job-calls.jsonl): CI Status 586; pr-merge.yml 727 over three jobs (PR-scoped runs ~92 calls each, mostly listing other workflows' runs); pr-dispatch-feedback 536; Check Intent 288; reconcile-lost-runs 200 (30-minute cron); Wait For Refine 161 (a poll duplicated in the two review workflows).

Done when

  • Spend is paced across the hour (no burst dispatch at BRAKE-CLEAR; running jobs check headroom before each costly phase), and a 24 h window shows at most a few trips.
  • Run lists are filtered by head SHA in pr-merge.yml; the review starts from Refine's event instead of polling it; reconcile-lost-runs runs hourly; pr-dispatch-feedback is gated with if: so it does not start for events it ignores. Expected saving ~330-570 calls/h.
  • The before/after calls per hour for each consumer are recorded from limits/gh-job-calls.jsonl.

Measurements and file:line basis: the API usage investigation of 2026-11-19 (read-only; numbers from logs the hub already writes).

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 19, 2026
  2. github-actions commented on Nov 19, 2026

    @github-actions
    Contributor

    Size: Large
    Complexity: High
    Uncertainty: Knowable
    Flags: none
    Blocked by: none
    Blast radius: CI workflows (pr-merge, pr-dispatch-feedback, review/refine, reconcile-lost-runs), the token-headroom brake script, and the hub dispatch logic; every PR check and lane run in this repository depends on them.
    Reversible: yes; each per-consumer change is a workflow or script edit that can be reverted.
    Open questions: none
    I understood the issue this way. The hourly GITHUB_TOKEN pool runs out in busy hours and trips the CI kill switch. The causes are burst dispatch at BRAKE-CLEAR, jobs that check headroom only at start, and a few heavy consumers. It asks for paced spend, trimmed calls in four named workflows, and before/after calls per hour for each consumer. The work spans several subsystems and the 24 h trip-count check needs time to pass, so it needs splitting into one PR per consumer, which the comments say is under way. PR #17533 merged with Refs:, so the issue stays open until the remaining done-when items are checked. Nothing in the merged-PR list I was given clearly fixes it. I found no duplicate among the open issues.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 19, 2026
  4. melodic-automation-lanes commented on Nov 19, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"b9d79e15394684868e80dd7492faaedb551014eb2b4db489528839f82149d267","body_sha256":"7c49fed3fd14618dbcda30edd82e1622d413fc637aa019dc1ad47e1a0cb4f825"}

  5. melodic-automation-lanes commented on Nov 19, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"b9d79e15394684868e80dd7492faaedb551014eb2b4db489528839f82149d267","body_sha256":"7c49fed3fd14618dbcda30edd82e1622d413fc637aa019dc1ad47e1a0cb4f825"}

  6. kyle-sexton commented on Nov 19, 2026

    @kyle-sexton
    MemberAuthor

    Claimed by the GitHub Optimization session (owner's local login), with the hub's agreement. It ships one small PR per consumer, each with before/after calls per hour from limits/gh-job-calls.jsonl.

    Correction to my first version of this comment: the per-consumer numbers in the issue body stand. The log writes one row per job per rate-limit reset window (limits-sampler.sh, the row function's comment), so repeated (run, job) rows are window parts, not duplicates. Over the last 24 h the rows sum to 104.5k calls, against 109.6k on the token pool's own counter across 57 windows.

  7. melodic-automation-lanes commented on Nov 20, 2026

    @melodic-automation-lanes
    Contributor

    Pull request #17533 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  8. github-actions commented on Nov 20, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  9. melodic-automation-lanes commented on Nov 20, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: merged pull request #17508, #17517, #17531, #17533 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  10. kyle-sexton commented on Nov 20, 2026

    @kyle-sexton
    MemberAuthor

    24 h after-window measurement: trips fell with traffic; the A1 check fails as written.

    Window: 2026-11-19T22:38:21Z (#17531 merge) to 2026-11-20T22:38:21Z. BEFORE is the 24 h from 2026-11-18T22:18:24Z. Source: the job-calls log, every row summed (70,241 calls, 15,351 rows), and the monitor's BRAKE-SET events. Busy hours are 14Z-08Z (18 full hours in each window); quiet hours are 09Z-13Z.

    Measure BEFORE 24 h AFTER 24 h
    Calls/h, mean 4,335 2,927
    Calls/h, busy hours 4,953 3,297
    Busy hours over 5,000 (all tokens) 10 2
    Jobs/h, mean 717 456 (-36%)
    Jobs/h, busy hours 862 521 (-40%)
    Calls/job, busy hours 5.75 6.33 (+10%)
    Token windows at core_remaining 0 14 of 25 2 of 21 (1 after the merge)
    BRAKE-SET events 15 2 (1 after the merge)
    BRAKE-SET per 1,000 jobs 0.87 0.18 (0.09 counting only the post-merge one)
    Ungated share of calls 84.8% 86.9%
    • Of the two AFTER zero windows, the one resetting 22:47:45Z ran from 21:47Z, so all but 9 minutes of it came before the merge. The 22:48Z REFUSED at 91% left event belongs to it. The one post-merge trip is the window resetting 23:47Z (23:25Z BRAKE-SET ... EXHAUSTED at 0% left).
    • Trips happen past a threshold, so a per-job rate overstates the improvement when traffic falls. A traffic-matched comparison: the three AFTER windows with BEFORE-level traffic (at least 800 jobs/h, resetting 23:47Z, 02:47Z and 03:47Z) ended with 0, 547 and 48 calls left. In BEFORE, nearly every hour at that traffic tripped.
    • Normalized: AFTER's busy-hour 6.33 calls/job at BEFORE's 862 jobs/h projects to about 5,460 calls/h, which is above the 5,000/h pool. This is a projection; no trip was observed at that traffic after the cuts landed.

    What merged when (UTC)

    • 2026-11-19 22:18 #17533: lost-run reconciler hourly. The reconciler fell from 176/h to 67/h.
    • 2026-11-19 22:38 #17531: route labels only.
    • 2026-11-19 23:01 #17508: re-fire at most one budget-stopped issue run per slot.
    • 2026-11-20 15:49 #17657: skip the Wait For Refine poll where no refine can start.
    • 2026-11-20 16:17 #17660: defer the pr-merge arm run when the projected budget runs out.
    • 2026-11-20 16:18 #17658: list each green-run cell from its earliest record.
    • 2026-11-20 16:51 #17655: pin the review completions rerun-ci-status admits.

    The deferral and the cuts landed in the last 6 h of the window. Traffic in those hours ran 120-792 jobs/h, and the windows after 16:51Z peaked at 1,065-2,994. No window came near the limit, so this window cannot show what those merges do under busy-hour load. Whether the arm-run deferral fired at all is unchecked.

    A1 check: FAIL. The rule is at least 3 zero windows AND an ungated share above 70%. The ungated share passes at 86.9%. The zero-window count fails at 2, and only 1 of those falls after the merge. The count tracks traffic: busy-hour jobs/h fell 40%. Per the plan, an A1 failure stops the build for a replan, so no decision comment is posted.

    Reading. The data does not contradict option B's basis. Ungated lanes still carry about 87% of calls, and calls per job rose. At last week's busy-hour traffic, demand projects to about 5,460/h. At this day's traffic, trips were rare: one after the merge, before any cut landed. The arm-run deferral and the cuts that merged today may be enough at this traffic, but nothing here tests them at the traffic that tripped.

    Open decision: recovery for deferred reviews and Check Intent runs. Today the only thing that re-runs them is the hourly lost-run reconciler, which keeps at most 3 re-runs in flight. That covers under 10% of the deferral load modeled for a tripping hour (about 9.5 review heads and 22.6 Check Intent runs an hour). The options:

    Decision (owner, 2026-11-20): hold the review and Check Intent deferral and its recovery pass (a) for now. Keep #17660 and today's cuts, and re-run this measurement over the next 24 h that has at least 6 busy hours at 800 or more jobs/h. Build (a) only if that run shows at least 3 zero windows. Basis: the table above. The only window at that traffic after the merge tripped, but it came before the cuts landed, and no busy hour has run since they did.

    Data coverage: the job log reaches done_at 22:40Z and the monitor's event log reaches 22:35Z, 3 minutes short of the window end. The token windows resetting 11:48Z-13:48Z and 20:49Z have no core_remaining rows, which fits their low traffic. The window resetting 22:50Z was still open when the data ends.

    🤖 Generated with Claude Code

  11. kyle-sexton commented on Nov 21, 2026

    @kyle-sexton
    MemberAuthor

    Busy-day re-measure, 24 h ending 2026-11-21T20:49:50Z, the first trailing 24 h with 6 busy hours (at least 800 jobs/h, counted 14Z-08Z). Source: gh-job-calls notices condensed locally and measured with the same script and --gated list as the Nov 20 measurement.

    Before (24 h to 2026-11-19T22:18Z) After (24 h to 2026-11-21T20:49Z)
    Token windows that reached core_remaining == 0 14 of 25 3 of 50 (resets 07:51Z, 18:55Z, 19:55Z)
    BRAKE-SETs 15 3 (07:44Z, 18:54Z, 19:47Z)
    Calls per hour, all tokens about 6,000-6,500 in tripping hours 3,671 average; 4 clock hours over 5,000
    Calls per job 6.64 (Nov 20 partial) 5.70
    Spend behind a budget start gate 15.2% 26.0% (954/h)

    Top spenders now: pr-merge 445/h, pr-dispatch-feedback 386/h, pr-require-checks 385/h (CI Status 349/h), intake-triage 368/h.

    Not measured: how many pipeline jobs deferred with cause=defer. Counting it needs the run logs downloaded, which costs API calls; it is left for the build decision.

    The pre-set rule was "at least 3 zero windows: build Phase 2 (review and Check Intent deferral) with the post-reset recovery pass; fewer: close with the Nov 20 merges". The result sits exactly at the threshold. Two of the three exhaustions (18:55Z and 19:55Z) happened during the same hours as a local host overload and a session restart. Whether the two are linked is UNVERIFIED. The decision is with the owner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    high

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions