Skip to content

The workflow token reaches its rate-limit floor each hour and its steps fail instead of waiting #16642

Open

Description

@kyle-sexton

Problem

On 2026-11-16 the workflow token (token=github-token in the api-budget notices) reached its core rate-limit floor in two consecutive hourly windows. Each time, steps that needed one call failed their run instead of waiting, and a person had to re-run them.

Window 1, reset 20:32:29Z:

  • Refine on PR 16631, 20:12Z: "left GitHub's core API 65 of 5000 requests, at or under its stop-floor share 0.02 … this step fails"; the pr-refine dispatch "was not sent".
  • Mark Lanes on PR 16631 head 0237d839 failed at 20:22Z (run 43949103070).
  • Both review lanes on that head did not run: CI Status reported "'PR Review / Status' did not run … a kill switch was on when it started" for runs 43949103692 and 43949103789.
  • 13 of the 15 PR Verify runs created between 20:22Z and 20:28Z concluded failure; the one opened (43950072774) failed at "Report the kill switch".
  • Recovery took three manual re-runs after the reset (gh run rerun 43949103070 --failed, and all jobs of the two lane runs).

Window 2, reset 21:32:30Z:

  • Refine on PR 16641, 21:24Z: limit=5000 remaining=88 used=4912 resource=core, "no run dispatched".

UNVERIFIED: what spends about 4,900 core requests an hour on this token. Nothing here measured the callers.

Done when

  • The callers that spent the token's core budget in these two windows are named, with their request counts, from the api-budget notices or the run logs.
  • Either the hourly spend is brought under the floor, or a step that finds the token at its floor waits for x-ratelimit-reset (or is re-run by the reconciler) instead of failing a run that then needs a person.
  • A pull request pushed while the token is at its floor gets its review lanes and Mark Lanes without a manual re-run.

Activity

  1. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Callers that spent the token, by workflow and job, from the hub's per-job call log (gh-job-calls.jsonl, bucketed by each job's finish time). The log does not attribute every call: in the 20:32Z window it recorded 4,082 core calls against 4,781 used.

    Window (reset) Top consumers (core calls, jobs)
    19:32Z CI Status 528 (65), PR Verify Check Intent 278 (41), the two review subscriptions 334 (39), reconcile-lost-runs 162 (1), rerun-ci-status 148 (58, of which 8 re-ran CI Status), pr-merge 148 (2)
    20:32Z pr-merge 876 (10: Merge 495 in 4, Enable Auto Merge 208, Dispatch post merge runs 173), CI Status 513 (53), Check Intent 284 (38), review subscriptions 307 (38), reconcile-lost-runs 251 (2)

    What the hub found:

    In flight on #15497: moving the heaviest repeated reads in CI Status, PR Verify and the review lanes to the GITHUB_TOKEN's separate GraphQL pool. That's estimated to raise the ceiling from about 21 to about 26 open PRs. Analysis: #15497 (comment) and the comments after it. This issue's ask (wait at the floor instead of failing) is a separate fix to the same problem, so link the two rather than merge them.

  2. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    First live test of PR #16650, after the 22:17Z brake (cleared 22:33Z): reconcile run https://github.com/melodic-software/claude-code-plugins/actions/runs/43971789832 (22:37Z).

    • Done-when item 2 is shown live: "chore(release): release 20 plugins #16652: re-ran the failed jobs of run 43968483728 (.github/workflows/pr-require-checks.yml) attempt 1, which failed only where the GITHUB_TOKEN hit its rate-limit floor". It also re-ran 2 failed pr-verify lane runs and 2 pr-dispatch-feedback runs whose verdict the kill switch had held.
    • Item 3 is not met at scale: the same pass logged 45 lane runs "deferred, all 3 re-run places taken" (pr-verify.yml, some from about 19:40Z). At 3 re-runs per pass and 2 passes an hour, a backlog that size takes hours to drain. A PR whose pr-verify is in that queue stays unmerged meanwhile. For example, docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602 left the merge queue at 22:15Z because pr-verify had not judged its head.
      Remaining: drain the post-brake backlog faster, keeping the stagger. Two options: a larger cap for the first pass after a brake clear, or re-running only the newest lost run per PR, since older heads are superseded. Basis: the reconcile log lines above.
  3. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  4. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none
    Blast radius: the lost-run reconciler script, its workflow and their tests; maintainers and the pipeline's recovery of stalled PR runs.
    Reversible: yes; revert the change, and the 30-minute cron pass resumes as the only recovery trigger.
    Open questions: none

    Decision comment, quoted:

    Decision recorded. What is left is done-when item 3 at scale: the lost-run reconciler drains a post-brake backlog at most 3 re-runs per 30-minute pass. The rest is done or moved.

    [quote truncated; the full decision is at the link]

    #16642 (comment)

    The decision settles the open questions. Items 1 and 2 are recorded as done (#16650, #16814, #17014, #17033, #17003, #17023, #17028), and the kill-switch part moved to #17285. What remains is a follow-up reconcile pass when a pass defers re-runs for want of a place. MAX_IN_FLIGHT stays 3, and the six done-when items in the decision define the scope. Related, not duplicates: #17285 and #16944.

    Automated triage. A maintainer reviews it.

  5. melodic-automation-lanes commented on Nov 16, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: merged pull request #16650 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. kyle-sexton commented on Nov 17, 2026

    @kyle-sexton
    MemberAuthor

    Item 3 seen live on today's first brake, 2026-11-17 (BRAKE-SET 17:27:45Z, REFUSED at 0% left; BRAKE-CLEAR 17:42Z at 89%).

    So the reconciler re-runs only runs lost to the brake, capped at 3 a pass. A brake that leaves more than 3 runs takes one extra scheduled pass, about 25 minutes, for each further 3.

  7. kyle-sexton commented on Nov 18, 2026

    @kyle-sexton
    MemberAuthor

    Done-when item 1: the callers, with request counts. Measured from the jobs' api-budget notices (core-by-reset, big counter only) over the two GITHUB_TOKEN brake hours of 2026-11-17, W1 20:42-21:42Z and W2 22:42-23:42Z.

    Rank Workflow / job Calls, W1 + W2
    1 pr-merge / Enable Auto Merge (PR-scoped "arm #N" wakes) 734
    2 pr-verify / Check Intent 555
    3 pr-dispatch-feedback / Dispatch 529
    4 maintenance-reconcile-lost-runs / Reconcile 512
    5 pr-require-checks / CI Status 501
    • The job notices account for 3,768 of the counter's 5,000 in W1 (75%) and 4,118 in W2 (82%).
    • Runs started per PR spent 60% and 52% of each hour.
    • The counter that refused is the one resetting at :42. A second counter, resetting at :52-:56, never passed 486 used.
    • The pool is not shared with ci-workflows: its token stayed at 2-6 used against different resets. For standards this is unverified, because its jobs post no api-budget notices.

    Fixes so far:

  8. kyle-sexton commented on Nov 18, 2026

    @kyle-sexton
    MemberAuthor

    Two more occurrences on 2026-11-18, details in #17248 (closed as a duplicate of this issue):

  9. kyle-sexton commented on Nov 18, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. What is left is done-when item 3 at scale: the lost-run reconciler drains a post-brake backlog at most 3 re-runs per 30-minute pass. The rest is done or moved.

    Done since the last refinement:

    Still open, evidence from today's post-clear pass (run 44384109700, 19:48:59Z): reconcile: lane run 44381721733 (.github/workflows/pr-verify.yml): deferred, all 3 re-run places taken, the same for 7 more pr-verify runs and #17117: deferred, all 3 re-run places taken. Passes run on cron 4,34 * * * *, so each further 3 waits 30 minutes. #16658 already drops lost runs that follow no open head, so the second option in #16642 (comment) is done; this records the first one's replacement.

    RECOMMENDED (vetoable, reversible): keep MAX_IN_FLIGHT at 3 and the burn-rate pacer; a pass that defers recovery work for want of a place starts one follow-up pass instead of waiting for the next scheduled tick. Basis: maintenance-reconcile-lost-runs.yml cron and MIN_GAP_MINUTES: "30", reconcile.mjs MAX_IN_FLIGHT = 3 and its header ("A pass that defers anything but a flake-only set writes no clean, so the pacer runs the next tick"), read at origin/main today.

    Done when:

    • A reconcile pass that defers any re-run other than a flake-only set for want of a place causes one follow-up pass to start before the next scheduled tick.
    • The follow-up pass runs the same kill-switch check, upstream-status check and burn-rate pace step as a scheduled pass before it reads anything.
    • A pass that defers nothing, or defers only flake-only sets, starts no follow-up pass.
    • No more than one follow-up pass is queued or running at a time: a deferring pass that finds one already pending starts none.
    • MAX_IN_FLIGHT stays 3.
    • The reconcile-lost-runs script and workflow tests cover each item above and pass.
  10. 3 remaining items

  11. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor
    No description provided.
  12. melodic-automation-lanes commented on Nov 18, 2026

    @melodic-automation-lanes
    Contributor

    This records a re-fire of an earlier stop, not a new one. A GitHub API pool's stop floor stopped run 44406300700, the work-resolve-issue run that agent-ready started, after its model ran, until 2026-11-18T22:46:50Z. work-unblock has now removed and added agent-ready back to start a fresh run early in a window. A floor stop is the lanes' load, not a fault in this issue. After limits.limit-stops-per-issue of these since a person last added agent-ready or a run last reported, work-unblock waits the limits.stranded-backoff-hours failed-again backoff instead.

  13. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor
    No description provided.
  14. github-actions commented on Nov 19, 2026

    @github-actions
    Contributor
    No description provided.
  15. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 19, 2026
  16. melodic-automation-lanes commented on Nov 19, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (patch-protected-path): the change needs a path the issue lane may not push, which a maintainer session builds by hand.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    The run stopped early (patch-protected-path): its change needs a path the issue lane may not push, which a person changes instead. In the run's words: patch-protected-path: .github/workflows/maintenance-reconcile-lost-runs.yml. The recorded decision (follow-up reconcile pass when a pass defers recovery work) needs this workflow changed. The follow-up has to be told apart from a person's dispatch and paced like a scheduled tick. It also has to wait until the re-runs the deferring pass started have left their places, and the workflow header and pace-step comment ("the tick after it always runs", "Every 30 minutes") have to describe the new pass. All of that lives in this workflow file, which the lane cannot change.

    What the run says it did not do:

    • Follow-up reconcile pass on deferral (all six done-when items of the recorded decision): not built, because it needs .github/workflows/maintenance-reconcile-lost-runs.yml, which this lane cannot change
    • Tidy, review, build and lint were not run: the run stopped before any edit and changed no file

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/44424531077

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  17. kyle-sexton commented on Nov 20, 2026

    @kyle-sexton
    MemberAuthor

    Callers in the two windows (done-when item 1), summed from every row of the api-budget call collector whose reset falls within 5 minutes of each window's reset. The collector attributes 3,540 and 3,753 of the roughly 4,900 calls spent; the rest are unattributed.

    Workflow 20:32:29Z window 21:32:30Z window
    pr-merge.yml 294 837
    pr-require-checks.yml 514 379
    pr-verify.yml 443 435
    pr-review-hosted.yml 474 396
    pr-review-security-hosted.yml 478 372
    pr-dispatch-feedback.yml 319 282
    pr-refine.yml 262 157
    maintenance-reconcile-lost-runs.yml under 179 249
    pr-review-main.yml 179 under 157

    Since then, 24 h core calls on this token fell from 116,767 to 99,218 (to 2026-11-20T05:20Z) through the GitHub Optimization cuts. The kill switch last tripped at 2026-11-19T23:25Z, and none has tripped on 2026-11-20 through 14:07Z. Today's busy hours have not run yet.

    Re-decision (GitHub Work's Q106, option A, handed to GitHub Optimization): items 2 and 3 are the paced start gate that #17460 builds, which defers optional PR jobs past the reset instead of failing them. So this issue stays open, tied to #17460, with no separate build. It closes when #17460's gate merges, or when a week of the hourly cadence shows no floor hit, whichever comes first.

  18. kyle-sexton commented on Nov 21, 2026

    @kyle-sexton
    MemberAuthor

    Evidence: the ci-lanes record is the largest single cause of red full runs on main.

    From the 306 full pr-require-checks.yml runs on main since 2026-11-14, 41 failed attempt 1. Twelve of them failed only in CI Status, all with the same annotation:

    could not record ci-lanes on <sha> (HTTP 403): rate limited past rate-limit-wait-seconds; re-run this job after the limit resets

    run attempt 1 (UTC)
    44135733765 2026-11-17 17:19
    44178536055 2026-11-17 21:19
    44222750671 2026-11-18 02:22
    44243430446 2026-11-18 05:19
    44284503435 2026-11-18 10:20
    44303940983 2026-11-18 12:26
    44411335376 2026-11-18 22:24
    44419777099 2026-11-18 23:19
    44428900294 2026-11-19 00:25
    44450832501 2026-11-19 03:21
    44617915696 2026-11-19 21:21
    44636223457 2026-11-19 23:16

    All twelve land at :16 to :26 past the hour, the end of an hourly window when the pool is lowest. None has come since 2026-11-19 23:16Z, after the spend cuts of 11-20 and 11-21. Each one turns a run whose tests passed red, which holds the post-merge breaker until a newer commit's full run passes or the lost-run reconciler's floor re-run lands.

    There was one more rate-limit red, of a different kind. In 44565037503 (2026-11-19 16:20), scripts/ci-queued-pr.test.sh in Test Bash (6) read Could not read queued pull request #7492: rate limited (every pool until 2026-11-19T17:31:25Z). Not checked yet: whether that suite read the runner's real rate-guard state. A 2026-11-14 codeload download timeout (43638508739) is not a pool failure.

    Where the record step lives: the report-ci-status composite of melodic-software/ci-workflows, pinned at v0.78.0 (pr-require-checks.yml:7278). Its rate-limit-wait-seconds is held to WAIT_BUDGET_SECONDS.step (.github/actions/log-api-budget/rate-guard.mjs:56-64), by outage posture element 1 (docs/conventions/pr-pipeline/README.md, owner decision Q677): a wait past a small per-step budget counts as unknown instead of a sleep. Waiting out the reset in this step would add a second exception beside work-resolve-issue's, so it is a change to that owner decision.

    Data: 306 runs listed, then attempt 1's failed jobs and their failure annotations for the 41 failed runs (~130 calls).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    high

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions