You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The workflow token reaches its rate-limit floor each hour and its steps fail instead of waiting #16642
On 2026-11-16 the workflow token (token=github-token in the api-budget notices) reached its core rate-limit floor in two consecutive hourly windows. Each time, steps that needed one call failed their run instead of waiting, and a person had to re-run them.
Window 1, reset 20:32:29Z:
Refine on PR 16631, 20:12Z: "left GitHub's core API 65 of 5000 requests, at or under its stop-floor share 0.02 … this step fails"; the pr-refine dispatch "was not sent".
Mark Lanes on PR 16631 head 0237d839 failed at 20:22Z (run 43949103070).
Both review lanes on that head did not run: CI Status reported "'PR Review / Status' did not run … a kill switch was on when it started" for runs 43949103692 and 43949103789.
13 of the 15 PR Verify runs created between 20:22Z and 20:28Z concluded failure; the one opened (43950072774) failed at "Report the kill switch".
Recovery took three manual re-runs after the reset (gh run rerun 43949103070 --failed, and all jobs of the two lane runs).
Window 2, reset 21:32:30Z:
Refine on PR 16641, 21:24Z: limit=5000 remaining=88 used=4912 resource=core, "no run dispatched".
UNVERIFIED: what spends about 4,900 core requests an hour on this token. Nothing here measured the callers.
Done when
The callers that spent the token's core budget in these two windows are named, with their request counts, from the api-budget notices or the run logs.
Either the hourly spend is brought under the floor, or a step that finds the token at its floor waits for x-ratelimit-reset (or is re-run by the reconciler) instead of failing a run that then needs a person.
A pull request pushed while the token is at its floor gets its review lanes and Mark Lanes without a manual re-run.
Callers that spent the token, by workflow and job, from the hub's per-job call log (gh-job-calls.jsonl, bucketed by each job's finish time). The log does not attribute every call: in the 20:32Z window it recorded 4,082 core calls against 4,781 used.
Window (reset)
Top consumers (core calls, jobs)
19:32Z
CI Status 528 (65), PR Verify Check Intent 278 (41), the two review subscriptions 334 (39), reconcile-lost-runs 162 (1), rerun-ci-status 148 (58, of which 8 re-ran CI Status), pr-merge 148 (2)
20:32Z
pr-merge 876 (10: Merge 495 in 4, Enable Auto Merge 208, Dispatch post merge runs 173), CI Status 513 (53), Check Intent 284 (38), review subscriptions 307 (38), reconcile-lost-runs 251 (2)
What the hub found:
No single workflow wastes calls. CI Status re-runs are needed: 6 of the 8 actual re-runs turned a PR from red to green. No CI Status job reads the same data twice, and reconcile reads each run once. The spend tracks how many PRs are open (about 18-21) times how often each one is pushed, labeled or edited.
pr-merge's jump to 876 in the 20:32Z window follows the post-brake merge sweep: about 124 calls per Merge job, against about 50 per run on 11-15. Which step spent them is not read yet.
In flight on #15497: moving the heaviest repeated reads in CI Status, PR Verify and the review lanes to the GITHUB_TOKEN's separate GraphQL pool. That's estimated to raise the ceiling from about 21 to about 26 open PRs. Analysis: #15497 (comment) and the comments after it. This issue's ask (wait at the floor instead of failing) is a separate fix to the same problem, so link the two rather than merge them.
Done-when item 2 is shown live: "chore(release): release 20 plugins #16652: re-ran the failed jobs of run 43968483728 (.github/workflows/pr-require-checks.yml) attempt 1, which failed only where the GITHUB_TOKEN hit its rate-limit floor". It also re-ran 2 failed pr-verify lane runs and 2 pr-dispatch-feedback runs whose verdict the kill switch had held.
Item 3 is not met at scale: the same pass logged 45 lane runs "deferred, all 3 re-run places taken" (pr-verify.yml, some from about 19:40Z). At 3 re-runs per pass and 2 passes an hour, a backlog that size takes hours to drain. A PR whose pr-verify is in that queue stays unmerged meanwhile. For example, docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602 left the merge queue at 22:15Z because pr-verify had not judged its head.
Remaining: drain the post-brake backlog faster, keeping the stagger. Two options: a larger cap for the first pass after a brake clear, or re-running only the newest lost run per PR, since older heads are superseded. Basis: the reconcile log lines above.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Blast radius: the lost-run reconciler script, its workflow and their tests; maintainers and the pipeline's recovery of stalled PR runs.
Reversible: yes; revert the change, and the 30-minute cron pass resumes as the only recovery trigger.
Open questions: none
Decision comment, quoted:
Decision recorded. What is left is done-when item 3 at scale: the lost-run reconciler drains a post-brake backlog at most 3 re-runs per 30-minute pass. The rest is done or moved.
[quote truncated; the full decision is at the link]
The decision settles the open questions. Items 1 and 2 are recorded as done (#16650, #16814, #17014, #17033, #17003, #17023, #17028), and the kill-switch part moved to #17285. What remains is a follow-up reconcile pass when a pass defers re-runs for want of a place. MAX_IN_FLIGHT stays 3, and the six done-when items in the decision define the scope. Related, not duplicates: #17285 and #16944.
Not admitted to the issue lane: merged pull request #16650 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Item 3 seen live on today's first brake, 2026-11-17 (BRAKE-SET 17:27:45Z, REFUSED at 0% left; BRAKE-CLEAR 17:42Z at 89%).
The post-clear forced reconcile, run 44139935464 at 17:42:33Z, re-ran three lost runs: 44136652172, 44135720053 and 44137137019, each with a reconcile-rerun-* artifact.
The next pass did not reach it in 20 minutes, so it was re-run by hand at 18:0xZ.
So the reconciler re-runs only runs lost to the brake, capped at 3 a pass. A brake that leaves more than 3 runs takes one extra scheduled pass, about 25 minutes, for each further 3.
Done-when item 1: the callers, with request counts. Measured from the jobs' api-budget notices (core-by-reset, big counter only) over the two GITHUB_TOKEN brake hours of 2026-11-17, W1 20:42-21:42Z and W2 22:42-23:42Z.
Rank
Workflow / job
Calls, W1 + W2
1
pr-merge / Enable Auto Merge (PR-scoped "arm #N" wakes)
734
2
pr-verify / Check Intent
555
3
pr-dispatch-feedback / Dispatch
529
4
maintenance-reconcile-lost-runs / Reconcile
512
5
pr-require-checks / CI Status
501
The job notices account for 3,768 of the counter's 5,000 in W1 (75%) and 4,118 in W2 (82%).
Runs started per PR spent 60% and 52% of each hour.
The counter that refused is the one resetting at :42. A second counter, resetting at :52-:56, never passed 486 used.
The pool is not shared with ci-workflows: its token stayed at 2-6 used against different resets. For standards this is unverified, because its jobs post no api-budget notices.
What followed: CLAUDE_LANES_REPO_DISABLED was on while both PRs' review lanes started, so all four lane runs skipped, and the variable was set back to false at 19:47:29Z. The lost-run reconciler re-ran only one of those four runs: PR Review Hosted 44382598651. Security 44382598493 on docs(upstream): record the owner's 2026-11-18 re-decision of the home-repo navigation rows #17225 and both lanes on feat: run improvement skills across several repositories #17228 (44383832798, 44383832780) stayed at attempt 1 until they were re-run by hand. The cause is UNVERIFIED: the reconciler may not have run yet, or it may skip runs whose conclusion is success. Lane runs that skipped this way conclude success, but the check reports that they did not run.
Decision recorded. What is left is done-when item 3 at scale: the lost-run reconciler drains a post-brake backlog at most 3 re-runs per 30-minute pass. The rest is done or moved.
Still open, evidence from today's post-clear pass (run 44384109700, 19:48:59Z): reconcile: lane run 44381721733 (.github/workflows/pr-verify.yml): deferred, all 3 re-run places taken, the same for 7 more pr-verify runs and #17117: deferred, all 3 re-run places taken. Passes run on cron 4,34 * * * *, so each further 3 waits 30 minutes. #16658 already drops lost runs that follow no open head, so the second option in #16642 (comment) is done; this records the first one's replacement.
RECOMMENDED (vetoable, reversible): keep MAX_IN_FLIGHT at 3 and the burn-rate pacer; a pass that defers recovery work for want of a place starts one follow-up pass instead of waiting for the next scheduled tick. Basis: maintenance-reconcile-lost-runs.yml cron and MIN_GAP_MINUTES: "30", reconcile.mjsMAX_IN_FLIGHT = 3 and its header ("A pass that defers anything but a flake-only set writes no clean, so the pacer runs the next tick"), read at origin/main today.
Done when:
A reconcile pass that defers any re-run other than a flake-only set for want of a place causes one follow-up pass to start before the next scheduled tick.
The follow-up pass runs the same kill-switch check, upstream-status check and burn-rate pace step as a scheduled pass before it reads anything.
A pass that defers nothing, or defers only flake-only sets, starts no follow-up pass.
No more than one follow-up pass is queued or running at a time: a deferring pass that finds one already pending starts none.
MAX_IN_FLIGHT stays 3.
The reconcile-lost-runs script and workflow tests cover each item above and pass.
This records a re-fire of an earlier stop, not a new one. A GitHub API pool's stop floor stopped run 44406300700, the work-resolve-issue run that agent-ready started, after its model ran, until 2026-11-18T22:46:50Z. work-unblock has now removed and added agent-ready back to start a fresh run early in a window. A floor stop is the lanes' load, not a fault in this issue. After limits.limit-stops-per-issue of these since a person last added agent-ready or a run last reported, work-unblock waits the limits.stranded-backoff-hours failed-again backoff instead.
The work-resolve-issue lane stopped on this issue (patch-protected-path): the change needs a path the issue lane may not push, which a maintainer session builds by hand.
It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.
In the run's words:
The run stopped early (patch-protected-path): its change needs a path the issue lane may not push, which a person changes instead. In the run's words: patch-protected-path: .github/workflows/maintenance-reconcile-lost-runs.yml. The recorded decision (follow-up reconcile pass when a pass defers recovery work) needs this workflow changed. The follow-up has to be told apart from a person's dispatch and paced like a scheduled tick. It also has to wait until the re-runs the deferring pass started have left their places, and the workflow header and pace-step comment ("the tick after it always runs", "Every 30 minutes") have to describe the new pass. All of that lives in this workflow file, which the lane cannot change.
What the run says it did not do:
Follow-up reconcile pass on deferral (all six done-when items of the recorded decision): not built, because it needs .github/workflows/maintenance-reconcile-lost-runs.yml, which this lane cannot change
Tidy, review, build and lint were not run: the run stopped before any edit and changed no file
Callers in the two windows (done-when item 1), summed from every row of the api-budget call collector whose reset falls within 5 minutes of each window's reset. The collector attributes 3,540 and 3,753 of the roughly 4,900 calls spent; the rest are unattributed.
Workflow
20:32:29Z window
21:32:30Z window
pr-merge.yml
294
837
pr-require-checks.yml
514
379
pr-verify.yml
443
435
pr-review-hosted.yml
474
396
pr-review-security-hosted.yml
478
372
pr-dispatch-feedback.yml
319
282
pr-refine.yml
262
157
maintenance-reconcile-lost-runs.yml
under 179
249
pr-review-main.yml
179
under 157
Since then, 24 h core calls on this token fell from 116,767 to 99,218 (to 2026-11-20T05:20Z) through the GitHub Optimization cuts. The kill switch last tripped at 2026-11-19T23:25Z, and none has tripped on 2026-11-20 through 14:07Z. Today's busy hours have not run yet.
Re-decision (GitHub Work's Q106, option A, handed to GitHub Optimization): items 2 and 3 are the paced start gate that #17460 builds, which defers optional PR jobs past the reset instead of failing them. So this issue stays open, tied to #17460, with no separate build. It closes when #17460's gate merges, or when a week of the hourly cadence shows no floor hit, whichever comes first.
Evidence: the ci-lanes record is the largest single cause of red full runs on main.
From the 306 full pr-require-checks.yml runs on main since 2026-11-14, 41 failed attempt 1. Twelve of them failed only in CI Status, all with the same annotation:
could not record ci-lanes on <sha> (HTTP 403): rate limited past rate-limit-wait-seconds; re-run this job after the limit resets
All twelve land at :16 to :26 past the hour, the end of an hourly window when the pool is lowest. None has come since 2026-11-19 23:16Z, after the spend cuts of 11-20 and 11-21. Each one turns a run whose tests passed red, which holds the post-merge breaker until a newer commit's full run passes or the lost-run reconciler's floor re-run lands.
There was one more rate-limit red, of a different kind. In 44565037503 (2026-11-19 16:20), scripts/ci-queued-pr.test.sh in Test Bash (6) read Could not read queued pull request #7492: rate limited (every pool until 2026-11-19T17:31:25Z). Not checked yet: whether that suite read the runner's real rate-guard state. A 2026-11-14 codeload download timeout (43638508739) is not a pool failure.
Where the record step lives: the report-ci-status composite of melodic-software/ci-workflows, pinned at v0.78.0 (pr-require-checks.yml:7278). Its rate-limit-wait-seconds is held to WAIT_BUDGET_SECONDS.step (.github/actions/log-api-budget/rate-guard.mjs:56-64), by outage posture element 1 (docs/conventions/pr-pipeline/README.md, owner decision Q677): a wait past a small per-step budget counts as unknown instead of a sleep. Waiting out the reset in this step would add a second exception beside work-resolve-issue's, so it is a change to that owner decision.
Data: 306 runs listed, then attempt 1's failed jobs and their failure annotations for the 41 failed runs (~130 calls).
Activity
kyle-sexton commentedon Nov 16, 2026
Callers that spent the token, by workflow and job, from the hub's per-job call log (
gh-job-calls.jsonl, bucketed by each job's finish time). The log does not attribute every call: in the 20:32Z window it recorded 4,082 core calls against 4,781 used.What the hub found:
In flight on #15497: moving the heaviest repeated reads in CI Status, PR Verify and the review lanes to the GITHUB_TOKEN's separate GraphQL pool. That's estimated to raise the ceiling from about 21 to about 26 open PRs. Analysis: #15497 (comment) and the comments after it. This issue's ask (wait at the floor instead of failing) is a separate fix to the same problem, so link the two rather than merge them.
kyle-sexton commentedon Nov 16, 2026
First live test of PR #16650, after the 22:17Z brake (cleared 22:33Z): reconcile run https://github.com/melodic-software/claude-code-plugins/actions/runs/43971789832 (22:37Z).
Remaining: drain the post-brake backlog faster, keeping the stagger. Two options: a larger cap for the first pass after a brake clear, or re-running only the newest lost run per PR, since older heads are superseded. Basis: the reconcile log lines above.
fix(ci): supersede a lost lane run only by checks that start its lane
github-actions commentedon Nov 16, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
github-actions commentedon Nov 16, 2026
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Blast radius: the lost-run reconciler script, its workflow and their tests; maintainers and the pipeline's recovery of stalled PR runs.
Reversible: yes; revert the change, and the 30-minute cron pass resumes as the only recovery trigger.
Open questions: none
Decision comment, quoted:
#16642 (comment)
The decision settles the open questions. Items 1 and 2 are recorded as done (#16650, #16814, #17014, #17033, #17003, #17023, #17028), and the kill-switch part moved to #17285. What remains is a follow-up reconcile pass when a pass defers re-runs for want of a place.
MAX_IN_FLIGHTstays 3, and the six done-when items in the decision define the scope. Related, not duplicates: #17285 and #16944.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 16, 2026
Not admitted to the issue lane: merged pull request #16650 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 17, 2026
Item 3 seen live on today's first brake, 2026-11-17 (BRAKE-SET 17:27:45Z, REFUSED at 0% left; BRAKE-CLEAR 17:42Z at 89%).
reconcile-rerun-*artifact.reconcile: #16892: deferred, all 3 re-run places taken.So the reconciler re-runs only runs lost to the brake, capped at 3 a pass. A brake that leaves more than 3 runs takes one extra scheduled pass, about 25 minutes, for each further 3.
test(source-control): cover the held leg's annotation read in collect…
kyle-sexton commentedon Nov 18, 2026
Done-when item 1: the callers, with request counts. Measured from the jobs' api-budget notices (
core-by-reset, big counter only) over the two GITHUB_TOKEN brake hours of 2026-11-17, W1 20:42-21:42Z and W2 22:42-23:42Z.Fixes so far:
kyle-sexton commentedon Nov 18, 2026
Two more occurrences on 2026-11-18, details in #17248 (closed as a duplicate of this issue):
limit=5000 remaining=0 used=5000 reset=1795031207(19:46:47Z). CI Status failed on reads and on theci-lanesstatus write in the same hour. feat: run improvement skills across several repositories #17228 failed the same way in run 44381351313.CLAUDE_LANES_REPO_DISABLEDwas on while both PRs' review lanes started, so all four lane runs skipped, and the variable was set back to false at 19:47:29Z. The lost-run reconciler re-ran only one of those four runs: PR Review Hosted 44382598651. Security 44382598493 on docs(upstream): record the owner's 2026-11-18 re-decision of the home-repo navigation rows #17225 and both lanes on feat: run improvement skills across several repositories #17228 (44383832798, 44383832780) stayed at attempt 1 until they were re-run by hand. The cause is UNVERIFIED: the reconciler may not have run yet, or it may skip runs whose conclusion issuccess. Lane runs that skipped this way concludesuccess, but the check reports that they did not run.kyle-sexton commentedon Nov 18, 2026
Decision recorded. What is left is done-when item 3 at scale: the lost-run reconciler drains a post-brake backlog at most 3 re-runs per 30-minute pass. The rest is done or moved.
Done since the last refinement:
CLAUDE_LANES_REPO_DISABLED, one re-run) moves to its own bug, Re-run every review lane a kill switch stopped once the switch is off, not one lane per head per pass #17285.Still open, evidence from today's post-clear pass (run 44384109700, 19:48:59Z):
reconcile: lane run 44381721733 (.github/workflows/pr-verify.yml): deferred, all 3 re-run places taken, the same for 7 more pr-verify runs and#17117: deferred, all 3 re-run places taken. Passes run on cron4,34 * * * *, so each further 3 waits 30 minutes. #16658 already drops lost runs that follow no open head, so the second option in #16642 (comment) is done; this records the first one's replacement.RECOMMENDED (vetoable, reversible): keep
MAX_IN_FLIGHTat 3 and the burn-rate pacer; a pass that defers recovery work for want of a place starts one follow-up pass instead of waiting for the next scheduled tick. Basis:maintenance-reconcile-lost-runs.ymlcron andMIN_GAP_MINUTES: "30",reconcile.mjsMAX_IN_FLIGHT = 3and its header ("A pass that defers anything but a flake-only set writes noclean, so the pacer runs the next tick"), read at origin/main today.Done when:
MAX_IN_FLIGHTstays 3.3 remaining items
github-actions commentedon Nov 18, 2026
melodic-automation-lanes commentedon Nov 18, 2026
This records a re-fire of an earlier stop, not a new one. A GitHub API pool's stop floor stopped run 44406300700, the work-resolve-issue run that
agent-readystarted, after its model ran, until 2026-11-18T22:46:50Z. work-unblock has now removed and addedagent-readyback to start a fresh run early in a window. A floor stop is the lanes' load, not a fault in this issue. After limits.limit-stops-per-issue of these since a person last addedagent-readyor a run last reported, work-unblock waits the limits.stranded-backoff-hoursfailed-againbackoff instead.github-actions commentedon Nov 18, 2026
fix(work-resolve-issue): keep the stop-floor no-fail to the longer wa…
github-actions commentedon Nov 19, 2026
fix(work-resolve-issue): keep the floor re-fire reading the named han…
melodic-automation-lanes commentedon Nov 19, 2026
The work-resolve-issue lane stopped on this issue (
patch-protected-path): the change needs a path the issue lane may not push, which a maintainer session builds by hand.It added
host-runand set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/44424531077
To hand it back to the lane, remove
host-runand set Work status to agent-ready again.kyle-sexton commentedon Nov 20, 2026
Callers in the two windows (done-when item 1), summed from every row of the api-budget call collector whose reset falls within 5 minutes of each window's reset. The collector attributes 3,540 and 3,753 of the roughly 4,900 calls spent; the rest are unattributed.
Since then, 24 h core calls on this token fell from 116,767 to 99,218 (to 2026-11-20T05:20Z) through the GitHub Optimization cuts. The kill switch last tripped at 2026-11-19T23:25Z, and none has tripped on 2026-11-20 through 14:07Z. Today's busy hours have not run yet.
Re-decision (GitHub Work's Q106, option A, handed to GitHub Optimization): items 2 and 3 are the paced start gate that #17460 builds, which defers optional PR jobs past the reset instead of failing them. So this issue stays open, tied to #17460, with no separate build. It closes when #17460's gate merges, or when a week of the hourly cadence shows no floor hit, whichever comes first.
kyle-sexton commentedon Nov 21, 2026
Evidence: the ci-lanes record is the largest single cause of red full runs on main.
From the 306 full
pr-require-checks.ymlruns on main since 2026-11-14, 41 failed attempt 1. Twelve of them failed only in CI Status, all with the same annotation:All twelve land at :16 to :26 past the hour, the end of an hourly window when the pool is lowest. None has come since 2026-11-19 23:16Z, after the spend cuts of 11-20 and 11-21. Each one turns a run whose tests passed red, which holds the post-merge breaker until a newer commit's full run passes or the lost-run reconciler's floor re-run lands.
There was one more rate-limit red, of a different kind. In 44565037503 (2026-11-19 16:20),
scripts/ci-queued-pr.test.shin Test Bash (6) readCould not read queued pull request #7492: rate limited (every pool until 2026-11-19T17:31:25Z). Not checked yet: whether that suite read the runner's real rate-guard state. A 2026-11-14 codeload download timeout (43638508739) is not a pool failure.Where the record step lives: the
report-ci-statuscomposite of melodic-software/ci-workflows, pinned at v0.78.0 (pr-require-checks.yml:7278). Itsrate-limit-wait-secondsis held toWAIT_BUDGET_SECONDS.step(.github/actions/log-api-budget/rate-guard.mjs:56-64), by outage posture element 1 (docs/conventions/pr-pipeline/README.md, owner decision Q677): a wait past a small per-step budget counts as unknown instead of a sleep. Waiting out the reset in this step would add a second exception beside work-resolve-issue's, so it is a change to that owner decision.Data: 306 runs listed, then attempt 1's failed jobs and their failure annotations for the 41 failed runs (~130 calls).