diff --git a/.changes/pr-pipeline/perf-17460-defer-reviews-948b2f21.md b/.changes/pr-pipeline/perf-17460-defer-reviews-948b2f21.md new file mode 100644 index 00000000000..2c0e2547ec6 --- /dev/null +++ b/.changes/pr-pipeline/perf-17460-defer-reviews-948b2f21.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- Both hosted review callers' gates defer a first attempt (`defer: "true"`, `defer-job: review`) while the GITHUB_TOKEN window is projected to run short before the review ends, reading their horizon from the new `defer-run-minutes-review` (github-token default 11, the p90 of 418 hosted review runs) through `pr-pipeline.yaml`'s `limits.api` over `rate-limits.json`. Their stop notice now prints `cause= reset=`. The lost-run reconciler reads that notice: it waits until a minute past the reset, then re-runs the deferred run once, recorded, in its own band of at most 12 re-runs a pass beside the 3 places other recovery takes; the reconciler workflow uploads up to 15 records a pass. The re-run's attempt 2 never defers ([#17460](https://github.com/melodic-software/claude-code-plugins/issues/17460)). diff --git a/.github/actions/check-kill-switch/README.md b/.github/actions/check-kill-switch/README.md index 24fc6dc1ea0..c7bbcccccf3 100644 --- a/.github/actions/check-kill-switch/README.md +++ b/.github/actions/check-kill-switch/README.md @@ -41,7 +41,9 @@ to the job's horizon, or to the reset when that comes first, leaves under github `defer-project` share of the header's limit. The horizon, the p90 of the job's measured run time, is the gate's `defer-minutes` when set, else github-token's `defer-run-minutes-`, else `defer-run-minutes`. `defer-run-minutes-pr-merge` is 6: the p90 of the 29 first-attempt pr-merge -arm runs listed on 2026-11-20 is 327 s. The github-token values are read from `resolved-config`'s `limits.api` (`pr-pipeline.yaml`) over `rate-limits.json` +arm runs listed on 2026-11-20 is 327 s. `defer-run-minutes-review` is 11, for both hosted review +callers' gates: the p90 of the 418 hosted review runs of 2 minutes or more among the 300 newest of +each caller, listed on 2026-11-21, is 10.4 minutes. The github-token values are read from `resolved-config`'s `limits.api` (`pr-pipeline.yaml`) over `rate-limits.json` (`defer-project` 0.3, `defer-run-minutes` 15: on the mean burn curve of the 16 windows that ran out between 2026-11-18 and 2026-11-20, at a median of minute 50, those values start deferring at about minute 23, and the mean curve of the 17 windows that did not run out never reaches them). `defer-project` 0 @@ -86,7 +88,10 @@ and the pr-unblock legs (its dispatcher and pr-refine), which fix existing pull which a person or a pull request starts, keep it too. Set `defer: "true"` instead on a gate that finishes existing work only where a later run that never -defers recovers it: pr-merge's PR-scoped arm run (`inputs.pr`), whose work the 30-minute sweep does. +defers recovers it: pr-merge's PR-scoped arm run (`inputs.pr`), whose work the 30-minute sweep does, +and the two hosted review callers' gates, whose stop notice prints `cause=defer reset=`, so +the lost-run reconciler re-runs the run once that reset is past (reconcile.mjs `DEFER_NOTE`, in its +own `DEFER_MAX` band). There a low budget delays the work rather than throwing it away, which is the reason the paragraph above keeps the budget off. CI Status and pr-dispatch-feedback never defer. A budget or defer stop ends the run neutral: a notice, no failed step. A job that sets `budget: "true"` or `defer: "true"` diff --git a/.github/actions/check-kill-switch/check-kill-switch.test.mjs b/.github/actions/check-kill-switch/check-kill-switch.test.mjs index b045d83f98d..5b70280742d 100644 --- a/.github/actions/check-kill-switch/check-kill-switch.test.mjs +++ b/.github/actions/check-kill-switch/check-kill-switch.test.mjs @@ -403,6 +403,11 @@ test("the resolved config overrides the shipped share and horizon", async () => assert.equal((await runMain(DEFER, reading(2000, 1800), { resolved: config({ "defer-project": 0.25, "defer-run-minutes": 5 }) })).outputs.cause, "ok", why); }); +test("the shipped horizon for defer-job review is 11 minutes, the hosted reviews' measured p90", async () => { + const gate = await deferGate({ rateLimits: SHIPPED_LIMITS, job: "review", log: () => {} }); + assert.deepEqual([gate.minutes, gate.labels.minutes], [11, "defer-run-minutes-review"]); +}); + test("the shipped horizon for defer-job pr-merge is 6 minutes, and an unset job key reads defer-run-minutes", async () => { assert.equal((await deferGate({ rateLimits: SHIPPED_LIMITS, job: "pr-merge", log: () => {} })).minutes, 6); const other = await deferGate({ rateLimits: SHIPPED_LIMITS, job: "pr-verify", log: () => {} }); @@ -506,9 +511,9 @@ const GATES = { "pr-dispatch-feedback.yml": { "post-verify-notes": "switch", dispatch: "switch", "rerun-ci-status": "switch" }, "pr-merge.yml": { "dispatch-post-merge-runs": "switch", merge: "switch", "enable-auto-merge": "defer" }, "pr-rate-merge-risk.yml": { "rate-merge-risk": "switch", "rate-merge-risk-subscription": "switch", "rate-merge-risk-fallback": "switch" }, - "pr-review-hosted.yml": { "check-kill-switch": "switch" }, + "pr-review-hosted.yml": { "check-kill-switch": "defer" }, "pr-review-main.yml": { "check-trigger": "switch", "collect-trusted-text": "switch", review: "switch", "review-subscription": "switch", "review-fallback": "switch" }, - "pr-review-security-hosted.yml": { "check-kill-switch": "switch" }, + "pr-review-security-hosted.yml": { "check-kill-switch": "defer" }, // check-broker reads the budget only for a caller passing start-gate, and // only on a call that is the first in its slot (FIRST_CALL). "pr-run-activity-read.yml": { "check-broker": "first-call", run: "switch" }, @@ -542,7 +547,11 @@ const GATES = { "work-unblock.yml": { "label-unblocked": "start", "publish-lane-level": "switch", "sync-status": "switch" }, }; -const DEFER_INPUT = /^ {10}defer: \$\{\{ inputs\.pr && 'true' \|\| 'false' \}\}$/m; +const DEFER_INPUTS = { + "pr-merge.yml": /^ {10}defer: \$\{\{ inputs\.pr && 'true' \|\| 'false' \}\}$/m, + "pr-review-hosted.yml": /^ {10}defer: "true"$/m, + "pr-review-security-hosted.yml": /^ {10}defer: "true"$/m, +}; const FIRST_CALL = /^ {10}budget: \$\{\{ inputs\.start-gate && \(inputs\.fallback == '' \|\| inputs\.fallback == 'credit-stop' \|\| inputs\.fallback == 'no-key'\) && 'true' \|\| 'false' \}\}$/m; const FIRST_CALL_PROJECT = /^ {10}project: \$\{\{ inputs\.start-gate && \(inputs\.fallback == '' \|\| inputs\.fallback == 'credit-stop' \|\| inputs\.fallback == 'no-key'\) && 'true' \|\| 'false' \}\}$/m; @@ -604,7 +613,8 @@ test("every gate in the workflows is classified, and only start gates read the b assert.match(step, FIRST_CALL, `${file} ${job} starts new work on a starting caller's first call only`); assert.match(step, FIRST_CALL_PROJECT, `${file} ${job} projects the window on a starting caller's first call only`); } else if (kind === "defer") { - assert.match(step, DEFER_INPUT, `${file} ${job} defers its PR-scoped run only`); + assert.match(step, DEFER_INPUTS[file], `${file} ${job} defers only the runs a later run recovers`); + assert.match(step, /^ {10}resolved-config: \$\{\{ runner\.temp \}\}\/defer-config\.json$/m, `${file} ${job} reads the repository's override`); assert.match(step, /^ {10}defer-job: [a-z0-9-]+$/m, `${file} ${job} names its horizon key`); assert.doesNotMatch(step, /^ {10}defer-minutes:/m, `${file} ${job} reads its horizon from the config`); assert.doesNotMatch(step, /budget:|project:/, `${file} ${job} reads no start-gate floor`); diff --git a/.github/actions/log-api-budget/rate-guard.mjs b/.github/actions/log-api-budget/rate-guard.mjs index 75822ea1d14..2cee2facad2 100644 --- a/.github/actions/log-api-budget/rate-guard.mjs +++ b/.github/actions/log-api-budget/rate-guard.mjs @@ -114,7 +114,7 @@ const DEFAULTS = new URL("./rate-limits.json", import.meta.url); // (rate-guard.test.mjs holds the two equal). const SHARES = { plan: 0.7, "writer-pause": 0.4, breaker: 0.2, "read-floor": 0.1, "stop-floor": 0.02 }; export const BUILT_IN_LIMITS = Object.freeze({ - "github-token": { ...SHARES, "limit-cap": 1000, pace: 0.2, "pace-pr-merge": 0.3, "start-gate": 0.1, "start-project": 0.3, "start-run-minutes": 21, "defer-project": 0.3, "defer-run-minutes": 15, "defer-run-minutes-pr-merge": 6 }, + "github-token": { ...SHARES, "limit-cap": 1000, pace: 0.2, "pace-pr-merge": 0.3, "start-gate": 0.1, "start-project": 0.3, "start-run-minutes": 21, "defer-project": 0.3, "defer-run-minutes": 15, "defer-run-minutes-pr-merge": 6, "defer-run-minutes-review": 11 }, "lanes-app": SHARES, graphql: SHARES, "owner-token": SHARES, @@ -129,7 +129,7 @@ const COUNT = (value) => Number.isInteger(value) && value >= 1; // optional, the share of the header's limit left at which the latch stops; // start-gate, optional, check-kill-switch's share under which a lane does not // start, which that gate reads from rate-limits.json alone. -const FIELDS = { plan: (value) => SHARE(value) && value > 0, "writer-pause": SHARE, breaker: SHARE, "read-floor": SHARE, "stop-floor": SHARE, "limit-cap": COUNT, pace: SHARE, "pace-pr-merge": SHARE, "start-gate": SHARE, "start-project": SHARE, "start-run-minutes": COUNT, "defer-project": SHARE, "defer-run-minutes": COUNT, "defer-run-minutes-pr-merge": COUNT }; +const FIELDS = { plan: (value) => SHARE(value) && value > 0, "writer-pause": SHARE, breaker: SHARE, "read-floor": SHARE, "stop-floor": SHARE, "limit-cap": COUNT, pace: SHARE, "pace-pr-merge": SHARE, "start-gate": SHARE, "start-project": SHARE, "start-run-minutes": COUNT, "defer-project": SHARE, "defer-run-minutes": COUNT, "defer-run-minutes-pr-merge": COUNT, "defer-run-minutes-review": COUNT }; const CONTENT_FIELDS = { ...FIELDS, "per-minute": COUNT, "per-hour": COUNT }; // Fields rate-limits.json holds that pr-pipeline.yaml's limits.api cannot set // (its schema rejects them): their one reader takes the file's value alone. diff --git a/.github/actions/log-api-budget/rate-limits.json b/.github/actions/log-api-budget/rate-limits.json index 688fda97278..e0efc211034 100644 --- a/.github/actions/log-api-budget/rate-limits.json +++ b/.github/actions/log-api-budget/rate-limits.json @@ -1,5 +1,5 @@ { - "github-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02, "limit-cap": 1000, "pace": 0.2, "pace-pr-merge": 0.3, "start-gate": 0.1, "start-project": 0.3, "start-run-minutes": 21, "defer-project": 0.3, "defer-run-minutes": 15, "defer-run-minutes-pr-merge": 6 }, + "github-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02, "limit-cap": 1000, "pace": 0.2, "pace-pr-merge": 0.3, "start-gate": 0.1, "start-project": 0.3, "start-run-minutes": 21, "defer-project": 0.3, "defer-run-minutes": 15, "defer-run-minutes-pr-merge": 6, "defer-run-minutes-review": 11 }, "lanes-app": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, "graphql": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, "owner-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, diff --git a/.github/actions/resolve-config/callers.test.mjs b/.github/actions/resolve-config/callers.test.mjs index ba58acf7d46..52d980d7209 100644 --- a/.github/actions/resolve-config/callers.test.mjs +++ b/.github/actions/resolve-config/callers.test.mjs @@ -172,7 +172,7 @@ for (const [file, job, lane, laneSwitch] of [ test(`${file}: the review runs only when the check-kill-switch job says proceed`, () => { const review = jobs[job]; const gate = jobs["check-kill-switch"]; - assertKillSwitchJob(gate); + assertKillSwitchJob(gate, { defers: true }); for (const proceed of ["true", "false", ""]) { const context = { "github.event.pull_request.draft": false, @@ -272,7 +272,7 @@ function importClosure(entry) { // The check-kill-switch job reads the action from the default branch with a // read-only token and passes it the org and repository switches; a gate // that starts new work also passes budget "true". -function assertKillSwitchJob(gate, { starts = false } = {}) { +function assertKillSwitchJob(gate, { starts = false, defers = false } = {}) { assert.deepEqual(gate.permissions, { contents: "read" }); const checkout = gate.steps.find((step) => step.uses?.startsWith("actions/checkout@")); assertMainTip(gate.steps, checkout); @@ -285,6 +285,7 @@ function assertKillSwitchJob(gate, { starts = false } = {}) { value: "${{ vars.CLAUDE_LANES_DISABLED }}", "repo-value": "${{ vars.CLAUDE_LANES_REPO_DISABLED }}", ...(starts && { budget: "true" }), + ...(defers && { defer: "true", "defer-job": "review", "resolved-config": "${{ runner.temp }}/defer-config.json" }), }); } diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs index c3281ca6b5f..49e16997709 100755 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs @@ -151,7 +151,12 @@ // API rate limit (`rateHeld`), named by ci-status's RATE_HELD_NOTE, goes the // same way, once RATE_RESET_GRACE_MS past the latest reset its refusals // name; the merge gate binds that attempt 2 (babysit_review_evidence -// `_rate_held`). +// `_rate_held`). A first attempt the defer gate stopped (its gate job's +// DEFER_NOTE, cause=defer with the window's reset) waits until +// RATE_RESET_GRACE_MS past that reset, so the first pass after it, a few +// minutes past the reset when the hourly tick follows it, re-runs it. Those +// re-runs take the DEFER_MAX band, not the places below; their attempt 2 +// never defers. // Other lost lane runs (a review lane run lost any other way, refine) are listed // in the summary for a person: a review re-run would trade one review for // another, and refine's lanes token broker refuses a re-run @@ -228,7 +233,8 @@ // SETTLED_PATH plan reads the last pass's settled runs and job reads, then writes // this pass's there for the upload; unset carries nothing // GITHUB_RUN_ID, RUNNER_TEMP this run, and where the last carry is downloaded -// GITHUB_OUTPUT plan writes record-1 to record-3 there (empty when unused) +// GITHUB_OUTPUT plan writes record-1 to record-15 there (empty when unused; +// https://github.com/melodic-software/claude-code-plugins/blob/main/.github/workflows/maintenance-reconcile-lost-runs.yml) // GITHUB_STEP_SUMMARY the outcome is listed there when set // GITHUB_API_URL import { appendFileSync, readFileSync, writeFileSync } from "node:fs"; @@ -420,6 +426,12 @@ export const FAULT_HELD_HOURS = 6; // for the next pass, so a pass after a long outage adds a bounded load. export const FAULT_HELD_READS = 100; export const MAX_IN_FLIGHT = 3; +// Re-runs of reviews the defer gate stopped, per pass, beside MAX_IN_FLIGHT: +// the 24 h to 2026-11-21T20:49Z would have deferred at most 9 review runs in +// one token window, so 12 leaves room for one more head, both lanes +// (https://github.com/melodic-software/claude-code-plugins/issues/17460). +export const DEFER_MAX = 12; +export const RECORD_SLOTS = MAX_IN_FLIGHT + DEFER_MAX; export const WRITE_GAP_MS = 1000; // Pages of 100 read per status query of a lane's run listing; a listing that // does not fit is a failure, never a partial read. @@ -1050,6 +1062,15 @@ export const rateHeld = async (jobs, notesOf) => { return undefined; }; export const RATE_RESET_GRACE_MS = 60_000; +// The hosted callers' stop notice when the gate deferred the review, with the +// window's reset in epoch seconds (https://github.com/melodic-software/claude-code-plugins/blob/main/.github/actions/check-kill-switch/README.md). +export const DEFER_NOTE = /\bcause=defer reset=([0-9]{9,12})\b/; +const DEFERRED = "success (deferred)"; +export const deferReset = async (jobs, notesOf) => { + const gate = jobs.find((job) => GATE_JOBS.has(job?.name) && Number.isSafeInteger(job.id) && job.steps?.some((step) => step?.name === REVIEW_STOP_STEP && step.conclusion === "success")); + const resets = gate ? (await notesOf(gate)).map((a) => DEFER_NOTE.exec(String(a?.message ?? ""))?.[1]).filter(Boolean).map(Number) : []; + return resets.length > 0 ? Math.max(...resets) : undefined; +}; /** * Why the run's first attempt must not be re-run, with whether that hold is @@ -1499,8 +1520,14 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp } const jobs = await readJobs(r); const stopped = reviewStopped(jobs); + const notesOf = (job) => job.annotations ?? call("GET", `/repos/${repo}/check-runs/${job.id}/annotations?per_page=100`); + const defer = stopped ? await deferReset(jobs, notesOf) : undefined; + if (beforeReset(defer === undefined ? undefined : defer * 1000, now)) { + waitReset(`#${pull.number}: review run ${r.id} (${r.path})`, defer * 1000, "was deferred on the GITHUB_TOKEN budget"); + continue; + } if (!stopped) { - const reset = await rateHeld(jobs, (job) => job.annotations ?? call("GET", `/repos/${repo}/check-runs/${job.id}/annotations?per_page=100`)); + const reset = await rateHeld(jobs, notesOf); if (reset === undefined) { continue; } @@ -1511,10 +1538,10 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp } const hold = await gateHold(); if (hold) { - say(`#${pull.number}: review run ${r.id} (${r.path}) ${stopped ? "was stopped by a kill switch" : "held on a GitHub API rate limit"}; waiting, ${hold}`); + say(`#${pull.number}: review run ${r.id} (${r.path}) ${defer !== undefined ? "was deferred on the GITHUB_TOKEN budget" : stopped ? "was stopped by a kill switch" : "held on a GitHub API rate limit"}; waiting, ${hold}`); continue; } - const conclusion = stopped ? "success (kill switch stop)" : "success (rate-limit hold)"; + const conclusion = defer !== undefined ? DEFERRED : stopped ? "success (kill switch stop)" : "success (rate-limit hold)"; items.push({ kind: "review", pr: pull.number, id: r.id, path: r.path, conclusion, attempt: 1, head: pull.head.sha, record: rerunRecordName(r.id, 2, pull.head.sha) }); } return items; @@ -1966,7 +1993,26 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp : places(set) > room ? `only ${room} of ${MAX_IN_FLIGHT} re-run places left` : ""; - const place = (what, set) => { + let deferRoom = DEFER_MAX; + const takeDeferred = (what, set) => { + const reviews = set.filter(isReview); + if (reviews.length === 0 || !reviews.every((item) => item.conclusion === DEFERRED)) { + return set; + } + if (reviews.length > deferRoom) { + deferred += 1; + say(`${what}: its deferred reviews wait, only ${deferRoom} of ${DEFER_MAX} recovery places left this pass`); + } else { + items.push(...reviews); + deferRoom -= reviews.length; + } + return set.filter((item) => !isReview(item)); + }; + const place = (what, whole) => { + const set = takeDeferred(what, whole); + if (set.length === 0) { + return; + } const core = set.filter((item) => !isReview(item)); const split = short(set) === NO_UPLOAD && places(set) <= room && core.length > 0 && core.length < set.length && short(core) === ""; if (split) { @@ -2062,7 +2108,7 @@ async function act({ repo, call, say, since, sleep, items }) { : item.kind === "too-old" ? `${where}: re-ran run ${item.id} (${item.path}) so pr-verify judges the head again: the pull request is past ${TOO_OLD_DAYS} days and its head has no verdict` : item.kind === "review" - ? `${where}: re-ran review run ${item.id} (${item.path}), which ${item.conclusion === "success (rate-limit hold)" ? "held on a GitHub API rate limit" : "a kill switch stopped"}; its completion re-runs ci-status` + ? `${where}: re-ran review run ${item.id} (${item.path}), which ${item.conclusion === "success (rate-limit hold)" ? "held on a GitHub API rate limit" : item.conclusion === DEFERRED ? "the defer gate stopped until the GITHUB_TOKEN window reset" : "a kill switch stopped"}; its completion re-runs ci-status` : item.kind === "fault-held" ? `${where}: re-ran run ${item.id} (${item.path}), which ${item.path === RATER ? "the lanes kill switch stopped before it rated" : "held on a GitHub fault"}` : `${where}: re-ran run ${item.id} (${item.path}), which ended ${item.conclusion}`, @@ -2083,10 +2129,10 @@ async function act({ repo, call, say, since, sleep, items }) { // The plan's record-N outputs, each a record name or empty. function recordOutputs(items) { const names = items.filter(recorded).map((item) => item.record); - if (names.length > MAX_IN_FLIGHT || !names.every(isRecordName)) { + if (names.length > RECORD_SLOTS || !names.every(isRecordName)) { throw new Error("the plan's record names are malformed"); } - return Object.fromEntries(Array.from({ length: MAX_IN_FLIGHT }, (_, i) => [`record-${i + 1}`, names[i] ?? ""])); + return Object.fromEntries(Array.from({ length: RECORD_SLOTS }, (_, i) => [`record-${i + 1}`, names[i] ?? ""])); } export async function main({ env = process.env, github, fetchStatus: readStatus = fetchStatus, sleep = delay, now = Date.now(), download } = {}) { diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs index 74fd6fefcc3..dea9f949469 100644 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs @@ -1,3 +1,4 @@ +// test-scope: .github/workflows/*.yml // Expected values come from the reconciler's spec: a CI run on a PR's current // head whose newest attempt ended cancelled, timed_out or startup_failure, // with no newer run of that workflow on the head, is re-run once; at most 3 @@ -19,7 +20,7 @@ import { test } from "node:test"; import { createGitHub, GitHubError } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; import { createRateLimitLatch, RateLimitStop } from "../../actions/log-api-budget/rate-guard.mjs"; -import { CI_WORKFLOWS, FLOOR_NOTE, floorNoted, floorReset, RATE_STEP, RATE_WRITE_STEP, RATER, RATER_LEGS, RATER_RESOLVE_STEP, RATER_SWITCH_STEP, RECORDED_LANES, GITHUB_STATUS_URL, JUDGE_PREDECESSOR, JUDGE_STEP, KILL_SWITCH_NOTE, LANE_TRIGGERS, main, NPM_STATUS_URL, MISSING_REVIEW_NOTE, PENDING_BACKOFF_MINUTES, PENDING_REVIEW_NOTE, prefilterSpec, RATE_HELD_NOTE, REVIEW_STEP, REVIEW_STOP_NOTE, REVIEW_STOP_STEP, REVIEW_SWITCHES, SETTLED_VERSION, TOO_OLD_DAYS, UPSTREAM_STOP_STEP, VERDICT_STOP_STEP, WRITE_STEP } from "./reconcile.mjs"; +import { CI_WORKFLOWS, DEFER_MAX, DEFER_NOTE, RECORD_SLOTS, FLOOR_NOTE, floorNoted, floorReset, RATE_STEP, RATE_WRITE_STEP, RATER, RATER_LEGS, RATER_RESOLVE_STEP, RATER_SWITCH_STEP, RECORDED_LANES, GITHUB_STATUS_URL, JUDGE_PREDECESSOR, JUDGE_STEP, KILL_SWITCH_NOTE, LANE_TRIGGERS, main, NPM_STATUS_URL, MISSING_REVIEW_NOTE, PENDING_BACKOFF_MINUTES, PENDING_REVIEW_NOTE, prefilterSpec, RATE_HELD_NOTE, REVIEW_STEP, REVIEW_STOP_NOTE, REVIEW_STOP_STEP, REVIEW_SWITCHES, SETTLED_VERSION, TOO_OLD_DAYS, UPSTREAM_STOP_STEP, VERDICT_STOP_STEP, WRITE_STEP } from "./reconcile.mjs"; import { switchHold } from "../pr-merge/merge-holds.mjs"; import { tempDir } from "../../../lib/test-tmp.mjs"; @@ -2817,6 +2818,75 @@ test("the second stopped lane goes on the next pass, after the first has its re- assert.deepEqual(reruns, [301]); }); +const RESET_S = NOW / 1000 + 600; +const deferNote = (reset = RESET_S) => [{ annotation_level: "notice", message: `a kill switch stops this review (lanes: kill-switch cause=defer reset=${reset}, CLAUDE_REVIEW_DISABLED: unset)` }]; +const deferredJobs = (reset) => stoppedJobs().map((job) => (job.id === 312 ? { ...job, notes: deferNote(reset) } : job)); +const deferredRun = (id, head, opts = {}, reset = RESET_S) => stoppedRun(id, head, opts, deferredJobs(reset)); +const bothDeferred = (n, reset) => [redChecks(100 + n, sha(n)), deferredRun(200 + n, sha(n), {}, reset), deferredRun(300 + n, sha(n), { path: SECURITY_REVIEW }, reset)]; + +test("a deferred review waits until RATE_RESET_GRACE_MS past the reset its notice names, then is re-run whole, recorded", async (t) => { + const log = t.mock.method(console, "log", () => {}); + const before = await reconcile({ pulls: [pr(1)], runs: bothDeferred(1), env: SWITCHES_OFF, now: RESET_S * 1000 + 60_000 }); + assert.equal(before.code, 0); + assert.deepEqual(before.reruns, [], "inside the grace, so the one re-run is not spent in the window that deferred it"); + assert.equal(before.outputs.clean, "", "the next tick runs, not paced"); + assert.ok(log.mock.calls.some((c) => String(c.arguments[0]).startsWith(`reconcile: #1: review run 201 (${REVIEW}) was deferred on the GITHUB_TOKEN budget; waiting for the rate-limit reset at`))); + assert.ok(before.calls.includes(`GET /repos/${REPO}/check-runs/312/annotations?per_page=100`), "the gate job's notice is read for the reset"); + const after = await reconcile({ pulls: [pr(1)], runs: bothDeferred(1), env: SWITCHES_OFF, now: RESET_S * 1000 + 61_000 }); + assert.deepEqual(after.reruns, [201, 301]); + assert.deepEqual(after.failedJobReruns, []); + assert.deepEqual(records(after.outputs), [`reconcile-rerun-201-2-${sha(1)}`, `reconcile-rerun-301-2-${sha(1)}`, ""]); +}); + +test("deferred reviews take their own band, so they go beside a full set of re-run places", async (t) => { + const log = t.mock.method(console, "log", () => {}); + const runs = [windowsLost(1), windowsLost(2), windowsLost(3), ...bothDeferred(4), ...bothDeferred(5)]; + const { code, reruns, outputs } = await reconcile({ pulls: [1, 2, 3, 4, 5].map((n) => pr(n)), runs, env: SWITCHES_OFF, now: RESET_S * 1000 + 61_000 }); + assert.equal(code, 0); + assert.deepEqual(reruns, [401, 402, 403, 204, 304, 205, 305]); + assert.equal(outputs["record-4"], `reconcile-rerun-305-2-${sha(5)}`, "past the three places' uploads"); + assert.ok(!log.mock.calls.some((c) => String(c.arguments[0]).includes("#4: deferred"))); +}); + +test("the defer band starts at most DEFER_MAX re-runs a pass, a head's two lanes together, and the rest wait with no clean", async (t) => { + const log = t.mock.method(console, "log", () => {}); + const heads = Array.from({ length: DEFER_MAX / 2 + 1 }, (_, i) => i + 1); + const { code, reruns, outputs } = await reconcile({ pulls: heads.map((n) => pr(n)), runs: heads.flatMap((n) => bothDeferred(n)), env: SWITCHES_OFF, now: RESET_S * 1000 + 61_000 }); + assert.equal(code, 0); + assert.equal(reruns.length, DEFER_MAX); + assert.deepEqual(records(outputs).length, 3); + assert.equal(outputs[`record-${DEFER_MAX}`], `reconcile-rerun-${300 + DEFER_MAX / 2}-2-${sha(DEFER_MAX / 2)}`); + assert.equal(outputs[`record-${DEFER_MAX + 1}`], ""); + assert.equal(outputs.clean, ""); + assert.ok(log.mock.calls.some((c) => c.arguments[0] === `reconcile: #${DEFER_MAX / 2 + 1}: its deferred reviews wait, only 0 of ${DEFER_MAX} recovery places left this pass`)); +}); + +test("the workflow has one scanned upload step for each record the plan can name", () => { + const text = readFileSync(new URL("../../workflows/maintenance-reconcile-lost-runs.yml", import.meta.url), "utf8"); + const uploads = [...text.matchAll(/^ {6}- name: Record re-run ([0-9]+)$/gm)].map((m) => Number(m[1])); + assert.deepEqual(uploads, Array.from({ length: RECORD_SLOTS }, (_, i) => i + 1)); + for (const n of uploads) { + assert.match(text, new RegExp(`name: \\$\\{\\{ steps\\.plan\\.outputs\\.record-${n} \\}\\}`), `record-${n}`); + } +}); + +test("a stopped review with no defer notice takes a re-run place now, as a switch stop does", async () => { + const runs = [windowsLost(1), windowsLost(2), windowsLost(3), ...bothLanesStopped(4)]; + const { reruns } = await reconcile({ pulls: [1, 2, 3, 4].map((n) => pr(n)), runs, env: SWITCHES_OFF }); + assert.deepEqual(reruns, [401, 402, 403]); +}); + +test("the hosted callers print the gate's cause and reset in the stop notice the reconciler reads", () => { + for (const file of [REVIEW, SECURITY_REVIEW]) { + const text = readFileSync(new URL(`../../../${file}`, import.meta.url), "utf8"); + const line = /echo "::notice::a kill switch stops this (?:security )?review \(.*"$/m.exec(text)?.[0]; + assert.ok(line, file); + const shown = line.replace("${CAUSE:-unreadable}", "defer").replace("${RESET:-none}", "1800000000"); + assert.equal(DEFER_NOTE.exec(shown)?.[1], "1800000000", file); + assert.match(text, /^ {10}RESET: \$\{\{ steps\.kill\.outputs\.reset \}\}$/m, file); + } +}); + test("a lane's own switch keeps its re-run, with no read, and leaves the other lane's", async () => { const runs = [redChecks(101, sha(1)), stoppedRun(201, sha(1)), stoppedRun(301, sha(1), { path: SECURITY_REVIEW })]; const { code, reruns, calls } = await reconcile({ pulls: [pr(1)], runs, env: { CLAUDE_REVIEW_DISABLED: "true" } }); diff --git a/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs b/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs index 7ff06a26fe6..459d0fd3ce9 100644 --- a/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs +++ b/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs @@ -340,6 +340,15 @@ test("a head whose ci-lanes passed, is pending, is missing or came from another assert.deepEqual(result.dispatched, []); }); +// A review the defer gate stopped turns ci-status red on its review-stop note +// while the lanes pass: no fix-ci round, since the reconciler re-runs the +// review after the reset (https://github.com/melodic-software/claude-code-plugins/issues/17460, assumption A4). +test("a head whose ci-status failed only on a stopped review, with ci-lanes passed, gets no fix-ci round", async () => { + const { api, result } = await run({ pages: [[pr(7, { checks: [check("ci-status", "FAILURE", { title: "a kill switch was on when it started" }), lanes("SUCCESS")] })]] }); + assert.deepEqual(result.dispatched, []); + assert.deepEqual(api.calls.dispatch, []); +}); + test("a head that already has a fix-ci run is skipped, unless every run failed within the restart count", async () => { const fix = (conclusion, extra) => check("pr-unblock / fix-ci", conclusion, extra); const failed = lanes("FAILURE"); diff --git a/.github/workflows/maintenance-reconcile-lost-runs.yml b/.github/workflows/maintenance-reconcile-lost-runs.yml index 3a36cd2395f..ddd1b9896af 100644 --- a/.github/workflows/maintenance-reconcile-lost-runs.yml +++ b/.github/workflows/maintenance-reconcile-lost-runs.yml @@ -57,11 +57,13 @@ name: Maintenance Reconcile Lost Runs # # It does nothing while githubstatus.com shows Actions as anything but # operational, or the page cannot be read, so it adds no load mid-incident. At -# most 3 re-runs are in flight at once, auto-merge PRs first, and its writes +# most 3 re-runs are in flight at once, auto-merge PRs first, beside at most +# 12 re-runs a pass of hosted reviews the defer gate stopped, each once its +# window's reset is a minute past, and its writes # are at least a second apart. A lost pull_request review or refine lane run is listed in # the run summary for a person: those lanes, or the lanes token broker, refuse # a re-run github-actions[bot] starts. No model, and no PR code is checked out. -# Rules: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs. +# Rules: https://github.com/melodic-software/claude-code-plugins/blob/main/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs. # # How to stop it: set the organization or repository lanes kill-switch # variable to anything but "false". @@ -244,6 +246,222 @@ jobs: if-no-files-found: error retention-days: 1 + - name: Scan the upload for credentials + id: scan-record-4 + if: >- + steps.plan.outputs.record-4 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 4 + if: >- + (steps.plan.outputs.record-4 != '') && steps.scan-record-4.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-4 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-5 + if: >- + steps.plan.outputs.record-5 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 5 + if: >- + (steps.plan.outputs.record-5 != '') && steps.scan-record-5.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-5 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-6 + if: >- + steps.plan.outputs.record-6 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 6 + if: >- + (steps.plan.outputs.record-6 != '') && steps.scan-record-6.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-6 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-7 + if: >- + steps.plan.outputs.record-7 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 7 + if: >- + (steps.plan.outputs.record-7 != '') && steps.scan-record-7.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-7 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-8 + if: >- + steps.plan.outputs.record-8 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 8 + if: >- + (steps.plan.outputs.record-8 != '') && steps.scan-record-8.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-8 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-9 + if: >- + steps.plan.outputs.record-9 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 9 + if: >- + (steps.plan.outputs.record-9 != '') && steps.scan-record-9.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-9 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-10 + if: >- + steps.plan.outputs.record-10 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 10 + if: >- + (steps.plan.outputs.record-10 != '') && steps.scan-record-10.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-10 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-11 + if: >- + steps.plan.outputs.record-11 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 11 + if: >- + (steps.plan.outputs.record-11 != '') && steps.scan-record-11.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-11 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-12 + if: >- + steps.plan.outputs.record-12 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 12 + if: >- + (steps.plan.outputs.record-12 != '') && steps.scan-record-12.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-12 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-13 + if: >- + steps.plan.outputs.record-13 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 13 + if: >- + (steps.plan.outputs.record-13 != '') && steps.scan-record-13.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-13 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-14 + if: >- + steps.plan.outputs.record-14 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 14 + if: >- + (steps.plan.outputs.record-14 != '') && steps.scan-record-14.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-14 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + + - name: Scan the upload for credentials + id: scan-record-15 + if: >- + steps.plan.outputs.record-15 != '' + uses: ./.base/.github/actions/scan-upload # zizmor: ignore[self-repository] + with: + path: ${{ runner.temp }}/reconcile-record/record.json + + - name: Record re-run 15 + if: >- + (steps.plan.outputs.record-15 != '') && steps.scan-record-15.outputs.clean == 'true' + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: ${{ steps.plan.outputs.record-15 }} + path: ${{ runner.temp }}/reconcile-record/record.json + if-no-files-found: error + retention-days: 1 + # A Verdict re-run uploads a pr-feedback-deferred record, so this fixed # name goes first: the dispatch tick reads the per-head records only # while it is listed (feedback-dispatch.mjs DEFERRAL_MARKER). Kept a day diff --git a/.github/workflows/pr-review-hosted.yml b/.github/workflows/pr-review-hosted.yml index a619713949f..e1ea73d3dd8 100644 --- a/.github/workflows/pr-review-hosted.yml +++ b/.github/workflows/pr-review-hosted.yml @@ -104,6 +104,7 @@ jobs: /.github/actions/setup-pinned-node/ /.node-version /.github/actions/log-api-budget/ + /docs/conventions/pr-pipeline.yaml sparse-checkout-cone-mode: false - name: Set up Node @@ -111,12 +112,25 @@ jobs: with: node-version-file: .base/.node-version + - name: Read the defer override from the base config + continue-on-error: true + run: | + yq -o=json '{"limits": {"api": (.limits.api // {})}}' .base/docs/conventions/pr-pipeline.yaml >"$RUNNER_TEMP/defer-config.json" + + # A first attempt defers (cause defer) while the GITHUB_TOKEN window is + # projected to run short before the review ends; the stop below names + # the reset, and the lost-run reconciler re-runs the run once after it + # (https://github.com/melodic-software/claude-code-plugins/blob/main/.github/actions/check-kill-switch/README.md). + # A re-run attempt never defers. - name: Check the kill switch id: kill uses: ./.base/.github/actions/check-kill-switch # zizmor: ignore[self-repository] with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + defer: "true" + defer-job: review + resolved-config: ${{ runner.temp }}/defer-config.json # Skips the run while Actions or npm installs list an outage, with a # logged reason; fails open (.github/actions/check-upstream-status). @@ -132,9 +146,10 @@ jobs: env: REASON: ${{ steps.kill.outputs.reason }} CAUSE: ${{ steps.kill.outputs.cause }} + RESET: ${{ steps.kill.outputs.reset }} LANE_SWITCH: ${{ vars.CLAUDE_REVIEW_DISABLED }} run: | - echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, CLAUDE_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" + echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause=${CAUSE:-unreadable} reset=${RESET:-none}, CLAUDE_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" # A skip fails this job, so the run is a failed first attempt: the # lost-run reconciler re-runs it once GitHub Actions is operational diff --git a/.github/workflows/pr-review-security-hosted.yml b/.github/workflows/pr-review-security-hosted.yml index ce7f102ea52..7cd5a79c7c2 100644 --- a/.github/workflows/pr-review-security-hosted.yml +++ b/.github/workflows/pr-review-security-hosted.yml @@ -106,6 +106,7 @@ jobs: /.github/actions/setup-pinned-node/ /.node-version /.github/actions/log-api-budget/ + /docs/conventions/pr-pipeline.yaml sparse-checkout-cone-mode: false - name: Set up Node @@ -113,12 +114,25 @@ jobs: with: node-version-file: .base/.node-version + - name: Read the defer override from the base config + continue-on-error: true + run: | + yq -o=json '{"limits": {"api": (.limits.api // {})}}' .base/docs/conventions/pr-pipeline.yaml >"$RUNNER_TEMP/defer-config.json" + + # A first attempt defers (cause defer) while the GITHUB_TOKEN window is + # projected to run short before the review ends; the stop below names + # the reset, and the lost-run reconciler re-runs the run once after it + # (https://github.com/melodic-software/claude-code-plugins/blob/main/.github/actions/check-kill-switch/README.md). + # A re-run attempt never defers. - name: Check the kill switch id: kill uses: ./.base/.github/actions/check-kill-switch # zizmor: ignore[self-repository] with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + defer: "true" + defer-job: review + resolved-config: ${{ runner.temp }}/defer-config.json # Skips the run while Actions or npm installs list an outage, with a # logged reason; fails open (.github/actions/check-upstream-status). @@ -134,9 +148,10 @@ jobs: env: REASON: ${{ steps.kill.outputs.reason }} CAUSE: ${{ steps.kill.outputs.cause }} + RESET: ${{ steps.kill.outputs.reset }} LANE_SWITCH: ${{ vars.CLAUDE_SECURITY_REVIEW_DISABLED }} run: | - echo "::notice::a kill switch stops this security review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, CLAUDE_SECURITY_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" + echo "::notice::a kill switch stops this security review (lanes: ${REASON:-unreadable} cause=${CAUSE:-unreadable} reset=${RESET:-none}, CLAUDE_SECURITY_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" # A skip fails this job, so the run is a failed first attempt: the # lost-run reconciler re-runs it once GitHub Actions is operational diff --git a/docs/conventions/pr-pipeline.yaml b/docs/conventions/pr-pipeline.yaml index ba49639c397..7032df82113 100644 --- a/docs/conventions/pr-pipeline.yaml +++ b/docs/conventions/pr-pipeline.yaml @@ -1279,7 +1279,7 @@ limits: # thresholds meant for the hub's limits sampler, which nothing here enforces. Their # defaults live in .github/actions/log-api-budget/rate-limits.json, held # for a person like this file; add an `api:` entry here only to override one. - # github-token defer-project, defer-run-minutes and defer-run-minutes-pr-merge: https://github.com/melodic-software/claude-code-plugins/blob/main/.github/actions/check-kill-switch/README.md + # github-token defer-project, defer-run-minutes, defer-run-minutes-pr-merge and defer-run-minutes-review: https://github.com/melodic-software/claude-code-plugins/blob/main/.github/actions/check-kill-switch/README.md # The adaptive lane level L (owner decision Q111, 2026-11-10): the issue # lane's concurrency .github/scripts/lib/lane-level.mjs would pick each # window; SLOTS stays default while shadow is set. L steps up 1 after diff --git a/docs/conventions/pr-pipeline/README.md b/docs/conventions/pr-pipeline/README.md index 39249d06e19..23668d2889f 100644 --- a/docs/conventions/pr-pipeline/README.md +++ b/docs/conventions/pr-pipeline/README.md @@ -3945,7 +3945,7 @@ with its default. | `fix-docs-per-run` | 2 | fix-docs dispatches one pr-unblock-dispatch `refine` run sends for its PR's event; 0 to 20. Each event offers one PR, so 0 turns CI fix-docs off and any other value turns it on. | | `admission-factor` | 0.8 | Share of the lane's hourly merge rate admitted per hour for the queued cap ([WIP cap](#wip-cap)); greater than 0, at most 1. | | `admission-min-per-hour` | 1 | Admissions per hour the queued cap never falls below, so a day with no lane merges cannot hold admissions shut; 0 to 100, and 0 removes the floor. | -| `api` | schema | GitHub API margins per pool (`github-token`, `lanes-app`, `graphql`, `owner-token`, `content-creation`). `limit-cap` sets the pool's planned budget: planned = min(header limit, `limit-cap`), used = header limit - header remaining, left = planned - used. `github-token` defaults it to GitHub's documented GITHUB_TOKEN limit, so a higher header limit is headroom, never a target; the other pools have no cap and plan to their header limit. `plan` and `read-floor` are shares of the planned budget; `writer-pause`, `breaker` and `stop-floor` are shares of the header's limit, by its remaining; `content-creation` adds `per-minute` and `per-hour`. Their defaults are in `.github/actions/log-api-budget/rate-limits.json`, the only copy, which every merge hold list floors, so a change to a default waits for a person. The rate guard (`.github/actions/log-api-budget/rate-guard.mjs`, `planOf`) reads `read-floor` and `limit-cap`, to warn once while calls continue, and stops on the header's remaining and GitHub's refusals: at a remaining of 0, or at or under `stop-floor` of the header's limit, it sends no further call to that pool before the header's reset, records the stop in the job's stop file, and holds the step under `API_BUDGET_HOLD`, else fails it; a response with no readable limit, remaining or future reset never stops on `stop-floor`; `pace` (github-token, default 0.2) is the share of the header's limit, projected left at the window's reset from the burn rate so far, under which a timer-driven lane skips a scheduled tick when the previous scheduled run, by run id, marked a clean pass of its work (its `Mark this tick's work done` step, which a held or red pass and a paced tick skip), so two scheduled ticks in a row are never both skipped and its cadence halves (`.github/actions/log-api-budget/pace.mjs`; the lost-run reconciler, pr-unblock-dispatch and work-unblock's sweep read it, and pr-merge's dispatch-post-merge-runs reads it for the merge and enable-auto-merge sweep jobs, which skip at job level while the post-merge check and cover dispatch run on every tick), so the timer lanes slow down before the breaker stops every lane; `0` turns pacing off; `pace-pr-merge` (github-token, 0.3, rate-limits.json alone) replaces `pace` for pr-merge, so its sweep halves once the window is projected to have under 0.3 of the header's limit left at the reset, sooner than the other timer lanes, and `pace` 0 turns it off too; `start-gate` (github-token, 0.1) is the share of the header's limit, by its remaining, under which a kill-switch gate that starts new work (`.github/actions/check-kill-switch`, `budget: "true"`) does not start its lane while the reset is still ahead (one GET /rate_limit per such gate, a stop with reason `kill-switch`, cause `budget` and the window's `reset`; a refused read, 403, 429, `retry-after` or a remaining of 0, also stops, and any other failed read proceeds); that gate reads it from rate-limits.json alone, and the schema rejects a `start-gate` set here; on the issue lane's first call (`project: "true"`) the same gate also stops, cause `budget` and the window's reset, while the pacer's projection (`.github/actions/check-kill-switch/projection.mjs`) of the window's burn rate to `start-run-minutes` (github-token, 21, the p90 of successful issue runs' check-broker start to run end) ahead, or to the reset when sooner, leaves under `start-project` (github-token, 0.3) of the header's limit, reading `start-project` and `start-run-minutes` from rate-limits.json alone (the schema rejects `start-project`, `start-run-minutes` and `pace-pr-merge` here); work-unblock's sweep re-fires that stop after the reset like a low-budget one; `defer-project` (github-token, 0.3, 0 turns it off) and `defer-run-minutes` (github-token, 15) drive the same gate's deferral (`defer: "true"`, set only on pr-merge's PR-scoped arm run, whose work the sweep recovers, with `defer-job: pr-merge`, so its horizon is `defer-run-minutes-pr-merge`, github-token, 6, the p90 of 29 first-attempt arm runs, 327 s, read 2026-11-20): on a first run attempt it stops the job, cause `defer` and the window's reset, while the projection of the window's burn rate to the gate's job key (else `defer-run-minutes`) ahead, or to the reset when sooner, leaves under `defer-project` of the header's limit, with no `start-gate` floor and no wait, and a re-run attempt never defers; both may be set here, and CI Status and pr-dispatch-feedback never defer; the others are thresholds meant for the hub's limits sampler, and nothing else in this repository enforces them. resolve-config passes the block through as set. | +| `api` | schema | GitHub API margins per pool (`github-token`, `lanes-app`, `graphql`, `owner-token`, `content-creation`). `limit-cap` sets the pool's planned budget: planned = min(header limit, `limit-cap`), used = header limit - header remaining, left = planned - used. `github-token` defaults it to GitHub's documented GITHUB_TOKEN limit, so a higher header limit is headroom, never a target; the other pools have no cap and plan to their header limit. `plan` and `read-floor` are shares of the planned budget; `writer-pause`, `breaker` and `stop-floor` are shares of the header's limit, by its remaining; `content-creation` adds `per-minute` and `per-hour`. Their defaults are in `.github/actions/log-api-budget/rate-limits.json`, the only copy, which every merge hold list floors, so a change to a default waits for a person. The rate guard (`.github/actions/log-api-budget/rate-guard.mjs`, `planOf`) reads `read-floor` and `limit-cap`, to warn once while calls continue, and stops on the header's remaining and GitHub's refusals: at a remaining of 0, or at or under `stop-floor` of the header's limit, it sends no further call to that pool before the header's reset, records the stop in the job's stop file, and holds the step under `API_BUDGET_HOLD`, else fails it; a response with no readable limit, remaining or future reset never stops on `stop-floor`; `pace` (github-token, default 0.2) is the share of the header's limit, projected left at the window's reset from the burn rate so far, under which a timer-driven lane skips a scheduled tick when the previous scheduled run, by run id, marked a clean pass of its work (its `Mark this tick's work done` step, which a held or red pass and a paced tick skip), so two scheduled ticks in a row are never both skipped and its cadence halves (`.github/actions/log-api-budget/pace.mjs`; the lost-run reconciler, pr-unblock-dispatch and work-unblock's sweep read it, and pr-merge's dispatch-post-merge-runs reads it for the merge and enable-auto-merge sweep jobs, which skip at job level while the post-merge check and cover dispatch run on every tick), so the timer lanes slow down before the breaker stops every lane; `0` turns pacing off; `pace-pr-merge` (github-token, 0.3, rate-limits.json alone) replaces `pace` for pr-merge, so its sweep halves once the window is projected to have under 0.3 of the header's limit left at the reset, sooner than the other timer lanes, and `pace` 0 turns it off too; `start-gate` (github-token, 0.1) is the share of the header's limit, by its remaining, under which a kill-switch gate that starts new work (`.github/actions/check-kill-switch`, `budget: "true"`) does not start its lane while the reset is still ahead (one GET /rate_limit per such gate, a stop with reason `kill-switch`, cause `budget` and the window's `reset`; a refused read, 403, 429, `retry-after` or a remaining of 0, also stops, and any other failed read proceeds); that gate reads it from rate-limits.json alone, and the schema rejects a `start-gate` set here; on the issue lane's first call (`project: "true"`) the same gate also stops, cause `budget` and the window's reset, while the pacer's projection (`.github/actions/check-kill-switch/projection.mjs`) of the window's burn rate to `start-run-minutes` (github-token, 21, the p90 of successful issue runs' check-broker start to run end) ahead, or to the reset when sooner, leaves under `start-project` (github-token, 0.3) of the header's limit, reading `start-project` and `start-run-minutes` from rate-limits.json alone (the schema rejects `start-project`, `start-run-minutes` and `pace-pr-merge` here); work-unblock's sweep re-fires that stop after the reset like a low-budget one; `defer-project` (github-token, 0.3, 0 turns it off) and `defer-run-minutes` (github-token, 15) drive the same gate's deferral (`defer: "true"`, set on pr-merge's PR-scoped arm run, whose work the sweep recovers, with `defer-job: pr-merge`, so its horizon is `defer-run-minutes-pr-merge`, github-token, 6, the p90 of 29 first-attempt arm runs, 327 s, read 2026-11-20, and on the two hosted review callers' gates, whose run the lost-run reconciler re-runs once the reset is past, with `defer-job: review`, so their horizon is `defer-run-minutes-review`, github-token, 11, the p90 of 418 hosted review runs of 2 minutes or more, 10.4 minutes, read 2026-11-21): on a first run attempt it stops the job, cause `defer` and the window's reset, while the projection of the window's burn rate to the gate's job key (else `defer-run-minutes`) ahead, or to the reset when sooner, leaves under `defer-project` of the header's limit, with no `start-gate` floor and no wait, and a re-run attempt never defers; both may be set here, and CI Status and pr-dispatch-feedback never defer; the others are thresholds meant for the hub's limits sampler, and nothing else in this repository enforces them. resolve-config passes the block through as set. | | `lane-level` | 2, 4, 3, 2, `[lanes-app]`, `true`, `CLAUDE_LANE_LEVEL`, 150 | The adaptive lane level L (owner decision Q111): `floor`, `ceiling`, `default`, `clean-windows` and `pools` for `decide()` in `.github/scripts/lib/lane-level.mjs`, whose truth table is `lane-level.vectors.json` beside it. `shadow` (default `true`), `variable` (default `CLAUDE_LANE_LEVEL`, must end in `_LANE_LEVEL`) and `valid-minutes` (default 150, 60 to 1440) govern publish and read: work-unblock's hourly `publish-lane-level` job publishes L as `@` in `variable`, and pick-slot and label-unblocked read it through `.github/scripts/lib/lane-level-value.mjs` (empty, expired or malformed reads as `default`, clamped to `floor` and `ceiling`). While `shadow` is set the job mints no token and writes nothing, `SLOTS` stays `default`, and both log the level they would use. Each window L steps up 1 after `clean-windows` windows in a row that each had demand (runs waiting, or every slot running) and no limit signal, when every pool in `pools` fits: peak used, as a share of the pool's planned budget, times (L+1)/L at most `api..plan`. A window where a pool has `api..writer-pause` of its header's limit left or less resets the streak. Any limit signal the lanes App's own calls meet (403, 429, a secondary-limit or GraphQL rate-limit error, `retry-after`) halves L, rounded down, to no less than `floor`, and blocks a step up until the reset it names. A breaker trip is not a signal: it mostly reflects GITHUB_TOKEN spend, which pacing controls, and the breaker stays the backstop. Missing readings or demand hold L and the streak; a missing or malformed previous level reads as `default`, and every level is clamped to `floor` and `ceiling`. A `floor` above the `ceiling`, as set or defaulted, is rejected. An `api` pool field left unset reads from `rate-limits.json`, as the rate guard reads it. `.github/scripts/lib/lane-level-replay.mjs` replays the rule over the hub's recorded history. | report-outcome and the work-unblock sweep read them from resolve-config's output; diff --git a/docs/conventions/pr-pipeline/pr-pipeline.schema.json b/docs/conventions/pr-pipeline/pr-pipeline.schema.json index db63b88e69c..b2af14127f6 100644 --- a/docs/conventions/pr-pipeline/pr-pipeline.schema.json +++ b/docs/conventions/pr-pipeline/pr-pipeline.schema.json @@ -576,6 +576,11 @@ "minimum": 1, "description": "github-token only. The horizon in whole minutes of pr-merge's PR-scoped arm run (its gate's defer-job pr-merge), the p90 of that run's measured time; replaces defer-run-minutes for it." }, + "defer-run-minutes-review": { + "type": "integer", + "minimum": 1, + "description": "github-token only. The horizon in whole minutes of a hosted review run (its callers' gates' defer-job review), the p90 of that run's measured time; replaces defer-run-minutes for it." + }, "read-floor": { "type": "number", "minimum": 0,