Skip to content

perf(pr-pipeline): defer hosted reviews on a short token window and recover them after the reset - #18054

Queued
kyle-sexton wants to merge 3 commits into
mainfrom
perf/17460-defer-reviews
Queued

kyle-sexton wants to merge 3 commits into
mainfrom
perf/17460-defer-reviews

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #17460

session: AI Hero

Summary

Phase 2 of the #17460 option B plan, under the owner's OQ1 decision (2026-11-21, about 4:35 PM EST: option (a), "Build Phase 2 (Recommended)"). Busy token windows now delay the two hosted reviews instead of letting them spend into the brake, and a dedicated recovery band in the lost-run reconciler re-runs each deferred review after the reset it names.

This changes what the unattended review lanes do under load. The owner decided that in OQ1 above. On 2026-11-21, about 4:50 PM EST, the owner also accepted the merge latency this adds on busy days ("Accept, then measure"). Phase 8 reports the deferral backlog and the p90 merge latency on #17460.

Fix

  • pr-review-hosted.yml and pr-review-security-hosted.yml: the gate sets defer: "true", defer-job: review and resolved-config, so pr-pipeline.yaml overrides apply. The stop notice keeps its prefix, which the merge gate matches, and now prints cause=<cause> reset=<epoch>.
  • rate-limits.json and rate-guard.mjs: defer-run-minutes-review is 11. Basis: the p90 of the 418 hosted review runs that ran 2 minutes or more, among the 300 newest of each caller, is 10.4 minutes (gh run list, read 2026-11-21). The schema, the pr-pipeline.yaml comment and both READMEs name the key.
  • reconcile.mjs (Review kind):
    • A stopped review whose gate notice matches DEFER_NOTE waits until RATE_RESET_GRACE_MS past its reset and counts as a wait, so the pass writes no clean. It is re-run once after that.
    • Those re-runs go in their own DEFER_MAX band of 12 per pass, a head's two lanes together, beside the 3 existing places. A full set of flake or floor re-runs no longer blocks them, and they do not block those.
    • recordOutputs writes record-1 to record-15, and the workflow gains upload steps 4 to 15. Record matching is unchanged: one artifact per record name.
  • Cap basis: .work replay of the gate (projection.mjs, share 0.3, 11-minute horizon) over the gh-job-calls log for the 24 h to 2026-11-21T20:49Z.
    • 12 of 49 windows would defer, 44 review runs in total, at most 9 in one window.
    • 12 is that peak plus one head with both lanes. The plan's estimate of about 9.5 heads/h (19 runs) came from the pre-fix tripping hours.
    • A full band costs about 6 × (2 × 18.4 + 16.8), about 320 calls, early in the next window.
  • The recovery pass is the hourly :04 tick. Every window that would have deferred reset between :50 and :55 (12 of 12), so that tick runs 9 to 14 minutes after the reset. A window that resets at another minute waits for the next tick.
  • Left out of this PR:
    • Check Intent: its gate runs before the resolver, which skips most runs. Replayed, it would defer 333 runs a day, of which about 86% only skip (661 of 765 pr-verify runs ended in under 2 minutes). Each one would need a recorded re-run and save nothing.
    • pr-review-main: it has no recovery path for a stopped run. Its notice-only stop leaves the lane unbound, and deferring it would save about 14 × 8.5 calls a day.

Verification

  • node --test --test-concurrency=4 over check-kill-switch/*.test.mjs, maintenance-reconcile-lost-runs/*.test.mjs and pr-unblock/fix-ci-dispatch.test.mjs: 450 pass. The run on the other readers of the changed workflows (pr-review-main workflow.test.mjs, check-upstream-status, log-api-budget, workflow-gates, lane-binding-wiring, check-trusted-trigger) also passed: 805 tests.
  • bash scripts/ci-review-lanes.test.sh (PASS=295) and bash scripts/ci-rerun-ci-status.test.sh (PASS=324).
  • New tests:
    • A deferred review is not re-run before reset + RATE_RESET_GRACE_MS and is re-run after it, including a run created before the last pass.
    • Deferred reviews go beside a full set of places, with at most DEFER_MAX per pass and the rest waiting.
    • A stop with no defer notice keeps the old path.
    • The callers' notice matches DEFER_NOTE.
    • The defer-run-minutes-review default is pinned, and both callers are classified as defer gates.
    • Assumption A4: fix-ci-dispatch sends nothing for a head whose ci-status failed only on the review-stop note while ci-lanes passed.
  • actionlint and zizmor are clean on the three workflows, and the changelog fragment check passes.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01RAoAtST8QdquPzTRsKGF4Y

…ecover them after the reset

Both hosted review callers' gates defer a first attempt while the
GITHUB_TOKEN window is projected to run short before the review ends
(defer-job review, defer-run-minutes-review 11, the measured p90). The
stop notice prints cause= and reset=. The lost-run reconciler reads that
reset, waits a minute past it, then re-runs the run once, recorded, in a
DEFER_MAX band of 12 re-runs a pass beside the 3 existing places; the
workflow gains record uploads 4 to 15.

Refs: #17460

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtST8QdquPzTRsKGF4Y
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T21:54:13.582754Z cdb6d67 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…reconciler suite's workflow reads

Each record upload 4 to 15 follows its own scan-upload step, as the
artifact-retention table requires. reconcile.test.mjs declares its
.github/workflows reads, and the callers test expects the hosted gates'
defer inputs.

Refs: #17460

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtST8QdquPzTRsKGF4Y
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed b35ac17d76bf92861456912e44ddea063e6fb3f0: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, blocking only): no findings

I reviewed the change since dd69adf852940a98f0a3b5aba865902b23d88aa1. One file changed: .github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs. I read its whole diff in .claude-lane/incremental.diff.

What the change does:

  • It adds RECORD_SLOTS to an import from reconcile.mjs.
  • It adds one test. The test reads maintenance-reconcile-lost-runs.yml from the repository through a fixed relative URL. It checks two things:
    • The workflow has exactly one Record re-run N upload step for each value of N from 1 to RECORD_SLOTS.
    • Each of those steps names steps.plan.outputs.record-N.

Why there is no security surface:

  • The test only reads a file, using a fixed path and fixed regular expressions.
  • No trust control, token scope, kill switch, merge hold or workflow logic changed.
  • The test makes the existing protection stricter: every record slot the plan can name must have its own upload step. I already checked in the earlier review that each of those steps passes the scan-upload scan.

The review was blocking only (this lane already reviewed an earlier head). My earlier summary comment had no findings, so I have nothing to restate. No settled threads apply, and the incremental diff contains no embedded instructions. This change adds no spend.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on dd69adf852940a98f0a3b5aba865902b23d88aa1; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed b35ac17d76bf92861456912e44ddea063e6fb3f0: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): b35ac17 since dd69adf

Scope. One file changed since the last review: .github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs. I read its whole change. It imports RECORD_SLOTS and adds the test "the workflow has one scanned upload step for each record the plan can name".

Overview. The new test does what the owner said it does when they resolved this lane's earlier finding on reconcile.mjs:434:

  • Step count. It collects every - name: Record re-run N step in maintenance-reconcile-lost-runs.yml (6-space indent, anchored per line) and asserts the list is exactly 1..RECORD_SLOTS in order. RECORD_SLOTS is MAX_IN_FLIGHT + DEFER_MAX, exported at reconcile.mjs:434, so raising DEFER_MAX without adding upload steps now fails the test.
  • Artifact names. For each N it checks that the file contains name: ${{ steps.plan.outputs.record-N }}. The pattern needs a space before }}, so the check for record-1 cannot be satisfied by record-10 through record-15.
  • Indentation. The workflow's step lines use the indentation the regex expects (for example Record re-run 1 at line 203).

The earlier thread is resolved and its lines have not changed since, so it stays settled. No blocking finding is new in this push.

Not checked by the new test (advisory, dropped under blocking only). The test name says "scanned" upload step, but it does not check that each upload has a scan step or reads that scan's clean output. The scan step ids also differ between steps (scan-upload, scan-upload-2, scan-record-4). The owner's reply limited the fix to step count and artifact name, so this is not raised as a finding.

Closing issue. The PR says only Refs: #17460 and closes no issue, so there are no acceptance criteria to check.

Loosened checks. The wrapper sent no loosened_check_verdict:, loosened_check_patterns: or loosened_check_decision_record: lines, so I used fail with no consumer entries and did not check the citation requirement. This push only adds a test and loosens no check.

Verification. I ran nothing; this lane has no shell. Whether the new test passes is the author's claim and unverified.

Review mode. Blocking only: this lane already reviewed an earlier head.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 21, 2026
Any commits made after this event will not be merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant