Repository navigation
fix(source-control): cap pr-verify repeated FAILs only at a FAILed head - #18028
kyle-sexton wants to merge 7 commits into
Conversation
Owner decision Q245 (option a): pr-verify's repeated-fail cap closes a pull request only when its current head itself has a FAIL verdict; FAILs at superseded heads no longer close it. Reader.evidence now returns verify:repeated-fail only when a counted FAIL that is not superseded stands at the vetted head and FAILs sit at max_fails or more heads, that one included. A head whose newest verdicts pass, or whose own FAIL is superseded, goes on to the earlier-fail or pass decision. Sibling-PR counting, the capped-close floor, draft-judged exclusions, verify:fail and VERIFY_FINAL stay. The cap tests that expected it at a passing head now put a FAIL at the head; new tests pin a pass after FAILs at four older heads and a FAIL at the head after them. The docstring, verify-cap.mjs, the convention README, schema and pr-pipeline.yaml state the new condition. Closes #17646 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
… heads Issue #17646's third acceptance item: the same history as the cap case, with the head's own FAIL superseded, gives the pass decision; left unanswered it still caps. Refs: #17646 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
|
claude-security-review reviewed Security review: PR #18028, changes since
|
|
claude-review reviewed Code review: PR #18028, incremental from aebeb11 to 496ffb8 (blocking only)What I read. Every hunk of all 15 files in the incremental diff. For context I also read the unchanged code these lines call: My earlier blocking finding is resolved (discussion_r4655093893: the cap is checked after supersession, but the dispatchers still recorded it as final for the head).
Reader change. Closing issue #17646 acceptance (at this head)
Spend (not a finding). Once a lanes App pull request is capped, Other notes
Verifier dropped 0 of 0 candidates. |
Lane run report
Run 44950399269, attempt 1
Failed GitHub API callsNo failed call recorded by the run job's scripts. |
|
pr-unblock / address-feedback finished round 1 of max 4 on a71c474. |
Lane run report
Run 44950399269, attempt 1
Failed GitHub API callsNo failed call recorded by the run job's scripts. |
Lane run report
Run 44950399269, attempt 1
Failed GitHub API callsNo failed call recorded by the run job's scripts. |
Lane-Activity: pr-unblock/address-feedback Lane-Thread: PRRT_kwDOTCGFQM66b6f7
Lane run report
Run 44950399269, attempt 1
Failed GitHub API callsNo failed call recorded by the run job's scripts. |
|
pr-unblock / address-feedback finished round 2 of max 4 on bd97747. |
| # The cap decides before supersession: a FAIL at the head at the cap | ||
| # stays capped there whatever supersedes it later (VERIFY_FINAL), as | ||
| # the pr-unblock dispatchers record a capped head as handled. | ||
| head_superseded = head in failed and superseded(head) | ||
| if head in failed and len(failed) >= self.max_fails: | ||
| refusal = REPEATED_FAIL | ||
| detail = f"fails at {len(failed)} heads ({_heads(failed)}), the cap is {self.max_fails}" | ||
| elif head in failed and not (head_superseded := superseded(head)): | ||
| head_superseded = False |
There was a problem hiding this comment.
Blocking: this push breaks the third acceptance criterion of #17646.
The criterion says:
A reader test: the same history where the current head's FAIL is superseded does not yield
verify:repeated-fail.
The cap now runs before superseded(head), so that history does yield verify:repeated-fail. This push also inverted the tests that pinned the criterion:
test_a_superseded_fail_at_the_head_at_the_cap_stays_capped(test file lines 1387-1388)test_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_not(lines 1401-1402)test_the_cap_is_judged_before_a_superseded_fail_at_the_head(lines 1584-1585)
They now assert verify:repeated-fail, with supersededAtHead False.
The issue's Change section asks for the opposite order in two places:
- The cap check "must evaluate
superseded(head)first". - "a superseded head FAIL counts as a clean head".
The PR body's Verification section still says "A superseded FAIL at the head with max_fails=1 gives the pass decision, with supersededAtHead true".
How it fails: take FAILs at 3 earlier heads plus a FAIL at head H. A person relinks or edits the body, and pr-verify passes again at H. This is exactly the history() case in the second test. The reader still refuses verify:repeated-fail, so capped_close closes the pull request. The issue says it should not.
My last review offered this order as one of two fixes for the markHandled finding. I'm sorry: that option conflicts with the closing issue. The other option keeps the criterion:
- Keep supersession ahead of the cap.
- Make the cap's
final/markHandledinfeedback-dispatch.mjsandfix-ci-dispatch.mjsnot final oncesupersededAtHeadturns true at the same head, for example by re-dispatching when the reader's answer changes.
The other way out is to change the issue's acceptance criterion through an owner decision, then update the PR body to match. Either way, the issue, the code and its tests must agree before this merges.
claude-lane: claude-review
There was a problem hiding this comment.
pr-unblock / address-feedback: UNCERTAIN, left open
Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4380, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4381, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1376, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1573, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/feedback-dispatch.mjs:783
Basis: verified for the code, not for the issue text. The code check holds: babysit_verify_evidence.py:4381-4392 return verify:repeated-fail before the superseded-head branch at 4393, and they set head_superseded to False. The tests at test_babysit_verify_evidence.py:1376, 1390 and 1573 pin that order. I could not read issue #17646 from this lane, so I have not checked the acceptance criterion quoted in the finding against the issue itself. One part of the finding is out of date: the PR body's Verification section now says a superseded FAIL at the head with max_fails=1 stays verify:repeated-fail with supersededAtHead false, so it matches the code and not the issue's criterion as quoted. I made no edit because neither way to resolve this can happen in this lane. Putting supersession back ahead of the cap would bring back the earlier markHandled finding: feedback-dispatch.mjs:782-783 marks a head handled once capFinal(capped) is true, so a head that later turns superseded never gets dispatched again. The other code fix, re-dispatching in feedback-dispatch.mjs and fix-ci-dispatch.mjs when supersededAtHead turns true at the same head, changes shared pr-unblock infrastructure, and check-paths refuses both files for this thread. The third way out, changing the criterion, needs an owner decision. A person needs to pick one: an owner decision to change #17646's third criterion, or a session that moves supersession back ahead of the cap together with the dispatcher change, then updates the tests and the PR body to match.
There was a problem hiding this comment.
pr-unblock / adjudicate-feedback: the finding stands
A fresh-context adjudicator read the review comment, the locations both sides cited and the code at bd9774734c49921ba417984ba337ba7e8fcb5f94, without the fixing lane's reasoning, and found that the finding holds. The next address-feedback round fixes it; a round that leaves this thread open again ends the lane in a failing status.
The linked issue settles this intent question (#17646):
A reader test: the same history where the current head's FAIL is superseded does not yield
verify:repeated-fail.
Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4380, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4381, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1376, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1573, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/feedback-dispatch.mjs:783
There was a problem hiding this comment.
pr-unblock / address-feedback: UNCERTAIN, left open
Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1405, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/fix-ci-dispatch.mjs:688
This is the same finding as the newer thread on line 4379, and the adjudicator found that it stands. The in-scope part of the fix is in the working tree. The reader now judges supersession before the cap: head_failed at babysit_verify_evidence.py:4378 excludes a superseded FAIL at the head, so neither cap refuses there and the passes decide, as issue 17646's third criterion and its Change section ask. The three inverted tests now assert that criterion, each with a paired case that is not superseded and stays verify:repeated-fail. Those tests are the cap-1 relink test at 1378, the history test with 3 earlier FAILed heads at 1405, and the body-fix test at 1591. The docs and the changelog fragments describe the new order too. Left open because the other half of the reviewer's fix is in files check-paths refuses for this lane: feedback-dispatch.mjs:782-783 and fix-ci-dispatch.mjs:688-689 still treat a capped answer as final and mark the head handled, so a head capped before its FAIL is superseded is never dispatched again. The verify-cap.mjs header also still says cap first. A session needs to change the dispatchers so they re-dispatch when supersededAtHead turns true at the same head, and to update that header. No tests were run in this lane; CI checks the change.
|
pr-unblock / address-feedback finished round 3 of max 4 on bd97747. |
… rejects Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
| head_superseded = head in failed and superseded(head) | ||
| if head in failed and len(failed) >= self.max_fails: | ||
| refusal = REPEATED_FAIL | ||
| detail = f"fails at {len(failed)} heads ({_heads(failed)}), the cap is {self.max_fails}" | ||
| elif head in failed and not (head_superseded := superseded(head)): | ||
| head_superseded = False |
There was a problem hiding this comment.
Blocking (still open): the third acceptance criterion of #17646 is still not met.
The criterion says:
A reader test: the same history where the current head's FAIL is superseded does not yield
verify:repeated-fail.
The latest push only removed comments. The order is the same as at bd977473: the cap at line 4376 runs before head_superseded is used. When the head has a FAIL and len(failed) >= max_fails, the reader returns REPEATED_FAIL and sets head_superseded = False, even if that FAIL is superseded. The issue's Change section asks for the opposite order in two places:
- the cap check "must evaluate
superseded(head)first" - "a superseded head FAIL counts as a clean head"
Three tests still assert the behaviour the criterion rules out:
test_a_superseded_fail_at_the_head_at_the_cap_stays_cappedtest_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_nottest_the_cap_is_judged_before_a_superseded_fail_at_the_head
How it fails: take FAILs at 3 earlier heads plus a FAIL at head H. A person relinks or edits the body, and pr-verify passes again at H. The reader still refuses with verify:repeated-fail, so capped_close closes the pull request. Under #17646 it should not be closed.
The PR body has been reworded to describe the cap-first order. That rewording does not change the issue's criterion, and the PR context records no owner decision that amends it.
Fix, either way:
- Keep supersession ahead of the cap, and stop the dispatchers (
feedback-dispatch.mjs/fix-ci-dispatch.mjsmarkHandled,verify-cap.mjsfinal) from treating the cap as final oncesupersededAtHeadturns true at the same head. - Or record an owner decision that amends criterion 3 of pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646.
Before this merges, the issue, the code and the tests must agree.
claude-lane: claude-review
There was a problem hiding this comment.
pr-unblock / address-feedback: UNCERTAIN, left open
Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4379, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1419, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/verify-cap.mjs:6
The finding holds, and the adjudicator's quote of issue 17646 confirms the criterion. The in-scope part of the fix is in the working tree. Reader: babysit_verify_evidence.py now works out whether the head's FAIL is superseded first. head_failed (line 4378) is true only for a FAIL at the head that is not superseded, and only head_failed reaches the two caps and verify:fail, so a superseded FAIL at the head counts as a clean head and the passes decide. Tests: the three tests now assert the criterion. With max_fails=1, a superseded head FAIL (relink at 1378, body fix at 1591) gives the pass decision with supersededAtHead true. The same FAIL with no trusted act, or with a lanes App edit, stays verify:repeated-fail with supersededAtHead false. The history test (1405) has 3 earlier FAILed heads plus a superseded head FAIL and asserts that the result is not verify:repeated-fail (1419) and that supersededAtHead is true. Its unanswered twin stays capped. I also updated the module docstring, the README Verifier verdict section, pr-pipeline.schema.json, the pr-pipeline.yaml comment, the babysit_merge_policy.py comment and both changelog fragments, and recorded a body refresh. Left open because two instances are in files check-paths refuses. First, feedback-dispatch.mjs:782-783 and fix-ci-dispatch.mjs:688-689 still mark a head handled once verify-cap.mjs final() says capped. A head capped before its FAIL is superseded therefore gets no later pass-only round, even though capped_close re-judges and will no longer close it. Second, the verify-cap.mjs header (lines 6-11) still describes the cap-first order. A session needs to make the dispatchers re-read or re-dispatch when supersededAtHead turns true at the same head, and to update that header. I ran no tests (this lane runs none); CI checks the change.
There was a problem hiding this comment.
pr-unblock / adjudicate-feedback: the finding stands
A fresh-context adjudicator read the review comment, the locations both sides cited and the code at 5dc6f4deee916e0af65e820cc8335dcb5e193481, without the fixing lane's reasoning, and found that the finding holds. The next address-feedback round fixes it; a round that leaves this thread open again ends the lane in a failing status.
The linked issue settles this intent question (#17646):
A reader test: the same history where the current head's FAIL is superseded does not yield
verify:repeated-fail.
Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4379, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1419, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/verify-cap.mjs:6
|
The pr-verify verifier failed head What it wrote (69 lines)This pull request leaves the merge queue, and ci-status holds it out while it carries |
Lane-Activity: pr-unblock/address-feedback
|
pr-unblock / address-feedback finished round 4 of max 4 on aebeb11. |
|
The pr-unblock lane stopped
The review rounds are not converging after the round cap and one fresh fixer round. Open review threads:
When a retry on this head can change the outcome, the lane runs one on its own, up to its per-head cap, and edits this comment if it stops again; a run that does not stop clears the status. When no retry can (a review thread the lane does not take or settle, or a failing check after the retries are spent), a person resolves the threads or gets the checks green, then dispatches the workflow below: that run finds the cause gone and clears the status. On a pull request the lanes App opened, the dispatchers start that run once per stop; on any other, a person does. A new commit to the branch starts clean. A reply on a thread, from an account with write access, starting To run it again on this head now, dispatch the gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=18028 -f activity=address-feedback |
| # Supersession is judged before the cap (#17646): a superseded FAIL | ||
| # here counts as a clean head, so neither cap refuses at it. | ||
| head_superseded = head in failed and superseded(head) | ||
| head_failed = head in failed and not head_superseded |
There was a problem hiding this comment.
Blocking: with supersession first, the cap can clear at its own head, but the dispatchers still record it as final for that head.
This order is right for #17646 criterion 3. The catch is that verify:repeated-fail at head H can now clear at H itself: a trusted relink or body edit plus a new pass supersedes H's FAIL. Line 147 now says so.
The callers still assume the old order:
verify-cap.mjs:6-8, changed in this PR, still says "The reader judges the cap before supersession, so at that head it never clears". That is now false.final()(verify-cap.mjs:162-165) treats every cap as one that never clears for the head.feedback-dispatch.mjs:780-784andfix-ci-dispatch.mjs:686callmarkHandled(record, …, head)on it. Later runs at H then skip before any read.
How it fails:
- A PR that
capped_closedoes not close is capped at H: a PR not opened by the lanes App, a PR undermerge.report-only, or any PR before pr-merge runs. - The dispatcher marks H handled.
- A person relinks or edits the body, and pr-verify passes again at H.
- The reader now answers
supersededAtHead: true, with no refusal. - The pass-only round with
verify-superseded-head, which resolves that FAIL's fail-note threads, is never sent, because H is already marked handled. - Those threads keep holding the merge until a person resolves them.
The base branch could not do this: there the cap applied "whatever a pass says".
This is the point of the thread settled at bd97747 (discussion_r4654807864). That thread was settled by the cap-first order, which this push reverts. My last review asked for both halves; only the first landed.
Fix:
- Key the handled record so a later supersession at H gets read again. Head plus body update time is one option. Another is to skip
markHandledfor a cap on a PR thatcapped_closewill not close. - Reword
verify-cap.mjs:6-11,final()and the dispatcher comments to say the cap clears if the head's FAIL is later superseded.
The issue's line "VERIFY_FINAL never clearing once the cap applies at a FAIL head" conflicts with criterion 3 under this order. If the owner accepts the clear, say so in the PR body. The body also still says a superseded FAIL "stays capped".
claude-lane: claude-review
Lane run report
Run 44952613087, attempt 1
Failed GitHub API callsNo failed call recorded by the run job's scripts. |
…expire a capped head's handled mark Restore issue #17646's order (owner decision Q245): Reader.evidence checks supersession first, so a superseded FAIL at the head counts as a clean head and gets the earlier-fail or pass decision, not verify:repeated-fail. The three reader tests, the README, pr-pipeline.yaml, the schema, the docstring and both changelog fragments say so again. A cap no longer holds a head for good, so feedback-dispatch.mjs and fix-ci-dispatch.mjs stop recording a capped head as handled: a head whose FAIL a later pass superseded is read again and dispatched. The dispatch record's version is 7, so cap marks written by version 6 are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
Keep the supersession-first reader and docs from the previous commit, and take aebeb11's reader tests, which pair each superseded head FAIL with the same FAIL unsuperseded, still capped. A superseded FAIL now clears verify:repeated-fail at its head, so the docstring says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
Closes #17646
Summary
Builds owner decision D457 (Q245 option (a), owner "Go w/ recommendations", 2026-11-20):
Before this change,
Reader.evidenceinbabysit_verify_evidence.pyreturnedverify:repeated-failonce counted FAILs sat atmax_failsor more distinct heads, "whatever a pass says". The pr-merge gate'scapped_closethen closed pull requests whose current head had already passed. Lanes PR #17926 was closed this way at 19:20Z on 2026-11-21: its head 9400abb has a pass-notes verdict, and its FAILs were all at older heads.Fix
Reader.evidencereturnsverify:repeated-failonly when a counted FAIL stands at the vetted head and FAILs sit atmax_failsor more heads, that head included. It judges whether the head's FAIL is superseded (superseded(head)) before either cap, as pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646 asks: a superseded FAIL at the head counts as a clean head and lets the passes decide, at the cap or under it. Only an unsuperseded FAIL at the head is capped, first on this pull request's own FAILed heads, then on the count with other pull requests for its issues.markHandledcalls on a capped answer are gone fromfeedback-dispatch.mjsandfix-ci-dispatch.mjs, andRECORD_VERSIONindispatch-record.mjsmoves to 7 so cap marks already saved are dropped.final()inverify-cap.mjsis unchanged; it still tells a cap the reader found from one it could not read. Cost:fix-ci-dispatchre-reads the verifier for a capped lanes head on each sweep until pr-merge closes the pull request.elsewhere()is read once per decision and shared with the pass branch.elsewhere()for the live issues_pass_refusalneeds.Refs:pull requests (at a FAILed head),_attempt_floor, draft-judged exclusions,verify:failat the head, andVERIFY_FINAL.verify-cap.mjsheader, the README "Verifier verdict" section and the arm conditions,pr-pipeline.schema.json, thepr-pipeline.yamlcomment, thebabysit_merge_policy.pyfield comment, and theenable-auto-merge.mjsheader.verify-cap.mjsandcapped_closecall the same reader, so their behavior follows without a code change.write-verdict.mjs:442stays as it is, because FAILs still count toward the cap..changes/source-control/and.changes/pr-pipeline/.Verification
None).verify:repeated-fail, with the detail naming all 5 heads.verify:repeated-fail(relink and body-edit cases).verify:repeated-failonly while the head FAIL is not superseded; superseded, it givesNone.feedback-dispatch.test.mjsandfix-ci-dispatch.test.mjs: a capped head whose FAIL a later pass superseded is read again and dispatched.Refs:, named-FAIL, fresh-attempt, draft-judged and by-id tests that expected the cap at a passing head now put a FAIL at the head and still expect the cap, as issue pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646 asks. Where a reset or exclusion used to clear toNone, the paired case now expectsverify:fail. The by-id fetch cases now expect one more download: the head FAIL's own record, which is read to decide whether it is superseded.python3 -I -m unittestontest_babysit_verify_evidence.py, 466 passed;node --teston the feedback-dispatch, fix-ci-dispatch, verify-cap and dispatch tests, 447 of 448 passed. The one failure (ownerAdoption) needs theajvpackage the local worktree lacks and passes in the main checkout; CI runs it.ruff checkis clean, and the pre-commit comment lint passes.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf