Skip to content

fix(source-control): cap pr-verify repeated FAILs only at a FAILed head - #18028

Open
kyle-sexton wants to merge 7 commits into
mainfrom
fix/pr-verify-cap-head-fail
Open

kyle-sexton wants to merge 7 commits into
mainfrom
fix/pr-verify-cap-head-fail

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Closes #17646

Summary

Builds owner decision D457 (Q245 option (a), owner "Go w/ recommendations", 2026-11-20):

pr-verify's repeated-fail cap closes a PR only when its current head itself has a FAIL verdict; FAILs at superseded heads no longer close it.

Before this change, Reader.evidence in babysit_verify_evidence.py returned verify:repeated-fail once counted FAILs sat at max_fails or more distinct heads, "whatever a pass says". The pr-merge gate's capped_close then closed pull requests whose current head had already passed. Lanes PR #17926 was closed this way at 19:20Z on 2026-11-21: its head 9400abb has a pass-notes verdict, and its FAILs were all at older heads.

Fix

  • Reader.evidence returns verify:repeated-fail only when a counted FAIL stands at the vetted head and FAILs sit at max_fails or more heads, that head included. It judges whether the head's FAIL is superseded (superseded(head)) before either cap, as pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646 asks: a superseded FAIL at the head counts as a clean head and lets the passes decide, at the cap or under it. Only an unsuperseded FAIL at the head is capped, first on this pull request's own FAILed heads, then on the count with other pull requests for its issues.
  • Since a cap can now clear at the same head (a person's body edit or relink, then a pass), the pr-unblock dispatchers no longer record a capped head as handled: the markHandled calls on a capped answer are gone from feedback-dispatch.mjs and fix-ci-dispatch.mjs, and RECORD_VERSION in dispatch-record.mjs moves to 7 so cap marks already saved are dropped. final() in verify-cap.mjs is unchanged; it still tells a cap the reader found from one it could not read. Cost: fix-ci-dispatch re-reads the verifier for a capped lanes head on each sweep until pr-merge closes the pull request. elsewhere() is read once per decision and shared with the pass branch.
  • The pass branch no longer caps on FAILs from other heads or sibling pull requests. It still reads elsewhere() for the live issues _pass_refusal needs.
  • Unchanged: FAIL counting across sibling and Refs: pull requests (at a FAILed head), _attempt_floor, draft-judged exclusions, verify:fail at the head, and VERIFY_FINAL.
  • Other copies of the rule now say the same thing: the module docstring, the verify-cap.mjs header, the README "Verifier verdict" section and the arm conditions, pr-pipeline.schema.json, the pr-pipeline.yaml comment, the babysit_merge_policy.py field comment, and the enable-auto-merge.mjs header. verify-cap.mjs and capped_close call the same reader, so their behavior follows without a code change. write-verdict.mjs:442 stays as it is, because FAILs still count toward the cap.
  • Changelog fragments: .changes/source-control/ and .changes/pr-pipeline/.

Verification

  • New tests:
    • A pass at the head after FAILs at 4 older heads gives the pass decision (None).
    • A FAIL at the head after FAILs at 4 older heads gives verify:repeated-fail, with the detail naming all 5 heads.
    • A superseded FAIL at the head at the cap lets the passes decide, paired with the same FAIL unsuperseded, which stays verify:repeated-fail (relink and body-edit cases).
    • FAILs at 3 older heads plus a FAIL at the head give verify:repeated-fail only while the head FAIL is not superseded; superseded, it gives None.
    • In feedback-dispatch.test.mjs and fix-ci-dispatch.test.mjs: a capped head whose FAIL a later pass superseded is read again and dispatched.
  • Rewritten tests: the sibling, Refs:, named-FAIL, fresh-attempt, draft-judged and by-id tests that expected the cap at a passing head now put a FAIL at the head and still expect the cap, as issue pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646 asks. Where a reset or exclusion used to clear to None, the paired case now expects verify:fail. The by-id fetch cases now expect one more download: the head FAIL's own record, which is read to decide whether it is superseded.
  • At 496ffb8: python3 -I -m unittest on test_babysit_verify_evidence.py, 466 passed; node --test on the feedback-dispatch, fix-ci-dispatch, verify-cap and dispatch tests, 447 of 448 passed. The one failure (ownerAdoption) needs the ajv package the local worktree lacks and passes in the main checkout; CI runs it.
  • ruff check is clean, and the pre-commit comment lint passes.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf

Owner decision Q245 (option a): pr-verify's repeated-fail cap closes a
pull request only when its current head itself has a FAIL verdict;
FAILs at superseded heads no longer close it.

Reader.evidence now returns verify:repeated-fail only when a counted
FAIL that is not superseded stands at the vetted head and FAILs sit at
max_fails or more heads, that one included. A head whose newest
verdicts pass, or whose own FAIL is superseded, goes on to the
earlier-fail or pass decision. Sibling-PR counting, the capped-close
floor, draft-judged exclusions, verify:fail and VERIFY_FINAL stay.

The cap tests that expected it at a passing head now put a FAIL at the
head; new tests pin a pass after FAILs at four older heads and a FAIL
at the head after them. The docstring, verify-cap.mjs, the convention
README, schema and pr-pipeline.yaml state the new condition.

Closes #17646

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T19:42:36.855063Z afb785c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

… heads

Issue #17646's third acceptance item: the same history as the cap case,
with the head's own FAIL superseded, gives the pass decision; left
unanswered it still caps.

Refs: #17646

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 496ffb87261139087dd331a8528d3b9939e2c727: read 15 of 15 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #18028, changes since aebeb11

I found no security issues. This lane already reviewed an earlier head, so this review only looked for CRITICAL and IMPORTANT findings.

What changed since the last review

This push is the merge commit 496ffb8. It reconciles the branch with fix/pr-verify-cap-head-fail. It changes four things:

  • The cap check in babysit_verify_evidence.py. The supersession-first logic is now one if with an assignment expression (:=). The two-step head_superseded / head_failed form is gone.
  • The elsewhere() cache. The memo list is removed.
  • The pr-unblock dispatchers. feedback-dispatch.mjs and fix-ci-dispatch.mjs no longer call markHandled on a capped head. RECORD_VERSION moves from 6 to 7.
  • Text only. Tests, changelog fragments, README, schema, pr-pipeline.yaml comment and babysit_merge_policy.py comment are updated to match.

What I checked

  • Merge hold (babysit_verify_evidence.py, trusted path). I read the whole change and the decision chain it sits in (L4366-L4423). The new form decides the same way as the head I reviewed last:
    • superseded(head) is evaluated only when head in failed. Otherwise head_superseded stays False.
    • An unsuperseded FAIL at the head still refuses verify:repeated-fail once FAILs sit at max_fails or more heads. Under the cap it refuses verify:fail, or verify:repeated-fail once FAILs on the other pull requests for its issues are counted too.
    • A superseded head still reaches the verify:earlier-fail, unreadable and pass branches. The pass branch still runs _pass_refusal for verify:spec-changed and verify:untrusted-spec (L4405-L4413), so the untrusted-provenance and untrusted-spec hold is unchanged.
    • Removing the elsewhere() memo costs nothing: the branches are exclusive, so each decision calls it at most once (L4379 or L4405).
  • Dispatcher record (dispatch-record.mjs, feedback-dispatch.mjs, fix-ci-dispatch.mjs, verify-cap.mjs; trusted paths). These dispatchers no longer record a capped head as handled, so later sweeps read the verifier again for that head. A capped head still gets no round. Both dispatchers still skip it, and the pass-only path still skips on a cap the reader found. Raising the record version only throws away cached marks, which forces more reads; nothing new gets through. No token, permission or trusted-actor logic changed.
  • The friction cap. Narrowing the repeated-fail friction cap is the owner's decision (Q245, pr-verify: close on the repeated-fail cap only when the current head itself FAILs #17646), as the owner threat model allows. It is named here and holds nothing.
  • Instruction text. The docs, schema, YAML comment and changelog edits describe the new behavior. None of them is a standing instruction, so no guardrail is removed.
  • Trust controls and manifests. Nothing changed in the trusted-actor list, token scope, the kill switch, the workflows or the plugin manifests.

Spend: fix-ci-dispatch and feedback-dispatch now read the verifier again on each sweep for a capped head, until pr-merge closes the pull request. That costs a few more GitHub API calls inside lane runs that already happen. It is not a billing change.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on a71c474d192a4030ec8e0cdc1cb2f2a1bdbd8501; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 496ffb87261139087dd331a8528d3b9939e2c727: read 15 of 15 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #18028, incremental from aebeb11 to 496ffb8 (blocking only)

What I read. Every hunk of all 15 files in the incremental diff. For context I also read the unchanged code these lines call: Reader.evidence around babysit_verify_evidence.py:4337-4423, the cap handling in feedback-dispatch.mjs:749-809, and the version check in dispatch-record.mjs:73.

My earlier blocking finding is resolved (discussion_r4655093893: the cap is checked after supersession, but the dispatchers still recorded it as final for the head).

  • feedback-dispatch.mjs and fix-ci-dispatch.mjs no longer call markHandled on a capped answer. A head whose FAIL is later superseded is now read again, and the pass-only round with verify-superseded-head can be sent.
  • RECORD_VERSION is now 7. readRecord returns an empty record for any other version, so cap marks saved by version 6 are dropped instead of being trusted.
  • The new tests in both dispatcher test files check both things: a capped head is not recorded, and a second run reads the verifier again and dispatches.
  • The verify-cap.mjs header, the README, the schema, pr-pipeline.yaml, the babysit_merge_policy.py field comment and both changelog fragments no longer say the cap never clears at its head.
  • final() is still used correctly. feedback-dispatch.mjs:778 and :786 call it to tell a cap the reader found from one it could not read. fix-ci-dispatch.mjs no longer imports it.

Reader change. elsewhere() no longer caches its result. That is safe: the branch for a FAIL at the head (line 4379) and the pass branch (line 4405) are separate branches of one if/elif chain, so a decision still reads it at most once. The new order does the same as before. superseded(head) runs before both caps. A FAIL at the head that is not superseded gives verify:repeated-fail at the cap and verify:fail under it. A superseded FAIL at the head goes on to the pass or earlier-fail decision. The tightened assertion in test_babysit_verify_evidence.py (assertIsNone) now pins criterion 3.

Closing issue #17646 acceptance (at this head)

  • Criterion 1, a clean head after FAILs at superseded heads is not capped: met, unchanged since my last review.
  • Criterion 2, an unsuperseded FAIL at the head is capped: met.
  • Criterion 3, a superseded FAIL at the head is not capped: met, and the test is now stricter.
  • Criterion 4, the sibling-FAIL tests are rewritten to put a FAIL at the head: unchanged since my last review.
  • Criterion 5, the tests pass: the author says this ("466 passed", "447 of 448") and I have not checked it. I ran nothing.
  • The issue's "Keep" line, "VERIFY_FINAL never clearing once the cap applies at a FAIL head", conflicts with criterion 3 under this order. The PR body now says the cap can clear at the same head. That conflict is in the issue itself, so this is a note, not a finding.

Spend (not a finding). Once a lanes App pull request is capped, fix-ci-dispatch, and feedback-dispatch for a PR that still has a review thread, read the verifier again on every sweep until pr-merge closes it. The PR body states this cost. It uses existing Actions minutes only.

Other notes

  • This review was blocking only, because this lane already reviewed an earlier head. Advisory findings were dropped.
  • Loosened checks would be marked: fail. The wrapper wrote no loosened_check_verdict: or loosened_check_patterns: lines, so this is the default and there are no consumer entries. It wrote no decision-record line, so I did not check whether the change cites a decision record. Nothing in this push loosens a check.
  • No plugin manifest is changed, so there are no manifest lines.
  • The settled-threads and prior-findings files contained no embedded instructions.

Verifier dropped 0 of 0 candidates.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/address-feedback

Run 44950399269, attempt 1

Field Value
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 44950399269

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

pr-unblock / address-feedback finished round 1 of max 4 on a71c474.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/publish-feedback

Run 44950399269, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed none recorded
Run 44950399269

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/resolve-feedback-threads

Run 44950399269, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed none recorded
Run 44950399269

Failed GitHub API calls

No failed call recorded by the run job's scripts.

Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM66b6f7
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/apply-feedback-patch

Run 44950399269, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed aebeb11
Run 44950399269

Failed GitHub API calls

No failed call recorded by the run job's scripts.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

pr-unblock / address-feedback finished round 2 of max 4 on bd97747.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on bd9774734c49921ba417984ba337ba7e8fcb5f94; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment on lines +4377 to +4384
# The cap decides before supersession: a FAIL at the head at the cap
# stays capped there whatever supersedes it later (VERIFY_FINAL), as
# the pr-unblock dispatchers record a capped head as handled.
head_superseded = head in failed and superseded(head)
if head in failed and len(failed) >= self.max_fails:
refusal = REPEATED_FAIL
detail = f"fails at {len(failed)} heads ({_heads(failed)}), the cap is {self.max_fails}"
elif head in failed and not (head_superseded := superseded(head)):
head_superseded = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this push breaks the third acceptance criterion of #17646.

The criterion says:

A reader test: the same history where the current head's FAIL is superseded does not yield verify:repeated-fail.

The cap now runs before superseded(head), so that history does yield verify:repeated-fail. This push also inverted the tests that pinned the criterion:

  • test_a_superseded_fail_at_the_head_at_the_cap_stays_capped (test file lines 1387-1388)
  • test_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_not (lines 1401-1402)
  • test_the_cap_is_judged_before_a_superseded_fail_at_the_head (lines 1584-1585)

They now assert verify:repeated-fail, with supersededAtHead False.

The issue's Change section asks for the opposite order in two places:

  • The cap check "must evaluate superseded(head) first".
  • "a superseded head FAIL counts as a clean head".

The PR body's Verification section still says "A superseded FAIL at the head with max_fails=1 gives the pass decision, with supersededAtHead true".

How it fails: take FAILs at 3 earlier heads plus a FAIL at head H. A person relinks or edits the body, and pr-verify passes again at H. This is exactly the history() case in the second test. The reader still refuses verify:repeated-fail, so capped_close closes the pull request. The issue says it should not.

My last review offered this order as one of two fixes for the markHandled finding. I'm sorry: that option conflicts with the closing issue. The other option keeps the criterion:

  • Keep supersession ahead of the cap.
  • Make the cap's final / markHandled in feedback-dispatch.mjs and fix-ci-dispatch.mjs not final once supersededAtHead turns true at the same head, for example by re-dispatching when the reader's answer changes.

The other way out is to change the issue's acceptance criterion through an owner decision, then update the PR body to match. Either way, the issue, the code and its tests must agree before this merges.

claude-lane: claude-review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / address-feedback: UNCERTAIN, left open

Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4380, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4381, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1376, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1573, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/feedback-dispatch.mjs:783

Basis: verified for the code, not for the issue text. The code check holds: babysit_verify_evidence.py:4381-4392 return verify:repeated-fail before the superseded-head branch at 4393, and they set head_superseded to False. The tests at test_babysit_verify_evidence.py:1376, 1390 and 1573 pin that order. I could not read issue #17646 from this lane, so I have not checked the acceptance criterion quoted in the finding against the issue itself. One part of the finding is out of date: the PR body's Verification section now says a superseded FAIL at the head with max_fails=1 stays verify:repeated-fail with supersededAtHead false, so it matches the code and not the issue's criterion as quoted. I made no edit because neither way to resolve this can happen in this lane. Putting supersession back ahead of the cap would bring back the earlier markHandled finding: feedback-dispatch.mjs:782-783 marks a head handled once capFinal(capped) is true, so a head that later turns superseded never gets dispatched again. The other code fix, re-dispatching in feedback-dispatch.mjs and fix-ci-dispatch.mjs when supersededAtHead turns true at the same head, changes shared pr-unblock infrastructure, and check-paths refuses both files for this thread. The third way out, changing the criterion, needs an owner decision. A person needs to pick one: an owner decision to change #17646's third criterion, or a session that moves supersession back ahead of the cap together with the dispatcher change, then updates the tests and the PR body to match.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / adjudicate-feedback: the finding stands

A fresh-context adjudicator read the review comment, the locations both sides cited and the code at bd9774734c49921ba417984ba337ba7e8fcb5f94, without the fixing lane's reasoning, and found that the finding holds. The next address-feedback round fixes it; a round that leaves this thread open again ends the lane in a failing status.

The linked issue settles this intent question (#17646):

A reader test: the same history where the current head's FAIL is superseded does not yield verify:repeated-fail.

Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4380, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4381, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1376, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1573, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/feedback-dispatch.mjs:783

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / address-feedback: UNCERTAIN, left open

Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1405, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/fix-ci-dispatch.mjs:688

This is the same finding as the newer thread on line 4379, and the adjudicator found that it stands. The in-scope part of the fix is in the working tree. The reader now judges supersession before the cap: head_failed at babysit_verify_evidence.py:4378 excludes a superseded FAIL at the head, so neither cap refuses there and the passes decide, as issue 17646's third criterion and its Change section ask. The three inverted tests now assert that criterion, each with a paired case that is not superseded and stays verify:repeated-fail. Those tests are the cap-1 relink test at 1378, the history test with 3 earlier FAILed heads at 1405, and the body-fix test at 1591. The docs and the changelog fragments describe the new order too. Left open because the other half of the reviewer's fix is in files check-paths refuses for this lane: feedback-dispatch.mjs:782-783 and fix-ci-dispatch.mjs:688-689 still treat a capped answer as final and mark the head handled, so a head capped before its FAIL is superseded is never dispatched again. The verify-cap.mjs header also still says cap first. A session needs to change the dispatchers so they re-dispatch when supersededAtHead turns true at the same head, and to update that header. No tests were run in this lane; CI checks the change.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

pr-unblock / address-feedback finished round 3 of max 4 on bd97747.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 5dc6f4deee916e0af65e820cc8335dcb5e193481; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment on lines +4375 to +4379
head_superseded = head in failed and superseded(head)
if head in failed and len(failed) >= self.max_fails:
refusal = REPEATED_FAIL
detail = f"fails at {len(failed)} heads ({_heads(failed)}), the cap is {self.max_fails}"
elif head in failed and not (head_superseded := superseded(head)):
head_superseded = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (still open): the third acceptance criterion of #17646 is still not met.

The criterion says:

A reader test: the same history where the current head's FAIL is superseded does not yield verify:repeated-fail.

The latest push only removed comments. The order is the same as at bd977473: the cap at line 4376 runs before head_superseded is used. When the head has a FAIL and len(failed) >= max_fails, the reader returns REPEATED_FAIL and sets head_superseded = False, even if that FAIL is superseded. The issue's Change section asks for the opposite order in two places:

  • the cap check "must evaluate superseded(head) first"
  • "a superseded head FAIL counts as a clean head"

Three tests still assert the behaviour the criterion rules out:

  • test_a_superseded_fail_at_the_head_at_the_cap_stays_capped
  • test_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_not
  • test_the_cap_is_judged_before_a_superseded_fail_at_the_head

How it fails: take FAILs at 3 earlier heads plus a FAIL at head H. A person relinks or edits the body, and pr-verify passes again at H. The reader still refuses with verify:repeated-fail, so capped_close closes the pull request. Under #17646 it should not be closed.

The PR body has been reworded to describe the cap-first order. That rewording does not change the issue's criterion, and the PR context records no owner decision that amends it.

Fix, either way:

Before this merges, the issue, the code and the tests must agree.

claude-lane: claude-review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / address-feedback: UNCERTAIN, left open

Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4379, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1419, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/verify-cap.mjs:6

The finding holds, and the adjudicator's quote of issue 17646 confirms the criterion. The in-scope part of the fix is in the working tree. Reader: babysit_verify_evidence.py now works out whether the head's FAIL is superseded first. head_failed (line 4378) is true only for a FAIL at the head that is not superseded, and only head_failed reaches the two caps and verify:fail, so a superseded FAIL at the head counts as a clean head and the passes decide. Tests: the three tests now assert the criterion. With max_fails=1, a superseded head FAIL (relink at 1378, body fix at 1591) gives the pass decision with supersededAtHead true. The same FAIL with no trusted act, or with a lanes App edit, stays verify:repeated-fail with supersededAtHead false. The history test (1405) has 3 earlier FAILed heads plus a superseded head FAIL and asserts that the result is not verify:repeated-fail (1419) and that supersededAtHead is true. Its unanswered twin stays capped. I also updated the module docstring, the README Verifier verdict section, pr-pipeline.schema.json, the pr-pipeline.yaml comment, the babysit_merge_policy.py comment and both changelog fragments, and recorded a body refresh. Left open because two instances are in files check-paths refuses. First, feedback-dispatch.mjs:782-783 and fix-ci-dispatch.mjs:688-689 still mark a head handled once verify-cap.mjs final() says capped. A head capped before its FAIL is superseded therefore gets no later pass-only round, even though capped_close re-judges and will no longer close it. Second, the verify-cap.mjs header (lines 6-11) still describes the cap-first order. A session needs to make the dispatchers re-read or re-dispatch when supersededAtHead turns true at the same head, and to update that header. I ran no tests (this lane runs none); CI checks the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / adjudicate-feedback: the finding stands

A fresh-context adjudicator read the review comment, the locations both sides cited and the code at 5dc6f4deee916e0af65e820cc8335dcb5e193481, without the fixing lane's reasoning, and found that the finding holds. The next address-feedback round fixes it; a round that leaves this thread open again ends the lane in a failing status.

The linked issue settles this intent question (#17646):

A reader test: the same history where the current head's FAIL is superseded does not yield verify:repeated-fail.

Evidence: plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4378, plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4379, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1378, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1419, plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1591, .github/scripts/pr-unblock/feedback-dispatch.mjs:782, .github/scripts/pr-unblock/verify-cap.mjs:6

@github-actions

Copy link
Copy Markdown
Contributor

The pr-verify verifier failed head 5dc6f4deee916e0af65e820cc8335dcb5e193481. What it wrote, as data:

What it wrote (69 lines)
Summary: Issue #17646 (owner decision Q245) asks that `Reader.evidence` return
  `verify:repeated-fail` only when the head has its own counted FAIL that has not been superseded. A
  head whose only FAIL is superseded should count as clean, so it gets the earlier-fail or pass
  decision. The acceptance list asks for a reader test showing that a superseded head FAIL at the
  cap does not yield `verify:repeated-fail`.

The diff does the opposite. Both cap branches in `Reader.evidence` run before supersession and set
  `head_superseded = False`, so a superseded FAIL at the head stays capped. The new and rewritten
  tests pin that order: `test_a_superseded_fail_at_the_head_at_the_cap_stays_capped`,
  `test_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_not` and
  `test_the_cap_is_judged_before_a_superseded_fail_at_the_head`. The README, `pr-pipeline.yaml`, the
  schema, the docstring, the `verify-cap.mjs` header and both changelog fragments all say
  "superseded or not".

These parts are delivered:
- A clean head after FAILs at earlier heads now gets the pass decision.
- An unsuperseded FAIL at the head at the cap still caps.
- The pass branch no longer caps on sibling FAILs.
- The sibling-FAIL tests now put a FAIL at the head and still expect the cap.

CI on the judged head is green, including Test Python and Test Node
  (https://github.com/melodic-software/claude-code-plugins/actions/runs/44950051834/job/136986081651,
  https://github.com/melodic-software/claude-code-plugins/actions/runs/44950051834/job/136986081829).
  That shows the tests pass, but they pin behaviour the issue rejects. Moving the cap ahead of
  supersession was the address-feedback lane's own choice. It changes the owner-decided rule rather
  than delivering it.
- plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:4375: check for
  supersession before either cap branch. Use `if head in failed and not head_superseded and
  len(failed) >= self.max_fails` here, and add `not head_superseded` to the sibling-count branch
  below it. A head whose only FAIL is superseded then goes on to the earlier-fail or pass decision,
  as #17646 requires ('one whose only FAIL is superseded, gets the verify:earlier-fail or pass
  decision'). Stop forcing `head_superseded = False`.
- plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1376:
  reverse `test_a_superseded_fail_at_the_head_at_the_cap_stays_capped` and the superseded case of
  `test_fails_at_three_earlier_heads_cap_a_head_fail_superseded_or_not`. Assert that a superseded
  head FAIL with FAILs at max_fails heads does NOT yield `verify:repeated-fail` (acceptance item 3).
  Keep the unanswered, unsuperseded case expecting the cap.
- plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_verify_evidence.py:1571:
  rewrite `test_the_cap_is_judged_before_a_superseded_fail_at_the_head` so the body-edit
  supersession at the cap gives no `verify:repeated-fail` and reports `supersededAtHead` true.
- .github/scripts/pr-unblock/verify-cap.mjs:54: the new header says a capped head never clears even
  after a later pass supersedes its FAIL. Restate the rule as #17646 gives it: the cap needs an
  unsuperseded FAIL at the head. The open review thread warns that feedback-dispatch.mjs and
  fix-ci-dispatch.mjs mark a capped head handled and never re-read it after a later supersession.
  Address that so a head that is superseded after it was capped gets re-read. One option is a
  handled key that includes the body-edit or relink time.
- docs/conventions/pr-pipeline/README.md:2965: change 'a counted `fail` at the head, superseded (the
  next item) or not' and 'The cap is judged before supersession, so a capped head stays capped' to
  say that a superseded `fail` at the head does not cap.
- docs/conventions/pr-pipeline.yaml:1638: drop 'superseded or not'. The cap holds only when an
  unsuperseded FAIL stands at the head.
- docs/conventions/pr-pipeline/pr-pipeline.schema.json:1434: drop 'superseded or not' from the
  `max-verify-fails` description. Say the cap applies only when an unsuperseded fail stands at the
  head.
- plugins/source-control/skills/babysit-prs/scripts/babysit_merge_policy.py:918: change 'when one
  stands at its head, superseded or not' to 'when an unsuperseded one stands at its head'.
- plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:24: fix the module
  docstring, which says 'superseded (the next item) or not' and 'The cap is judged before
  supersession, so a capped head stays capped'. Match the rule in #17646: a superseded head FAIL
  counts as a clean head.
- .changes/source-control/fix-pr-verify-cap-head-fail-d4570a1c.md:26: remove 'The cap is still
  judged before supersession, so a capped head stays capped'. Say that a superseded FAIL at the head
  no longer caps.
- .changes/pr-pipeline/fix-pr-verify-cap-head-fail-d4570a1c.md:13: remove 'The cap is judged before
  that FAIL's supersession, so a capped head stays capped'. Say that a superseded FAIL at the head
  no longer caps.
- PR body: the Fix and Verification sections say a superseded FAIL at the head stays capped. Update
  them to match the corrected behaviour. Also say whether tests ran on the final head, since the
  body says the latest round ran no tests.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass removes the label, at a new commit or at this same head: once a trusted person edits the body, and only trusted people have edited it since the fail, this head is judged again, so a fail on the body or the issues it links needs only the body fixed, no new commit.

@github-actions github-actions Bot added the pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it. label Nov 21, 2026
Lane-Activity: pr-unblock/address-feedback
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

pr-unblock / address-feedback finished round 4 of max 4 on aebeb11.

@melodic-automation-lanes

melodic-automation-lanes Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped address-feedback on this pull request: review rounds are not converging: after the round cap and one fresh fixer round the code review still has open findings, so the fixer and the reviewer disagree on what the change should be; on a lanes App pull request the pr-merge lane then closes it and re-admits its issue for a fresh attempt. The lane sets no label: the run ends in a failing pr-unblock/address-feedback status on the head the lane gated.

The review rounds are not converging after the round cap and one fresh fixer round. Open review threads:

When a retry on this head can change the outcome, the lane runs one on its own, up to its per-head cap, and edits this comment if it stops again; a run that does not stop clears the status. When no retry can (a review thread the lane does not take or settle, or a failing check after the retries are spent), a person resolves the threads or gets the checks green, then dispatches the workflow below: that run finds the cause gone and clears the status. On a pull request the lanes App opened, the dispatchers start that run once per stop; on any other, a person does. A new commit to the branch starts clean. A reply on a thread, from an account with write access, starting PR-fix lane: please fix this finding grants one more round on that thread.

To run it again on this head now, dispatch the pr-unblock workflow:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=18028 -f activity=address-feedback

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on aebeb1114eb571ec8fbb3aa2fc8814021b9cfd0b; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment on lines +4375 to +4378
# Supersession is judged before the cap (#17646): a superseded FAIL
# here counts as a clean head, so neither cap refuses at it.
head_superseded = head in failed and superseded(head)
head_failed = head in failed and not head_superseded

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: with supersession first, the cap can clear at its own head, but the dispatchers still record it as final for that head.

This order is right for #17646 criterion 3. The catch is that verify:repeated-fail at head H can now clear at H itself: a trusted relink or body edit plus a new pass supersedes H's FAIL. Line 147 now says so.

The callers still assume the old order:

  • verify-cap.mjs:6-8, changed in this PR, still says "The reader judges the cap before supersession, so at that head it never clears". That is now false.
  • final() (verify-cap.mjs:162-165) treats every cap as one that never clears for the head.
  • feedback-dispatch.mjs:780-784 and fix-ci-dispatch.mjs:686 call markHandled(record, …, head) on it. Later runs at H then skip before any read.

How it fails:

  1. A PR that capped_close does not close is capped at H: a PR not opened by the lanes App, a PR under merge.report-only, or any PR before pr-merge runs.
  2. The dispatcher marks H handled.
  3. A person relinks or edits the body, and pr-verify passes again at H.
  4. The reader now answers supersededAtHead: true, with no refusal.
  5. The pass-only round with verify-superseded-head, which resolves that FAIL's fail-note threads, is never sent, because H is already marked handled.
  6. Those threads keep holding the merge until a person resolves them.

The base branch could not do this: there the cap applied "whatever a pass says".

This is the point of the thread settled at bd97747 (discussion_r4654807864). That thread was settled by the cap-first order, which this push reverts. My last review asked for both halves; only the first landed.

Fix:

  • Key the handled record so a later supersession at H gets read again. Head plus body update time is one option. Another is to skip markHandled for a cap on a PR that capped_close will not close.
  • Reword verify-cap.mjs:6-11, final() and the dispatcher comments to say the cap clears if the head's FAIL is later superseded.

The issue's line "VERIFY_FINAL never clearing once the cap applies at a FAIL head" conflicts with criterion 3 under this order. If the owner accepts the clear, say so in the PR body. The body also still says a superseded FAIL "stays capped".

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/report-escalation

Run 44952613087, attempt 1

Field Value
Model rung 0
Leg base
Run job success
Activity step skipped
Commits pushed none recorded
Run 44952613087

Failed GitHub API calls

No failed call recorded by the run job's scripts.

kyle-sexton and others added 2 commits November 21, 2026 16:32
…expire a capped head's handled mark

Restore issue #17646's order (owner decision Q245): Reader.evidence checks
supersession first, so a superseded FAIL at the head counts as a clean head
and gets the earlier-fail or pass decision, not verify:repeated-fail. The
three reader tests, the README, pr-pipeline.yaml, the schema, the docstring
and both changelog fragments say so again.

A cap no longer holds a head for good, so feedback-dispatch.mjs and
fix-ci-dispatch.mjs stop recording a capped head as handled: a head whose
FAIL a later pass superseded is read again and dispatched. The dispatch
record's version is 7, so cap marks written by version 6 are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
Keep the supersession-first reader and docs from the previous commit, and
take aebeb11's reader tests, which pair each superseded head FAIL with
the same FAIL unsuperseded, still capped. A superseded FAIL now clears
verify:repeated-fail at its head, so the docstring says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtVYpJGw3s2scXJV5jf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pr-verify: close on the repeated-fail cap only when the current head itself FAILs

1 participant