Skip to content

fix(pr-verify): trust intake-triage-briefed issues the owner has not vetoed - #18089

Open
kyle-sexton wants to merge 3 commits into
mainfrom
fix/pr-verify-trust-briefed-issues
Open

kyle-sexton wants to merge 3 commits into
mainfrom
fix/pr-verify-trust-briefed-issues

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

No related issue: implements the owner's standing rule at https://github.com/melodic-software/architecture/blob/main/docs/owner-decisions.md?plain=1#L318

Summary

pr-verify gave a bot-filed issue no spec text, so a pull request for it failed with verify:untrusted-spec (#16032 and #18063 did). The owner chose a standing rule (owner-decisions L318, owner, verbatim: "Standing rule (Recommended)"): an issue intake-triage has briefed, and that the owner has not vetoed, counts as trusted spec, with no case-by-case confirmation.

Fix

Definitions, both agent-made from what the code already treats as admission and as a stop:

  • Briefed: the lanes App filed the issue, and the lanes App's newest admission record names its current title and body by sha256. The record is the <!-- issue-admission-record --> comment admission writes when it admits an issue (.github/scripts/intake-triage/admit.mjs:2880). It counts only when the lanes App's account id posted it (never its login), its text is canonical, and it was never edited (.github/scripts/lib/admission-record.mjs:30, :62, :109). Any edit to the title or body after the record breaks the hash, so the brief no longer counts.
  • Vetoed: the issue carries a hold label (needs-human, needs-owner-decision, needs-reporter-info or needs-spec, .github/actions/resolve-config/read-status.mjs:21) or provenance: untrusted (.github/scripts/pr-merge/merge-holds.mjs:1701), or its body has a line opening with the word hold and a colon (admit.mjs:253 BODY_HOLD). A label list that cannot be read whole also counts as vetoed. The check does not ask who set a hold. Any hold keeps the body out of the spec, which is the stricter reading for a widening of trust.

Where it is enforced:

  • admit.mjs briefedIssue (:2032) is the one rule, beside ownAgentText and refinementSpec, and carries the one code comment that links the owner's row. resolve-pr.mjs linkedIssues calls it (:1119) for an issue no listed human wrote that is not a split child or an own-agent filing. The issue's body then reaches the verifier as spec. Its comments still count only when a listed human wrote them. The issue query now reads the labels' totalCount.
  • The merge gate cannot import JavaScript, so babysit_verify_evidence.py briefed_issue (:1089, with admission_record at :1037) is its twin, as own_agent_text is for ownAgentText. _trusted_spec calls it last (:3696) through _briefed, which reads the issue live once per Reader (BRIEFED_ISSUE_QUERY: title, body, author, labels, newest 100 comments). If only pr-verify had changed, a pass would still read verify:untrusted-spec at the gate, which needs a person.
  • docs/conventions/pr-pipeline/README.md states the rule beside the own-agent paragraph. Fragments: pr-pipeline (minor), source-control (minor).

Agent-made design choices:

  1. The marker is the admission record. The <!-- intake-triage refinement --> comment was not chosen, because a refinement that asks questions is not a brief. The <!-- claude-lane intake-triage --> verdict was not chosen either: github-actions[bot] posts it, and any workflow can act as that account.
  2. A veto is any hold or untrusted label, or a body hold: line, whoever set it. A hold set only in the Work status field is not read by this check. It still stops a lane merge through issue_hold_refusal (babysit_merge_policy.py:5425, :5965). A Work-status-only hold on a non-lane pull request blocks neither pr-verify nor this gate.
  3. The record does not have to predate the pull request: the hash binds it to the text as written.
  4. An owner's own body edit after admission also breaks the brief. Admission has to run again.
  5. The gate reads the newest 100 comments, which always hold the newest record. pr-verify reads all of them.
  6. Outcome for the two motivating issues: neither becomes trusted under this rule. Quarantined suite fails on main: lib/cli-version-check.test.sh #18063 has no admission record, because admission refused it for the missing source: label. Delete toolchain's bundled ecosystem defaults - part 2 of 3 #16032's newest record (2026-11-21T20:25:53Z) predates the owner's body edit (21:22:57Z), so its hash is expected not to match (not computed). Rows L316 and L317 already cover both.
  7. Left unchanged: write-issue.mjs already passes a lanes App filing's body to the issue lane as written; it is the lane's input, not a spec-trust decision. trustedSpecTexts (admission's own read) is unchanged, so admission does not trust its own record.

Verification

  • node --test --test-concurrency=4 .github/scripts/pr-verify/resolve-pr.test.mjs: 195 pass, 0 fail. The new test covers 13 cases:
    • trusted: a briefed issue with no hold;
    • untrusted:
      • no record;
      • the record in a listed human's comment or in github-actions[bot]'s comment (the forged-marker cases);
      • the body edited after the record;
      • a record naming another title;
      • an edited record;
      • a hold label (needs-human, or Needs-Owner-Decision in another case);
      • provenance: untrusted;
      • a hold: body line;
      • labels not read whole;
      • another bot as the author.
  • The "briefed, no hold" case was red on base: the own-agent test's "no source label" case expects no spec for the same issue.
  • node --test --test-concurrency=4 .github/scripts/intake-triage/admit.test.mjs: 752 pass, 0 fail.
  • python3 -m unittest over test_babysit_*_trust.py: 87 tests OK. The new classes cover:
    • the same cases, plus an older record for other text, a newer record for other text, and a malformed App record;
    • a node run of briefedIssue against briefed_issue on eight issues, which agree.
  • test_babysit_split_child_trust.py's fake now answers the new query.
  • The other suites that reach _trusted_spec (verify_evidence, capped_close, verify_index, guards, merge_policy and others): OK, except 29 failures in test_babysit_merge and test_babysit_merge_unattended and failures in the two review-evidence suites. The same 29 fail on main's checkout without this change.
  • scripts/run-ruff.sh check passes on the three Python files.
  • scripts/check-affected.sh: 19 passed, 1 skipped.
  • scripts/selection-audit.sh trace --strict over the three changed suites found that resolve-pr.test.mjs reads docs/upstream pages (upstreamPages' default directory) without selecting them (58 gaps). The suite now declares // test-scope: docs/upstream/*, and the trace at f8449d48ba reports no gap.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits November 21, 2026 19:48
…vetoed

Claim commit: the changelog fragment for the owner's standing rule at
https://github.com/melodic-software/architecture/blob/main/docs/owner-decisions.md?plain=1#L318.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtViDMW2P121uLvSb35
…vetoed

pr-verify (admit.mjs briefedIssue, called from resolve-pr.mjs
linkedIssues) and the merge gate (babysit_verify_evidence.py
briefed_issue) take the body of a lanes App filing as spec when the
lanes App's newest admission record names its current title and body by
sha256 and the issue carries no hold label, no provenance: untrusted and
no hold: body line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtViDMW2P121uLvSb35
…eads

The test-selection trace found the suite reading docs/upstream pages
(upstreamPages' default directory) that the selector did not map to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RAoAtViDMW2P121uLvSb35
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 22, 2026 01:16
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T01:17:43.813146Z f8449d4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8449d48ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"query($o:String!,$r:String!,$n:Int!){repository(owner:$o,name:$r){"
"issue(number:$n){title body author{...actor} "
"labels(first:100){totalCount nodes{name}} "
"comments(last:100){nodes{createdAt lastEditedAt body author{...actor}}}}}}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Paginate comments when locating the newest admission record

If an issue accumulates 100 comments after its latest admission record, this query excludes that record and _briefed returns false even though the issue's title, body, and holds have not changed. Unlike this query, resolve-pr.mjs reads the complete comment history, so pr-verify can approve the spec while the merge gate rejects the same issue as verify:untrusted-spec. Using the trust-test fixture, an admission record followed by 100 ordinary App comments returns true with the whole history and false with this query's last-100 slice. Read earlier pages until the newest admission record is found rather than treating its absence from this slice as no admission.

Useful? React with 👍 / 👎.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed f8449d48bacda02a06fc85a91b315696cca42faa: read 9 of 9 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #18089 (head f8449d4)

No security issues found. I read all 9 changed files in full. Three are on trusted paths: admit.mjs, resolve-pr.mjs and babysit_verify_evidence.py. For each I read the whole change and the code it gates.

What changed

pr-verify and the babysit merge gate now accept an issue the lanes App filed as trusted spec when all of these hold:

  • The lanes App's newest admission record names the issue's current title and body by sha256.
  • The record was posted under the App's numeric account id, its text is in canonical form, and it was never edited.
  • The labels can be read in full and include no HOLD_STATES label and no provenance: untrusted.
  • The body has no hold: line.

This removes the untrusted-spec stop for that kind of issue. That stop is a security control and not one of the owner's friction holds, so I judged it like any other merge hold.

What I checked

  • Who can control the trusted text. Only the issue's author (the App) or someone with repository write access can change an issue body. Text that people outside the repository put in comments still counts toward spec only when a listed human wrote it. The comments: filter in resolve-pr.mjs is unchanged. Any edit to the title or body after the record changes the hash, so the record no longer matches (admission-record.mjs matchesRecord, L109).

  • Faked records. A record counts only when the App's numeric id posted it (briefedIssue maps user.id from databaseId only for a Bot; admission_record checks __typename == "Bot" and databaseId == app_id). A human or github-actions[bot] posting the same marker does not count, and both test suites cover that case. An App comment that opens with the marker but is malformed or edited throws, and both the JS and Python code then refuse.

  • Who writes records. Records are written in three places:

    • admit.mjs L2880, which writes only after decide() has not refused the issue. decide() refuses an App filing that is not a trusted split child or own-agent filing and holds it as a security hold (L1469-1471).
    • label-unblocked.mjs L1051, which runs only for issues a trusted actor marked ready and that carry no hold.
    • report-outcome.mjs L1168, which runs only for an issue that is already agent-ready and has no hold.

    None of these lets an untrusted actor create a record.

  • Untrusted text reaching a credentialed lane. This does not change. write-issue.mjs already passes the body of a lanes App filing to the issue lane as written. This change only affects whether the verifier counts that body as spec. It does not grow any token's scope or trust any new actor.

  • Do the JS and Python versions agree? Both build the same canonical form, check v/actor_id/hex the same way and treat an edited record the same way. Python's BRIEF_HOLD_RE matches JS BODY_HOLD (/^[^\S\r\n]*hold:/im), including the U+2028/U+2029 line breaks. Both refuse when the label list cannot be read in full. The gate reads only the newest 100 comments, while pr-verify reads all of them. Where that differs, it only affects how older malformed App records are handled; it never changes which record is newest.

  • Gaps the PR states. A hold set only in the Work status field is not seen by this check. It still stops a lane merge through issue_hold_refusal. A non-lane PR comes from the owner or the owner's agents, so it adds no outside attacker path.

Embedded text: none. The PR context contains no instruction to the review lane.

No spend change: the gate makes one extra GraphQL read per linked issue it checks.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed f8449d48bacda02a06fc85a91b315696cca42faa: read 9 of 9 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #18089, trust intake-briefed lanes App filings as spec

No blocking or advisory findings.

I read all 9 changed files in full in the diff. I compared the new JS rule, admit.mjs briefedIssue, with its Python twin, babysit_verify_evidence.py briefed_issue and admission_record, against lib/admission-record.mjs admissionRecords/matchesRecord and admit.mjs BODY_HOLD. These points match on both sides:

  • Who posted the record: both require the lanes App's account id on a Bot, never the login.
  • Record shape: canonical JSON, v === 1 as an int (Python's type(...) is int rejects True), a safe-integer actor_id and 64-character hex hashes.
  • Edit check: a record counts only when it was never edited. JS maps updated_at = lastEditedAt ?? createdAt; Python checks lastEditedAt is None or == createdAt.
  • Hash inputs: sha256 over the title and body, with null read as "" on both sides.
  • Labels: both refuse a label list that is not read whole (totalCount against the node count).
  • Body hold: line: Python's lookbehind on \n \r U+2028 U+2029 plus JS_SPACE without \r\n stands in for JS's multiline ^[^\S\r\n]*.

In resolve-pr.mjs linkedIssues, the new briefed branch can't take over the recurring, drift or incident specs. Those all need github-actions[bot] as author, and briefed needs the lanes App, so the two sets of issues can't overlap. Comments still count only when a listed human wrote them. The new tests cover the forged-marker, edited-record, edited-body, hold-label, provenance: untrusted, body hold: line, cut-short label list and wrong-author cases on both sides. A node cross-check also runs briefedIssue against briefed_issue on eight issues. I ran nothing myself. The PR's test results (195 and 752 node tests passing, 87 Python tests OK, 29 failures it says are already on main) are the author's claims and I haven't verified them.

Small parity edges I checked and didn't count as findings, because realistic data won't hit them and none adds trust on its own:

  • On a same-second tie between two records, Python's max keeps the first and JS's sort keeps the last.
  • pr-verify reads 50 labels and the gate reads 100.
  • The gate reads only the newest 100 comments; pr-verify reads all of them.

Closing issue: none. The PR says "No related issue" and cites the owner's standing rule, so there are no acceptance criteria to check.

Path-scoped rules: .claude/rules/ruff-pin.md (**/*.py), skill-bodies-state-current-rules.md and cost-claims.md match changed files. None is broken by this diff. I found no paved-path row in the standards index.

Loosened checks: The change widens what pr-verify and the merge gate accept as trusted spec. I classed that as a bot-author/trust hold (a friction hold: no finding under the criterion) and a security control, which the security lane (pr-review-security) owns. So this lane made no finding for it. Loosened checks would be marked: fail (default: the prompt had no loosened_check_verdict: line); patterns: 0 (no loosened_check_patterns: line). The prompt also had no loosened_check_decision_record: line, so I didn't check whether a citation is required.

Scope: I left security findings to the separate security lane, as REVIEW.md splits the work. This was a full review, not blocking-only: there are no earlier heads and no prior findings.

Spend: the gate's _briefed adds one GraphQL issue read per lanes App linked issue per Reader, under the Actions token. It starts no billing.

Verifier dropped 0 of 0 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

The pr-verify verifier failed head f8449d48bacda02a06fc85a91b315696cca42faa. What it wrote, as data:

What it wrote (27 lines)
Summary: PR #18089 links no issue, so its only spec is its own title and body (spec_kind
  `pull-request`). The diff loosens the untrusted-spec check in two places. In pr-verify,
  `resolve-pr.mjs` `linkedIssues` calls the new `briefedIssue`. In the babysit merge gate,
  `babysit_verify_evidence.py` `_trusted_spec` calls `_briefed`/`briefed_issue`. Both now accept the
  body of an issue the lanes App filed, which no listed human wrote, as trusted spec. A spec counts
  as asking for a weakened check only when a `closes` or `refs` entry names it and carries the
  owner-decision line link and the owner's verbatim words. The PR body has both, but a
  `pull-request` entry cannot authorize a loosened check, so that cannot excuse this one. The code
  itself looks careful. The record is matched on the App's account id, must be canonical and never
  edited, and must hash the current title and body. Hold labels, `provenance: untrusted`, a `hold:`
  body line or a label list that cannot be read whole all fail closed. The JS and Python versions
  have a parity test. All Test Node, Test Python and Test Bash jobs passed (e.g.
  https://github.com/melodic-software/claude-code-plugins/actions/runs/44970658059/job/137049688094,
  https://github.com/melodic-software/claude-code-plugins/actions/runs/44970658059/job/137049688086).
  The failure is about authorization, not about how the code is written.
- PR body: this change loosens the untrusted-spec check, which is a widening of trust. Only a linked
  issue can authorize that: add a `Closes #<n>` or `Refs: #<n>` line for an issue whose trusted body
  names this loosened check and holds the owner-decisions line link plus the owner's words after
  `verbatim:` in double quotes. The owner-decision citation in the body of a pull request that links
  no issue does not count.
- .github/scripts/pr-verify/resolve-pr.mjs:1119: the new `briefed` branch makes a lanes App filing's
  body trusted spec without a closes/refs spec authorizing the loosened check. Keep it only once a
  linked issue as above authorizes it, or remove it.
- plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py:3696: `_trusted_spec`
  now also trusts `_briefed` issues, which loosens the merge gate's `verify:untrusted-spec` refusal
  without a closes/refs spec authorizing it. Keep it only once a linked issue as above authorizes
  it, or remove it.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass removes the label, at a new commit or at this same head: once a trusted person edits the body, and only trusted people have edited it since the fail, this head is judged again, so a fail on the body or the issues it links needs only the body fixed, no new commit.

@github-actions github-actions Bot added the pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it. label Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant