Skip to content

feat(hook): gate PR-claims lacking same-turn mutation - #880

Merged
devseunggwan merged 4 commits into
mainfrom
issue-868-feat-pr-claim-gate
Jul 28, 2026
Merged

devseunggwan merged 4 commits into
mainfrom
issue-868-feat-pr-claim-gate

Conversation

@devseunggwan

@devseunggwan devseunggwan commented Jul 27, 2026 •

Copy link
Copy Markdown
Owner

배경

2026-07-27 세션 retrospect finding #1 (HIGH, repeat 3x). CodeRabbit 리뷰 3건을 local worktree 에만 적용한 상태에서 사용자에게 처리 완료로 보고했다. 그 시점에 push / gh pr comment / 리뷰 코멘트 API / thread resolve 는 전부 0건이었다.

기존 completion-signal-gate (#392) 는 이 케이스를 잡지 못한다: (a) advisory 티어가 이 세션 fire ledger 기준 42건 발화 대비 행동 변화 0건, (b) 증거 검사가 "턴 내 임의의 Bash/Read 호출"이라 PR 을 전혀 건드리지 않은 git status 도 통과시킨다.

변경 사항

신규 sibling 훅 hooks/completion-verify/pr-claim-mutation-gate/impl.py (Stop, completion-verify role) 추가:

  • 트리거: PR/리뷰 SUBJECT(PR, 리뷰 코멘트, 피드백, coderabbit 등) + 처리-완료 CLAIM 동사(처리했, 반영했, resolved, applied, handled, fixed, ...)가 같은 줄에 공존
  • 억제자: 부정형, hedge(것 같다/might have), 질문형, 인용줄(>) — 각각 test case 로 인코딩
  • 증거: 같은 턴 내 PR-surface mutation tool_use (git push, gh pr comment, gh pr review, write-method gh api, GraphQL resolveReviewThread, GitHub MCP comment/review 툴) — read-only gh api 목록 조회는 증거로 인정하지 않음
  • 알려진 accepted gap 으로 문서화: 이중 부정("처리하지 않은 게 아닙니다")은 내부 부정 토큰 때문에 silent 처리됨 — line-level regex gate 의 한계로 spec.md 에 명시

입력 표면 enumeration (praxis:surface-enumeration)

축 커버된 변형
polarity 긍정 claim(5), 부정 claim EN/KR(2), 이중 부정(1, known gap)
언어 EN 단독, KR 단독, mixed 없음(리뷰 도메인 특성상 대부분 단일 스크립트)
hedge 것 같습니다, might have addressed (2) — 결정: silent 처리
quoted/reported > 인용줄 — 결정: silent (claim 을 보고하는 것과 claim 자체를 구분)
부재해야 할 negative subject 없음, subject+claim 없음(subject only), 질문형 EN/KR (4)
증거 turn-scope 같은 턴 push/comment/review/api-write/mcp(5, silent) vs read-only api / Write tool / 이전 턴(3, advisory 유지)

27 fixture case 전부 tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh 에 인코딩.

검증

$ bash tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh
... (27 cases)
pr-claim-mutation-gate: PASS=27 FAIL=0

$ ruff check hooks/completion-verify/pr-claim-mutation-gate/
All checks passed!

$ shellcheck --severity=warning tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh
(no output)

$ python3 -m pytest tests/ -q
1 failed, 499 passed  # 예상된 실패: test_check_dispatch_invariant.py::test_baseline_check_clean
                       # (신규 훅이 hooks/manifest.json 에 미등록 — 오케스트레이터가 Phase 2 통합 시 일괄 등록 예정)

검증하지 못한 것

  • hooks/manifest.json / 생성된 hooks.json 등록은 이번 PR 범위 밖 (Phase 2 lane 병렬 작업 — 오케스트레이터가 통합 시 일괄 등록). scripts/check-plugin-manifests.py 실패는 예상된 것.
  • 실제 Stop 이벤트 라이브 트리거(실제 세션에서 fire)는 검증하지 않음 — fixture 기반 unit 검증만 수행.

Pre-commit verified: bash tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh → PASS=27 FAIL=0; ruff check hooks/completion-verify/pr-claim-mutation-gate/ → All checks passed
Caller chain verified: N/A -- new hook, no caller expected (Stop hook invoked by harness via hooks.json registration, deferred to Phase 2 통합)

Closes #868

Summary by CodeRabbit

  • New Features

    • Added a new Stop-stage verification that blocks when the final response claims PR/review updates but no successful PR-surface mutation occurred in the same turn.
    • Added controls: PRAXIS_PR_CLAIM_BYPASS to silence the gate, and PRAXIS_PR_CLAIM_ADVISORY to downgrade it to advisory behavior.
    • The gate runs with a 10s timeout.
  • Documentation

    • Updated architecture, hook index, bypass registry, operating matrix, and the gate spec location.
  • Tests

    • Added scenario-based coverage for claim detection, evidence timing, advisory/bypass modes, and fail-open behavior.
  • Chores

    • Updated Stop hook ordering to run this gate before strike-counter.

@devseunggwan devseunggwan self-assigned this Jul 27, 2026
@coderabbitai

coderabbitai Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 116a3eb2-194f-43e8-922c-82c4f8a7121b

📥 Commits

Reviewing files that changed from the base of the PR and between eab1da7 and e862f0e.

📒 Files selected for processing (2)
  • hooks/completion-verify/pr-claim-mutation-gate/spec.md
  • tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh
🚧 Files skipped from review as they are similar to previous changes (2)
  • hooks/completion-verify/pr-claim-mutation-gate/spec.md
  • tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh

📝 Walkthrough

Walkthrough

Adds a Stop hook that detects PR completion claims and checks for same-turn PR mutation evidence, with advisory, strict-block, bypass, fail-open, documentation, registration, and scenario-test coverage.

Changes

PR claim mutation verification

Layer / File(s) Summary
Claim detection and decision flow
hooks/completion-verify/pr-claim-mutation-gate/*, hooks/pr-claim-mutation-gate.sh
Detects line-local PR completion claims, checks current-turn mutation commands, and emits advisory or strict-block responses with fail-open handling.
Hook registration and Stop ordering
hooks/manifest.json, .claude-plugin/hooks/hooks.json, .cursor-plugin/hooks/hooks.json, .opencode/hooks/hooks.json, plugins/praxis/..., scripts/check-plugin-manifests.py
Registers the hook and inserts it into the Stop chain after pr-report-destination-gate; updates ordering validation.
Scenario coverage and operational documentation
tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh, docs/..., ARCHITECTURE.md
Covers claim variants, mutation evidence, suppressors, strict and bypass modes, loop guards, and fail-open behavior; documents the hook and environment variables.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant StopEvent
  participant PRClaimMutationGate
  participant Transcript
  participant HookLedger
  StopEvent->>PRClaimMutationGate: invoke Stop hook
  PRClaimMutationGate->>Transcript: read final message and current-turn tool calls
  Transcript-->>PRClaimMutationGate: claim text and mutation evidence
  PRClaimMutationGate->>HookLedger: record advisory or block decision
  PRClaimMutationGate-->>StopEvent: emit JSON decision or silent pass
Loading

Possibly related PRs

  • devseunggwan/praxis#673: Updates the operating-matrix artifact and drift-checking context related to the new hook metadata.

Suggested labels: enhancement

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the new hook that blocks PR claims without same-turn mutation evidence.
Linked Issues check ✅ Passed The PR implements the requested stop hook, same-turn mutation checks, advisory fallback, and fixture/docs updates for issue #868.
Out of Scope Changes check ✅ Passed The added manifests, docs, tests, and scripts all support the new gate; no unrelated code changes stand out.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-868-feat-pr-claim-gate

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Confidence: high
Not-tested: not registered in hooks/manifest.json (orchestrator integration step)
Session-Id: e43a3532-3c9d-47e1-a23d-4f160433db90
The hook shipped unregistered, which is not a deferrable step: CI's
test_baseline_check_clean runs the plugin-manifest check, so an
unregistered hook fails the `test` job and the PR can never go green.
Registration has to live in the PR that adds the hook.

Adds the manifest entry (Stop, after pr-report-destination-gate), the
matching Stop-ordering entry in the check script, the INDEX.md and
ARCHITECTURE.md rows, the strict/opt-out rows in bypass-vars.md, and the
regenerated per-platform hooks.json / wrapper / hook doc.

Pre-commit verified: python3 scripts/check-plugin-manifests.py -> plugin-
manifest check OK (rc=0); python3 -m pytest tests/ -q -> 503 passed;
shellcheck --severity=warning on the generated wrapper -> clean.

Confidence: high
Scope-risk: narrow
Refs #868

Session-Id: e43a3532-3c9d-47e1-a23d-4f160433db90
@devseunggwan
devseunggwan force-pushed the issue-868-feat-pr-claim-gate branch from a22e7b0 to 287e3f5 Compare July 28, 2026 01:21

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hooks/completion-verify/pr-claim-mutation-gate/impl.py`:
- Around line 19-26: Update _GH_API_WRITE_RE so it matches only explicit
write-method gh api calls whose endpoint contains a comments, reviews, or
threads path segment. Preserve the existing method detection and
case-insensitive behavior, while ensuring unrelated write endpoints such as
labels or milestones do not satisfy the PR-surface mutation gate.
- Line 152: Update _MCP_GH_MUTATION_RE and its mutation-detection usage to
recognize only explicitly write-oriented GitHub MCP tool names or predicates,
rather than matching comment, review, merge, or thread substrings anywhere in a
raw name. Ensure consolidated read-only tools such as pull_request_read do not
suppress the PR-mutation claim.

In `@hooks/completion-verify/pr-claim-mutation-gate/spec.md`:
- Around line 96-101: Update the documented PR claim gate behavior so normal
installations enforce blocking by default rather than emitting only an advisory.
Ensure the manifest registration or equivalent deployment configuration enables
PRAXIS_PR_CLAIM_STRICT=1 by default, while preserving PRAXIS_PR_CLAIM_BYPASS=1
as the explicit opt-out.
- Around line 78-87: Update the completion gate logic described in the spec so
it clears only for successful tool calls with verified PR-scoped mutations.
Correlate each matching tool_use with its successful tool result, reject
dry-run/help/echoed or otherwise non-mutating Bash commands, and replace the
broad GitHub MCP name match with an explicit allowlist whose arguments validate
the target PR. Preserve the existing mutation categories while excluding
read-only and issue-scoped operations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d014cbe8-e535-48b2-ab45-ebf37ea208d2

📥 Commits

Reviewing files that changed from the base of the PR and between 337d7c6 and 287e3f5.

📒 Files selected for processing (15)
  • .claude-plugin/hooks/hooks.json
  • .cursor-plugin/hooks/hooks.json
  • .opencode/hooks/hooks.json
  • ARCHITECTURE.md
  • docs/bypass-vars.md
  • docs/hook-operating-matrix.md
  • docs/hook/INDEX.md
  • docs/hook/pr-claim-mutation-gate.md
  • hooks/completion-verify/pr-claim-mutation-gate/impl.py
  • hooks/completion-verify/pr-claim-mutation-gate/spec.md
  • hooks/manifest.json
  • hooks/pr-claim-mutation-gate.sh
  • plugins/praxis/.codex-plugin/hooks/hooks.json
  • scripts/check-plugin-manifests.py
  • tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh

Four CodeRabbit findings, all applied. Each one let something that is not a
PR-surface mutation clear the gate -- the precise failure the hook exists to
prevent.

1. `_GH_API_WRITE_RE` matched any write-method `gh api`, though the docstring
   promised a comments/reviews/threads endpoint restriction. Probe: the old
   regex accepted `gh api repos/o/r/issues/1/labels --method POST`. The
   endpoint segment is now required.

2. `_MCP_GH_MUTATION_RE` matched `comment|review|merge|thread` anywhere in a
   raw tool name, so consolidated READ tools cleared the claim. Probe: the old
   regex accepted `mcp__github__get_review_comments`. Matching now runs on
   write verbs, and a get/list/search/read prefix always loses.

3. Only the tool_use text was checked, so a failed, dry-run, help or echoed
   command counted as a mutation. tool_use `id` is now correlated with its
   tool_result `is_error` (same mechanism as pr-report-destination-gate), and
   rehearsal/echo forms are excluded. A tool_use with no result yet counts as
   success -- the gate is the accuser, so it biases to silence.

4. The hook shipped advisory-by-default while issue #868 asks for the
   opposite verbatim: "부재 시 차단 ... advisory 티어의 실측 효과가 0 이므로
   advisory 로는 같은 실패가 반복된다", and its AC reads "mutation 없음 ->
   block". Default flipped to block; `PRAXIS_PR_CLAIM_STRICT` is replaced by
   `PRAXIS_PR_CLAIM_ADVISORY` as a demote switch, mirroring
   negative-existence-verdict-gate, which already blocks-by-default with an
   advisory escape hatch.

Six fixtures added, one per hole: labels-endpoint write, consolidated MCP
reader, dry-run push, echoed push, failed push, succeeded push. The last two
are a matched pair -- identical commands, opposite is_error -- so the
correlation is what the test distinguishes, not the command text.

Also fixes the two reviewdog markdownlint errors in spec.md (h1->h3 jump,
unlabelled fence).

Pre-commit verified: bash tests/hooks/completion-verify/test_pr_claim_
mutation_gate.sh -> PASS=33 FAIL=0; python3 -m pytest tests/ -q -> 503
passed; scripts/check-plugin-manifests.py -> OK (rc=0); ruff check hooks/ ->
clean; shellcheck --severity=warning on the changed test -> clean;
markdownlint-cli2 on the changed docs -> 0 issues.

Raised by CodeRabbit on PR #880 (1 Critical, 3 Major).

Confidence: high
Not-tested: the block tier's live re-prompt behaviour is exercised only
through the fixture's stdout JSON, not against a running Stop lane.
Scope-risk: moderate
Directive: the endpoint segment in _GH_API_WRITE_RE and the read-prefix
loss in _is_mcp_mutation_tool are the gate, not decoration -- widening
either one restores the defect this commit closes.
Refs #868

Session-Id: e43a3532-3c9d-47e1-a23d-4f160433db90

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
hooks/completion-verify/pr-claim-mutation-gate/impl.py (1)

165-215: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Treat consolidated GitHub MCP PR-review write tools as mutations.

_is_mcp_mutation_tool rejects "mcp__github__pull_request_review_write" because its suffix has no ^verb_ prefix or _verb_ infix from the current list. Since that tool’s write methods include create, submit_pending, resolve_thread, and unresolve_thread, a claim backed by that tool should clear the gate instead of being blocked. Match consolidated write tool names and inspect input.method for write-only method values.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hooks/completion-verify/pr-claim-mutation-gate/impl.py` around lines 165 -
215, Update _is_mcp_mutation_tool to recognize the consolidated
pull_request_review_write tool as mutation-capable, including its fully
qualified MCP name. For that tool, inspect input.method and return true only for
write methods such as create, submit_pending, resolve_thread, and
unresolve_thread; preserve read-tool precedence and existing verb matching for
other tools.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh`:
- Around line 180-186: Update the inline comments for the
“write-is-not-pr-mutation” and “prev-turn-mutation-not-evidence” cases to state
that the hook blocks by default, matching their existing run_case block
assertions. Do not change the test commands or expectations.

---

Outside diff comments:
In `@hooks/completion-verify/pr-claim-mutation-gate/impl.py`:
- Around line 165-215: Update _is_mcp_mutation_tool to recognize the
consolidated pull_request_review_write tool as mutation-capable, including its
fully qualified MCP name. For that tool, inspect input.method and return true
only for write methods such as create, submit_pending, resolve_thread, and
unresolve_thread; preserve read-tool precedence and existing verb matching for
other tools.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 973deea9-3595-4722-b6ac-2c7b41bff140

📥 Commits

Reviewing files that changed from the base of the PR and between 287e3f5 and eab1da7.

📒 Files selected for processing (8)
  • ARCHITECTURE.md
  • docs/bypass-vars.md
  • docs/hook-operating-matrix.md
  • docs/hook/INDEX.md
  • hooks/completion-verify/pr-claim-mutation-gate/impl.py
  • hooks/completion-verify/pr-claim-mutation-gate/spec.md
  • hooks/manifest.json
  • tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • docs/hook-operating-matrix.md

The block-by-default flip left three case headers and the spec's Tests
paragraph still describing advisory outcomes, so the prose contradicted the
assertions directly beneath it -- the shape that invites a future reader to
"fix" a correct assertion. Also updates the count (27 -> 33) and describes
what the six new cases pin.

Pre-commit verified: bash tests/hooks/completion-verify/test_pr_claim_mutation_gate.sh -> PASS=33 FAIL=0; markdownlint-cli2 on the spec -> 0 issues; shellcheck --severity=warning -> clean.

Confidence: high
Scope-risk: narrow
Refs #868

Session-Id: e43a3532-3c9d-47e1-a23d-4f160433db90
@devseunggwan
devseunggwan merged commit 7ba3a03 into main Jul 28, 2026
10 checks passed
@devseunggwan
devseunggwan deleted the issue-868-feat-pr-claim-gate branch July 28, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(hook): gate PR-surface completion claims on same-turn mutation

1 participant