Test the verdict parser against the reviews that broke it

[deonmenezes]

Closes #19.

Why

Every defect in the merge gate was found after it shipped — four by Qodo reviewing #14, and a fifth by hand, when it turned out Qodo edits its verdict into an existing comment so the merge step had never once run.

Four rounds of review examined what the gate does when it runs. None established that it runs.

The parser is the part where a bug merges unreviewed code, and it was an inline heredoc inside the workflow — impossible to run without opening a PR.

What

Moves it to .github/scripts/qodo_verdict.py with fixtures captured from real comments in this repo:

Fixture	Must be
Qodo's 4-bug review of the gate	not clean
Qodo's clean review (0/0/0)	clean
"Qodo is busy working" placeholder	not clean
PR summary comment	not clean
A human reply quoting Bugs (4) in prose	not clean
Clean counters, no /commit/<sha> marker	not clean

The important one is Qodo's own 4-bug review: it contains the words "no issues found" twice, inside its findings, while reporting four bugs. The first version of the parser matched that phrase anywhere in the body and would have merged the PR that broke it.

That review no longer exists through the API — Qodo edited the comment in place once the findings were fixed — so it's preserved here from a copy taken while it was live.

The human-comment fixture came from getting the first one wrong: selecting comments matching Bugs (4) captured my own reply quoting the phrase in prose. Same free-text mistake, one layer up. So it's a test case now.

Also

• The workflow checks out the default branch for the parser, not the PR's copy, so a PR can't edit the rules that decide whether it merges.
• bun-ci.yml runs the parser tests, so a future Qodo wording change fails a test rather than a merge.
• 8 cases, all passing locally alongside the 23 existing tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EqKPWQMhP7XrA3THHV1hkM

[qodo-code-review]

PR Summary by Qodo

Add regression coverage for Qodo verdict parsing

🧪 Tests ✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Extract Qodo verdict parsing into a fail-closed, testable Python script.
• Cover real clean, malformed, quoted, placeholder, and summary comments with regression fixtures.
• Run parser tests in CI and load trusted rules from the default branch.

Diagram

sequenceDiagram
  actor Qodo
  participant GitHub
  participant Gate as Auto-merge gate
  participant Repo as Default branch
  participant Parser as Verdict parser
  participant CI as Bun CI
  participant Merge as Merge step
  Qodo->>GitHub: Edit review comment
  GitHub->>Gate: Send comment event
  Gate->>Repo: Checkout trusted parser
  Gate->>Parser: Parse comment body
  Parser-->>Gate: Return verdict and SHA
  Gate->>Merge: Authorize matching commit
  CI->>Parser: Run fixture tests

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Publish a required Qodo status

• ➕ Lets branch protection enforce the gate natively
• ➕ Creates an auditable verdict status for each commit
• ➕ Separates verdict publication from merge execution
• ➖ Requires branch-protection configuration and status publishing
• ➖ Adds more workflow permissions and operational components
• ➖ Is disproportionate for the repository's current scope

Recommendation: Keep the extracted fail-closed parser and fixture-driven tests for the current direct-merge gate; this is the smallest change that makes the critical logic reproducible and protects it from pull-request modification. If the gate becomes long-lived or expands, migrate the verdict into a required commit status so GitHub owns final enforcement.

Files changed (10) +685 / -31

Refactor (1) +48 / -0

qodo_verdict.pyExtract fail-closed Qodo verdict parsing +48/-0

Extract fail-closed Qodo verdict parsing

• Moves verdict parsing into a reusable Python function and CLI. It recognizes structured counter chips and reviewed commit markers, authorizing only all-zero, understood verdicts bound to a SHA.

.github/scripts/qodo_verdict.py

Tests (6) +621 / -0

human-quoting-counters.mdCapture human prose quoting Qodo counters +20/-0

Capture human prose quoting Qodo counters

• Adds a real human reply that mentions counter text in prose, ensuring unstructured quotations cannot be mistaken for a verdict.

tests/fixtures/human-quoting-counters.md

qodo-clean.htmlCapture a clean Qodo review verdict +35/-0

Capture a clean Qodo review verdict

• Adds a real all-zero Qodo review with a commit marker as the positive parser fixture.

tests/fixtures/qodo-clean.html

qodo-four-bugs.htmlPreserve the four-bug parser regression review +319/-0

Preserve the four-bug parser regression review

• Captures the Qodo review that reports four bugs while containing misleading “no issues found” prose. This verifies structured nonzero counters override free-text phrases.

tests/fixtures/qodo-four-bugs.html

qodo-placeholder.htmlCapture Qodo's in-progress placeholder +3/-0

Capture Qodo's in-progress placeholder

• Adds the pre-verdict placeholder comment to confirm incomplete reviews fail closed.

tests/fixtures/qodo-placeholder.html

qodo-summary.htmlCapture a non-verdict Qodo summary +137/-0

Capture a non-verdict Qodo summary

• Adds a real PR summary comment to verify unrelated Qodo content is not treated as merge authorization.

tests/fixtures/qodo-summary.html

test_qodo_verdict.pyAdd fixture-driven verdict parser regressions +107/-0

Add fixture-driven verdict parser regressions

• Adds eight dependency-free parser tests covering clean reviews, nonzero findings, quoted counters, placeholders, summaries, missing Bugs chips, and absent commit markers. A lightweight built-in runner supports direct CI execution without pytest.

tests/test_qodo_verdict.py

Other (3) +16 / -31

bun-ci.ymlRun verdict parser regressions in CI +5/-0

Run verdict parser regressions in CI

• Adds the standalone Python regression suite to the existing Bun CI job so parser or Qodo-format changes fail before merge.

.github/workflows/bun-ci.yml

qodo-automerge.ymlExecute the trusted standalone verdict parser +7/-31

Execute the trusted standalone verdict parser

• Replaces the inline Python heredoc with the extracted script. The workflow checks out the default branch first, preventing a pull request from changing its own merge authorization rules.

.github/workflows/qodo-automerge.yml

.gitignoreIgnore Python parser test artifacts +4/-0

Ignore Python parser test artifacts

• Excludes Python bytecode and cache directories generated by the parser and its tests.

.gitignore

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗