Test the verdict parser against the reviews that broke it

.github/scripts/qodo_verdict.py

@@ -0,0 +1,48 @@
+"""Read a Qodo review comment and decide whether it authorises a merge.
+
+Kept out of the workflow so it can be tested. Every rule here exists because
+something got past an earlier version of it, so the tests in
+tests/test_qodo_verdict.py are the specification, not an afterthought.
+"""
+
+import re
+
+# Qodo renders its counters as <code> chips. Everything else in the comment is
+# free text that quotes findings and diff hunks verbatim, so a phrase like
+# "no issues found" appears inside reviews that are not clean and cannot be
+# used as a signal.
+_COUNTER = re.compile(r"<code>[^<]*?\((\d+)\)</code>")
+_BUGS_CHIP = re.compile(r"<code>[^<]*Bugs \(\d+\)</code>")
+_COMMIT = re.compile(r"/commit/([0-9a-f]{40})")
+
+
+def parse(body: str) -> tuple[bool, str, list[str]]:
+    """Return (clean, reviewed_sha, counters).
+
+    clean is True only when every counter is zero, the Bugs chip is present, and
+    the comment names the commit it reviewed. Anything unrecognised is not clean:
+    a comment shape this does not understand must stall a merge, never allow one.
+    """
+    counters = _COUNTER.findall(body)
+    has_bugs = _BUGS_CHIP.search(body) is not None
+    match = _COMMIT.search(body)
+    sha = match.group(1) if match else ""
+
+    clean = bool(counters) and has_bugs and all(c == "0" for c in counters) and bool(sha)
+    return clean, sha, counters
+
+
+if __name__ == "__main__":
+    import os
+    import sys
+
+    clean, sha, counters = parse(os.environ.get("BODY", ""))
+    out = [
+        f"clean={'true' if clean else 'false'}",
+        f"reviewed_sha={sha}",
+        f"counters={','.join(counters) if counters else 'none'}",
+    ]
+    print("\n".join(out))
+    # Also echo to stderr so the run log shows the decision without needing the
+    # step output, which is written to a file.
+    print(f"verdict clean={clean} sha={sha[:12]} counters={counters}", file=sys.stderr)

.github/workflows/bun-ci.yml

@@ -41,3 +41,8 @@ jobs:
 
       - name: Test
         run: bun run test
+
+      # The merge gate's parser decides whether unreviewed code can land, so it
+      # is tested here rather than only in the workflow that uses it.
+      - name: Test the Qodo verdict parser
+        run: python3 tests/test_qodo_verdict.py

.github/workflows/qodo-automerge.yml

@@ -33,41 +33,17 @@ jobs:
       github.event.sender.login == 'qodo-code-review[bot]'
     runs-on: ubuntu-latest
     steps:
+      # issue_comment runs against the default branch, which is where the
+      # parser lives; the pull request's own copy is deliberately not used, so a
+      # pull request cannot change the rules that decide whether it merges.
+      - name: Check out the parser
+        uses: actions/checkout@v4
+
       - name: Read the verdict and the commit it applies to
         id: verdict
         env:
           BODY: ${{ github.event.comment.body }}
-        run: |
-          python3 - <<'PY' >> "$GITHUB_OUTPUT"
-          import os, re
-
-          body = os.environ["BODY"]
-
-          # Parse only the structured counters Qodo renders as <code> chips, never
-          # free text. Qodo quotes findings and diff hunks verbatim, so a phrase
-          # like "no issues found" appears inside the body of reviews that are not
-          # clean, and any substring test on the whole comment is forgeable by the
-          # PR's own content.
-          counters = re.findall(r"<code>[^<]*?\((\d+)\)</code>", body)
-
-          # The third chip is named differently across review types ("Requirement
-          # gaps", "Skill insights"), so require the one that is always present
-          # rather than a fixed set.
-          has_bugs = re.search(r"<code>[^<]*Bugs \(\d+\)</code>", body) is not None
-
-          clean = bool(counters) and has_bugs and all(c == "0" for c in counters)
-
-          # Bind the verdict to the revision Qodo actually read. Qodo footers the
-          # comment with the reviewed commit; with no such marker there is no
-          # verifiable revision identity and this must not merge.
-          m = re.search(r"/commit/([0-9a-f]{40})", body)
-          sha = m.group(1) if m else ""
-
-          print(f"clean={'true' if (clean and sha) else 'false'}")
-          print(f"reviewed_sha={sha}")
-          print(f"counters={','.join(counters) if counters else 'none'}")
-          PY
-          echo "Parsed verdict -> clean=${{ steps.verdict.outputs.clean }}" || true
+        run: python3 .github/scripts/qodo_verdict.py >> "$GITHUB_OUTPUT"
 
       - name: Merge the reviewed commit
         if: steps.verdict.outputs.clean == 'true'

.gitignore

@@ -9,3 +9,7 @@ node_modules/
 
 # TrueForge local data (SQLite lives in the OS app data dir; this is just a run dir)
 .trueforge/
+
+# python bytecode from the merge-gate parser and its tests
+__pycache__/
+*.pyc

tests/fixtures/human-quoting-counters.md

@@ -0,0 +1,20 @@
+Acted on all four findings. Every one was real, and the first was demonstrated on this very PR.
+
+**1. Free-text verdict matching.** `grep -qF 'no issues found'` tested the whole comment body. Qodo quotes findings and diff hunks verbatim, so that phrase appears inside reviews that are *not* clean — including the review above, which contains it while reporting `Bugs (4)`. The gate would have merged the PR that broke it. Now only the structured `<code>` counter chips are parsed; every counter must be `0`, and the Bugs chip must be present so an unrecognised comment shape can't pass by having no counters at all.
+
+**2. No binding to the reviewed revision.** This was live, not theoretical: Qodo reviewed `eccd352` while the head had already moved to `947095f`. The reviewed SHA now comes from the commit marker Qodo footers in the comment, must equal `headRefOid`, and the merge passes `--match-head-commit` so a push racing the comparison is rejected by GitHub rather than slipping in. No marker means no verifiable revision identity, so it fails closed.
+
+**3. Read failures counted as clean.** Both queries ended in `|| echo 0`, so an outage or auth error became "nothing failed, nothing pending" and merged without confirming the build ran. Check state is now read per-commit via the API, a read failure exits non-zero, and the `build` check must be present *and* successful rather than merely not failing.
+
+**4. Unreachable pending branch.** Correct on the exit code: `gh pr checks` exits 8 while pending, so `|| echo 0` appended a second zero, making the count `0\n0` — not equal to `0` — so pending checks took the failure exit and the branch meant to handle them never ran. Replaced with a bounded poll against the check-runs API, whose transport status is independent of check conclusions.
+
+Parser verified against the real comments on this repo:
+
+| Case | Result |
+| --- | --- |
+| This review (4 bugs, prose contains "no issues found") | not clean |
+| #13 review (0/0/0) | clean |
+| "Qodo is busy working" | not clean |
+| Clean counters, no commit marker | not clean |
+
+On the suggested alternative of publishing a required status: agreed it's the better shape, and worth doing if this outlives the hackathon. It needs branch protection plus a status-publishing step, which is more moving parts than a one-day repo warrants — the trade recorded here rather than left implicit.

tests/fixtures/qodo-clean.html

@@ -0,0 +1,35 @@
+
+<h3>Code Review by Qodo</h3>
+<code>🐞 Bugs (0)</code>  <code>📘 Rule violations (0)</code>  <code>📎 Requirement gaps (0)</code>
+
+<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
+
+<img src="https://www.qodo.ai/wp-content/uploads/2025/06/qodo-anteater.svg" width="20%">
+
+<h3>Great, no issues found!</h3>
+Qodo reviewed your code and found no material issues that require review
+
+<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
+
+
+
+<!-- qodo-daily-tip:start -->
+
+<details>
+<summary><strong>Tip of the day</strong></summary>
+
+<br/>
+
+<pre>💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full</pre>
+
+<a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a>
+
+</details>
+
+<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
+<!-- qodo-daily-tip:end -->
+
+
+<!-- https://github.com/deonmenezes/edit-ai/commit/d026201dfb2fb25e53c422c872305f6bf8e82902 -->
+
+<a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>