The pattern
Every defect in .github/workflows/qodo-automerge.yml was found after it shipped:
| Found by |
Defect |
| Qodo (#14) |
free-text grep 'no issues found' matched quoted findings, so a non-clean review could authorize its own merge |
| Qodo (#14) |
verdict not bound to the reviewed commit; a later push merged unreviewed |
| Qodo (#14) |
|| echo 0 turned API failures into "nothing failed" |
| Qodo (#14) |
gh pr checks exits 8 when pending, so || echo 0 produced "0\n0" and the pending branch was unreachable |
| Manual (#16) |
Qodo edits its verdict in, so types: [created] meant the merge step never ran at all |
The last one is the telling one: four rounds of review examined what the gate does when it runs, and none established that it runs.
What's missing
The verdict parser is pure — comment body in, (clean, sha) out — and is the part where a bug merges unreviewed code. It has no test.
Proposed
Extract the parser out of the inline heredoc into .github/scripts/qodo_verdict.py, and cover with fixtures captured from real comments in this repo:
These fixtures already exist as ad-hoc checks that were run by hand before shipping #14. They should be in CI so the next wording change from Qodo fails a test rather than a merge.
Co-Authored-By: qodo-code-review[bot] <151058649+qodo-code-review[bot]@users.noreply.github.com>
The pattern
Every defect in
.github/workflows/qodo-automerge.ymlwas found after it shipped:grep 'no issues found'matched quoted findings, so a non-clean review could authorize its own merge|| echo 0turned API failures into "nothing failed"gh pr checksexits 8 when pending, so|| echo 0produced"0\n0"and the pending branch was unreachabletypes: [created]meant the merge step never ran at allThe last one is the telling one: four rounds of review examined what the gate does when it runs, and none established that it runs.
What's missing
The verdict parser is pure — comment body in,
(clean, sha)out — and is the part where a bug merges unreviewed code. It has no test.Proposed
Extract the parser out of the inline heredoc into
.github/scripts/qodo_verdict.py, and cover with fixtures captured from real comments in this repo:no issues found-> not cleanBugs (0) / Rule violations (0) / Requirement gaps (0)-> cleanQodo is busy workingplaceholder -> not clean/commit/<sha>marker -> not clean, no verifiable revisionThese fixtures already exist as ad-hoc checks that were run by hand before shipping #14. They should be in CI so the next wording change from Qodo fails a test rather than a merge.
Co-Authored-By: qodo-code-review[bot] <151058649+qodo-code-review[bot]@users.noreply.github.com>