Skip to content

Auto-merge gate has no test, and its bugs were all found in production #19

Description

@deonmenezes

The pattern

Every defect in .github/workflows/qodo-automerge.yml was found after it shipped:

Found by Defect
Qodo (#14) free-text grep 'no issues found' matched quoted findings, so a non-clean review could authorize its own merge
Qodo (#14) verdict not bound to the reviewed commit; a later push merged unreviewed
Qodo (#14) || echo 0 turned API failures into "nothing failed"
Qodo (#14) gh pr checks exits 8 when pending, so || echo 0 produced "0\n0" and the pending branch was unreachable
Manual (#16) Qodo edits its verdict in, so types: [created] meant the merge step never ran at all

The last one is the telling one: four rounds of review examined what the gate does when it runs, and none established that it runs.

What's missing

The verdict parser is pure — comment body in, (clean, sha) out — and is the part where a bug merges unreviewed code. It has no test.

Proposed

Extract the parser out of the inline heredoc into .github/scripts/qodo_verdict.py, and cover with fixtures captured from real comments in this repo:

These fixtures already exist as ad-hoc checks that were run by hand before shipping #14. They should be in CI so the next wording change from Qodo fails a test rather than a merge.

Co-Authored-By: qodo-code-review[bot] <151058649+qodo-code-review[bot]@users.noreply.github.com>

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions