Skip to content

Fail closed when model review output remains empty or unparseable #6

Description

@dymoo

What happened

CommitReview completed green and posted “0 findings / No defects found” even though
every model review part remained empty or unparseable after repair.

Reproduction:

The job log records, for each of parts 1–3:

  1. Model returned an empty message.
  2. Could not parse JSON ...; asking it to repair.
  3. Model review ... still unparseable; treating as empty.

The action then emitted Verification kept 0 finding(s); 0 refuted, exited success,
and posted a sticky summary claiming no defects across 9 files.

A smaller run also returned an empty message during finding verification and still
completed green: https://github.com/byline-fyi/byline/actions/runs/30709977568

Why this is unsafe

reviewed=true and a green job are used as coverage evidence. A model response that
cannot be parsed after the bounded repair attempt is not a valid empty findings list;
it is missing review output. Treating it as empty makes a provider/schema failure look
like a clean review.

This is distinct from deliberately ignored paths, which are correctly reported and
remain compatible with fail-closed coverage.

Expected behavior

  • If any primary review part remains empty or unparseable after repair, fail the action
    and do not set reviewed=true.
  • Do not post “No defects found” for an incomplete review.
  • The sticky/job summary should identify the failed part without including repository
    content or secrets.
  • Add a regression test for empty and malformed primary-review responses after repair.
  • Decide and test the verification-stage policy separately: an unparseable verification
    must not silently convert a potentially valid finding into a clean-review claim.

Non-goals

  • No provider/model preflight.
  • No alternate model or fallback route.
  • No loosening of context completeness checks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions