What happened
CommitReview completed green and posted “0 findings / No defects found” even though
every model review part remained empty or unparseable after repair.
Reproduction:
The job log records, for each of parts 1–3:
Model returned an empty message.
Could not parse JSON ...; asking it to repair.
Model review ... still unparseable; treating as empty.
The action then emitted Verification kept 0 finding(s); 0 refuted, exited success,
and posted a sticky summary claiming no defects across 9 files.
A smaller run also returned an empty message during finding verification and still
completed green: https://github.com/byline-fyi/byline/actions/runs/30709977568
Why this is unsafe
reviewed=true and a green job are used as coverage evidence. A model response that
cannot be parsed after the bounded repair attempt is not a valid empty findings list;
it is missing review output. Treating it as empty makes a provider/schema failure look
like a clean review.
This is distinct from deliberately ignored paths, which are correctly reported and
remain compatible with fail-closed coverage.
Expected behavior
- If any primary review part remains empty or unparseable after repair, fail the action
and do not set reviewed=true.
- Do not post “No defects found” for an incomplete review.
- The sticky/job summary should identify the failed part without including repository
content or secrets.
- Add a regression test for empty and malformed primary-review responses after repair.
- Decide and test the verification-stage policy separately: an unparseable verification
must not silently convert a potentially valid finding into a clean-review claim.
Non-goals
- No provider/model preflight.
- No alternate model or fallback route.
- No loosening of context completeness checks.
What happened
CommitReview completed green and posted “0 findings / No defects found” even though
every model review part remained empty or unparseable after repair.
Reproduction:
d3199faff91214efca50a651ff9e6671245e837a(v2.0.2)deepseek/deepseek-v4-flash-0731through an OpenAI-compatible endpointThe job log records, for each of parts 1–3:
Model returned an empty message.Could not parse JSON ...; asking it to repair.Model review ... still unparseable; treating as empty.The action then emitted
Verification kept 0 finding(s); 0 refuted, exited success,and posted a sticky summary claiming no defects across 9 files.
A smaller run also returned an empty message during finding verification and still
completed green: https://github.com/byline-fyi/byline/actions/runs/30709977568
Why this is unsafe
reviewed=trueand a green job are used as coverage evidence. A model response thatcannot be parsed after the bounded repair attempt is not a valid empty findings list;
it is missing review output. Treating it as empty makes a provider/schema failure look
like a clean review.
This is distinct from deliberately ignored paths, which are correctly reported and
remain compatible with fail-closed coverage.
Expected behavior
and do not set
reviewed=true.content or secrets.
must not silently convert a potentially valid finding into a clean-review claim.
Non-goals