Skip to content

Codewhale review gate fails closed on large diffs: reasoning consumes the whole output budget, review never posts #6285

Description

@Hmbown

What happens

The Codewhale review required check fails on PRs with large diffs, and it fails
without producing a review — so a required gate blocks the merge while telling
the author nothing about their code.

Observed on PR #6281 and PR #6284 (both currently blocked by this, both with
otherwise-green required checks):

Review key: BYOK provider secret (provider: deepseek)
Review limits: 200000 characters per pass, at most 1 passes
Output budget: CLI automatic cap (no override set)
error: Review pass 1/1 incomplete: provider stop reason `length`;
       the partial review was not accepted or posted.;
       publication: not_attempted; completed review passes: 0/1;
       accumulated usage: {"input_tokens":35839,"output_tokens":65536,
       "prompt_cache_hit_tokens":640,"prompt_cache_miss_tokens":35199,
       "reasoning_tokens":65536}

Run: https://github.com/Hmbown/Codewhale/actions/runs/35069645353

The actual defect

output_tokens: 65536 and reasoning_tokens: 65536 are the same number. The
model spent its entire output allowance on reasoning and emitted zero
review text, then hit length and the action discarded the (empty) partial.

So the budget is being accounted in the wrong unit for a reasoning model: the
cap has to leave room for the answer after reasoning, or reasoning has to be
bounded separately. This is the same shape as the sub-agent hand-back reserve
problem in #6194 — a reserve sized as if only the visible output costs
anything.

Two independent things are wrong:

  1. It fails closed with no signal. A gate that blocks a merge must say why
    the code is wrong. "The reviewer ran out of budget" is the gate's problem,
    not the author's, and it should not read as a failed review. Either surface
    it as a distinct neutral/skipped outcome, or retry with a reduced scope.
  2. One pass, no degradation path. at most 1 passes with a 200k-character
    input and no chunking means the gate gets less reliable exactly as a PR
    gets bigger — the case where review matters most.

Acceptance criteria

  • A reasoning-model review reserves output capacity for the review text
    rather than letting reasoning consume the whole cap; the reserve is sized
    off the model's reasoning behaviour, not a flat constant
  • A budget exhaustion is reported as an infrastructure outcome distinct
    from "this PR failed review", and does not silently block a merge with no
    findings
  • A too-large diff degrades (chunked passes, or reviews the highest-risk
    files and says what it skipped) instead of returning nothing
  • No silent caps: whatever the gate did not read, it names

Notes

Copilot and Codex reviewers on the same PRs are also unavailable (quota), so
this gate is currently the only automated reviewer, which raises the cost of it
failing closed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    • Status
      Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions