Repository navigation
Report the real cause when a review fails before producing a verdict - #154
Merged
Merged
Conversation
A review that stopped for any reason other than a recorded scorer error was published as "Model output could not be validated" with the detail "scorer output did not satisfy the admitted contract". That branch is reached by admission limits, configuration problems, and unexpected internal errors, and it discarded the error it was handed. The result named a component that may never have run and hid the only text that identified the fault, so every reader began by investigating the model. Failures carrying a recorded scorer error keep the existing wording. Everything else now reports itself through an operational finding that states the review could not be completed and carries the underlying error, on the same path and with the same fail-closed severity, so gate behaviour and operational metrics are unchanged.
RUSTSEC-2026-0258 reports that h2 0.4.14 accepts unbounded empty DATA frames, which lets a peer hold a connection open and consume processing without advancing a stream. The advisory landed after the current lockfile, so audit scanning fails on every branch until the dependency moves to 0.4.16.
Hosted admission prices a worst case rather than an estimate: the full bounded selection of batches, every logical call the watchdog plan allows, and a hostile maximum-size retry payload on each one, at the admitted price bounds. The shipped profile projects about 15.6 dollars that way, while a review actually spends on the order of a cent. Both quantities were compared against one constant, so the projection could not clear it and every hosted review was refused before provider contact regardless of how small the change was. The spend cap keeps its value and its meaning: accumulated provider cost within one operation still stops at the per-review reservation the service holds, so a review cannot consume more of an organization's allowance than was set aside for it. Admission now has its own ceiling above the planning envelope. The existing plan test priced every token at one micro per million, so it could not observe either limit. A new test prices the shipped profile at its real bounds, which fails when the projection outgrows the ceiling instead of letting every production review fail.
The runtime-shaped preflight case proved that an over-budget plan is refused before it runs, using bounds that sat above the old shared cap. With admission measured against its own ceiling those bounds are admissible, so the case stopped exercising a refusal. Pricing it an order of magnitude above any admissible profile restores what it was written to check.
morgaesis
added a commit
that referenced
this pull request
Aug 18, 2026
Release commit for `v0.8.22`, carrying the admission and failure-reporting fixes from #154.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A review that stopped for any reason other than a recorded scorer error was published as
Model output could not be validated, with the detailscorer output did not satisfy the admitted contract. That branch inerror_envelopeis reached by admission limits, configuration problems and unexpected internal errors, and it discarded theerrit was handed. The published finding therefore named a component that may never have run, and dropped the only text identifying the fault.This is not hypothetical. Every hosted review currently fails admission before provider contact, with zero prompt and completion tokens recorded, and reports that scorer message on the pull request. The actual cause is
hosted review admission projects 11455328 micro-dollars of provider exposure across 54 attempts, exceeding the 1000000 micro-dollar operation cap, which was visible nowhere until the binary was run by hand.Failures carrying a recorded scorer error keep the existing wording, so a genuine scoring fault still reads as one. Everything else reports itself through an operational finding stating that the review could not be completed and carrying the underlying error. The path, severity and fail-closed behaviour are unchanged, so
gateconclusions and theoperational_failuremetric keep working. Two regression tests pin both directions, and each fails if the branch is restored.