Repository navigation
Give a single-wave hosted review a slot it can finish in - #157
Merged
Merged
Conversation
The per-model slot exists so a large review's batch waves all fit inside the generator phase, but it was pinned to the large-diff constant for every hosted review rather than to the timeout the review itself selected. A single-wave review therefore ran under a many-wave bound: each model call was cut at sixty seconds, and because the slot then equalled the phase remainder the timeout retry was unreachable, so one slow answer ended the review with a provider-unavailable finding. The review model's own latency sits well above that cut, so a tenth of hosted reviews were failing on answers that were still arriving.
The generator phase admits a model operation only when the slot still fits the deadline that remains once the diff has been fetched, less the scheduling reserve. A slot set to the full hosted request timeout equals the phase itself and is refused before any provider request, so the single-wave slot is its own bound, above the review model's tail latency and clear of the reserve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
About a tenth of hosted reviews were ending with a
Model provider unavailablefinding on answers the provider was still delivering. The per-model slot exists so a large review's batch waves all fit inside the generator phase, but it was pinned toLARGE_DIFF_LLM_REQUEST_TIMEOUT_SECSfor every hosted review rather than to the timeout the review itself selected. A review that fits in one wave therefore ran under a many-wave bound, and because the slot then equalled the phase remainder the timeout retry was unreachable: one attempt, no retry, terminal.Measured over the last 800 production model calls,
z-ai/glm-5.2answers at a median of 16s but a p95 of 85s and a p99 of 142s, so the cut landed just past p90.SINGLE_WAVE_LLM_REQUEST_TIMEOUT_SECSsits above that tail and clear of the scheduling reserve thatensure_hosted_review_scheduleenforces against the deadline remaining once the diff has been fetched; a slot equal to the phase is refused there before any provider request, which the evaluator suite now covers.The request is not streamed, so a model that is thinking and a socket that is dead are indistinguishable to the caller. Raising the bound buys headroom rather than removing that blind spot.