Repository navigation
Run hosted reviews on gpt-5.6-luna through Azure - #160
Merged
Merged
Conversation
The hosted generator moves to `openai/gpt-5.6-luna`, pinned to the Azure zero-retention endpoint with a price ceiling of $0.22 and $1.32 per million input and output tokens. Seven scored runs of the 70-fixture suite separate it from the previous generator on the metrics that decide whether a review costs someone a merge. Gate-verdict correctness is 85.5% against 73.7%, no run produced a false finding, and every run stayed silent on all thirteen clean fixtures. Detection rate is level at 89.5%; the suite cannot resolve a difference smaller than its own run-to-run spread, so that column is reported rather than relied on. Reviews cost roughly a seventh as much and the ninety-fifth-percentile latency falls from 29.1 to 17.6 seconds, which matters because the per-model slot a hosted review runs under has to be sized for that tail. Three of the seven runs enforce the deployed provider contract: one pinned upstream provider, zero data retention, and the manifest price ceiling. The recorded baseline comes from one of those rather than from unconstrained screening, and it is the median run on detection and the lowest of the three on gate correctness, so the release floor sits inside the observed spread instead of on top of it. `conservative_context_tokens` gains the new generator, without which its review budget would silently fall to the unknown-model floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hosted reviews move to
openai/gpt-5.6-luna, pinned to the Azure zero-retention endpoint with a price ceiling of $0.22 and $1.32 per million input and output tokens. The change is worth making for one number: gate-verdict correctness, how often the merge gate blocks exactly when it should, goes from 73.7% to 85.5% across repeated runs of the 70-fixture suite. That is the metric that decides whether a review costs somebody a merge, and it has been our weakest for months.Seven scored runs support it. No run produced a false finding and every run stayed silent on all thirteen clean fixtures. Detection rate is level at 89.5%, because the suite cannot resolve a difference smaller than its own run-to-run spread; it is reported rather than relied on. Reviews cost roughly a seventh as much, and ninety-fifth-percentile latency falls from 29.1 to 17.6 seconds, which matters because the per-model slot a hosted review runs under has to be sized for that tail.
Three of the seven runs enforce the contract the deployment actually uses: one pinned upstream provider, zero data retention, and the manifest price ceiling. That distinction turned out to be load-bearing. The other finalist looked steadier on unconstrained screening and lost its entire latency advantage once pinned, because the endpoint the router had been choosing for it is not one a zero-retention deployment may use. The recorded baseline therefore comes from a contract-enforced run, and it is the median on detection and the lowest of the three on gate correctness, so the release floor sits inside the observed spread instead of on top of it.
conservative_context_tokensgains the new generator; without that entry its review budget would silently drop to the unknown-model floor.