Skip to content

Run hosted reviews on gpt-5.6-luna through Azure - #160

Merged
morgaesis merged 1 commit into
mainfrom
swap/gpt-5.6-luna
Aug 20, 2026
Merged

morgaesis merged 1 commit into
mainfrom
swap/gpt-5.6-luna

Conversation

@morgaesis

Copy link
Copy Markdown
Contributor

Hosted reviews move to openai/gpt-5.6-luna, pinned to the Azure zero-retention endpoint with a price ceiling of $0.22 and $1.32 per million input and output tokens. The change is worth making for one number: gate-verdict correctness, how often the merge gate blocks exactly when it should, goes from 73.7% to 85.5% across repeated runs of the 70-fixture suite. That is the metric that decides whether a review costs somebody a merge, and it has been our weakest for months.

Seven scored runs support it. No run produced a false finding and every run stayed silent on all thirteen clean fixtures. Detection rate is level at 89.5%, because the suite cannot resolve a difference smaller than its own run-to-run spread; it is reported rather than relied on. Reviews cost roughly a seventh as much, and ninety-fifth-percentile latency falls from 29.1 to 17.6 seconds, which matters because the per-model slot a hosted review runs under has to be sized for that tail.

Three of the seven runs enforce the contract the deployment actually uses: one pinned upstream provider, zero data retention, and the manifest price ceiling. That distinction turned out to be load-bearing. The other finalist looked steadier on unconstrained screening and lost its entire latency advantage once pinned, because the endpoint the router had been choosing for it is not one a zero-retention deployment may use. The recorded baseline therefore comes from a contract-enforced run, and it is the median on detection and the lowest of the three on gate correctness, so the release floor sits inside the observed spread instead of on top of it.

conservative_context_tokens gains the new generator; without that entry its review budget would silently drop to the unknown-model floor.

The hosted generator moves to `openai/gpt-5.6-luna`, pinned to the Azure
zero-retention endpoint with a price ceiling of $0.22 and $1.32 per million
input and output tokens.

Seven scored runs of the 70-fixture suite separate it from the previous
generator on the metrics that decide whether a review costs someone a merge.
Gate-verdict correctness is 85.5% against 73.7%, no run produced a false
finding, and every run stayed silent on all thirteen clean fixtures. Detection
rate is level at 89.5%; the suite cannot resolve a difference smaller than its
own run-to-run spread, so that column is reported rather than relied on.
Reviews cost roughly a seventh as much and the ninety-fifth-percentile latency
falls from 29.1 to 17.6 seconds, which matters because the per-model slot a
hosted review runs under has to be sized for that tail.

Three of the seven runs enforce the deployed provider contract: one pinned
upstream provider, zero data retention, and the manifest price ceiling. The
recorded baseline comes from one of those rather than from unconstrained
screening, and it is the median run on detection and the lowest of the three
on gate correctness, so the release floor sits inside the observed spread
instead of on top of it.

`conservative_context_tokens` gains the new generator, without which its
review budget would silently fall to the unknown-model floor.
@morgaesis
morgaesis merged commit 96ccd76 into main Aug 20, 2026
15 of 16 checks passed
@morgaesis
morgaesis deleted the swap/gpt-5.6-luna branch August 20, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant