Repository navigation
Benchmark the model the release actually ships - #161
Merged
Merged
Conversation
The release gate set `REVIEW_MODEL: z-ai/glm-5.2` in workflow YAML, and a test asserted that literal was present. Changing the shipped generator therefore left the gate scoring the previous model against the previous model's baseline: v0.8.27 moved the hosted default to `openai/gpt-5.6-luna` and its gate still measured `z-ai/glm-5.2` at 93.0% against a 92.7% floor, reporting a pass for a model the release does not contain. The diff-file benchmark now falls back to the `default_model` in the same `config.toml` the build bakes into the binary, so the id cannot drift from what ships, and the workflow no longer restates it. The assertion is inverted to the invariant that matters: the release workflow must not name a model. An empty `default_model` still resolves to nothing, so a caller with no model from any source keeps failing with the explicit error rather than silently benchmarking a default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release gate named the review model in workflow YAML as
REVIEW_MODEL: z-ai/glm-5.2, and a test asserted that literal string was present. Changing the shipped generator therefore left the gate measuring the previous model against the previous model's baseline.v0.8.27moved the hosted default toopenai/gpt-5.6-luna, and its gate scoredz-ai/glm-5.2at 93.0% against a 92.7% floor, reporting a pass for a model that release does not contain. The failure is silent by construction: it certifies the old model indefinitely and never errors.The diff-file benchmark now falls back to the
default_modelin the sameconfig.tomlthe build bakes into the binary, and the workflow no longer restates it, so the benchmarked id cannot drift from the shipped one. The test assertion is inverted to the invariant that matters: the release workflow must not name a model at all.An empty
default_modelstill resolves to nothing, so a caller with no model from any source keeps failing with the explicit error rather than quietly benchmarking a default it inferred.