preflight: generate a per-release 'what tested green' matrix - #237
Merged
Conversation
Renders surface x capability from recorded preflight run artifacts: build identity, image size ladder, thinking on/off, output quality, the fp16 overflow canary, runner isolation. Generated, never hand-written. A hand-written green badge is worse than none -- it claims a surface was validated when nobody ran it, and nothing goes back to correct it later. Two properties that make it honest rather than decorative: - The six supported surfaces are listed explicitly, so one with no run appears as a full 'not run' row instead of vanishing. A missing row reads as 'not applicable', which is not the same claim and is the more dangerous of the two. mlx-cuda, rocm and cpu currently have no runs at all and now say so. - --version drops runs from other builds, so a release matrix cannot borrow a green cell from an older one. Filtered to 0.33.2-dynres every cell reads 'not run', which is exactly true: the image is still building. A group reports at its weakest check, so one skipped probe never reads as a pass. Attaching the matrix is added to the sync acceptance criteria, to be regenerated as each surface is preflighted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fixes to the generated matrix. The 'Image size ladder' column bundled token_ladder with the pinned-budget probe and reported the group at its weakest check, so CUDA read 'skipped' even though the ladder passed on all three arches -- the skips belong to the narrower pinned probe, which only nemotron has an expectation for. They answer different questions and now get their own columns. It also maps both the new pinned_image_token_budget name and the old pinned_budget, because runs recorded before the rename carry the old one and a matrix that silently stopped seeing the check would render it as green rather than missing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The harness deliberately separates two skips that mean different things: an arch structurally unable to run a probe, and one nobody has measured. Both arrive as SKIP and the distinction survives only in the summary text, so the matrix was collapsing them and reporting a column as "skipped" when two of three arches had passed and the third simply cannot run it. N/A is now recognised from the summary and treated as NEUTRAL -- dropped from the worst-wins comparison, reported only when every cell is N/A. Ranking it weak reports "n/a" for a column that passed; ranking it strong hides a real gap behind it; neither is honest. Checked in both directions on real runs: the post-fix CUDA run reads green, and the run from before the gaps were closed still reads "skipped". The change distinguishes not-applicable from unmeasured rather than making things look green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renders surface × capability from recorded preflight runs — build identity, image size ladder, thinking on/off, output quality, the fp16 overflow canary, runner isolation.
Generated, never hand-written. A hand-written green badge is worse than none: it claims a surface was validated when nobody ran it, and nothing goes back to correct it.
Two design choices that keep it honest:
mlx-cuda,rocmandcpuhave no runs at all and now say so out loud.--versiondrops runs from other builds, so a release matrix cannot borrow a green cell from an older one. Filtered to0.33.2-dynrestoday, every cell readsnot run— which is exactly true, the image is still building.A group reports at its weakest check, so one skipped probe never renders as a pass.
Attaching the matrix is added to the sync acceptance criteria, regenerated as each surface is preflighted.