driver: decide whether a run regressed - #594
Merged
Merged
Conversation
This was referenced Sep 25, 2026
MarcusKainth
force-pushed
the
driver/diff-record
branch
from
September 25, 2026 13:32
3ef55ff to
499390c
Compare
`clickdoom native regress NEW --history PATH` reads the lines `native diff --record` wrote and judges each line of NEW, in order, against what came before it. It exits 3 when anything regressed, and --findings writes each regression as a JSON line for whatever files the issue. Correctness is judged against the latest line with no error. It regressed when the first refused tic moves earlier or appears, when it names other bits at the same tic, or when a divergence appears or moves earlier. A refusal that moves later names new bits by construction, so bits are only compared at an unchanged tic. A line that compared nothing (compared_through null) says nothing about divergence either way. Cost is judged only against the line directly before it in NEW, and only when both carry the same run_id, CPU model and server version. Hosted runners differ by CPU model from job to job, and analysis times taken on two machines are not comparable, so history is never the cost baseline. A line for a commit already in the history is not judged again and becomes the next line's parent: a nightly that re-measures the last recorded commit first gets a same-VM baseline for the one after it. The limits default to 1.25x for stage1's analysis and 1.3x for the median tic. An error line is reported, judged for nothing, and breaks the cost chain. An empty NEW or a history path that does not exist fails with exit 1, since either would otherwise read as a clean night.
MarcusKainth
force-pushed
the
driver/regress
branch
from
September 25, 2026 13:43
25158fc to
e7d7aed
Compare
MarcusKainth
marked this pull request as ready for review
September 25, 2026 13:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
Stacked on #593, which adds the
native diff --recordline this reads.clickdoom native regress NEW --history PATHjudges each line of NEW, in order,against the lines before it, and exits 3 when anything regressed.
--findingswrites each regression as one JSON line, which the nightly renders into an issue.
The command talks to no server.
Correctness is judged against the latest line with no error. It regressed when
the first refused tic moves earlier or appears, when it names other bits at the
same tic, or when a divergence appears or moves earlier. A refusal that moves
later is progress, and names new bits by construction, so bits are compared only
at an unchanged tic. A line that compared nothing (
compared_throughnull) saysnothing about divergence.
Cost is judged only against the line directly before it in NEW, and only when
both carry the same
run_id, CPU model and server version. History is never thecost baseline, because hosted runners change CPU model between jobs. A line for a
commit already in the history is not judged again and becomes the next line's
parent, so a nightly that re-measures the last recorded commit first has a
same-VM baseline for the commit after it. Limits default to 1.25x for
stage1_analysis_sand 1.3x fortic_ms_p50.An error line is reported, judged for nothing, and breaks the cost chain. An
empty NEW or a
--historypath that does not exist fails with exit 1.The diff is 668 lines, of which the tests and fixtures are about half.
Evidence
The fixtures, one of which must report a regression:
A real line from #593's local run (
native diff 2000andnative diff 274against a regenerated trace, the refusal fields of the first merged into the
second with
jq) as history, and a night holding the same commit re-measuredplus a copy of it with the refusal moved to 270 and stage1's analysis at 1.5x:
Unit tests, then each rule broken once and restored with
git checkout HEAD -- driver/src/native/regress.rs:Invariants
None. The command reads JSON files and compares numbers the diff recorded. It
computes nothing the simulation or the renderer produces.
Spec impact
Checks
make gates. Not run under that name. The driver's unit tests, clippy,fmt and
check_purity.shwere run by exit code, above. The command has nolive side
make native-smoke, unaffectedWritten mostly by Claude Opus 5.5.