Skip to content

driver: decide whether a run regressed - #594

Merged
MarcusKainth merged 1 commit into
mainfrom
driver/regress
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
driver/regress

Conversation

@MarcusKainth

Copy link
Copy Markdown
Owner

What this changes, and why

Stacked on #593, which adds the native diff --record line this reads.

clickdoom native regress NEW --history PATH judges each line of NEW, in order,
against the lines before it, and exits 3 when anything regressed. --findings
writes each regression as one JSON line, which the nightly renders into an issue.
The command talks to no server.

Correctness is judged against the latest line with no error. It regressed when
the first refused tic moves earlier or appears, when it names other bits at the
same tic, or when a divergence appears or moves earlier. A refusal that moves
later is progress, and names new bits by construction, so bits are compared only
at an unchanged tic. A line that compared nothing (compared_through null) says
nothing about divergence.

Cost is judged only against the line directly before it in NEW, and only when
both carry the same run_id, CPU model and server version. History is never the
cost baseline, because hosted runners change CPU model between jobs. A line for a
commit already in the history is not judged again and becomes the next line's
parent, so a nightly that re-measures the last recorded commit first has a
same-VM baseline for the commit after it. Limits default to 1.25x for
stage1_analysis_s and 1.3x for tic_ms_p50.

An error line is reported, judged for nothing, and breaks the cost chain. An
empty NEW or a --history path that does not exist fails with exit 1.

The diff is 668 lines, of which the tests and fixtures are about half.

Evidence

The fixtures, one of which must report a regression:

$ clickdoom native regress driver/tests/fixtures/regress/night-steady.jsonl \
    --history driver/tests/fixtures/regress/history.jsonl
111111111111 already recorded, not judged again
222222222222 against 111111111111, cost against 111111111111: no regression
333333333333 not judged: native diff 2000 exited 1
exit=0

A real line from #593's local run (native diff 2000 and native diff 274
against a regenerated trace, the refusal fields of the first merged into the
second with jq) as history, and a night holding the same commit re-measured
plus a copy of it with the refusal moved to 270 and stage1's analysis at 1.5x:

$ clickdoom native regress local-night.jsonl --history local-history.jsonl --findings findings.jsonl
3ef55ff88993 already recorded, not judged again
feedfacefeed against 3ef55ff88993, cost against 3ef55ff88993: 2 regression(s)
  correctness: first_refused_tic 270 against 275 at 3ef55ff88993
  cost: stage1_analysis_s 1.905 (1.50x) against 1.270 at 3ef55ff88993
clickdoom: error: 2 regression(s) over 2 line(s)
exit=3
$ cat findings.jsonl
{"kind":"correctness","commit":"feedfacefeedfacefeedfacefeedfacefeedface","against":"3ef55ff88993d70ef7206fadfe7a490997fa30ad","metric":"first_refused_tic","before":"275","after":"270"}
{"kind":"cost","commit":"feedfacefeedfacefeedfacefeedfacefeedface","against":"3ef55ff88993d70ef7206fadfe7a490997fa30ad","metric":"stage1_analysis_s","before":"1.270","after":"1.905 (1.50x)"}

Unit tests, then each rule broken once and restored with
git checkout HEAD -- driver/src/native/regress.rs:

$ cargo test -p clickdoom-driver --lib
test result: ok. 127 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# the refusal rule inverted: `l < b` -> `l > b`
test cli::native::regress::tests::a_refusal_that_moved_earlier_exits_3_and_is_written ... FAILED
test native::regress::tests::a_refusal_that_moves_earlier_is_a_regression ... FAILED
test native::regress::tests::a_refusal_that_moves_later_is_progress_whatever_bits_it_names ... FAILED
test result: FAILED. 8 passed; 3 failed; 0 ignored; 0 measured; 116 filtered out

# cost judged against history: `parent.filter(|parent| same_machine(parent, line))` -> `parent.or(baseline)`
test native::regress::tests::a_refusal_that_moves_earlier_is_a_regression ... FAILED
test native::regress::tests::an_error_line_is_reported_and_breaks_the_cost_chain ... FAILED
test native::regress::tests::cost_is_judged_against_the_parent_measured_in_the_same_run ... FAILED
test result: FAILED. 8 passed; 3 failed; 0 ignored; 0 measured; 116 filtered out
$ cargo clippy -p clickdoom-driver --all-targets --all-features -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.65s
$ ./scripts/check_purity.sh >/dev/null 2>&1; echo "purity exit=$?"
purity exit=0
$ cargo fmt --all -- --check >/dev/null 2>&1; echo "fmt exit=$?"
fmt exit=0

Invariants

None. The command reads JSON files and compares numbers the diff recorded. It
computes nothing the simulation or the renderer produces.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates. Not run under that name. The driver's unit tests, clippy,
    fmt and check_purity.sh were run by exit code, above. The command has no
    live side
  • make native-smoke, unaffected
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added the area: driver The client loop that ticks the batch statement and blits frames label Sep 25, 2026
Base automatically changed from driver/diff-record to main September 25, 2026 13:43
`clickdoom native regress NEW --history PATH` reads the lines
`native diff --record` wrote and judges each line of NEW, in order,
against what came before it. It exits 3 when anything regressed, and
--findings writes each regression as a JSON line for whatever files the
issue.

Correctness is judged against the latest line with no error. It regressed
when the first refused tic moves earlier or appears, when it names other
bits at the same tic, or when a divergence appears or moves earlier. A
refusal that moves later names new bits by construction, so bits are only
compared at an unchanged tic. A line that compared nothing
(compared_through null) says nothing about divergence either way.

Cost is judged only against the line directly before it in NEW, and only
when both carry the same run_id, CPU model and server version. Hosted
runners differ by CPU model from job to job, and analysis times taken on
two machines are not comparable, so history is never the cost baseline.
A line for a commit already in the history is not judged again and
becomes the next line's parent: a nightly that re-measures the last
recorded commit first gets a same-VM baseline for the one after it. The
limits default to 1.25x for stage1's analysis and 1.3x for the median
tic.

An error line is reported, judged for nothing, and breaks the cost chain.
An empty NEW or a history path that does not exist fails with exit 1,
since either would otherwise read as a clean night.
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 13:51
@MarcusKainth
MarcusKainth merged commit f903251 into main Sep 25, 2026
20 checks passed
@MarcusKainth
MarcusKainth deleted the driver/regress branch September 25, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: driver The client loop that ticks the batch statement and blits frames

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant