driver: record what a differential run found - #593
Merged
Merged
Conversation
This was referenced Sep 25, 2026
`clickdoom native diff --record PATH` appends one JSON line per run: the commit, the server version, the CPU model, the first refused tic and the bits it named, the first divergent tic and field, the last tic compared and how many both sides held, each simulation statement's QueryAnalysisMicroseconds, the median and 95th percentile tic time, the tic count and GITHUB_RUN_ID when it is set. A nightly job appends these lines per main commit and compares each one with the one before it, so the line has to say which of refusal, divergence and agreement the run found, with nulls where the run could not tell: a run that stopped at a refusal compared nothing, and its compared_through is null. The analysis times come from system.query_log, keyed by the query ids the session already gives both statements. The server writes a statement's QueryFinish row after it has answered the close, so a single flush can miss it; the read flushes and polls for up to 30 s. The first tic is left out of the tic percentiles because it pays for the analysis, which is recorded on its own. The comparison is split from the printing so the record and the exit code read the same result. What the command prints and its exit codes are unchanged.
MarcusKainth
force-pushed
the
driver/diff-record
branch
from
September 25, 2026 13:32
3ef55ff to
499390c
Compare
MarcusKainth
marked this pull request as ready for review
September 25, 2026 13:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
clickdoom native diff --record PATHappends one JSON line per run saying whatthe run found and what it cost. A nightly job records one line per main commit
and compares each with the one before it, so a commit that moves the first
refusal earlier, adds a divergence or slows the tic statement's analysis gets
noticed on the commit that did it.
The line carries:
commit,clickhouse,runner_cpu,first_refused_tic,first_refused_bits,first_divergent_tic,first_divergent_field,compared_through,compared_tics,stage1_analysis_s,stage2_analysis_s,tic_ms_p50,tic_ms_p95,tics,run_id(GITHUB_RUN_ID) anderror.A field the run could not tell is
null. A run that stops at a refusal comparesnothing, and its
compared_throughisnull, which tells it apart from a runthat compared and agreed.
The residents already run under their own query ids, so the analysis times are
read from
system.query_logby those ids. The server writes a statement'sQueryFinishrow after it has answered the close, and a singleSYSTEM FLUSH LOGSmissed it on the first live run, so the read flushes and polls for up to30 s. The first tic is left out of the tic percentiles.
What the command prints and its exit codes are unchanged.
compareis splitfrom
reportso the record and the exit code read the same result.Evidence
Throwaway container
clickhouse/clickhouse-server:26.8.2.7on port 18139, AppleM5 Max, no machine lock (a correctness run).
The live suite, with the new assertions:
The same suite with the
record::appendcall replaced bylet _ = (path, found);,restored afterwards with
git checkout HEAD -- driver/src/cli/native/diff.rs:The command against a full probe trace regenerated from this tree
(
refemu probe ... --stop-at halt -n 4000000000, 2172 rows), the two runs thenightly makes: one to find the refusal, one over the span before it.
Unit tests, lint and purity, statuses captured before any pipe:
Invariants
PUR-10 and PUR-12. The percentiles and the wait for
system.query_logarereporting on timings the driver already measured with
Monotonic. Nothing theyproduce reaches a statement or a value the simulation computes, and the
analysis times are read back from the server's own log after the session has
closed.
check_purity.shpasses, above.Spec impact
NATIVE.mddoes not describenative diff's flags, and the parity behaviour itdoes describe is unchanged.
Checks
make gates. Not run under that name.native_diff_live, the driver'sunit tests, clippy, fmt and
check_purity.shwere run by exit code, abovemake native-smoke, unaffected: it renders a frame and does not diffWritten mostly by Claude Opus 5.5.