ci: record each main commit's parity and cost nightly - #595
Merged
Merged
Conversation
2 of 4 tasks
MarcusKainth
force-pushed
the
driver/regress
branch
from
September 25, 2026 13:43
25158fc to
e7d7aed
Compare
The nightly gains two jobs. native-regression (contents: read) walks the main commits not yet recorded, oldest first and first parents only, at most eight a night (max_commits), with scripts/native-regression-walk.sh. Each commit is built in its own worktree, loaded into a fresh database and run with `native diff --record` against a probe trace generated from that commit's ROM and probe; traces are cached by the git hashes of rom/PINNED_HASH and refemu/probe/. `clickdoom native regress` then judges the night against the recorded lines. native diff compares nothing when its span holds a refusal, so the walk diffs twice: SEARCH_TICS tics (2000) to find the first refused tic T, then T-1 tics, and keeps the second line with the first one's refusal. It refuses a second line that reaches T or compared no tic. A first line that did compare is kept as it is. The walk measures the last recorded commit again before the new ones, so the first new commit has a parent measured in the same job on the same VM; regress judges cost only against such a parent. A commit that fails to build, load or run becomes an error line and the walk goes on. The walk stops starting commits after BUDGET_MINUTES (240), so a slow night still uploads what it did. native-record (contents: write) downloads the artifact and appends the lines whose commit is not yet recorded to native/bench/regression/results.jsonl on the regression-data branch, creating it as an orphan branch through the git data API the first time. It checks nothing out and runs no repository code. The branch is never merged into main. nightly.yml's header, CONTRIBUTING.md and DEVELOPING.md describe the new jobs in the same change.
MarcusKainth
force-pushed
the
ci/nightly-regression
branch
from
September 25, 2026 13:51
3dcfe7e to
a129fdc
Compare
MarcusKainth
marked this pull request as ready for review
September 25, 2026 13:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
Stacked on #594 (
native regress), which is stacked on #593 (native diff --record).A commit that moves demo3's first refusal earlier, adds a divergence before it,
or slows the tic statement's analysis is caught today only when somebody happens
to run the right span. The nightly now records one line per main commit and
judges each against the one before it.
native-regression(contents: read) runsscripts/native-regression-walk.sh.It walks the main commits after the last recorded one, oldest first and first
parents only, at most
max_commitsa night (default 8). Each commit is builtin its own worktree, loaded fresh, and run with
native diff --recordagainsta probe trace generated from that commit's ROM and probe. Traces are cached
with
actions/cacheby the git hashes ofrom/PINNED_HASHandrefemu/probe/.clickdoom native regressjudges the night and the lines areuploaded as an artifact.
native diffcompares nothing when its span holds a refusal, so thewalk runs
SEARCH_TICS(2000) tics to find the refused tic T, then T-1 tics,and keeps the second line with the first one's refusal. It rejects a second
line that reaches T or compared zero tics. When the first line did compare, it
is kept as it is, so the walk needs no change when
native diffstartscomparing the tics before a refusal.
has a parent measured in the same job on the same VM.
regressjudges costonly against that parent.
goes on. No commit starts after
BUDGET_MINUTES(240), so a slow night stilluploads what it did.
native-record(contents: write) checks nothing out and runs no repositorycode. It downloads the artifact, drops lines whose commit is already recorded,
and appends the rest to
native/bench/regression/results.jsonlonregression-data, creating that as an orphan branch through the git data APIthe first time. The branch is never merged into main.
Issue filing and the
workflow_dispatchself-test are the next PR in the stack.Here a regression is a
::warning::on the judge step.Evidence
The walk run locally against a throwaway container (26.8.2.7, port 18139), with
a hand-seeded history whose last line is
mainat3159d52, andTIP=HEADonthis stack before the split into two PRs. So it measured
3159d52again as theparent (it predates
--record, so it becomes an error line), then #593's commit,#594's commit and the stack's head:
The first commit walked also generated its probe trace through
make gen-probe-traceinto the trace directory (2172 rows in 14.4 s), and thelater commits reused it by key.
The judge step's command on that night:
native-record's dedupe filter on the same files drops the error line for3159d52, which the history already holds, and keeps the three new commits:Lint, statuses captured before any pipe:
Not verified
the first real run is the first nightly after this merges.
native-record'sgh apicalls (branch probe, raw read,PUT contents, andthe blob, tree, commit and ref calls that create the orphan branch) were not
run. Running them would write a branch to this repository.
tics. At the CI tic cost in this repository's notes (about 0.5 s) that is
roughly 20 to 25 minutes a commit, so eight commits fit inside the 240-minute
budget only just.
max_commitsandBUDGET_MINUTESare the knobs.Invariants
None. The walk and the jobs orchestrate the driver. They compute nothing the
simulation or the renderer produces.
Spec impact
Checks
make gates. Not run.actionlint,zizmorandshellcheckwere run byexit code, above. The workflow has no local runner
make native-smoke, unaffectedWritten mostly by Claude Opus 5.5.