Skip to content

ci: record each main commit's parity and cost nightly - #595

Merged
MarcusKainth merged 1 commit into
mainfrom
ci/nightly-regression
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
ci/nightly-regression

Conversation

@MarcusKainth

Copy link
Copy Markdown
Owner

What this changes, and why

Stacked on #594 (native regress), which is stacked on #593 (native diff --record).

A commit that moves demo3's first refusal earlier, adds a divergence before it,
or slows the tic statement's analysis is caught today only when somebody happens
to run the right span. The nightly now records one line per main commit and
judges each against the one before it.

  • native-regression (contents: read) runs scripts/native-regression-walk.sh.
    It walks the main commits after the last recorded one, oldest first and first
    parents only, at most max_commits a night (default 8). Each commit is built
    in its own worktree, loaded fresh, and run with native diff --record against
    a probe trace generated from that commit's ROM and probe. Traces are cached
    with actions/cache by the git hashes of rom/PINNED_HASH and
    refemu/probe/. clickdoom native regress judges the night and the lines are
    uploaded as an artifact.
  • The span: native diff compares nothing when its span holds a refusal, so the
    walk runs SEARCH_TICS (2000) tics to find the refused tic T, then T-1 tics,
    and keeps the second line with the first one's refusal. It rejects a second
    line that reaches T or compared zero tics. When the first line did compare, it
    is kept as it is, so the walk needs no change when native diff starts
    comparing the tics before a refusal.
  • Cost: the walk measures the last recorded commit again first, so the next one
    has a parent measured in the same job on the same VM. regress judges cost
    only against that parent.
  • A commit that fails to build, load or run becomes an error line, and the walk
    goes on. No commit starts after BUDGET_MINUTES (240), so a slow night still
    uploads what it did.
  • native-record (contents: write) checks nothing out and runs no repository
    code. It downloads the artifact, drops lines whose commit is already recorded,
    and appends the rest to native/bench/regression/results.jsonl on
    regression-data, creating that as an orphan branch through the git data API
    the first time. The branch is never merged into main.

Issue filing and the workflow_dispatch self-test are the next PR in the stack.
Here a regression is a ::warning:: on the judge step.

Evidence

The walk run locally against a throwaway container (26.8.2.7, port 18139), with
a hand-seeded history whose last line is main at 3159d52, and TIP=HEAD on
this stack before the split into two PRs. So it measured 3159d52 again as the
parent (it predates --record, so it becomes an error line), then #593's commit,
#594's commit and the stack's head:

$ TIP=HEAD CH_HTTP_PORT=18139 CLICKHOUSE_PASSWORD=clickdoom GITHUB_RUN_ID=local-walk \
    scripts/native-regression-walk.sh walk-history.jsonl walk
::group::3159d5226c14
...
error: unexpected argument '--record' found
::error::3159d5226c14: native diff 2000 exited 2
::endgroup::
::group::3ef55ff88993
3ef55ff88993: {"commit":"3ef55ff88993d70ef7206fadfe7a490997fa30ad","clickhouse":"26.8.2.7","runner_cpu":"Apple M5 Max","first_refused_tic":275,"first_refused_bits":"unresolved: CHASE_STUCK","first_divergent_tic":null,"first_divergent_field":null,"compared_through":274,"compared_tics":273,"stage1_analysis_s":1.176,"stage2_analysis_s":0.352214,"tic_ms_p50":46.530042,"tic_ms_p95":86.933459,"tics":274,"run_id":"local-walk","error":null}
::group::25158fcc2fb5
25158fcc2fb5: {"commit":"25158fcc2fb57702504f81bbec99fc50dd785aac",...,"first_refused_tic":275,...,"compared_through":274,"compared_tics":273,"stage1_analysis_s":1.205761,...}
::group::971936a30b97
971936a30b97: {"commit":"971936a30b97c5662dafe7562516d9b67b9e36ab",...,"first_refused_tic":275,...,"compared_through":274,"compared_tics":273,"stage1_analysis_s":1.238711,...}
4 line(s) in .../walk/night.jsonl
walk exit=0

The first commit walked also generated its probe trace through
make gen-probe-trace into the trace directory (2172 rows in 14.4 s), and the
later commits reused it by key.

The judge step's command on that night:

$ clickdoom native regress walk/night.jsonl --history walk-history.jsonl
3159d5226c14 not judged: native diff 2000 exited 2
3ef55ff88993 against nothing before it, cost not judged: no parent measured by this run on this machine: no regression
25158fcc2fb5 against 3ef55ff88993, cost against 3ef55ff88993: no regression
971936a30b97 against 25158fcc2fb5, cost against 25158fcc2fb5: no regression
regress exit=0

native-record's dedupe filter on the same files drops the error line for
3159d52, which the history already holds, and keeps the three new commits:

$ jq -c -n --slurpfile old walk-history.jsonl --slurpfile new walk/night.jsonl '...' | cut -c1-60
{"commit":"3ef55ff88993d70ef7206fadfe7a490997fa30ad","clickh
{"commit":"25158fcc2fb57702504f81bbec99fc50dd785aac","clickh
{"commit":"971936a30b97c5662dafe7562516d9b67b9e36ab","clickh

Lint, statuses captured before any pipe:

$ actionlint .github/workflows/*.yml; echo "actionlint exit=$?"
actionlint exit=0
$ zizmor --persona=regular .github/ ; echo "zizmor exit=$?"
No findings to report. Good job! (14 suppressed)
zizmor exit=0
$ shellcheck scripts/native-regression-walk.sh; echo "shellcheck exit=$?"
shellcheck exit=0

Not verified

  • No run of the workflow itself. The workflow runs from the default branch, so
    the first real run is the first nightly after this merges.
  • native-record's gh api calls (branch probe, raw read, PUT contents, and
    the blob, tree, commit and ref calls that create the orphan branch) were not
    run. Running them would write a branch to this repository.
  • The runner's per-commit cost: each commit is a build, a load and 2000 + T-1
    tics. At the CI tic cost in this repository's notes (about 0.5 s) that is
    roughly 20 to 25 minutes a commit, so eight commits fit inside the 240-minute
    budget only just. max_commits and BUDGET_MINUTES are the knobs.

Invariants

None. The walk and the jobs orchestrate the driver. They compute nothing the
simulation or the renderer produces.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates. Not run. actionlint, zizmor and shellcheck were run by
    exit code, above. The workflow has no local runner
  • make native-smoke, unaffected
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents labels Sep 25, 2026
Base automatically changed from driver/regress to main September 25, 2026 13:51
The nightly gains two jobs. native-regression (contents: read) walks the
main commits not yet recorded, oldest first and first parents only, at
most eight a night (max_commits), with scripts/native-regression-walk.sh.
Each commit is built in its own worktree, loaded into a fresh database
and run with `native diff --record` against a probe trace generated from
that commit's ROM and probe; traces are cached by the git hashes of
rom/PINNED_HASH and refemu/probe/. `clickdoom native regress` then
judges the night against the recorded lines.

native diff compares nothing when its span holds a refusal, so the walk
diffs twice: SEARCH_TICS tics (2000) to find the first refused tic T,
then T-1 tics, and keeps the second line with the first one's refusal.
It refuses a second line that reaches T or compared no tic. A first line
that did compare is kept as it is.

The walk measures the last recorded commit again before the new ones,
so the first new commit has a parent measured in the same job on the
same VM; regress judges cost only against such a parent. A commit that
fails to build, load or run becomes an error line and the walk goes on.
The walk stops starting commits after BUDGET_MINUTES (240), so a slow
night still uploads what it did.

native-record (contents: write) downloads the artifact and appends the
lines whose commit is not yet recorded to
native/bench/regression/results.jsonl on the regression-data branch,
creating it as an orphan branch through the git data API the first
time. It checks nothing out and runs no repository code. The branch is
never merged into main.

nightly.yml's header, CONTRIBUTING.md and DEVELOPING.md describe the new
jobs in the same change.
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 13:59
@MarcusKainth
MarcusKainth merged commit 0830059 into main Sep 25, 2026
20 checks passed
@MarcusKainth
MarcusKainth deleted the ci/nightly-regression branch September 25, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant