Skip to content

ci: file an issue when a nightly line regressed - #596

Merged
MarcusKainth merged 1 commit into
mainfrom
ci/nightly-regression-issues
Sep 25, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
ci/nightly-regression-issues

Conversation

@MarcusKainth

Copy link
Copy Markdown
Owner

What this changes, and why

Stacked on #595, which records the lines.

A regression the nightly finds becomes an issue on the commit that caused it.

  • The read-only job renders each native regress --findings line with
    scripts/native-regression-issues.sh. gh cannot fill a .yml form, so the
    body follows the form by hand: each field's label as a ### heading, and a
    render: field fenced in its language. Correctness findings use
    1b-native-divergence.yml (divergence, area: native). Cost findings use
    3-performance.yml (performance, area: native).
  • native-record gains issues: write and files what was rendered. It still
    checks nothing out and runs no repository code. A title ends with
    (metric at commit), and an open or closed issue whose title ends the same way
    is not filed again. --label is dropped silently when the token cannot apply
    it, so the labels are read back and a mismatch fails the step.
  • inject_regression (a workflow_dispatch boolean, off by default) judges one
    more line: a copy of the last measured line, under the commit name
    self-test-<run id>, with its refusal one tic earlier. The judge step fails
    unless regress reports it. The issue says it was injected. The copy is not in
    the night's lines, so it is never recorded. Normal runs never inject. With
    max_commits at 0, a dispatch measures only the last recorded commit.

Evidence

The injection and the rendering run locally on the night from #595's walk
(the same commands the judge step runs, with self-test-local for the run id):

$ cat walk-history.jsonl walk/night.jsonl | jq -c 'select(.error == null)' | tail -n 1 \
    | jq -c --arg commit self-test-local '.commit = $commit | .first_refused_tic = ((.first_refused_tic // 2001) - 1)' \
    > walk/injected.jsonl
$ cat walk/night.jsonl walk/injected.jsonl > walk/judged.jsonl
$ clickdoom native regress walk/judged.jsonl --history walk-history.jsonl --findings walk/findings.jsonl
3159d5226c14 not judged: native diff 2000 exited 2
3ef55ff88993 against nothing before it, cost not judged: no parent measured by this run on this machine: no regression
25158fcc2fb5 against 3ef55ff88993, cost against 3ef55ff88993: no regression
971936a30b97 against 25158fcc2fb5, cost against 25158fcc2fb5: no regression
self-test-lo against 971936a30b97, cost against 971936a30b97: 1 regression(s)
  correctness: first_refused_tic 274 against 275 at 971936a30b97
clickdoom: error: 1 regression(s) over 5 line(s)
regress exit=3
$ jq -e --arg commit self-test-local 'select(.commit == $commit)' walk/findings.jsonl >/dev/null; echo "found exit=$?"
found exit=0

$ RUN_URL=https://example.invalid/run scripts/native-regression-issues.sh walk/findings.jsonl walk/lines.jsonl walk/issues
1 issue(s) rendered into .../walk/issues
render exit=0
$ cat walk/issues/1/title walk/issues/1/labels
native: divergence at tic 274 (first_refused_tic at self-test-local)
divergence
area: native
$ bash -c 'title=$(cat walk/issues/1/title); echo "marker=(${title##*(}"'
marker=(first_refused_tic at self-test-local)

The body opens with the self-test note, then one ### section per form field:

This finding was injected by the nightly's `inject_regression` input to show that filing works. Nothing regressed. Close it.

Filed by the nightly native regression walk in https://example.invalid/run.

### ROM sha256
...

A correctness and a cost finding rendered from #594's local findings (refusal
moved to 270, stage1 analysis at 1.5x):

native: divergence at tic 270 (first_refused_tic at feedfacefeed)
perf: stage1_analysis_s up 1.50x on its parent (stage1_analysis_s at feedfacefeed)

Lint, statuses captured before any pipe:

actionlint exit=0
zizmor exit=0      No findings to report. Good job! (14 suppressed)
shellcheck exit=0  (scripts/native-regression-issues.sh)

Not verified

  • The filing step's gh issue list, gh issue create and label read-back were
    not run, since that files a real issue. The first workflow_dispatch with
    inject_regression checked, after this merges, is that proof.
  • GitHub's search for a phrase holding parentheses. The step filters the search
    result by exact title suffix, so a loose search can only return more
    candidates. A search that returns nothing for an existing title would file a
    duplicate.

Invariants

None. The scripts and jobs render and file reports. They compute nothing the
simulation or the renderer produces.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates. Not run. actionlint, zizmor and shellcheck were run by
    exit code, above
  • make native-smoke, unaffected
  • No AI attribution trailers in the commits

Written mostly by Claude Opus 5.5.

@github-actions github-actions Bot added area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents labels Sep 25, 2026
Base automatically changed from ci/nightly-regression to main September 25, 2026 13:59
native regress's findings are rendered by
scripts/native-regression-issues.sh in the read-only job, one directory
per finding with a title, labels and a body. gh cannot fill a .yml issue
form, so the body follows the form by hand: each field's label as a ###
heading and a render: field fenced in its language. A correctness
finding uses 1b-native-divergence.yml (labels divergence and area:
native), a cost finding 3-performance.yml (performance and area:
native).

native-record gains issues: write and files each rendered issue. It
still runs no repository code: the rendering happened in the read-only
job and the filing step only passes the files to gh. A title ends with
"(metric at commit)", and an issue whose title ends the same way, open
or closed, is not filed again. --label is dropped silently when the
token cannot apply it, so each issue's labels are read back and a
mismatch fails the step.

workflow_dispatch's inject_regression input shows the filing path
working without a real regression. It judges one more line: a copy of
the last measured line, tonight's or the history's, under the commit
name self-test-<run id> with its first refusal one tic earlier. The judge
step fails unless regress reports it, and the issue says it was
injected. The copy is not in the night's lines, so nothing records it.
@MarcusKainth
MarcusKainth force-pushed the ci/nightly-regression-issues branch from c4d9020 to 3901c11 Compare September 25, 2026 13:59
@MarcusKainth
MarcusKainth marked this pull request as ready for review September 25, 2026 14:07
@MarcusKainth
MarcusKainth merged commit 70c1d38 into main Sep 25, 2026
20 checks passed
@MarcusKainth
MarcusKainth deleted the ci/nightly-regression-issues branch September 25, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, the Makefile, and the scripts they run area: docs The prose: READMEs, ADRs, and the contributor documents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant