ci(pr-body): citations are verified against the head tree; template and fixer contract aligned (refs #2904)

[apmantza]

Round 3 state-space table

This table records the citation grammar axes and verdicts required by #2904.
Each cell names an existing test in tests/scripts/check-pr-body.test.ts.

Form	Placement	Test-id form	Quote form	Verdict	Test
file:line	prose	none	none	accept	accepts a plain citation without a quote
file:line	table cell	none	none	accept	accepts a citation in a table cell without a quote
file:line-line	prose	none	none	accept	accepts range citations by their first line
file:~line	prose	none	none	accept	accepts approximate-line citations by their hinted line
file:line	any	none	none	reject-missing-file/line	rejects a citation to a missing or out-of-range head file
none	table cell	backticked title	none	accept/reject-missing-test-id	normalizes canonical it titles in table cells
none	prose	it("…")	none	accept/reject-missing-test-id	checks canonical it titles in prose
none	table cell	bare quoted title	none	accept/reject-missing-test-id	checks bare test titles in table cells
file:line	fenced block	none	fenced block	ignore	ignores citations in fences and accepts the canonical it title in a table
file:line	prose	none	fenced next line	accept/reject-missing-line	pins the ±20 citation quote window
file:line	prose	none	none	accept	accepts a plain citation without a quote
file:line	prose	none	none	accept	accepts a plain citation without a quote
repeated file:line	prose	none	fabricated fenced next line	reject-missing-line	checks every repeated citation quote
file:line	prose	none	real vitest/npm transcript fence	ignore	recognizes only real transcript quote shapes
file:line	prose	none	incidental pass/fail fence	reject-missing-line	does not treat incidental pass or fail words as transcripts

Round 5 cell-predicate state space

lintTestReferences checks a cell only when its column header matches /test|probe|case|witness|id/i; header cells, bullets, and prose are ignored.

Token shape	Matching test header cell	Other-header cell	Header cell itself	Bullet	Prose
Backticked it("…") title	normalizes canonical it titles in table cells	ignores non-test table cells	ignores table header cells	checks canonical it titles in prose	checks canonical it titles in prose
Backticked path under tests/	accepts a test path in a test column	ignores non-test table cells	ignores table header cells	accepts a test path in a test column	accepts a test path in a test column
Backticked command	rejects a command-shaped test cell without a real title	ignores non-test table cells	ignores table header cells	rejects a command-shaped test cell without a real title	rejects a command-shaped test cell without a real title
Bare word	rejects a fabricated bare test title	ignores non-test table cells	ignores table header cells	checks bare test titles in table cells	checks bare test titles in table cells
SHA	ignores SHA cells in test columns	ignores non-test table cells	ignores table header cells	ignores SHA cells in test columns	ignores SHA cells in test columns

Checked cells are the matching-header cells in every row. Ignored cells are every other placement. The named tests are added in this round and each title is grep-able in the tree.

Summary

Refs #2904. Round 3 fixes citation range handling, conditional quote checking,
canonical test-title extraction, transcript recognition, repeated-citation
validation, and the local-lint declaration.

Tests

Round 3 adds state-space and mutation-sensitive coverage in
tests/scripts/check-pr-body.test.ts.

Blast radius

The production blast radius is scripts/check-pr-body.mjs: local preflight,
the pull-request event entrypoint, and the advisory PR-body workflow.

Class sweep

The citation and test-reference sweep covers clients/, tools/, mcp/,
scripts/, index.ts, and tests/. The open-PR population sweep appears
under Round 3 below.

Observability

No new failure path; no record added.

Test assessment

tests/scripts/check-pr-body.test.ts uniquely pins the parser and its
mutation-sensitive boundaries. No touched test became redundant.

Round 2

The prior round fixed the missing #2877 fixture, fence-aware scanning, local
working-tree reads, and the ±20 source window. Its population claims that
ranges and missing quotes were defects were incorrect and are corrected below.

Round 3

Operating rule: validate citation file and start-line existence, and validate
only an offered source quote without requiring one.

Kept: the lane remains advisory with a ten-green-merged-PR bound. The #2896
uncited prose shape remains outside citation validation; the remainder is named
on #2904.

Population sweep at each member's own head, using --lint-local from an
isolated worktree and diffing against origin/master:

PR	New-rule findings	Verdict
#2917	12	0 true positives; ranges and unquoted citations accepted
#2900	19	0 true positives; unquoted citations accepted
#2911	2	0 true positives; unquoted citations accepted
#2908	1	0 true positives; table citation accepted

The four-member v2 population sweep is therefore zero findings after the
round-3 rules. The whole-tree pattern sweep found one citation parser and no
second local-lint option seam. Consolidation verdict: citation parsing remains
on the existing pathLineReferences seam; no follow-up fold is required.

Mutation proofs:

Mutation	Result
M1: disable offered-quote mismatch	RED: requires an adjacent quote to match source text within twenty lines
M2: disable master-claim transcript guard	RED: requires origin/master transcripts for master-red claims
M3: disable missing/out-of-range citation guard	RED: rejects a citation to a missing or out-of-range head file
M4: disable short table-id guard	RED: rejects a fabricated short table identifier
M5: change ±20 to ±2000	RED: pins the ±20 citation quote window
M6: disable canonical table-title normalization	RED: normalizes canonical it titles in table cells
M7: disable transcript recognition	RED: recognizes only real transcript quote shapes
M8: restore mandatory quote rejection	RED: accepts a plain citation without a quote

The targeted suite passed 135 tests after the round-3 additions.

Preflight:

gate	mirrored CI job	pass/fail	first red line
build	Lint & type-check	pass
lint	Lint & type-check	pass
fmt:check	oxfmt format check	pass
changelog:check	Unit tests	pass
check-changelog-fragments	Changelog fragment (fast-fail)	pass
check:lockfile	Lint & type-check	pass
lockfile:complete	Lint & type-check	pass
tests/config	Unit tests	pass
generation-guard	Unit tests	pass
flake-shape-ratchet	Unit tests	pass
lsp-spawn-heavy-coverage	Unit tests	pass
ci-verdict	Unit tests	pass
knip	knip (advisory)	pass

Round 4

Round 4 fixes the v3 blocker and both live false positives. Test-reference
validation now harvests declaration titles and test paths from the checked-out
tree, excludes body fixtures from the corpus, and matches short ids exactly.
Markdown claim checks use one block-aware sentence splitter; code spans are
atoms, table rows are blocks, and only the next non-blank block can satisfy a
master transcript.

The state-space table above names the actual placement and quote form for each
test. The previous row-8 bullet claim was corrected to the table placement
implemented by checks bare test titles in table cells.

Round 4 tests edited in tests/scripts/check-pr-body.test.ts pin the committed
#2877 fixture through the real reader, ±20 and ±21 on both sides, first-line
range resolution, origin/master transcript false positives, preflight command
cells, trailing canonical it() table cells, next-block transcript adjacency,
and dots inside code spans.

Mutation checks: M10 range-last-line, M11 dropped tilde normalization, M5c
backward-window deletion, M12 transcript-info bypass, M13 origin/master quote
bypass, M15 table-block exemption deletion, and M16 reviewer attribution
deletion each red the targeted suite. Vacuous M14 and M17 branches were
deleted.

Current local verification:

Check	Result
tests/scripts/check-pr-body.test.ts	151 passed
tests/config	458 passed, 1 skipped
flake-shape ratchet	57 passed
npm run fmt:check	pass, exit 0
npx tsc --noEmit	pass, exit 0
npm run preflight	all 13 gates pass

The whole tests/scripts run reports 1,758 passed, 9 environment reds, and 11
skips. The reds are outside this diff: the worker Git guard blocks temporary
fixture repositories, and the known bare-node rekey-hook-await-exemptions
case reports ERR_UNKNOWN_FILE_EXTENSION for hook-await-scan.ts.

Round 4 population sweep at each live open PR head, using isolated git archive checkouts and the same checker:

PR	Head	New-rule findings	True positives
#2933	368f85437c20fc39709d2400e39e41c4155eaa82	0	0
#2929	c5db1b69b7ee94d97c177a7d4fd2fb2eb4a319e0	0	0
#2921	f443aef31fcac3ff16b4a311d26f2a6c9aa6af4e	0	0
#2913	7c07ff1a7808309959efbefb086b90c5f09b3654	0	0
#2900	aaf9f1c6537b2e4ba485cb8d19212fc900328cbd	0	0
#2849	6c5adcda6d37e0fcc37fdb59df1ec9660cdc78ae	0	0

The population result is zero new-rule findings across all six live open PRs.

Round 5

Operating rule: only backticked cells below a /test|probe|case|witness|id/i column header are test-reference candidates; declaration titles come from lexer string spans.

Kept: header cells, other table columns, bullets, prose, commands, and SHAs do not enter the table-cell candidate set. The existing prose it("…") check remains separate.

Round 5 fixes the four-round live false-positive seam and the .each corpus gap. The existing comment/string lexer now emits string spans, and the corpus derives titles from those spans. This captures regular and .each declaration titles without a second regex over the same bytes. Local mode includes untracked files under tests/ through git ls-files --others --exclude-standard.

The corpus builder was measured at 1.73 seconds for each isolated lint process on this tree. Within one lint invocation, the corpus is cached, so repeated checks reuse the first scan. The v4 baseline measured 256 ms for the first corpus build, including approximately 240 ms in the prior blanking pass; the new span pass removes the second full-source harvest regex and is covered by the targeted suite.

Round 5 tests edited in tests/scripts/check-pr-body.test.ts add placement coverage for matching and non-matching headers, header cells, command cells, bare titles, SHAs, .each declaration titles, and untracked local files. The targeted suite passes 151/151.

Population sweep at each open PR head used isolated temporary archive checkouts outside the main checkout. Every open head produced zero findings:

PR	Head	Findings
#2929	8f2bf562f397	0
#2921	41b2c5033951	0
#2913	e1a269616d47	0
#2849	6c5adcda6d37	0

Local verification: tests/scripts reports 1,758 passed, 11 skipped, and 9 environment failures outside this diff. tests/config reports 458 passed and 1 skipped. The flake-shape ratchet reports 57/57. npm run fmt:check exits 0, npx tsc --noEmit exits 0, npm run preflight exits 0, and npm run build exits 0.

The explicit code-scanning read reports no master alerts. It still reports one open js/redos alert attached to the prior head commit e1a269616d47be96ae646da8a71641e4837b14dd; the local source no longer contains that harvest regex. A fresh GitHub analysis on the pushed head is required to close the stale alert.

[github-actions]

ci-classifier: real — first failure: tests/config/gitignore-tracked-shadow.test.ts > gitignore does not shadow tracked files (#2250) > no git-tracked file is reported ignored by git check-ignore --no-index

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[apmantza]

Held as draft after verify round 5 (2026-09-11 ~03:00Z). Five fix rounds each fixed the previous verify's findings and introduced new ones on the same predicate; round 5 additionally retired a guard silently (M4 red → green) and edited its own fixture assertion to stay green, and the lexer-span corpus regressed real titles (194 lost) because blankCommentsAndStrings has no regex-literal state. The five verify records (review-2913 v1–v5 in the orchestrator scratchpad, summarised on #2904) are the design input for the next attempt, which is not a sixth patch round on this branch.

Proposed re-scope into three PRs, each tests-first with its state-space table in the body BEFORE code:

1. blankCommentsAndStrings regex-literal state + string-span export (shared lexer seam, own PR, census test: every it(/test(/describe(/.each title under tests/ is in the corpus — derived from git grep, not enumerated).
2. Test-reference placement rule: short ids checked in any column, free text only under a test-shaped header (the verifier's probed remedy: 11/11 on the test(config): the spawn-cwd sweep resolves each cwd to its origin and covers every child spawn (closes #2872, refs #2777) #2877 fixture, 0 new findings on open PRs); fabricated it('…') in prose rejected.
3. Markdown-block sentence splitter (code spans and cells atomic) with the M22 prose-form pin, and the refs #2904 remainder (±3 → ±20) stated.
   Corpus cost is a gate for (1): the mutation lane's 5 s dry-run must pass (round 5 measured 1,175 ms cold, quadratic per-span slicing).

[apmantza]

Proceeding with the three-slice re-scope (maintainer, 2026-09-11 05:0xZ): slice 1 ci/2904-slice-1-lexer-string-spans and slice 2 ci/2904-slice-2-test-reference-placement are in flight as separate PRs from master; slice 3 (sentence splitter) follows slice 2. This draft closes as superseded once they land.

[github-actions]

This PR is merge-conflicted; required checks are silently skipped until resolved.

[apmantza]

Measured false-positive census for scripts/check-pr-body.mjs, from two PRs this session. Posting it here rather than filing separately because this is the open lane on that checker.

The "test reference is missing under tests/" rule fires on backticked prose that is not a test title. Eleven hits across two bodies, zero of them real:

PR #2955 — 4 hits:

PR body test reference is missing under tests/: npm test
PR body test reference is missing under tests/: git init
PR body test reference is missing under tests/: PI_LENS_HOME=$PWD/.probe-home npm test
PR body test reference is missing under tests/: recordDegradationOnce({ kind: "sgconfig-baseline-cap-evict" })

PR #2997 — 7 hits:

PR body test reference is missing under tests/: typos (advisory)
PR body test reference is missing under tests/: The suite passed locally
PR body test reference is missing under tests/: Orchestrator invariants
PR body test reference is missing under tests/: Verification rounds
PR body test reference is missing under tests/: tests/config
PR body test reference is missing under tests/: No test files found
PR body test reference is missing under tests/: rg -l 'merge-train/SKILL|pi-lens-reviewer\.md' tests/ scripts/

What the census shows

The eleven split into four kinds, and they are not equally hard:

1. Shell commands — npm test, git init, an rg invocation with flags, an env-prefixed command. ci(pr-body): recognise test references by token shape and markdown placement (refs #2904) #2946's review already cleared a related case (npm run build in a Test column correctly reported), so there is an existing command notion; it does not cover a bare npm test, an env prefix, or a command with quoted arguments.
2. A code identifier with arguments — recordDegradationOnce({ kind: "sgconfig-baseline-cap-evict" }). A call expression is never a test title.
3. Quoted section headings from the files under discussion — Orchestrator invariants, Verification rounds.
4. Quoted tool output and check names — typos (advisory), No test files found, The suite passed locally, tests/config.

Kind 4 is the interesting one, because it is self-inflicted: a PR body that quotes a check name or a tool's output is doing exactly what the honesty rules ask for — reporting what ran and what it said. The checker currently penalises the bodies that quote their evidence most precisely.

Why this matters beyond noise

It is advisory, so nothing was blocked. But this session also added an orchestrator rule that advisory rows are read and disposed of before every merge, precisely because typos (advisory) caught a genuine defect in #2955's round-12 diff that no gating check had an opinion about. A lane with a high false-positive rate is what trains people to stop reading it — so precision here has a cost beyond tidiness.

No remedy prescribed; #2904's slices own the classifier design and this is data for whoever takes the next one. If it helps, the cheapest discriminator for kinds 1 and 2 looks like "a token containing whitespace and a shell/call shape (leading known binary, = prefix, or trailing (...)) is not a test title", and kind 3/4 would need placement context rather than token shape.

This is also another member of catalogued shape 50 — a permissive syntax heuristic classifying an untrusted reference — at the accept-too-much end rather than the fail-open end.