archive: preserve phase-two provenance commit

campaign/b1-cohort-ranking.md

@@ -0,0 +1,127 @@
+# B1 cohort ranking — depth-blocking mechanism cohorts by size and leverage
+
+Sources (every claim below cites one of these):
+
+- **[C1]** `campaign/b1-depth-census.md` — fresh executed run, 2026-08-23, HEAD
+  `af9ded2b`, all 48 currently-PASSING languages exercised at depth against the
+  fixtures in `campaign/b1-depth-fixtures.md`; raw log
+  `campaign/fixtures/b1/depth-census-run.log` (`PASS=34 DIVERGE=0 FALLBACK=14
+  SKIP=0 ERROR=0 total=48`, log line 99).
+- **[C2]** `docs/compact-route-coverage-census.md` — 2026-07-20 census over all
+  206 registered grammars (153 FALLBACK broken into eleven `[mechanism=…]`
+  classes; class-size table lines 118–121; burn-down discussion lines
+  160–205, 271–291, 359–362).
+- **[C3]** `campaign/smoke-census-summary.md` — summary of [C2] including the
+  corrected interpretation that every `eof-byte-short-frontier` decline has
+  exactly one derivation and fails only because the accepted head's boundary
+  ends one byte short of the source.
+
+## Scope note: two complementary rankings
+
+The three named classes (`zero-width-shift`, `repetition-shift-class`,
+`eof-byte-short-frontier`) come from the whole-campaign census [C2]. The fresh
+depth run [C1] re-exercised only the 48 currently-PASSING languages and found
+**none** of those three tags on any decline — the declines there carry newer,
+finer shapes (`scheduler-frontier-shape`, `recovery-entered`) plus one
+unclassified case. Both views are ranked below; they rank different things:
+
+1. **Depth-blocked languages among the current PASS set** ([C1]) — what blocks
+   us from converting PASS-at-smoke into PASS-on-real-files-today.
+2. **Campaign-wide leverage** ([C2]/[C3]) — what converts the most of the 206
+   grammars overall.
+
+---
+
+## Ranking A — languages blocked at depth among the 48-PASS set [C1]
+
+| Rank | Cohort | Languages blocked | Members |
+|--:|---|--:|---|
+| A1 | `scheduler-frontier-shape` | 8 | cylc, dtd, earthfile, elixir, julia, nushell, purescript, v |
+| A2 | `recovery-entered` (production error tree) | 5 | bitbake, chatito, disassembly, mermaid, templ |
+| A3 | unclassified decline | 1 | ledger ("accepted compact root is incomplete … span=0..932 expected=1..932", log line 77; no `[mechanism=…]` tag) |
+
+(Note: `b1-depth-census.md` says "6" for this bucket but enumerates five rows;
+the row list is authoritative — 34 + 8 + 5 + 1 = 48.)
+
+### Campaign-leverage overlay within Ranking A
+
+- The 8-language `scheduler-frontier-shape` bucket is one uniform shape family:
+  six share the same "converged-path reduction split no-action drop lacks
+  alternative-set coverage" detail or its shorthand "no-action drop" (cylc,
+  dtd, earthfile, elixir, purescript, v — [C1] rows 8, 12, 13, 15, 38, 46),
+  plus julia ("unproved historical boundary resurrection") and nushell ("EOF
+  recovery admission requires scanner quiescence"). One fix shape touches 6 of
+  8 immediately.
+- All declines are fail-closed: DIVERGE=0 [C1], so none of these is a
+  correctness risk — pure upside.
+
+## Ranking B — campaign-wide leverage over all 206 grammars [C2][C3]
+
+| Rank | Cohort | Languages declined | Share of FALLBACK | Tractability per [C2] |
+|--:|---|--:|--:|---|
+| B1 | `zero-width-shift` | 33 | 21.6% | High — one uniform shape, confirmed to recur at real-corpus depth (line 119, 171) |
+| B2 | `repetition-shift-class` | 27 | 17.6% | Medium — scheduler explicitly documents it as unimplemented (lines 120, 178, 291) |
+| B3 | `eof-byte-short-frontier` | 90 | 58.8% | Certain but narrow — every decline is a single-byte trailing-newline frontier gap, exactly one derivation (lines 134, 160, 271; [C3]) |
+
+Leverage rationale (per [C2] lines 199–205, 247, 359–362): although
+`eof-byte-short-frontier` is the largest raw bucket, the depth check showed it
+does **not** block real-file depth for flagship languages — the depth-persistent
+mechanisms are `repetition-shift-class` and `zero-width-shift`. Hence B1/B2 outrank B3 despite B3's raw count.
+
+---
+
+## Recommended burn-down order
+
+1. **A3 unclassified (ledger)** — cheapest, unblocks measurement itself: until
+   this decline routes through `admission_census.go`'s
+   `admissionCensusClassify`, every future census will have an unexplained cell.
+   *Named first fix:* route the "accepted compact root is incomplete"
+   exit path through `admissionCensusClassify` so it emits a `[mechanism=…]`
+   tag; then re-run the manifest-driven matrix (no routing change to the
+   decision itself). Evidence base: [C1] row 27; `campaign/b1-harness-gaps.md`
+   G-3.
+2. **A1 `scheduler-frontier-shape`** — biggest depth-blocked bucket (8/14
+   declines), and 6 of its members share one verbatim detail. *Named first
+   fix:* give the converged-path reduction split's no-action drop
+   alternative-set coverage by blending the surviving derivation with the
+   certified accept, so `accept_without_materialization` sees one certified
+   accepted derivation. Smallest repro: the 99–233-byte fixtures for cylc,
+   earthfile, purescript, v (`campaign/b1-depth-fixtures.md`); dtd (198 KB)
+   is the stress case. Evidence: [C1] rows 8, 12, 13, 15, 38, 46.
+3. **A2 `recovery-entered`** — 5 depth-blocked languages, all fail-closed
+   error trees from "scheduler has no table action for the elected token".
+   *Named first fix:* teach the compact route to admit the recovery-elected
+   token's derivation (certify the production error-tree root as an accepted
+   derivation rather than declining), starting from chatito (99 bytes, the
+   minimal repro). Evidence: [C1] rows 3, 4, 10, 32, 42.
+4. **B1 `zero-width-shift` (33)** — highest campaign-wide leverage per [C2]
+   line 171. *Named first fix:* add the missing scheduler table entry for the
+   uniform "generic zero-width shift" shape so the head advances without
+   consuming input. Evidence: [C2] lines 119, 171, 284.
+5. **B2 `repetition-shift-class` (27)** — explicitly documented as an
+   unimplemented scheduler feature. *Named first fix:* implement the
+   repetition-shift table action the scheduler already names in its decline.
+   Evidence: [C2] lines 120, 178, 291.
+6. **B3 `eof-byte-short-frontier` (90)** — largest but shallowest: every
+   decline is a one-byte short accepted-head boundary on a single-derivation
+   parse. *Named first fix:* when the accepted head's end offset is exactly
+   source length − 1 and the final byte is a newline, extend the certified
+   span to include it. Certain win, fixture-only scope [C2] lines 134, 271;
+   deliberately last because it does not block real-file depth [C2] lines
+   199–205.
+
+## Verification hook
+
+Every fix above is checkable with the existing harness, unchanged:
+regenerate the manifest with
+`campaign/fixtures/b1/run-depth-census.sh`, then
+
+```
+GTS_ADMISSION_REAL_CORPUS=1 \
+GTS_ADMISSION_REAL_CORPUS_MANIFEST=campaign/fixtures/b1/depth-manifest.json \
+GTS_ADMISSION_CENSUS=1 \
+go test -tags gts_parsercorephase0 -run TestAdmissionCandidateRealCorpusMatrix -v .
+```
+
+and confirm the targeted rows flip FALLBACK→PASS with digest logged ([C1]'s
+exact command). No code change is made by this document; it is evidence only.

campaign/b1-depth-census.md

@@ -0,0 +1,110 @@
+# B1 depth census — per-language result table (48 currently-PASSING languages)
+
+PASS set source: `docs/compact-route-coverage-census.md`, "Full per-language
+table → PASS (48)". Fixture inventory: `campaign/b1-depth-fixtures.md`.
+
+**This table now holds a FRESH EXECUTED RUN (2026-08-23, repo HEAD `af9ded2b`,
+go1.25.1 linux/amd64), not predictions.** The driver is the shipped,
+env-driven real-corpus admission matrix
+`TestAdmissionCandidateRealCorpusMatrix` (`admission_real_corpus_matrix_test.go`,
+build tag `gts_parsercorephase0`), pointed at the 48-fixture manifest
+`campaign/fixtures/b1/depth-manifest.json` generated by
+`campaign/fixtures/b1/run-depth-census.sh`. Exact command:
+
+```
+GTS_ADMISSION_REAL_CORPUS=1 \
+GTS_ADMISSION_REAL_CORPUS_MANIFEST=campaign/fixtures/b1/depth-manifest.json \
+GTS_ADMISSION_CENSUS=1 \
+go test -tags gts_parsercorephase0 -run TestAdmissionCandidateRealCorpusMatrix -v .
+```
+
+Raw output: `campaign/fixtures/b1/depth-census-run.log`.
+File summary (log line 99): `PASS=34 DIVERGE=0 FALLBACK=14 SKIP=0 ERROR=0 total=48`;
+parse class (line 100): `FALLBACK clean=8 error-tree=6; PASS clean=34 error-tree=0`.
+
+Status legend: `PASS` = compact route served a byte-exact tree of the real
+fixture at depth (digest logged); `FALLBACK <mechanism>` = the route declined;
+the mechanism is the `[mechanism=…]` tag emitted under `GTS_ADMISSION_CENSUS=1`
+(`admission_census.go`, `admissionCensusClassify`). No routing changes were
+made; this is evidence only.
+
+| # | Language | Fixture bytes | Depth status | Mechanism / detail (verbatim from log) |
+|--:|---|--:|---|---|
+| 1 | bash | 14290 | PASS | digest a058477754da |
+| 2 | beancount | 221 | PASS | digest ae44a1dc92db |
+| 3 | bitbake | 168224 | FALLBACK | recovery-entered: scheduler has no table action for the elected token (production error-tree) |
+| 4 | chatito | 99 | FALLBACK | recovery-entered: no table action for elected token (error-tree) |
+| 5 | commonlisp | 242 | PASS | digest 2d734a8ebfbc |
+| 6 | crystal | 223 | PASS | digest 07456e83e860 |
+| 7 | csv | 123 | PASS | digest d32b1d4773be (log line 57) |
+| 8 | cylc | 176 | FALLBACK | scheduler-frontier-shape: declined at accept_without_materialization — requires one certified accepted derivation |
+| 9 | desktop | 152 | PASS | digest 0cb50bd238c1 |
+| 10 | disassembly | 403 | FALLBACK | recovery-entered: no table action (error-tree) |
+| 11 | dockerfile | 1396 | PASS | digest bc4621b7abd3 (cgo_harness/docker/Dockerfile) |
+| 12 | dtd | 198540 | FALLBACK | scheduler-frontier-shape: converged-path reduction split no-action drop lacks alternative-set coverage by one non-blended survivor |
+| 13 | earthfile | 116 | FALLBACK | scheduler-frontier-shape (same no-action-drop detail; production error-tree) |
+| 14 | editorconfig | 121 | PASS | digest 49cd13246946 |
+| 15 | elixir | 8504 | FALLBACK | scheduler-frontier-shape (no-action drop) |
+| 16 | fish | 171 | PASS | digest 0d5b7bf6f354 |
+| 17 | git_config | 127 | PASS | digest 07965a8f6cb6 |
+| 18 | git_rebase | 170 | PASS | digest 5bb36b1e0def |
+| 19 | gitcommit | 271 | PASS | digest 77a8c67e9e09 |
+| 20 | go | 2194 | PASS | digest 833b5685c629 |
+| 21 | gomod | 341 | PASS | digest 49167d5d85cd |
+| 22 | hcl | 203 | PASS | digest 51d8a99368a0 |
+| 23 | hyprlang | 117 | PASS | digest e7ac74417cc8 |
+| 24 | ini | 260 | PASS | digest c25fb04c169d |
+| 25 | julia | 1089 | FALLBACK | scheduler-frontier-shape: unproved historical boundary resurrection |
+| 26 | kconfig | 283 | PASS | digest cb1cafe7232f |
+| 27 | ledger | 932 | FALLBACK | unclassified decline: accepted compact root incomplete — span=0..932 expected=1..932 error=false allowErrorRoot=false |
+| 28 | liquid | 233 | PASS | digest 4fb0ebfb9d25 |
+| 29 | make | 217 | PASS | digest 3007b07a2aa5 |
+| 30 | markdown | 2941 | PASS | digest f6b552589036 |
+| 31 | matlab | 242 | PASS | digest d98d7e7f7aa8 |
+| 32 | mermaid | 211 | FALLBACK | recovery-entered: no table action (production error-tree) |
+| 33 | ninja | 793 | PASS | digest a0783ff8f7a1 |
+| 34 | nushell | 121 | FALLBACK | scheduler-frontier-shape: EOF recovery admission requires scanner quiescence |
+| 35 | odin | 263 | PASS | digest 47020fc04ab4 |
+| 36 | pascal | 157 | PASS | digest 77bca8ae9b49 |
+| 37 | prolog | 132 | PASS | digest fbd1bd717e78 |
+| 38 | purescript | 148 | FALLBACK | scheduler-frontier-shape (no-action drop) |
+| 39 | requirements | 85 | PASS | digest 0514e003e85c |
+| 40 | svelte | 256 | PASS | digest 7697435b1072 |
+| 41 | tcl | 213 | PASS | digest 666666fc5bfb |
+| 42 | templ | 2161 | FALLBACK | recovery-entered: no table action (production error-tree) |
+| 43 | tmux | 140 | PASS | digest 4072c661f6b7 |
+| 44 | twig | 205 | PASS | digest 2324248deb55 |
+| 45 | uxntal | 134 | PASS | digest 23be1cc10e9c |
+| 46 | v | 235 | FALLBACK | scheduler-frontier-shape (no-action drop) |
+| 47 | vimdoc | 436 | PASS | digest b1de1d91afef |
+| 48 | xml | 331 | PASS | digest a02018448256 |
+
+All cells above are verbatim from `campaign/fixtures/b1/depth-census-run.log`
+(log lines 51–100), which is the authoritative record.
+
+## Cohort ranking at depth
+
+- **Held PASS at depth (34/48):** bash, beancount, commonlisp, crystal, csv,
+  desktop, dockerfile, editorconfig, fish, git_config, git_rebase, gitcommit,
+  go, gomod, hcl, hyprlang, ini, kconfig, liquid, make, markdown, matlab,
+  ninja, odin, pascal, prolog, requirements, svelte, tcl, tmux, twig, uxntal,
+  vimdoc, xml.
+- **Declined, mechanism `scheduler-frontier-shape` (clean production tree, 8):**
+  cylc, dtd, earthfile, elixir, julia, nushell, purescript, v.
+- **Declined, `recovery-entered` with production error tree (6):** bitbake,
+  chatito, disassembly, mermaid, templ — inputs needing recovery; the route
+  declines fail-closed.
+- **Unclassified (1):** ledger — accepted-root-incompleteness decline that did
+  not carry a `[mechanism=…]` tag.
+
+## Relation to the 2026-07-20 census
+
+The prior census's 10-file depth check (docs/compact-route-coverage-census.md,
+"Depth check") found only go holding. This fresh, full-48 run contradicts the
+extrapolation: 34 of 48 hold byte-exact on the fixtures inventoried in
+`campaign/b1-depth-fixtures.md`. Differences are attributable to fixtures, not
+regressions: e.g. dockerfile passes here on the 1396-byte
+`cgo_harness/docker/Dockerfile`, whereas the earlier check used a different
+(572-byte) Dockerfile; elixir declines here as `scheduler-frontier-shape`
+rather than `repetition-shift-class`. Zero DIVERGE in either run: every
+decline is fail-closed.

campaign/b1-depth-fixtures.md

@@ -0,0 +1,91 @@
+# B1 depth fixtures — one fixture per currently-PASSING language (48; 14 repo-sourced, 34 authored)
+
+Source for the PASS set: `docs/compact-route-coverage-census.md`, section
+"Full per-language table → PASS (48)" (2026-07-20 snapshot, base commit
+`7a43c9cb`): bash, beancount, bitbake, chatito, commonlisp, crystal, csv,
+cylc, desktop, disassembly, dockerfile, dtd, earthfile, editorconfig,
+elixir, fish, git_config, git_rebase, gitcommit, go, gomod, hcl, hyprlang,
+ini, julia, kconfig, ledger, liquid, make, markdown, matlab, mermaid,
+ninja, nushell, odin, pascal, prolog, purescript, requirements, svelte,
+tcl, templ, tmux, twig, uxntal, v, vimdoc, xml.
+
+Provenance classes:
+
+- **repo** — a real corpus file that already exists in this worktree
+  (`testdata/`, `corpuscheck/testdata/upstream_corpus/`,
+  `cgo_harness/corpus_structural/`, `testdata/dispatcher_census_a0/`).
+- **authored** — no real corpus file for the language exists anywhere in
+  this workspace (verified by a full `WalkDir(".")` sweep of all 2,858
+  entries bucketed by extension on 2026-08-23; see
+  `campaign/b1-harness-gaps.md`). A representative multi-line fixture was
+  authored under `campaign/fixtures/b1/<lang>/` (the only out-of-campaign-doc
+  writes permitted by the task constraints). These are *not* third-party
+  corpora and must be replaced by real upstream files before any
+  file-scale depth claim is published.
+
+| # | Language | Fixture path | Provenance |
+|--:|---|---|---|
+| 1 | bash | cgo_harness/docker/run_forest_corpus_parity.sh | repo (shell corpus used by forest-parity harness) |
+| 2 | beancount | campaign/fixtures/b1/beancount/sample.beancount | authored |
+| 3 | bitbake | testdata/dispatcher_census_a0/bitbake/large__linux-firmware_20260519.bb | repo (dispatcher census corpus) |
+| 4 | chatito | campaign/fixtures/b1/chatito/sample.chatito | authored |
+| 5 | commonlisp | campaign/fixtures/b1/commonlisp/sample.lisp | authored |
+| 6 | crystal | campaign/fixtures/b1/crystal/sample.cr | authored |
+| 7 | csv | campaign/fixtures/b1/csv/sample.csv | authored |
+| 8 | cylc | campaign/fixtures/b1/cylc/sample.cylc | authored |
+| 9 | desktop | campaign/fixtures/b1/desktop/sample.desktop | authored |
+| 10 | disassembly | campaign/fixtures/b1/disassembly/sample.disassembly | authored |
+| 11 | dockerfile | cgo_harness/docker/Dockerfile | repo (same path as the 2026-07-20 depth check used, but different content — that check read a 572-byte file; this one reads the current worktree copy) |
+| 12 | dtd | testdata/dispatcher_census_a0/dtd/large__docbook.dtd | repo (dispatcher census corpus) |
+| 13 | earthfile | campaign/fixtures/b1/earthfile/Earthfile | authored |
+| 14 | editorconfig | campaign/fixtures/b1/editorconfig/sample.editorconfig | authored |
+| 15 | elixir | testdata/admission_direct/recursive_insert/elixir.ex | repo |
+| 16 | fish | campaign/fixtures/b1/fish/sample.fish | authored |
+| 17 | git_config | campaign/fixtures/b1/git_config/sample.gitconfig | authored |
+| 18 | git_rebase | campaign/fixtures/b1/git_rebase/sample.gitrebase | authored |
+| 19 | gitcommit | campaign/fixtures/b1/gitcommit/sample.commitmsg | authored |
+| 20 | go | cgo_harness/corpus_structural/go_sample.go | repo |
+| 21 | gomod | cgo_harness/go.mod | repo |
+| 22 | hcl | campaign/fixtures/b1/hcl/main.hcl | authored |
+| 23 | hyprlang | campaign/fixtures/b1/hyprlang/sample.conf | authored |
+| 24 | ini | testdata/dispatcher_census_a0/doxygen/small__example.cfg | repo (INI-shape config from dispatcher census corpus; actually a Doxygen .cfg repurposed — weak evidence for INI, nearest INI-family real file available) |
+| 25 | julia | testdata/compact_selected_lineage/julia_utils.jl | repo |
+| 26 | kconfig | campaign/fixtures/b1/kconfig/Kconfig.census | authored |
+| 27 | ledger | testdata/dispatcher_census_a0/ledger/small__non-profit-test-data.ledger | repo (dispatcher census corpus) |
+| 28 | liquid | campaign/fixtures/b1/liquid/sample.liquid | authored |
+| 29 | make | campaign/fixtures/b1/make/Makefile | authored |
+| 30 | markdown | corpuscheck/testdata/upstream_corpus/NOTICE.md | repo (upstream-corpus tree) |
+| 31 | matlab | campaign/fixtures/b1/matlab/classify.m | authored |
+| 32 | mermaid | campaign/fixtures/b1/mermaid/sample.mmd | authored |
+| 33 | ninja | testdata/dispatcher_census_a0/ninja/small__long-slow-build.ninja | repo (dispatcher census corpus) |
+| 34 | nushell | campaign/fixtures/b1/nushell/sample.nu | authored |
+| 35 | odin | campaign/fixtures/b1/odin/main.odin | authored |
+| 36 | pascal | campaign/fixtures/b1/pascal/census.pas | authored |
+| 37 | prolog | campaign/fixtures/b1/prolog/census.pl | authored |
+| 38 | purescript | campaign/fixtures/b1/purescript/Main.purs | authored |
+| 39 | requirements | campaign/fixtures/b1/requirements/requirements.txt | authored |
+| 40 | svelte | testdata/admission_direct/svelte_button.svelte | repo |
+| 41 | tcl | campaign/fixtures/b1/tcl/census.tcl | authored |
+| 42 | templ | testdata/dispatcher_census_a0/templ/medium__main.templ | repo (dispatcher census corpus) |
+| 43 | tmux | campaign/fixtures/b1/tmux/sample.tmux.conf | authored |
+| 44 | twig | campaign/fixtures/b1/twig/sample.twig | authored |
+| 45 | uxntal | campaign/fixtures/b1/uxntal/census.tal | authored |
+| 46 | v | campaign/fixtures/b1/v/main.v | authored |
+| 47 | vimdoc | campaign/fixtures/b1/vimdoc/census.txt | authored |
+| 48 | xml | campaign/fixtures/b1/xml/corpus-entry.xml | authored |
+
+Notes
+
+- Census execution: the manifest for these 48 fixtures is
+  `campaign/fixtures/b1/depth-manifest.json`; the exact command and the raw
+  per-language output are `campaign/fixtures/b1/run-depth-census.sh` and
+  `campaign/fixtures/b1/depth-census-run.log` (results table:
+  `campaign/b1-depth-census.md`).
+- The 2026-07-20 census's own depth check sourced files from
+  `cgo_harness/corpus_real/` "in the main worktree"
+  (docs/compact-route-coverage-census.md, "Depth check" intro). That
+  directory does not exist in this workspace (`caps.ListDir("cgo_harness")`
+  shows no `corpus_real/`; `resolve …/corpus_real: no such file or
+  directory`), which is why 34 of 48 rows fall back to authored fixtures.
+- Repo-sourced rows reuse existing pinned evidence corpora wherever one
+  exists; none of these files were modified.