Add AC1a reproduce-artifacts and AC1b reproduce with a comparison artifact

[drewOrc]

What

Splits AC1 as Drew decided on 2026-09-29 and adds the code for both levels. This PR does not run AC1b; the status stays "RQ1 to RQ5 complete; Tier 1 acceptance not complete".

• docs/PLAN.md section 5: AC1 keeps its wording, renamed AC1b and marked "definition unchanged". New AC1a. New section 5.1 records the decision, the six frozen AC1b pass criteria and the budget rule (original AC6 spend fixed at US$3.19; the AC1b Haiku run is reproduction-validation with its own US$5 cap, never summed with the original). No other AC or protocol changed.
• make reproduce-artifacts (AC1a): release.py lists the 76 Release files from the committed manifests, downloads each to a temporary name and moves it into place only after its SHA-256 matches; then verify-logits, verify-llm, analysis, llm-latency, cost, figures, report, and a final check that results/ and README.md are byte-identical to HEAD. CPU latency is not remeasured (machine-dependent). New workflow reproduce-artifacts.yml: weekly, on demand, and on PRs touching the files it exercises; not a required check; actions pinned by full SHA; permissions: contents: read.
• make reproduce (AC1b): AC2, pilot-lr, pilot-steps, baselines, both curves, OOS ablation, verify-logits, Haiku (MAX_USD=5), verify-llm, analysis, bench-cpu, llm-latency, cost, figures, report, then a check that the originals are untouched. Everything goes under reproduction/<HEAD[:12]>/ through new RESULTS_ROOT, CHECKPOINT_ROOT and README_OUT Makefile variables (location fields only, so run identities are unchanged). Preflight: clean tree, HEAD merged into origin/main, uv sync --locked, API key present (recorded as true/false only), 8.8 GiB free. Each step passes only on its whole completion line(s) and exit 0; the first failure stops the run and still writes the comparison. Resumes: passed steps whose outputs keep their SHA-256 are skipped until one step reruns.
• Comparison (reproduction/<id>/comparison.{json,md}): per-step flow result, AC2 per seed against 95.7%, Haiku rows, identity, spend, per-row prediction differences on (split, index) and parse failures, the two budgets side by side, pilot choices, the 14 README first-screen numbers, 8-way accuracy and OOS recall at every k for both encoders, and every threshold-diagnostic statistic. Outside the original mean ± std is REVIEW REQUIRED; with no spread (single run, std 0 or null) any difference is. Verdict FAIL only when the flow or AC2 fails.

Checks

• make lint, make test (632 passed), make smoke: pass locally.
• make reproduce-artifacts run twice in a fresh clone: verified 76/76 release files, then results/ and README.md byte-identical to HEAD. In CI it first ran on ubuntu-latest: every JSON file, report.md and the README were identical but the four PNGs were not, so the workflow runs on macos-15 (the platform the committed figures came from) and passes there.
• make reproduce without a key on this unmerged branch: stops in preflight (NOT STARTED), no step runs.
• Mutation check: 29 mutations to the verdict, comparison, driver, Makefile, location plumbing and release download; all 29 caught by the tests.

Review fixes (R1 to R6)

• R1: every step records its identity (HEAD, SHA-256 of uv.lock and configs/, Python, torch, transformers). A reproduction directory started under another identity is refused in preflight (NOT STARTED, asks for a new REPRO_ID) and again in run_steps; the review's case (same id, one more empty commit) now runs no step. The comparison lists the commit of every step (flow.commits and a column in the flow table).
• R2: scope fixed before the first AC1b run and written into PLAN 5.1 ("實跑前補充"): the README router table (35 rows: LLM-only; ModernBERT k=10 and k=100 small-only, hybrid at 2% and 5%, oracle; accuracy, OOS recall, high-confidence OOS misroute, Haiku call rate) and every statistic of ablation_comparison (22 rows) are judged. Latency, k=100 training time and peak memory, and the cost model are listed with differences but never judged (LISTED, NOT JUDGED).
• R3: more PR paths for the AC1a workflow. R4: check-originals --snapshot also compares every file under results/, gitignored ones included, with a snapshot taken before the run (AC1b before the first step, AC1a after verify-llm). R5: a failing git status or git rev-parse in preflight is a problem, not a clean tree. R6: README notes the shared Hugging Face cache and HF_HOME.
• Mutations for these fixes: 16 of 17 caught, including dropping HEAD from the identity. The one that survives puts the listed-only rows into the REVIEW count; they can only ever be LISTED, so that change does nothing (an equivalent mutant). The mutation that would matter, judging those rows, is caught.
• AC1a rerun in a fresh clone: still byte-identical, with the snapshot check.