bench/: a replicate-based reliability instrument for the document write paths (models × settings × paths)

[mmcky]

Nothing in the estate can currently answer "for this operation, on this content class, with these settings and this model, what fraction of runs writes a structurally correct file?" — and 2026-07-27 showed what that costs.

Full plan, with the measurement matrix, metric table, cost model and staged delivery: PROJECT-PLAN-ENGINE-BENCHMARK.md in QuantEcon/project-translation.

Why

That session spent a day on one question — does the model tier matter for new lectures? — and produced three results that make the case better than an argument would.

Two wrong causal hypotheses were formed and discarded before the right one. First "Sonnet 5 is worse at full-document translation" (refuted: Sonnet 4.6 kept the head block on 13/13 seeds, Sonnet 5 on 7/7 via sync). Then "the localisation rules are the trigger" (declared unsupported at 1/3 vs 2/3, then reinstated). Each reversal came from reading a single run, or three, as if it were a rate.

The real answer needed 15 replicates per cell. The design that started the day had 7 — a post-hoc power calculation puts that at ~13% power. The question was not hard; the instrument was absent.

A correct-looking fix nearly shipped a regression. #225 wires the localisation rules into the sync path, exactly as #178 asks. Validation then showed those rules cut the structural pass rate from 93.3% to 33.3% (p = 0.002) on the one path that is currently clean. Nothing in CI, review or the E2E harness would have caught it — the harness fixtures contain zero {raw} jupyter blocks and zero (label)= anchors, so a harness run would have returned green while being structurally incapable of exercising the defect.

What the measurements were

Same lecture, model, engine and glossary via translate init; only --localize differs:

Configuration	Structural pass rate	95% CI (Wilson)
--localize default	5/15 = 33.3%	[15.2%, 58.3%]
--localize none	14/15 = 93.3%	[70.2%, 98.8%]

Fisher exact p = 0.002; all nine failures byte-identical — directive #1 name changed: source line 15 has {raw}, output line 45 has {index}. That is #118, with an identified cause for the first time.

Meanwhile the model contrast that motivated the day (Opus 5 7/7 vs Sonnet 5 4/7) sits at p = 0.192 and remains unresolved after 26 invocations.

Where it should live

bench/ in this repo, not a new one and not benchmark.translate-zh-cn.

The metrics are this engine's guards — checkStructuralParity, extractStructuralTokens, buildHeadingMap, diff-checks. A separate repo forces a published-package dependency, which lags the release being gated (so the gate would certify v0.25.0 using v0.24.0's parity rules), or a vendored copy. The vendored-copy rot is not hypothetical: experiments/thinking-sonnet5/scripts/lib.mjs already re-implements buildTranslatePrompt and carries a hand-written "re-sync this" warning.

benchmark.translate-zh-cn is ruled out on two grounds: it is zh-cn by construction (config.yaml, glossary pin, domain list all single-language) while this spans zh-cn/fa/fr/ml, and document-structure numbers filed under a linguistic-quality Elo leaderboard will be read as one number despite measuring different things.

experiments/ is ruled out because it holds finished one-offs, none re-run. This has to be re-answered on every release and every model.

Milestone 1 — one person, one day

Scope deliberately minimal, and it is the contrast that is already fully specified:

• bench/corpus.yml pinning (repo, commit_sha, path, sha256, class) plus a sparse-checkout fetch step. A pinned manifest rather than vendored bytes because init -f refuses any file absent from _toc.yml, so vendoring individual files forces a synthetic _toc.yml and loses _static/, _admonition/ and quant-econ.bib.
• One cell: init × {default, none} × prospects_bounded_rationality × n=15.
• Scored on write-completion and head-block survival, importing checkStructuralParity and extractStructuralTokens from dist/ — no re-implementation.
• Reported as rates with Wilson intervals and a Fisher contrast. Never a bare count: a cell reported as "9/10" with no interval is the failure being corrected.
• A Tier-0 static test that fails the build if bench/ calls client.messages outside a single wrapper, or re-implements anything it could import.

Had this existed on 2026-07-27 it would have blocked #225 automatically. Reproducing that day's result from a committed harness rather than a scratch directory is the acceptance criterion.

Design constraints worth fixing now

• No LLM-judged metric gates anything, ever. Not a preference: Review mode grades the engine's own i18n font localisation as unauthorised code modification — a gating major finding on every zh-cn plotting lecture #224 established that reviewer error is itself a function of the localisation treatment, i.e. differential misclassification that biases the primary contrast in a known direction.
• Anti-gaming: frontmatter, title-coherence and heading-level metrics live outside structural-parity.ts, so a release cannot raise the headline number by weakening the guard. Every run records the sha256 of dist/structural-parity.js.
• Infra failures are excluded from numerator and denominator, and >10% marks a run UNRELIABLE and withholds the verdict.
• Power is stated up front. Cells sized below their stated power are advisory and cannot gate.

Cost is not the obstacle

At the ~$0.21/file measured in #194: Tier 0 is $0, a Tier 1 release gate is 190 attempts ($40–60), a Tier 2 model matrix 780 ($150–250). The powered comparison on 2026-07-27 was 30 invocations for under $10 and settled a question that had consumed a day. The obstacle is that nothing is set up to run a cell fifteen times and report a rate.

Deliberate non-goals

Not this	Covered by
Phrase-level linguistic quality, Elo, human judging	benchmark.translate-zh-cn
Native-speaker reference comparison per language	#189 / #194
Gating on any LLM-judged score	excluded by design
End-to-end workflow wiring	the test-translation-sync harness

The harness validates wiring; this validates behaviour rates. Separately, and regardless of this issue: the harness needs a fixture carrying a QuantEcon-style head block ({raw} jupyter + (label)=) before it can be informative about this defect class at all. Production prevalence is 97/138 lectures in lecture-python.myst.

Open decision

Does this get built before or after the #118 rule-prompt fix? Building first means the fix is validated by the instrument rather than by another ad-hoc run; fixing first unblocks #178 sooner. Milestone 1 is one day, which argues for first.

Related: #118, #178, #187, #189, #194, #222, #223, #224.

[mmcky]

Method validation from an adjacent surface: #248 applied this plan's replicate approach to review mode (13 re-reviews of a fixed translation + 5 fresh syncs × 3 lanes, on the harness) and it worked exactly as the bench design intends — single-run conclusions were overturned (a routing 'signal' formed from three single runs was falsified by replicates showing a deterministic reviewer and a ~1/9 translation-side draw), and the informative statistic turned out to be distributional (severity×category assignment), not scalar. When bench/ lands, review-mode replicates belong in scope alongside the write paths; the #248 report is a worked example of the harness-driven variant, including its two operational gotchas (label re-add is a silent no-op if the remove did not land; verdict comments are overwritten in place, so capture must precede each re-fire).