Release gate takes one draw per scenario: a ~40% model-side defect passed three gates (scenario 17, .ml)

[mmcky]

The §4a release gate takes one draw per scenario, so it cannot see a model-side defect that fails less than always. Found 2026-09-21 on the v0.29.1 gate (tally on #316; diagnosis and repair in #317).

Evidence

Scenario 17 (new-document-toc) on the .ml lane failed the v0.29.1 gate, and failed again on a re-run of the same tagged bytes: the model wrapped the fixture's plain ## Exercises list in {exercise-start} … {exercise-end}, and the structural-parity guard refused the file. Reproduced locally on the fixture, twelve draws per arm, refusals by the guard:

Engine	Refused
v0.29.0	5 / 12
v0.29.1	11 / 12
v0.29.1 + the #317 repair	0 / 24

So the defect had been in every release since the exercise-verbatim rule arrived in v0.28.0, at roughly 40%, and the gates of v0.28.0, v0.28.1 and v0.29.0 each passed scenario 17 on a single draw. Three passes in a row at that rate is about a one-in-five event — the gate did not so much verify those releases as fail to catch them. It caught v0.29.1 only because the rate had risen to about 90%.

A second lesson from the same diagnosis: six draws per arm read 1/6 against 5/6 and looked like a clean regression; the next six moved the baseline to 5/12. Small samples gave the right direction and the wrong story.

What to consider

• Where the model writes a whole new document (scenario 17, and any other new-file or full-document scenario), one draw is the weakest evidence in the suite: the output is long, the structural-parity guard is fail-closed, and a refusal leaves the sync PR with a dangling _toc.yml entry (the sync silently first-translates lectures missing from the target — unreachable when it succeeds, invisible when it fails #222 / Remove failed-new-file TOC entries before opening sync PR #289 class, W1 W1 — v0.27.0: make partial delivery loud (declared-vs-delivered, TOC merge) [P0] #259). Options: N draws for those scenarios only; or an in-action retry when structural parity refuses a new document (the guard writes nothing, so a retry is safe and cheap — one more call against a cached prompt); or both.
• A production-side view of the same thing: an in-action retry would also help live editions, where a refusal today means a missing lecture and a failure issue. That is a behaviour change and wants its own decision; the harness-side change does not.
• Cost: the gate is already 84 sync + 84 review runs. Extra draws confined to the handful of full-document scenarios on three lanes is a small addition; extra draws everywhere is not.
• A local recipe exists for measuring a rate when the gate does flag something: init -f <fixture>.md -d . --localize none against a git archive of the test/NN-… branch, with a v0.X.Y worktree for the old-version arm and --glossary to cross rules with glossary. Count by whether the output file was written — the CLI's failure wording differs from the action's.

Related: #260 (deterministic repair for rule misses the prompt does not deliver), #259 (W1, loud partial delivery), #203 (an earlier one-in-three structural transposition on init).

[mmcky]

Decision (Matt, 2026-09-23): the local N-draw rate check. The gate stays as it is — one honest action-path draw per scenario at the tag — and §4a gains a scripted step that runs the CLI at the tag N ≥ 12 times on each fixture that introduces a whole new document, and reports the refusal rate per fixture and language. It costs no GitHub runs, measures the rate directly, and would have shown 5/12 at v0.28.0. The in-action retry on a parity refusal is not adopted here: it is a production behaviour change, and it would hide exactly the rate this step exists to see (a 40% defect passes a two-retry gate at ~6%). If it is ever wanted for delivery, it is its own decision under W1 (#259) / #260. Script to follow on a PR that closes this.