harness: rate-check.sh — N-draw refusal-rate check on full-document fixtures, release step 4b (closes #320)

[mmcky]

Closes #320 — Matt's decision of 2026-09-23 (recorded on the issue): keep the gate as it is and add a local N-draw rate check as release checklist step 4b.

Why

The gate takes one action-path draw per scenario. Scenario 17's new document was refused by the structural-parity guard on about 40% of draws from v0.28.0 to v0.29.0 and passed three release gates; v0.29.1's gate caught it only because the rate had risen to ~90%. One draw cannot see a rate. Twelve can.

What

tool-test-action-on-github/rate-check.sh [--ref <tag|branch>] [--draws N] [--languages …] [--fixtures …] [--parallel J] [--max-refusals K] [--model M] [--summarize DIR]

• Runs the CLI (init -f <fixture> --localize none) N times (default 12) per full-document fixture and language, from a two-file source dir (fixture + a generated _toc.yml). A draw is refused when no output file is written — the parity guard's fail-closed behaviour.
• Prints one line per cell with the distinct refusal reasons beneath it; exits 1 if any cell has more than --max-refusals (default 1). Writes meta.txt and summary.txt under .dev/scratch/rate-check/<stamp>-<pid>/; --summarize DIR re-reads a run without new draws.
• --ref compiles the requested version in a temporary worktree and runs the CLI from it, so that version's own glossary is used — an old release can be measured. Languages derive from the main script's LANGUAGES array; fixtures default to game-theory.md, the only file the scenarios introduce as a whole new document today (scenario 20 is a git mv with unchanged content).
• Local only: N × fixtures × languages translations against a cached prompt — about 36 for a full run; no GitHub runs.

AGENTS.md gains step 4b; the tool README a "Phase 1b" section; CHANGELOG under [Unreleased]. Session log .dev/log/2026-09-23-close-out.md.

Validated

Engine	game-theory.md, ml, 12 draws	Exit
v0.29.2 (this checkout)	0 / 12	0
v0.29.0 (--ref, worktree)	4 / 12 — 3× source [none] vs output [exercise-start], 1× [exercise]	1

That is the defect that passed three gates, read off in one command. The first v0.29.0 run printed FAIL and then exited 0: the verdict flag was being set inside a { … } | tee pipeline — a subshell. The tally is now a function run in the current shell, and --summarize on the same two run directories reproduced both exit codes without new draws.

Not adopted

The in-action retry on a parity refusal. It is a production behaviour change, and it would hide exactly the rate this step exists to see (a 40% defect passes a two-retry gate at ~6%). If wanted for delivery it is its own decision under W1 (#259) / #260.

Notes for merging

Rebased onto main after #322 merged (2026-09-23); [Unreleased] now carries this PR's ### Added above #322's ### Fixed. bash -n and shellcheck clean.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

rate-check.sh currently references an unset $SUMMARY variable (fails under set -u), and the README examples use an inconsistent invocation path relative to the preceding cd instructions.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Unset SUMMARY variable causes set -u failure · New
• Incorrect script path after cd in example commands · New
• --help omits the implemented --summarize DIR option · New

What changed in this PR

Adds a local, multi-draw “refusal-rate” check (rate-check.sh) to measure how often full-document fixtures are rejected by the structural-parity guard, addressing the limitation that the existing GitHub-based harness gate observes only one draw per scenario.

Changes:

• Add tool-test-action-on-github/rate-check.sh to run N draws per (fixture × language), summarize refusal rates/reasons, and fail above a threshold.
• Document the new “Phase 1b” rate-check step in the harness README and add release checklist step 4b in AGENTS.md.
• Add an [Unreleased] changelog entry describing the new tool.

File	Description
tool-test-action-on-github/​rate-check.sh	New script that repeatedly runs the CLI against full-document fixtures, logs draws, and summarizes refusal counts with a pass/fail verdict.
tool-test-action-on-github/​README.md	Adds “Phase 1b” documentation and example invocations for the new rate-check script.
AGENTS.md	Adds release checklist step 4b to run the rate-check at the tag before moving floating tags.
CHANGELOG.md	Records the addition of rate-check.sh under [Unreleased].
.dev/​log/​2026-09-23-close-out.md	Session log documenting the decision and validation results for the new rate-check step.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mmcky]

Fixed in 7a2c3d1 — a real bug, thank you. SUMMARY had become a local inside summarize_run when the tally moved into a function, and the closing line still read the global, so under set -u a passing full run would have exited 1. It is set globally in the function now. My earlier validation only re-ran --summarize after that refactor, which exits before this line; this time the full-run path was exercised with one real draw (--languages ml --draws 1): 0/1, closing line printed, exit 0.