QuantEcon · mmcky · Sep 23, 2026 · Sep 23, 2026 · Sep 23, 2026 · mmcky
diff --git a/.dev/log/2026-09-23-close-out.md b/.dev/log/2026-09-23-close-out.md
@@ -0,0 +1,13 @@
+# 2026-09-23 — close-out of the 2026-09-21 session: #321 fixed, #320 decided and scripted
+
+**Trigger**: working the session-leftovers board for the 2026-09-21 Malayalam session. Nothing had moved on the editor's side (ml#23, ml#24, ml#14 all unreviewed); #319 was still merge-ready (CLEAN, #314 still a draft).
+
+**#321 → #322.** The harness reset's three `gh pr list` calls had no `--limit` (default 30). Fix: `--limit 200` on all three, plus a post-reset check that re-lists every repo and stops the run with the survivors' numbers if any is still open — a failed listing counts as a failure now (the target loop's `2>/dev/null || echo ""` had made it read as "nothing to close"), and so would a close that did not take. README and CHANGELOG. Not run end-to-end: the harness sits at `@v0` with one PR per repo, so a run would not exercise the >30 path; the next gate is its first real exercise.
+
+**#320 — decision (Matt): the local N-draw rate check.** The gate stays one action-path draw per scenario; §4a gains step **4b**: `tool-test-action-on-github/rate-check.sh [--ref vX.Y.Z]` runs the CLI N (default 12) times per full-document fixture and language, counts refusals (no output file), prints the table with the distinct refusal reasons, writes `meta.txt` + `summary.txt` under `.dev/scratch/rate-check/`, and exits 1 above `--max-refusals` (default 1). `--ref` builds any version in a temporary worktree with its own glossary (cwd there, so the CLI loads *that* glossary); languages derive from the main script's `LANGUAGES` array; fixtures default to the one file the scenarios introduce as a new document (`game-theory.md`, scenario 17). `--summarize DIR` re-reads a run. The in-action retry on parity refusal was not adopted: production behaviour change, and it would hide the rate this step exists to see.
+
+**Validated** (ml only, 12 draws each): v0.29.2 → 0/12, exit 0; `--ref v0.29.0` → 4/12 (3× `[exercise-start]`, 1× `[exercise]`), exit 1. The first v0.29.0 run printed FAIL and then exited 0 — `FAIL=1` was being set inside a `{ … } | tee` pipeline, i.e. a subshell. The tally is now a function run in the current shell; `--summarize` on the same run directories confirmed both exit codes without new draws. Lesson recorded in the script's comment.
+
+**Housekeeping**: a `git worktree` at `…/e473c662…/scratchpad/at-main` (detached at `5f74d74`) belongs to another session and was left alone.
+
+**Next**: Matt merges #319, #322 and the rate-check PR; AGENTS 4b is then part of every release. Waiting on the editor: ml#23 (round 4), ml#24, ml#14. Then the rule-2 arm.
diff --git a/AGENTS.md b/AGENTS.md
@@ -238,6 +238,19 @@ Before creating a release, verify the following:
 
    The harness defaults to `main`, so the release itself is only gated if you ask for it by
    name. This step verifies the **tagged bytes**, not a branch that may have moved since.
+4b. **Rate-check the full-document fixtures at the tag** — alongside 4a, before moving the floating tags:
+
+   ```bash
+   ./tool-test-action-on-github/rate-check.sh --ref vX.Y.Z
+   ```
+
+   The gate takes one draw per scenario, so it cannot see a model-side defect that fails
+   less than always: scenario 17's new document was refused on ~40% of draws from v0.28.0 to
+   v0.29.0 and passed three gates (#320). This runs the CLI at the tag twelve times per
+   full-document fixture and language and fails if any cell shows more than one refusal.
+   Local only, ~36 translations against a cached prompt; the summary goes on the release PR
+   beside the gate tally.
+
 5. **Commit, tag, push** — commit all changes, create git tag `vX.Y.Z`, push with `--tags`; **then move both floating tags** to the release commit:
 
    ```bash

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **`tool-test-action-on-github/rate-check.sh`** — release checklist step 4b (#320): the release gate takes one draw per scenario, so a model-side defect that fails less than always can pass it (scenario 17's new document was refused on ~40% of draws from v0.28.0 to v0.29.0 and passed three gates). The script runs the CLI at a tag or checkout N times (default 12) per full-document fixture and language, prints refusals per cell with their reasons, and fails above `--max-refusals` (default 1). Local only; `--ref` builds any version in a temporary worktree with its own glossary.
+
 ### Fixed
 
 - **Harness reset no longer misses open PRs beyond the first page** (#321): the three `gh pr list` calls in `test-action-on-github.sh` had no `--limit`, so a repo with more than 30 open PRs kept its oldest through the reset — on 2026-09-21 the `.ml` lane had 31 after diagnosis work between two gates, and the survivor read as a missing verdict in the v0.29.2 tally. Now `--limit 200`, and a post-reset check that stops the run if any repo still has an open PR (a failed listing counts too: it no longer reads as "nothing to close").

diff --git a/tool-test-action-on-github/README.md b/tool-test-action-on-github/README.md
@@ -84,6 +84,15 @@ The script will:
 
 Both source and target PRs remain **open** for evaluation.
 
+### Phase 1b: Rate-check the full-document fixtures
+
+```bash
+./rate-check.sh --ref v0.29.2               # at a tag (release checklist 4b)
+./rate-check.sh --languages ml --draws 12   # this checkout's build
+```
+
+Phase 1 takes **one draw per scenario**. That is the right test of whether the tagged bytes deliver through the action path, but a model-side defect that fails less than always slips through it: scenario 17's new document (`game-theory.md`) was refused by the structural-parity guard on about 40% of draws from v0.28.0 to v0.29.0 and passed three release gates before the rate rose enough to be caught (QuantEcon/action-translation#320). `rate-check.sh` measures the rate directly: it runs the CLI N times (default 12) per fixture and language, counts the draws that produced no output file, prints a table with the refusal reasons, and exits non-zero if any cell shows more than `--max-refusals` (default 1). It costs no GitHub runs — N × fixtures × languages CLI translations against a cached prompt — and `--ref` builds any tag or branch in a temporary worktree with that version's own glossary, so an old release can be measured too. Fixtures default to the files the scenarios introduce as whole new documents (only `game-theory.md` today); add to `--fixtures` when a scenario adds another. Logs, `meta.txt` and `summary.txt` land in `.dev/scratch/rate-check/<timestamp>-<pid>/`; `--summarize <that dir>` re-prints a finished run's table and verdict without new draws. Validated on 2026-09-23: v0.29.2 reads 0/12 on `game-theory.md` for `ml`, v0.29.0 (via `--ref`) reads 4/12 and fails — the defect that passed three gates.
+
 ### Phase 2: Evaluate Translation Quality
 
 ```bash

diff --git a/tool-test-action-on-github/rate-check.sh b/tool-test-action-on-github/rate-check.sh
@@ -0,0 +1,205 @@
+#!/usr/bin/env bash
+# rate-check.sh — N-draw refusal-rate check on the full-document fixtures (release checklist 4b).
+#
+# The release gate (test-action-on-github.sh) takes ONE draw per scenario through the action
+# path, so a model-side defect that fails less than always can pass it: scenario 17's fixture
+# was refused by the structural-parity guard on ~40% of draws from v0.28.0 to v0.29.0 and
+# passed three gates (#320). This script measures the rate directly. It runs the CLI at the
+# requested engine N times per fixture and language, counts the draws the guard refused (no
+# output file written), and fails if any cell shows more than --max-refusals. No GitHub runs:
+# the cost is draws x fixtures x languages CLI translations against a cached prompt.
+#
+# Usage: ./tool-test-action-on-github/rate-check.sh [--ref <tag|branch>] [--draws N]
+#            [--languages ml,fa,zh-cn] [--fixtures game-theory.md] [--parallel J]
+#            [--max-refusals K] [--model M] [--summarize DIR]
+#
+#   --ref            engine to test. Default: this checkout's dist/ and glossary (build first).
+#                    With a ref, a temporary worktree at that ref is compiled with `npx tsc`
+#                    and its own glossary is used, so old versions can be measured too.
+#   --draws          draws per fixture x language (default 12 — six per arm gave the right
+#                    direction and the wrong story on 2026-09-21)
+#   --languages      comma list; default: every language in test-action-on-github.sh
+#   --fixtures       comma list of files in test-action-on-github-data/ that the gate introduces
+#                    as NEW documents (default: game-theory.md, scenario 17)
+#   --parallel       concurrent CLI runs (default 6)
+#   --max-refusals   most refusals a cell may show and still pass (default 1)
+#   --model          Claude model (default claude-sonnet-5, the production default)
+#   --summarize DIR  re-read a finished run directory (no new draws) and print its table
+#
+# Per-draw logs, meta.txt and summary.txt land in .dev/scratch/rate-check/<UTC timestamp>-<pid>/.
+# Exit 0 = every cell within --max-refusals; 1 = a cell over it; 2 = usage or setup error.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DATA_DIR="$SCRIPT_DIR/test-action-on-github-data"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+MAIN_SCRIPT="$SCRIPT_DIR/test-action-on-github.sh"
+
+REF=""; DRAWS=12; LANGS=""; FIXTURES="game-theory.md"; PARALLEL=6; MAX_REFUSALS=1; MODEL="claude-sonnet-5"; SUMMARIZE=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --ref) REF="$2"; shift 2 ;;
+        --draws) DRAWS="$2"; shift 2 ;;
+        --languages) LANGS="$2"; shift 2 ;;
+        --fixtures) FIXTURES="$2"; shift 2 ;;
+        --parallel) PARALLEL="$2"; shift 2 ;;
+        --max-refusals) MAX_REFUSALS="$2"; shift 2 ;;
+        --model) MODEL="$2"; shift 2 ;;
+        --summarize) SUMMARIZE="$2"; shift 2 ;;
+        -h|--help) sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+# Tally a run directory: one line per fixture x language, the distinct refusal reasons under
+# any cell with refusals, written to summary.txt. Runs in the current shell on purpose — a
+# `{ … } | tee` here would put FAIL in a subshell and the exit code would always be 0 (it was,
+# on the first validation run).
+summarize_run() {
+    local lang fx i out refused reasons rate verdict
+    SUMMARY="$RUN_DIR/summary.txt"   # global: the closing line reports it
+    FAIL=0
+    {
+        echo "engine ${ENGINE_TAG:-(untagged)} $ENGINE_SHA · model $MODEL · $DRAWS draws per cell · threshold $MAX_REFUSALS"
+        echo ""
+        printf '%-24s %-8s %-14s %-6s %s\n' fixture language refused/draws rate verdict
+        for lang in "${LANG_LIST[@]}"; do
+            for fx in "${FIXTURE_LIST[@]}"; do
+                refused=0
+                reasons="$RUN_DIR/$lang/${fx%.md}/reasons.txt"
+                : > "$reasons"
+                for i in $(seq 1 "$DRAWS"); do
+                    out="$RUN_DIR/$lang/${fx%.md}/$i"
+                    if [ ! -f "$out/$fx" ]; then
+                        refused=$((refused+1))
+                        sed 's/\x1b\[[0-9;]*m//g' "$out.log" | grep -m1 -E "differs|parity|Failed|Error|error" | sed 's/^[[:space:]]*//' >> "$reasons" || echo "(no reason line in log)" >> "$reasons"
+                    fi
+                done
+                rate=$(( refused * 100 / DRAWS ))
+                if [ "$refused" -gt "$MAX_REFUSALS" ]; then verdict="FAIL"; FAIL=1; else verdict="ok"; fi
+                printf '%-24s %-8s %-14s %-6s %s\n' "$fx" "$lang" "$refused/$DRAWS" "${rate}%" "$verdict"
+                if [ "$refused" -gt 0 ]; then
+                    sort "$reasons" | uniq -c | sort -rn | sed 's/^/      /'
+                fi
+            done
+        done
+    } > "$SUMMARY"
+    cat "$SUMMARY"
+}
+
+# --summarize: re-read a finished run (its meta.txt carries what the table needs) and exit
+# with the same verdict the run would have given. No API key, no draws.
+if [ -n "$SUMMARIZE" ]; then
+    RUN_DIR="$SUMMARIZE"
+    [ -f "$RUN_DIR/meta.txt" ] || { echo "no meta.txt in $RUN_DIR" >&2; exit 2; }
+    # shellcheck disable=SC1091
+    . "$RUN_DIR/meta.txt"
+    IFS=',' read -ra LANG_LIST <<< "$LANGS"
+    IFS=',' read -ra FIXTURE_LIST <<< "$FIXTURES"
+    summarize_run
+    [ "$FAIL" -eq 1 ] && exit 1
+    exit 0
+fi
+
+: "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY must be set}"
+command -v node >/dev/null || { echo "node not found" >&2; exit 2; }
+
+# Languages come from the main script's LANGUAGES array ("code|Name" per line), so there is
+# one list to maintain.
+if [ -z "$LANGS" ]; then
+    LANGS=$(awk '/^LANGUAGES=\(/{f=1;next} f&&/^\)/{exit} f{gsub(/[" ]/,""); sub(/\|.*/,""); print}' "$MAIN_SCRIPT" | paste -sd, -)
+fi
+IFS=',' read -ra LANG_LIST <<< "$LANGS"
+IFS=',' read -ra FIXTURE_LIST <<< "$FIXTURES"
+for fx in "${FIXTURE_LIST[@]}"; do
+    [ -f "$DATA_DIR/$fx" ] || { echo "fixture not found: $DATA_DIR/$fx" >&2; exit 2; }
+done
+
+# Engine under test: this checkout, or a worktree at --ref compiled in place.
+WORKTREE=""
+SRC_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rate-check-src.XXXXXX")"
+cleanup() {
+    rm -rf "$SRC_DIR"
+    if [ -n "$WORKTREE" ]; then
+        rm -f "$WORKTREE/node_modules"
+        git -C "$REPO_ROOT" worktree remove --force "$WORKTREE" >/dev/null 2>&1 || true
+    fi
+}
+trap cleanup EXIT
+
+ENGINE_DIR="$REPO_ROOT"
+if [ -n "$REF" ]; then
+    WORKTREE="${TMPDIR:-/tmp}/rate-check-wt.$$"
+    git -C "$REPO_ROOT" worktree add -q "$WORKTREE" "$REF"
+    ln -s "$REPO_ROOT/node_modules" "$WORKTREE/node_modules"
+    echo "Compiling $REF in a temporary worktree..."
+    (cd "$WORKTREE" && npx tsc)
+    ENGINE_DIR="$WORKTREE"
+fi
+CLI="$ENGINE_DIR/dist/cli/index.js"
+[ -f "$CLI" ] || { echo "no $CLI — run 'npm run build' first (or pass --ref)" >&2; exit 2; }
+ENGINE_SHA=$(git -C "$ENGINE_DIR" rev-parse --short HEAD)
+ENGINE_TAG=$(git -C "$ENGINE_DIR" tag --points-at HEAD | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1 || true)
+ENGINE_VER=$(node -p "require('$ENGINE_DIR/package.json').version")
+
+# Source directory: the fixtures plus a TOC that lists them (init copies non-markdown files
+# and translates only the -f file).
+for fx in "${FIXTURE_LIST[@]}"; do cp "$DATA_DIR/$fx" "$SRC_DIR/"; done
+{
+    echo "format: jb-book"
+    echo "root: ${FIXTURE_LIST[0]%.md}"
+    if [ "${#FIXTURE_LIST[@]}" -gt 1 ]; then
+        echo "chapters:"
+        for fx in "${FIXTURE_LIST[@]:1}"; do echo "  - file: ${fx%.md}"; done
+    fi
+} > "$SRC_DIR/_toc.yml"
+
+STAMP="$(date -u +%Y-%m-%dT%H-%M-%SZ)-$$"   # PID keeps two runs in one second apart
+RUN_DIR="$REPO_ROOT/.dev/scratch/rate-check/$STAMP"
+mkdir -p "$RUN_DIR"
+{
+    echo "ENGINE_TAG='${ENGINE_TAG}'"; echo "ENGINE_SHA='${ENGINE_SHA}'"; echo "MODEL='${MODEL}'"
+    echo "DRAWS='${DRAWS}'"; echo "MAX_REFUSALS='${MAX_REFUSALS}'"; echo "LANGS='${LANGS}'"; echo "FIXTURES='${FIXTURES}'"
+} > "$RUN_DIR/meta.txt"
+
+TOTAL=$(( DRAWS * ${#FIXTURE_LIST[@]} * ${#LANG_LIST[@]} ))
+echo "========================================"
+echo "Full-document refusal-rate check"
+echo "========================================"
+echo "  engine     ${ENGINE_TAG:-(untagged)} ${ENGINE_SHA} package.json ${ENGINE_VER}${REF:+ (from --ref $REF)}"
+echo "  model      $MODEL"
+echo "  fixtures   ${FIXTURES}"
+echo "  languages  ${LANGS}"
+echo "  draws      $DRAWS per cell — $TOTAL CLI translations, $PARALLEL at a time"
+echo "  threshold  a cell fails above $MAX_REFUSALS refusal(s)"
+echo "  logs       $RUN_DIR"
+echo ""
+
+run_one() {
+    local lang=$1 fx=$2 i=$3
+    local out="$RUN_DIR/$lang/${fx%.md}/$i"
+    mkdir -p "$(dirname "$out")"
+    # cwd is the engine dir so the CLI picks up THAT checkout's glossary.
+    (cd "$ENGINE_DIR" && node "$CLI" init -s "$SRC_DIR" -t "$out" --target-language "$lang" \
+        -d . -f "$fx" --localize none -m "$MODEL" > "$out.log" 2>&1) || true
+}
+
+for lang in "${LANG_LIST[@]}"; do
+    for fx in "${FIXTURE_LIST[@]}"; do
+        for i in $(seq 1 "$DRAWS"); do
+            while [ "$(jobs -rp | wc -l)" -ge "$PARALLEL" ]; do sleep 1; done
+            run_one "$lang" "$fx" "$i" &
+        done
+    done
+done
+wait
+
+summarize_run   # sets FAIL; prints and writes summary.txt
+
+echo ""
+if [ "$FAIL" -eq 1 ]; then
+    echo "✗ A cell is over the threshold. One draw per scenario would have passed this at ~$((100 - 100 * MAX_REFUSALS / DRAWS))% or better; the gate cannot see it — do not release on the gate alone."
+    exit 1
+fi
+echo "✓ Every cell within threshold. Summary: $SUMMARY"