From a8fbe61bb6897363167788ac0b0a6c1a2b8dee24 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 23 Sep 2026 13:24:32 +1000 Subject: [PATCH 1/2] =?UTF-8?q?harness:=20rate-check.sh=20=E2=80=94=20N-dr?= =?UTF-8?q?aw=20refusal-rate=20check=20on=20the=20full-document=20fixtures?= =?UTF-8?q?=20(release=20step=204b)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate takes one draw per scenario and passed a ~40% defect three times. This runs the CLI at a tag or checkout N times per full-document fixture and language, prints refusals per cell with reasons, and fails above a threshold. Validated: v0.29.2 0/12, v0.29.0 4/12 (FAIL) on game-theory.md for ml. Closes #320. Co-Authored-By: Claude Fable 5.1 --- .dev/log/2026-09-23-close-out.md | 13 ++ AGENTS.md | 13 ++ CHANGELOG.md | 4 + tool-test-action-on-github/README.md | 9 + tool-test-action-on-github/rate-check.sh | 204 +++++++++++++++++++++++ 5 files changed, 243 insertions(+) create mode 100644 .dev/log/2026-09-23-close-out.md create mode 100755 tool-test-action-on-github/rate-check.sh diff --git a/.dev/log/2026-09-23-close-out.md b/.dev/log/2026-09-23-close-out.md new file mode 100644 index 0000000..5f8e50d --- /dev/null +++ b/.dev/log/2026-09-23-close-out.md @@ -0,0 +1,13 @@ +# 2026-09-23 — close-out of the 2026-09-21 session: #321 fixed, #320 decided and scripted + +**Trigger**: working the session-leftovers board for the 2026-09-21 Malayalam session. Nothing had moved on the editor's side (ml#23, ml#24, ml#14 all unreviewed); #319 was still merge-ready (CLEAN, #314 still a draft). + +**#321 → #322.** The harness reset's three `gh pr list` calls had no `--limit` (default 30). Fix: `--limit 200` on all three, plus a post-reset check that re-lists every repo and stops the run with the survivors' numbers if any is still open — a failed listing counts as a failure now (the target loop's `2>/dev/null || echo ""` had made it read as "nothing to close"), and so would a close that did not take. README and CHANGELOG. Not run end-to-end: the harness sits at `@v0` with one PR per repo, so a run would not exercise the >30 path; the next gate is its first real exercise. + +**#320 — decision (Matt): the local N-draw rate check.** The gate stays one action-path draw per scenario; §4a gains step **4b**: `tool-test-action-on-github/rate-check.sh [--ref vX.Y.Z]` runs the CLI N (default 12) times per full-document fixture and language, counts refusals (no output file), prints the table with the distinct refusal reasons, writes `meta.txt` + `summary.txt` under `.dev/scratch/rate-check/`, and exits 1 above `--max-refusals` (default 1). `--ref` builds any version in a temporary worktree with its own glossary (cwd there, so the CLI loads *that* glossary); languages derive from the main script's `LANGUAGES` array; fixtures default to the one file the scenarios introduce as a new document (`game-theory.md`, scenario 17). `--summarize DIR` re-reads a run. The in-action retry on parity refusal was not adopted: production behaviour change, and it would hide the rate this step exists to see. + +**Validated** (ml only, 12 draws each): v0.29.2 → 0/12, exit 0; `--ref v0.29.0` → 4/12 (3× `[exercise-start]`, 1× `[exercise]`), exit 1. The first v0.29.0 run printed FAIL and then exited 0 — `FAIL=1` was being set inside a `{ … } | tee` pipeline, i.e. a subshell. The tally is now a function run in the current shell; `--summarize` on the same run directories confirmed both exit codes without new draws. Lesson recorded in the script's comment. + +**Housekeeping**: a `git worktree` at `…/e473c662…/scratchpad/at-main` (detached at `5f74d74`) belongs to another session and was left alone. + +**Next**: Matt merges #319, #322 and the rate-check PR; AGENTS 4b is then part of every release. Waiting on the editor: ml#23 (round 4), ml#24, ml#14. Then the rule-2 arm. diff --git a/AGENTS.md b/AGENTS.md index 612579e..cf2f9b2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -238,6 +238,19 @@ Before creating a release, verify the following: The harness defaults to `main`, so the release itself is only gated if you ask for it by name. This step verifies the **tagged bytes**, not a branch that may have moved since. +4b. **Rate-check the full-document fixtures at the tag** — alongside 4a, before moving the floating tags: + + ```bash + ./tool-test-action-on-github/rate-check.sh --ref vX.Y.Z + ``` + + The gate takes one draw per scenario, so it cannot see a model-side defect that fails + less than always: scenario 17's new document was refused on ~40% of draws from v0.28.0 to + v0.29.0 and passed three gates (#320). This runs the CLI at the tag twelve times per + full-document fixture and language and fails if any cell shows more than one refusal. + Local only, ~36 translations against a cached prompt; the summary goes on the release PR + beside the gate tally. + 5. **Commit, tag, push** — commit all changes, create git tag `vX.Y.Z`, push with `--tags`; **then move both floating tags** to the release commit: ```bash diff --git a/CHANGELOG.md b/CHANGELOG.md index ba26c6d..a39ab23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **`tool-test-action-on-github/rate-check.sh`** — release checklist step 4b (#320): the release gate takes one draw per scenario, so a model-side defect that fails less than always can pass it (scenario 17's new document was refused on ~40% of draws from v0.28.0 to v0.29.0 and passed three gates). The script runs the CLI at a tag or checkout N times (default 12) per full-document fixture and language, prints refusals per cell with their reasons, and fails above `--max-refusals` (default 1). Local only; `--ref` builds any version in a temporary worktree with its own glossary. + ### Fixed - **Harness reset no longer misses open PRs beyond the first page** (#321): the three `gh pr list` calls in `test-action-on-github.sh` had no `--limit`, so a repo with more than 30 open PRs kept its oldest through the reset — on 2026-09-21 the `.ml` lane had 31 after diagnosis work between two gates, and the survivor read as a missing verdict in the v0.29.2 tally. Now `--limit 200`, and a post-reset check that stops the run if any repo still has an open PR (a failed listing counts too: it no longer reads as "nothing to close"). diff --git a/tool-test-action-on-github/README.md b/tool-test-action-on-github/README.md index 55f9346..63fdf91 100644 --- a/tool-test-action-on-github/README.md +++ b/tool-test-action-on-github/README.md @@ -84,6 +84,15 @@ The script will: Both source and target PRs remain **open** for evaluation. +### Phase 1b: Rate-check the full-document fixtures + +```bash +./tool-test-action-on-github/rate-check.sh --ref v0.29.2 # at a tag (release checklist 4b) +./tool-test-action-on-github/rate-check.sh --languages ml --draws 12 # this checkout's build +``` + +Phase 1 takes **one draw per scenario**. That is the right test of whether the tagged bytes deliver through the action path, but a model-side defect that fails less than always slips through it: scenario 17's new document (`game-theory.md`) was refused by the structural-parity guard on about 40% of draws from v0.28.0 to v0.29.0 and passed three release gates before the rate rose enough to be caught (QuantEcon/action-translation#320). `rate-check.sh` measures the rate directly: it runs the CLI N times (default 12) per fixture and language, counts the draws that produced no output file, prints a table with the refusal reasons, and exits non-zero if any cell shows more than `--max-refusals` (default 1). It costs no GitHub runs — N × fixtures × languages CLI translations against a cached prompt — and `--ref` builds any tag or branch in a temporary worktree with that version's own glossary, so an old release can be measured too. Fixtures default to the files the scenarios introduce as whole new documents (only `game-theory.md` today); add to `--fixtures` when a scenario adds another. Logs, `meta.txt` and `summary.txt` land in `.dev/scratch/rate-check/-/`; `--summarize ` re-prints a finished run's table and verdict without new draws. Validated on 2026-09-23: v0.29.2 reads 0/12 on `game-theory.md` for `ml`, v0.29.0 (via `--ref`) reads 4/12 and fails — the defect that passed three gates. + ### Phase 2: Evaluate Translation Quality ```bash diff --git a/tool-test-action-on-github/rate-check.sh b/tool-test-action-on-github/rate-check.sh new file mode 100755 index 0000000..e7ce1a7 --- /dev/null +++ b/tool-test-action-on-github/rate-check.sh @@ -0,0 +1,204 @@ +#!/usr/bin/env bash +# rate-check.sh — N-draw refusal-rate check on the full-document fixtures (release checklist 4b). +# +# The release gate (test-action-on-github.sh) takes ONE draw per scenario through the action +# path, so a model-side defect that fails less than always can pass it: scenario 17's fixture +# was refused by the structural-parity guard on ~40% of draws from v0.28.0 to v0.29.0 and +# passed three gates (#320). This script measures the rate directly. It runs the CLI at the +# requested engine N times per fixture and language, counts the draws the guard refused (no +# output file written), and fails if any cell shows more than --max-refusals. No GitHub runs: +# the cost is draws x fixtures x languages CLI translations against a cached prompt. +# +# Usage: ./tool-test-action-on-github/rate-check.sh [--ref ] [--draws N] +# [--languages ml,fa,zh-cn] [--fixtures game-theory.md] [--parallel J] +# [--max-refusals K] [--model M] +# +# --ref engine to test. Default: this checkout's dist/ and glossary (build first). +# With a ref, a temporary worktree at that ref is compiled with `npx tsc` +# and its own glossary is used, so old versions can be measured too. +# --draws draws per fixture x language (default 12 — six per arm gave the right +# direction and the wrong story on 2026-09-21) +# --languages comma list; default: every language in test-action-on-github.sh +# --fixtures comma list of files in test-action-on-github-data/ that the gate introduces +# as NEW documents (default: game-theory.md, scenario 17) +# --parallel concurrent CLI runs (default 6) +# --max-refusals most refusals a cell may show and still pass (default 1) +# --model Claude model (default claude-sonnet-5, the production default) +# --summarize DIR re-read a finished run directory (no new draws) and print its table +# +# Per-draw logs, meta.txt and summary.txt land in .dev/scratch/rate-check/-/. +# Exit 0 = every cell within --max-refusals; 1 = a cell over it; 2 = usage or setup error. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +DATA_DIR="$SCRIPT_DIR/test-action-on-github-data" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +MAIN_SCRIPT="$SCRIPT_DIR/test-action-on-github.sh" + +REF=""; DRAWS=12; LANGS=""; FIXTURES="game-theory.md"; PARALLEL=6; MAX_REFUSALS=1; MODEL="claude-sonnet-5"; SUMMARIZE="" +while [ $# -gt 0 ]; do + case "$1" in + --ref) REF="$2"; shift 2 ;; + --draws) DRAWS="$2"; shift 2 ;; + --languages) LANGS="$2"; shift 2 ;; + --fixtures) FIXTURES="$2"; shift 2 ;; + --parallel) PARALLEL="$2"; shift 2 ;; + --max-refusals) MAX_REFUSALS="$2"; shift 2 ;; + --model) MODEL="$2"; shift 2 ;; + --summarize) SUMMARIZE="$2"; shift 2 ;; + -h|--help) sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;; + esac +done + +# Tally a run directory: one line per fixture x language, the distinct refusal reasons under +# any cell with refusals, written to summary.txt. Runs in the current shell on purpose — a +# `{ … } | tee` here would put FAIL in a subshell and the exit code would always be 0 (it was, +# on the first validation run). +summarize_run() { + local summary="$RUN_DIR/summary.txt" lang fx i out refused reasons rate verdict + FAIL=0 + { + echo "engine ${ENGINE_TAG:-(untagged)} $ENGINE_SHA · model $MODEL · $DRAWS draws per cell · threshold $MAX_REFUSALS" + echo "" + printf '%-24s %-8s %-14s %-6s %s\n' fixture language refused/draws rate verdict + for lang in "${LANG_LIST[@]}"; do + for fx in "${FIXTURE_LIST[@]}"; do + refused=0 + reasons="$RUN_DIR/$lang/${fx%.md}/reasons.txt" + : > "$reasons" + for i in $(seq 1 "$DRAWS"); do + out="$RUN_DIR/$lang/${fx%.md}/$i" + if [ ! -f "$out/$fx" ]; then + refused=$((refused+1)) + sed 's/\x1b\[[0-9;]*m//g' "$out.log" | grep -m1 -E "differs|parity|Failed|Error|error" | sed 's/^[[:space:]]*//' >> "$reasons" || echo "(no reason line in log)" >> "$reasons" + fi + done + rate=$(( refused * 100 / DRAWS )) + if [ "$refused" -gt "$MAX_REFUSALS" ]; then verdict="FAIL"; FAIL=1; else verdict="ok"; fi + printf '%-24s %-8s %-14s %-6s %s\n' "$fx" "$lang" "$refused/$DRAWS" "${rate}%" "$verdict" + if [ "$refused" -gt 0 ]; then + sort "$reasons" | uniq -c | sort -rn | sed 's/^/ /' + fi + done + done + } > "$summary" + cat "$summary" +} + +# --summarize: re-read a finished run (its meta.txt carries what the table needs) and exit +# with the same verdict the run would have given. No API key, no draws. +if [ -n "$SUMMARIZE" ]; then + RUN_DIR="$SUMMARIZE" + [ -f "$RUN_DIR/meta.txt" ] || { echo "no meta.txt in $RUN_DIR" >&2; exit 2; } + # shellcheck disable=SC1091 + . "$RUN_DIR/meta.txt" + IFS=',' read -ra LANG_LIST <<< "$LANGS" + IFS=',' read -ra FIXTURE_LIST <<< "$FIXTURES" + summarize_run + [ "$FAIL" -eq 1 ] && exit 1 + exit 0 +fi + +: "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY must be set}" +command -v node >/dev/null || { echo "node not found" >&2; exit 2; } + +# Languages come from the main script's LANGUAGES array ("code|Name" per line), so there is +# one list to maintain. +if [ -z "$LANGS" ]; then + LANGS=$(awk '/^LANGUAGES=\(/{f=1;next} f&&/^\)/{exit} f{gsub(/[" ]/,""); sub(/\|.*/,""); print}' "$MAIN_SCRIPT" | paste -sd, -) +fi +IFS=',' read -ra LANG_LIST <<< "$LANGS" +IFS=',' read -ra FIXTURE_LIST <<< "$FIXTURES" +for fx in "${FIXTURE_LIST[@]}"; do + [ -f "$DATA_DIR/$fx" ] || { echo "fixture not found: $DATA_DIR/$fx" >&2; exit 2; } +done + +# Engine under test: this checkout, or a worktree at --ref compiled in place. +WORKTREE="" +SRC_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rate-check-src.XXXXXX")" +cleanup() { + rm -rf "$SRC_DIR" + if [ -n "$WORKTREE" ]; then + rm -f "$WORKTREE/node_modules" + git -C "$REPO_ROOT" worktree remove --force "$WORKTREE" >/dev/null 2>&1 || true + fi +} +trap cleanup EXIT + +ENGINE_DIR="$REPO_ROOT" +if [ -n "$REF" ]; then + WORKTREE="${TMPDIR:-/tmp}/rate-check-wt.$$" + git -C "$REPO_ROOT" worktree add -q "$WORKTREE" "$REF" + ln -s "$REPO_ROOT/node_modules" "$WORKTREE/node_modules" + echo "Compiling $REF in a temporary worktree..." + (cd "$WORKTREE" && npx tsc) + ENGINE_DIR="$WORKTREE" +fi +CLI="$ENGINE_DIR/dist/cli/index.js" +[ -f "$CLI" ] || { echo "no $CLI — run 'npm run build' first (or pass --ref)" >&2; exit 2; } +ENGINE_SHA=$(git -C "$ENGINE_DIR" rev-parse --short HEAD) +ENGINE_TAG=$(git -C "$ENGINE_DIR" tag --points-at HEAD | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1 || true) +ENGINE_VER=$(node -p "require('$ENGINE_DIR/package.json').version") + +# Source directory: the fixtures plus a TOC that lists them (init copies non-markdown files +# and translates only the -f file). +for fx in "${FIXTURE_LIST[@]}"; do cp "$DATA_DIR/$fx" "$SRC_DIR/"; done +{ + echo "format: jb-book" + echo "root: ${FIXTURE_LIST[0]%.md}" + if [ "${#FIXTURE_LIST[@]}" -gt 1 ]; then + echo "chapters:" + for fx in "${FIXTURE_LIST[@]:1}"; do echo " - file: ${fx%.md}"; done + fi +} > "$SRC_DIR/_toc.yml" + +STAMP="$(date -u +%Y-%m-%dT%H-%M-%SZ)-$$" # PID keeps two runs in one second apart +RUN_DIR="$REPO_ROOT/.dev/scratch/rate-check/$STAMP" +mkdir -p "$RUN_DIR" +{ + echo "ENGINE_TAG='${ENGINE_TAG}'"; echo "ENGINE_SHA='${ENGINE_SHA}'"; echo "MODEL='${MODEL}'" + echo "DRAWS='${DRAWS}'"; echo "MAX_REFUSALS='${MAX_REFUSALS}'"; echo "LANGS='${LANGS}'"; echo "FIXTURES='${FIXTURES}'" +} > "$RUN_DIR/meta.txt" + +TOTAL=$(( DRAWS * ${#FIXTURE_LIST[@]} * ${#LANG_LIST[@]} )) +echo "========================================" +echo "Full-document refusal-rate check" +echo "========================================" +echo " engine ${ENGINE_TAG:-(untagged)} ${ENGINE_SHA} package.json ${ENGINE_VER}${REF:+ (from --ref $REF)}" +echo " model $MODEL" +echo " fixtures ${FIXTURES}" +echo " languages ${LANGS}" +echo " draws $DRAWS per cell — $TOTAL CLI translations, $PARALLEL at a time" +echo " threshold a cell fails above $MAX_REFUSALS refusal(s)" +echo " logs $RUN_DIR" +echo "" + +run_one() { + local lang=$1 fx=$2 i=$3 + local out="$RUN_DIR/$lang/${fx%.md}/$i" + mkdir -p "$(dirname "$out")" + # cwd is the engine dir so the CLI picks up THAT checkout's glossary. + (cd "$ENGINE_DIR" && node "$CLI" init -s "$SRC_DIR" -t "$out" --target-language "$lang" \ + -d . -f "$fx" --localize none -m "$MODEL" > "$out.log" 2>&1) || true +} + +for lang in "${LANG_LIST[@]}"; do + for fx in "${FIXTURE_LIST[@]}"; do + for i in $(seq 1 "$DRAWS"); do + while [ "$(jobs -rp | wc -l)" -ge "$PARALLEL" ]; do sleep 1; done + run_one "$lang" "$fx" "$i" & + done + done +done +wait + +summarize_run # sets FAIL; prints and writes summary.txt + +echo "" +if [ "$FAIL" -eq 1 ]; then + echo "✗ A cell is over the threshold. One draw per scenario would have passed this at ~$((100 - 100 * MAX_REFUSALS / DRAWS))% or better; the gate cannot see it — do not release on the gate alone." + exit 1 +fi +echo "✓ Every cell within threshold. Summary: $SUMMARY" From 7a2c3d1fc9e9e0e4219aac08a8b08d467fb8b6da Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 23 Sep 2026 14:02:42 +1000 Subject: [PATCH 2/2] =?UTF-8?q?harness:=20rate-check.sh=20=E2=80=94=20SUMM?= =?UTF-8?q?ARY=20is=20global=20again=20(a=20passing=20run=20exited=201=20u?= =?UTF-8?q?nder=20set=20-u),=20--summarize=20in=20the=20synopsis,=20README?= =?UTF-8?q?=20paths=20match=20the=20cd=20above?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- tool-test-action-on-github/README.md | 4 ++-- tool-test-action-on-github/rate-check.sh | 9 +++++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/tool-test-action-on-github/README.md b/tool-test-action-on-github/README.md index 63fdf91..cf47d26 100644 --- a/tool-test-action-on-github/README.md +++ b/tool-test-action-on-github/README.md @@ -87,8 +87,8 @@ Both source and target PRs remain **open** for evaluation. ### Phase 1b: Rate-check the full-document fixtures ```bash -./tool-test-action-on-github/rate-check.sh --ref v0.29.2 # at a tag (release checklist 4b) -./tool-test-action-on-github/rate-check.sh --languages ml --draws 12 # this checkout's build +./rate-check.sh --ref v0.29.2 # at a tag (release checklist 4b) +./rate-check.sh --languages ml --draws 12 # this checkout's build ``` Phase 1 takes **one draw per scenario**. That is the right test of whether the tagged bytes deliver through the action path, but a model-side defect that fails less than always slips through it: scenario 17's new document (`game-theory.md`) was refused by the structural-parity guard on about 40% of draws from v0.28.0 to v0.29.0 and passed three release gates before the rate rose enough to be caught (QuantEcon/action-translation#320). `rate-check.sh` measures the rate directly: it runs the CLI N times (default 12) per fixture and language, counts the draws that produced no output file, prints a table with the refusal reasons, and exits non-zero if any cell shows more than `--max-refusals` (default 1). It costs no GitHub runs — N × fixtures × languages CLI translations against a cached prompt — and `--ref` builds any tag or branch in a temporary worktree with that version's own glossary, so an old release can be measured too. Fixtures default to the files the scenarios introduce as whole new documents (only `game-theory.md` today); add to `--fixtures` when a scenario adds another. Logs, `meta.txt` and `summary.txt` land in `.dev/scratch/rate-check/-/`; `--summarize ` re-prints a finished run's table and verdict without new draws. Validated on 2026-09-23: v0.29.2 reads 0/12 on `game-theory.md` for `ml`, v0.29.0 (via `--ref`) reads 4/12 and fails — the defect that passed three gates. diff --git a/tool-test-action-on-github/rate-check.sh b/tool-test-action-on-github/rate-check.sh index e7ce1a7..acc3adb 100755 --- a/tool-test-action-on-github/rate-check.sh +++ b/tool-test-action-on-github/rate-check.sh @@ -11,7 +11,7 @@ # # Usage: ./tool-test-action-on-github/rate-check.sh [--ref ] [--draws N] # [--languages ml,fa,zh-cn] [--fixtures game-theory.md] [--parallel J] -# [--max-refusals K] [--model M] +# [--max-refusals K] [--model M] [--summarize DIR] # # --ref engine to test. Default: this checkout's dist/ and glossary (build first). # With a ref, a temporary worktree at that ref is compiled with `npx tsc` @@ -57,7 +57,8 @@ done # `{ … } | tee` here would put FAIL in a subshell and the exit code would always be 0 (it was, # on the first validation run). summarize_run() { - local summary="$RUN_DIR/summary.txt" lang fx i out refused reasons rate verdict + local lang fx i out refused reasons rate verdict + SUMMARY="$RUN_DIR/summary.txt" # global: the closing line reports it FAIL=0 { echo "engine ${ENGINE_TAG:-(untagged)} $ENGINE_SHA · model $MODEL · $DRAWS draws per cell · threshold $MAX_REFUSALS" @@ -83,8 +84,8 @@ summarize_run() { fi done done - } > "$summary" - cat "$summary" + } > "$SUMMARY" + cat "$SUMMARY" } # --summarize: re-read a finished run (its meta.txt carries what the table needs) and exit