Design: route sync PRs by review verdict (auto-merge vs editor label) and make human review measurable

[mmcky]

Context

The translation dashboard (status-translations) carried a "Human review" column that counted open issues labelled translation-review in each edition repo. We removed it on 2026-07-18 because it measured whether anyone had seeded labels, not whether review was happening: programming.fa rendered "—" (no labelled issues) while still needing widespread native review, and programming.zh-cn's "24 open" was a fixed April backlog with no burn-down signal. This issue is the design discussion for collecting a metric worth showing, and it lands here because the mechanisms below live in the engine.

Today's flow: sync opens a PR → review-translations.yml posts a PASS/WARN/FAIL quality report as a comment → a human merges. The verdict is purely advisory, and nothing in the pipeline records whether a human ever looked at the content.

Proposal A — route sync PRs by review verdict

Make the reviewer a gatekeeper instead of a commentator. On each auto-sync PR, the review classifies quality and the workflow acts on it:

• High quality → auto-merge (or auto-approve, leaving merge to branch protection), with the verdict recorded in the PR.
• Below threshold → apply an editor label and request human review.

The dashboard then counts open/closed editor-labelled PRs per edition — a live measure of human-review demand and burn-down exactly where the pipeline judged a human necessary, rather than a hand-seeded issue count.

Prerequisites and open questions:

• Verdict reliability is the gate for auto-merge. Review mode: missing criterion score from the model yields Overall NaN/10 and an automatic FAIL verdict #102 (a missing criterion score from the model becomes Overall NaN/10 → automatic FAIL) shows the verdict path needs schema validation and retries before any merge decision can hang off it. A false FAIL wastes an editor's time; a false PASS auto-merges a bad translation.
• Threshold policy: is PASS + zero syntax errors sufficient, or should auto-merge require a stricter bar (e.g. score ≥ 9 and no suggestions touching changed lines)?
• Interaction with required checks and with the rebase workflow when several syncs queue on one file (we hit exactly this across zh-cn/fa/fr this week — sequential source PRs touching the same lines, where the newer sync superseded the older).
• Provenance: auto-merged PRs should carry a clear marker (label such as auto-merged, plus the verdict in the merge commit or PR body) so a human can always audit what the machine merged.

Proposal B — labelled human audits of the existing corpus

Verdict routing only measures review of incoming changes. For the standing corpus, track deliberate human audits as issues with a dedicated label (one issue per lecture per campaign — the shape zh-cn's #25–#48 already have, formalised): a human-audit (or the existing translation-review) label that dashboards can filter on for coverage and burn-down per edition. Audit outcomes (error classes found: terminology drift, meaning inversion as in lecture-python-programming.fa#106, typography) feed back into glossary and prompt improvements — the audit label becomes the collection point for pipeline-quality data, not just a progress bar.

How they compose

A measures flow (are humans reviewing what the machine flags?); B measures stock (how much of the corpus has a human actually audited?). Both produce label-based signals a collector can read cheaply, and both should be specified in the public metadata/reporting contract (#66) rather than scraped ad hoc.

Origin: program discussion in QuantEcon/project-translation, 2026-07-18, following the sync-PR review pass across the three programming editions.

[mmcky]

The threshold for auto-merge should be configurable. For example requiring 9 or above. If the threshold is set to 10 then ALL PRs require human review (no automerge).

[mmcky]

Design session 2026-07-22 (program repo) settled the open questions here and adopted a staged delivery plan — recording the decisions so this issue reflects current direction. Program tracking: QuantEcon/project-translation#15 (private); Proposal B is now its own feature issue, #136.

Threshold policy (open question 2): the gate becomes a composite, not a scalar threshold. The weighted overall can average away exactly the failure that matters — a 7 on accuracy under 10s on fluency/formatting still clears 9.0, and the fluent-but-wrong class (meaning inversions, lecture-python-programming.fa#106) is what humans keep catching. Auto-merge requires ALL of: (1) deterministic diff risk class eligible (v1: prose-only edits inside existing sections; code cells, math, directives, anchors, headings, new sections/files, frontmatter and config never eligible regardless of score); (2) deterministic validators green (v0.20.0 structural parity guard, strict -n -W build, URL identity vs source hunk, glossary compliance); (3) a categorical verdict-v2 recommendation with per-criterion floors and zero findings ≥ minor severity on changed lines — floors set empirically from a shadow-mode phase (every sync PR logs "would auto-merge", precision measured against "human merged unchanged", target ≥99%) rather than a priori; (4) caps — max diff size, daily cap, and a breaker: any revert or human flag of an auto-merged PR trips the feature off for that repo until re-enabled. Config is per edition; enabled: false and maximal strictness are first-class states (per the earlier comment here). Fail-closed polarity as established by the #102 fix: malformed or missing verdict routes to a human, never to a merge.

Rebase/supersession (open question 3): resolved by an invariant — the verdict records the head SHA it reviewed, and merge requires verdict SHA == current head SHA. Any rebase or force-push invalidates the verdict until re-review completes (the post-#125 guard already re-triggers review on rebase-pushed heads). An older sync with a newer open sibling touching the same file can never satisfy it.

Provenance (open question 4): permanent auto-merged label, verdict block retained on the PR, merge-commit trailer naming gate version + verdict SHA, a recurring digest of auto-merges, and a dashboard lane showing per-edition mode (off / shadow / active / tripped) and spot-check pass rate. A sampled fraction of auto-merged PRs get retrospective spot-check review — a permanent live precision measurement. Maintainer visibility is a hard requirement of the program plan.

Verdict v2 (prerequisite work): today the review comment contains nothing machine-readable (only the dedup marker; the 0.7/0.3 composite is neither printed nor exposed as an output). v2 adds a structured block — per-criterion scores, findings with severity/category/file, categorical recommendation, reviewer model + engine version, reviewed head SHA — specified in #66.

Also decided: SYNC↔REVIEW model separation is to be introduced — both modes currently default to claude-sonnet-5 through the shared claude-model input (review never ran Opus; Opus is bulk-seed only per D-2026-07-14) — starting with a separate review-model input, then provider abstraction so review can run non-Anthropic models (Gemini a leading candidate; related: #2, #82), benchmarked before production. No back-translation in the gate (round-trip adds noise; translate backward stays human-driven) — instead, divergence reporting: an issue when significant target-side content appears in an edition. Editor routing via an editors.primary GitHub handle in .translate/config.yml (single primary editor during development; secondary RA pool at production; the existing pr-reviewers plumbing in pr-creator.ts is reusable). Note the config writers (init, setup, status --write-state) currently rebuild config.yml from three fields and would drop an editors: block — writeConfig needs a merge fix first.

Still open (bundled, decided during Stage 5 development): merge identity — the PAT that opens the PR should probably not be the identity that approves/merges it (couples to #61) — plus whether gate-passed PRs merge immediately or after a quiet delay window.

Sequencing (headline last, deliberately): labels/contract → editor routing → audits (manual, #136) → shadow gate fleet-wide → active auto-merge (harness-first, then programming.zh-cn) → scale/tune. Stages 1–4 generate the calibration data that makes Stage 5 defensible.

[mmcky]

Design-relevant data from #248 (replicate experiment, harness, v0.25.0): in 27 observed verdicts across two arms, the per-criterion floors never once bound — every score profile passed them and 100% of routing was decided by the minor-finding-in-gating-category rule. Score profiles are also quantised: the same lane+fixture yields byte-identical scores across different translated texts (9/8/9/10 in 18 of 18 ml verdicts), so floors behave as step functions and small adjustments can produce discontinuous routing changes. The reviewer's decision layer is deterministic on fixed text (13/13 identical). Implication for this issue's design: the gate's real tunable is severity/category assignment, not the floor values — the verdict-v2 rubric decision (findings gate, blend demoted) is validated, and the Stage 4 close is likely confirm-or-simplify floors plus severity-rubric calibration. The one instability found sits in finding descriptions, not decisions — filed as #249, deferred past the shadow window.

[mmcky]

Measured input to the gate design: the category assignment is doing the damage, not the severity threshold

The M0 certification (project-translation#28, report) put 140 reviews of 45 fixtures through the deployed gate at @v0/v0.26.0. Three results bear directly on this design.

1. computeRecommendation has no notion of what the PR changed, and that is the largest hole. The plan's Decision 2 — "eligible diff class v1 = prose-only edits inside existing sections; code cells, math, directives, labels/anchors, new sections, new files, frontmatter, _toc.yml/_config.yml/.translate/ are never eligible, regardless of score" — is not implemented anywhere. The function gates on verdict, syntax errors, diff checks, findings, source-missing, suppression and floors; nothing about the change's shape.

The certification shows what that costs. reviewPR filters the PR's files to .md before either model call, so a PR that de-localises _toc.yml is judged only on its markdown. A fixture pairing a byte-identical clean translation with an English-captioned _toc.yml reached auto-merge at 9/9/9/9 with recommendationReasons: [] in 2 of 3 replicates, and 0 of 3 findings mentioned the table of contents. That is lecture-intro.zh-cn#278 reproduced on demand. No severity or rubric change reaches it — the reviewer is never shown the file. Decision 2 is the fix, and it is cheap and deterministic.

2. The open question recorded for Stage 4 is answered, and the answer is not the one anticipated. The plan asks whether the gate approximately never fires because the prompt defines minor as "correct but awkward" while minor in accuracy/terminology gates absolutely — with the pre-registered response being sharpen the severity definitions. The data says the binding constraint is category, not severity: of 36 injected reviews that were not caught, 24 (67%) detected the defect and filed it minor/formatting (18) or minor/structure (13), neither of which gates. Only 7% of reviews failed to see the defect at all.

Widening the gating set to admit formatting at ≥minor would lift catch 68% → 83% — but also lift clean-control gating from 11/20 to 15/20, dropping auto-merge on genuinely clean translations from 45% to 25%. So the repair is the prompt's category definitions, moving localisation and CJK-policy defects into terminology/accuracy, not a wider gate.

3. Two rule details worth pinning in the design. blocker and major gate through blockers > 0 / majors > 0 in every category, not only the gating ones — easy to mis-state as "any ≥minor finding in a gating category", which is right only for minor. And the floors are near-inert rather than inert: 2 of 140 reviews gated on floors alone (1.4%), one of them a de-localisation review the finding rule missed.

Also relevant: 35 gating findings fired on injected fixtures without pointing at the injected defect (#249's class), so a meaningful share of editor routings are correct outcomes reached for unrelated reasons. A gate certified on route alone would have looked considerably better than it is.