Validate verdict v2 (v0.22.0) on real production sync PRs — nothing has run through the Actions path yet

[mmcky]

v0.22.0 shipped reviewer verdict v2 (#144) and @v0 has moved, so every edition now runs it. The released artifact was smoke-tested by driving the built bundle against a harness PR — it emitted a complete 17-field block with engineVersion: 0.22.0 — but nothing has yet run through the real GitHub Actions path, because no edition had an open translation PR at release time.

A local bundle run cannot exercise the things most likely to break in production: @v0 resolution inside a workflow, the deployed workflow config and its label filter, the estate's secrets, or comment rendering as GitHub actually displays it. This issue is the reminder to close that gap once real PRs exist, and to analyse the first batch rather than glancing at one.

This program has now been caught twice by treating one step as implying the next — #131's fix was released but silently skipped review on every CLI resync PR for days, and the v0.22.0 release PR merged without anyone cutting the tag. Shipped is not the same as working.

What will produce the first real PRs

Sync runs in the source repo when a PR merges there. Currently wired:

Source repo	Editions it syncs to
lecture-python-programming	zh-cn, fa, fr
lecture-python-intro	zh-cn
lecture-python.myst	none yet — myst#979 still open

So the first organic verdict-v2 review will come from a merge into lecture-python-programming or lecture-python-intro. CLI resync PRs (translate forward --github) are a second, distinct shape worth covering — see below.

What to check on the first real sync PR

The review ran at all. The job triggers, is not skipped, and posts a comment. This is the #131 failure mode: a label filter that does not match means the workflow completes green having done nothing.

The block is present and parseable. A translation-review-verdict HTML comment at the end of the review comment, valid JSON, all 17 fields, schemaVersion: 1.

engineVersion reads 0.22.0, not unknown. If it reads unknown the ESM resolution path is being taken — getEngineVersion() relies on __dirname, which does not exist under "type": "module". Harmless to the gate but it degrades a contract field, and src/cli/translate-state.ts's getToolVersion() already has ESM-safe fallbacks to copy.

reviewedHeadSha equals the PR head. The whole supersession design rests on this binding.

autoMergeMode is off and wouldAutoMerge is absent. No edition has opted into shadow yet; if the field appears, something is enabling it unintentionally.

The comment renders cleanly. Findings are now rendered from structured objects with severity and category labels instead of prose strings — this is the first time real model output flows through that rendering path in production.

What to analyse across the first batch (~10 PRs)

Beyond "it works", the first batch is the earliest real data on the question Stage 4 exists to answer:

• What fraction would qualify for auto-merge, and what gates the rest? Collect recommendation and recommendationReasons across the batch. All six harness runs recommended editor, and an auditor argued the gate may approximately never fire, because the prompt defines minor as "correct but awkward" while a minor finding in accuracy or terminology gates absolutely. If the eligible rate is near zero, the fix is to sharpen the prompt's severity definitions — not to lower the floors, which are provisional placeholders rather than policy.
• Are the findings well-formed in practice? How often does findingsMalformed gate, how often is file attributed on multi-file PRs, and do severities look consistently applied.
• Do the diff checks behave? headingMapCorrect failed on most harness fixtures, but those carry a legacy heading-map key — production editions should not, so a high failure rate there would indicate something real.
• Comment size. Observed 4.8–7.2 KB on the harness against GitHub's 65,536-char cap; worth confirming production stays well clear.

Also still unproven: the #131 path

Separately from verdict v2, the #131 fix shipped in v0.21.0 and no CLI resync PR has been opened anywhere in the fleet since, so "resync PRs receive AI review" remains unobserved in production. The next resync wave is its first real test. Worth checking in the same pass, since a resync PR also exercises the reviewer's getSourceAtCommit path rather than the source-PR-diff path.

Done when

A handful of real sync PRs (and at least one CLI resync PR) have been inspected against the checklist above, the results recorded on QuantEcon/project-translation#15, and either the shadow rollout proceeds or a prompt/severity adjustment is raised off the back of the numbers.

---

Context: program plan and Stage 1 delivery notes in QuantEcon/project-translation (PROJECT-PLAN-HUMAN-REVIEWS.md); handover QuantEcon/project-translation#16; contract spec docs/user/metadata-contract.md.

[mmcky]

Verdict v2 validated through the Actions path — on the harness, not a live edition

Step 2 has been run on the test harness rather than waiting for an organic production sync. That is a deliberate substitution and worth justifying, because this issue was written on the premise that only production would do.

The review path is not an approximation of production — it is the same file. The harness targets' review-translations.yml was diffed against all five production editions: the action-translation label filter and the uses: QuantEcon/action-translation@v0 pin are identical in all five, differing only in source-repo, target-language and docs-folder. So a harness review run exercises the same tag resolution, the same filter, real secrets and real GitHub comment rendering.

Result — first organic verdict v2 block, posted by Actions

A fixture PR was fired, both targets received translation PRs, and the zh-cn review ran @v0 and posted a complete block. Against this issue's checklist:

Check	Result
Review job ran, not skipped	✅ ran; a second run correctly skipped by the label filter
Block present, parses, schemaVersion: 1	✅ 16 fields
engineVersion	✅ 0.22.0 — not unknown, so the ESM concern does not bite the action bundle
reviewedHeadSha matches PR head	✅
autoMergeMode	✅ off, and wouldAutoMerge absent
Comment renders cleanly from structured findings	✅
Comment size	4.2 KB, well clear of the 65,536 cap

The verdict was PASS with overall 8.9, and the recommendation was editor — a scalar threshold merges that; the rubric did not. Decision 1 demonstrated again, this time through the deployed workflow rather than a local bundle.

The finding that matters more than the pass

The sole reason for the editor routing was diff check failed: headingMapCorrect. This issue anticipated exactly that and flagged it as a harness-fixture artefact — it is, and the consequence is larger than "noisy fixture":

The harness lectures still carried the legacy heading-map: frontmatter key. Production is entirely on translation.title + translation.headings — 0 legacy files across 196 in the three zh-cn editions. A failed headingMapCorrect routes to editor on its own, regardless of translation quality, so the harness could never have produced an auto-merge recommendation. That would have quietly made it useless for measuring the auto-merge base rate — the one number Stage 4 depends on, and the question this issue asks about the first batch.

The fixtures have been migrated to the current schema, and the same fixture was re-fired to confirm the gate clears. It does — and the result is worth its own line:

recommendation: "auto-merge", with recommendationReasons: [], all four diff checks true, overall 10.

That is the first auto-merge recommendation this program has ever seen. It matters because of the standing objection recorded in QuantEcon/project-translation#16: an auditor argued the branch may approximately never fire, since the prompt defines minor as "correct but awkward phrasing" while a minor finding in accuracy or terminology gates absolutely. All six earlier harness runs recommended editor, which was consistent with that worry — but those runs were against fixtures whose legacy frontmatter was gating them unconditionally, so they were never evidence about the rubric at all.

The auto-merge branch is therefore reachable, and the "never fires" hypothesis is disproved as an absolute. It is not yet evidence about the rate: this is one synthetic single-file change with a clean diff, which is close to the best case. The real question — what fraction of genuine lecture diffs qualify — is unchanged and still needs production PRs.

Other harness drift found and fixed the same pass

Three defects, all the same shape — stale config that fails without complaining:

The source sync workflows were pinned ref: v0.16.1 and built the action locally, six releases behind the estate. The local-build form also never exercises whether @v0 resolves to the intended release, which is the change the estate most often makes. They also never checked out the source repo and had no \translate-resync trigger.

Both targets' .translate/ metadata was hand-written with key names the engine does not recognise (source_commit/translated_at against source-sha/synced-at, with model, mode and section-count absent). readFileState() validates all five keys and returns undefined on any mismatch, silently — so both targets read as untranslated, and every state-dependent path had been running against a target the engine believed had never been translated. That includes the resync path of #131, which is the other thing this issue wants proven.

The fa target's rebase workflow still pushed with GITHUB_TOKEN, giving rebased branches run-less heads — the #125 defect whose both-ways validation was run on this harness on 2026-07-21, and whose fix never reached fa.

What this still does not prove

The production sync half. Each source repo's own sync workflow and secrets are separate files in separate repos; only a real source merge covers those. The review half is now closed by inspection plus execution.

The base rate. Harness fixtures are synthetic and few. Real recommendation / recommendationReasons distribution still needs real lecture diffs, so the first-batch analysis in this issue remains outstanding. Now that headingMapCorrect no longer fires spuriously, the harness can at least show whether auto-merge is reachable at all.

Suggest splitting this issue's scope explicitly: 2a mechanism — done, here; 2b base rate — still needs production PRs.

Both target languages now covered

The fa half of the harness was inert during the first pass: test-translation-sync.fa could not run its review workflow because the ANTHROPIC_API_KEY org secret was not visible to that repo (QUANTECON_SERVICES_PAT was). It failed with Input required and not supplied: anthropic-api-key — a clean confirmation of the diagnosis, and the same class of gap as the PAT grants of 2026-07-21.

The grant has since been made, and the fixture was re-fired end-to-end. The fa review now runs and posts a valid block: schemaVersion: 1, engineVersion: 0.22.0, verdict: PASS, recommendation: auto-merge with no reasons, all four diff checks true, overall 9.6, syntaxErrorCount: 0.

So verdict v2 is verified through the deployed Actions path in two target languages, one CJK and one RTL, both reaching the auto-merge branch on a clean diff.

Harness reference: infrastructure/harness.md in QuantEcon/project-translation; reset via make harness-reset.

[mmcky]

Now validated on real production sync PRs — the thing this issue was actually opened for

The earlier comment today covered the harness. This one covers production, which is what this issue asks for.

QuantEcon/lecture-python.myst#979 merged, wiring the last unwired source repo, and two \translate-resync triggers produced the first organic production sync PRs ever reviewed under verdict v2: QuantEcon/lecture-python.zh-cn#198 (inventory_dynamics, from myst#623) and #199 (perm_income, from myst#975). Both source PRs had merged before the workflow existed, so their drift had never reached the target — the resyncs both verified the path and did real reconciliation work.

Checklist results, on production

Check	Result
Review job ran, not skipped	✅ — but see the trap below
Block present, parses, schemaVersion: 1	✅ both
engineVersion	✅ 0.22.0 on both
reviewedHeadSha matches PR head	✅
autoMergeMode / wouldAutoMerge	✅ off / absent
Comment renders cleanly from structured findings	✅
Comment size	7.2 KB — consistent with the 4.8–7.2 KB harness range, far clear of the 65,536 cap

The trap this issue should record: skipped runs are not failure

A sync applies its labels in a single addLabels call, GitHub emits one labeled event per label, and on #198 the labels were applied twice. The result is ten Review Translations runs, of which nine are correctly skipped by the label filter and exactly one is success. Querying with --limit 1 returns a skipped one, and the run list at a glance reads as total failure — the same surface as #131. I briefly concluded production was broken. Find the success run before drawing any conclusion.

First base-rate data (n=2): both editor, but only one gate was valid

This is the part that matters for Stage 4, and it complicates the measurement rather than advancing it.

#198 — a genuine catch. Routed editor on "1 minor finding in gating categories" plus "terminology 8 below floor 9". The finding is real: the glossary specifies 边缘分布 for "marginal distribution" and the translation used 边际分布 throughout. verdict: PASS, overall 9.0 — any scalar threshold merges this; the rubric did not. Decision 1 demonstrated on production content.

#199 — a false gate. Routed editor on diff check failed: positionCorrect alone, with three confident structure findings asserting the source's code change was never applied. Verified false: the target carries it identically (def time_path(T, rng), rng.standard_normal, the added rng = np.random.default_rng(), both call sites). Filed as #148 — the four diffChecks are model output but gate absolutely, so a hallucinated boolean routes a correct PR to a human.

The implication for this issue's first-batch analysis plan. The plan here is to collect recommendation and recommendationReasons across ~10 PRs to get the auto-merge base rate. On n=2 that already needs a caveat: a gate driven by reviewer error is indistinguishable, in the collected data, from a gate driven by a real translation defect. Anyone reading the batch as a base rate will read it low, and floors calibrated on it are calibrated partly against noise. Suggest the batch analysis record, per PR, whether the gating reason was verified by hand — otherwise the number this issue produces cannot be trusted for the purpose it exists to serve. Distinguishing diff-check gates from findings gates in the collected data would let their precisions be reported separately.

Still outstanding on this issue

A larger sample. n=2 says nothing about the rate. Organic PRs will now arrive from three wired source repos rather than two.

The #131 resync path. Partly addressed but not cleanly: #198 and #199 were triggered by \translate-resync, which is the source-repo sync path, not the CLI translate forward --github path #131 fixed. Both carried the action-translation label and received review, but a CLI-created resync/* PR remains unobserved in production.

The harness gap that would have hidden this. Recorded in the earlier comment: the harness fixtures carried legacy heading-map: frontmatter, so headingMapCorrect failed unconditionally and no harness run could ever recommend auto-merge. Fixed. Worth noting the symmetry — that was a false gate too, from a different cause, and it also would have depressed a measured base rate.

[mmcky]

Status update — the premise here has been overtaken, and v0.23.0 changes the method this issue proposes.

The gap this issue was filed to close is now closed twice over. v0.22.0 ran through the real Actions path on the first organic production PRs, QuantEcon/lecture-python.zh-cn#198 and #199 (2026-07-23), and v0.23.0 was verified the same way on the harness today — a fixture fired the deployed sync → review path and the posted verdict block reads engineVersion: 0.23.0. @v0 resolution inside a workflow, the deployed label filter, estate secrets and GitHub comment rendering have all now been exercised for real.

What remains is the part this issue is actually valuable for: the auto-merge base rate on a batch of real PRs. n=2 organic PRs is not a rate.

v0.23.0 changes how that batch must be analysed (#148 → #152). The four diffChecks are no longer homogeneous: structurePreserved and headingMapCorrect are now computed by the engine, while scopeCorrect and positionCorrect remain the reviewer's judgement, and the new diffCheckSources field says which is which. A model-asserted check that fails now arrives as a minor/diff-check finding rather than a false boolean.

So the first-batch analysis should report gates split by provenance, not blended:

Gate source	Read from	Interpretation
Deterministic check	diffChecks where diffCheckSources says deterministic	measured fact — a real structural defect
Model-asserted check	a diff-check finding	reviewer opinion — must be hand-verified before counting as a defect
Findings	findings[] in gating categories	the model's quality judgement
Criterion floors	scores vs CRITERION_FLOORS	provisional, and the thing Stage 4 is calibrating

Blending these is exactly the error #148 was filed about: it makes reviewer error indistinguishable from translation defects in the calibration data. The precision of the model-asserted checks should be reported separately, because that number is what decides whether a deterministic positionCorrect is worth building.

One caveat for whoever runs the batch: on the two organic PRs so far, one of the two editor routings was a false gate (#199's positionCorrect, verified false by hand-diffing the source against the target). positionCorrect is still model-driven in v0.23.0 — made attributable, not fixed — so that class can recur.

[mmcky]

First organic production data point: QuantEcon/lecture-python.zh-cn#202 (source QuantEcon/lecture-python.myst#992, 2026-07-23, nine new lectures).

v0.23.0 is confirmed running through the Actions path: "engineVersion": "0.23.0" with diffCheckSources present and populated — structurePreserved and headingMapCorrect sourced deterministic, scopeCorrect and positionCorrect sourced model, exactly as #148 intended.

The verdict

Verdict	PASS
Routing	editor
Accuracy / Fluency / Terminology / Formatting	8 / 8 / 8 / 9
Diff quality	10/10
Findings	1 major, 3 minor, 1 nit
Stated reason for routing	1 major finding; 2 minor in gating categories; accuracy 8 below floor 9; terminology 8 below floor 9

Two observations that bear directly on whether these numbers can be used as a base rate.

1. The major finding that gated the PR is self-refuting

The single major/accuracy finding — the one doing most of the routing work — reads, in full:

在 phillips_learning.md 的 translation.title 元数据中未出现，但检查内容发现该文档的翻译标题正确译为'适应性学习与逃逸动态'，与英文原标题 'Adaptive Learning and Escape Dynamics' 一致，未发现问题。实际检查未发现该问题，此为误报占位符。

The model emitted a major-severity finding whose own description concludes "no issue found on inspection; this is a false-positive placeholder", and that finding then routed the PR to a human. suggestion is null, which is a usable deterministic signal — a major finding with no suggestion is a reasonable thing to be suspicious of.

n=1, so this is not yet a rate. But if it recurs it inflates the human-review rate for reasons that have nothing to do with translation quality, and floors fitted to it will be fitted to noise.

2. The terminology score is not measuring glossary adherence

Of the four terms the review flagged under terminology and fluency consistency — 逃离/逃逸 (escape), 运气不好/运气变化 (bad luck), 自然率假说 (natural-rate hypothesis), 卢卡斯批判 (Lucas critique) — none appears in the 357-term zh-cn glossary. Probing the glossary for phillips, natural rate, escape, luck, nash, self-confirming and lucas returns nothing; only rational expectations is present.

So on this PR the gating terminology category is measuring intra-document consistency of un-glossaried terms, not adherence to the shipped glossary. Both are legitimate quality signals — see #133, which proposes exactly the first as a first-class check — but they are different quantities with different fixes, and #154 should measure them separately rather than assume the terminology score is an adherence proxy.

Caveat on using this PR at all

The PR does not build (see #117 and #156), while the review scored it PASS with diff 10/10. That miss is tracked separately in #157. For the purposes of this issue it means the first production data point is not a clean sample: the verdict was produced against an artefact that a reviewer would reject on sight, so it tells us more about the review's blind spots than about the auto-merge base rate. Worth recording as data point one, not worth fitting anything to.

[mmcky]

Closing — validated twice over, and the remaining value has moved to #103 and #154.

Mechanism — done. Verdict v2 was exercised through the deployed Actions path on the harness at v0.22.0, then on genuinely organic production syncs: QuantEcon/lecture-python.zh-cn#198 and #199, and then #202 with engineVersion: 0.23.0 and diffCheckSources populated exactly as #148 intended. @v0 resolution inside a workflow, the deployed label filter, estate secrets and GitHub comment rendering are all now exercised for real, on more than one engine version.

Base rate — no longer this issue's job. Three organic PRs is not a rate, and the calibration this issue was reaching for — what fraction of real lecture diffs qualify for auto-merge — is #103 Stage 4's shadow-mode work, fed by #154's adherence data. The 2026-07-23 comment above already says the premise here has been overtaken; this closure just makes the tracker agree with it.

What this issue actually produced. The validation window is where three live reviewer defects were found, each now carrying its own issue: #157 (review returns PASS 10/10 on a PR that cannot build), #187 (out-of-diff prose grading forces editor routing on code-only PRs), and #135 (review report language drift). It also caught the legacy heading-map: fixture drift that had been making the harness structurally incapable of ever recommending auto-merge — which would have quietly invalidated the base-rate measurement this issue was written to enable.

Read as a validation log rather than an open task, this thread did its job.