Align the test-translation-sync harness repos with production config (labels, workflows, .translate bootstrap, frontmatter)

[mmcky]

The #108 smoke test (test-translation-sync.fa#78, 2026-07-18) surfaced that the harness repos have drifted from what production target repos actually run: the very first forward --github run errored on PR creation because the .fa repo lacked the action-translation-sync and resync labels — a failure no production repo wired per the tutorials would hit. The harness only earns its keep if its config mirrors production closely enough that e2e results transfer, and the periodic "Reset: base state for testing" commits reset content but not config, so this drift accumulates silently.

Current state vs production

	test-translation-sync (source)	test-translation-sync.fa	test-translation-sync.zh-cn	Production wiring
Sync/forward labels	old translate:* taxonomy	action-translation-sync + resync created manually 2026-07-18 during the smoke (missing before; gh pr create --label fails hard)	present	present per tutorials
Review workflow	n/a	none — review can only be driven locally with INPUT_* env vars	review-translations.yml, heavily drifted (below)	v0.17.0 template
Action version	sync workflows pin ref: v0.16.1, checkout + npm ci + uses: ./action	n/a	review checks out the default branch (unpinned actions/checkout of the action repo)	QuantEcon/action-translation@v0
Model pin	workflow default	n/a	claude-sonnet-4-5-20250929 (two generations old)	default (claude-sonnet-5)
.translate/ bootstrap	n/a	absent	absent	status --write-state after connect
Base-state frontmatter	n/a	legacy heading-map:	legacy heading-map:	translation: block (new format)

Specific drift in test-translation-sync.zh-cn/review-translations.yml

Compared with the v0.17.0 template it is missing the labeled trigger type (labels are applied after the PR opens, so [opened, synchronize] misses label-gated PRs), the per-PR concurrency group with cancel-in-progress, and the explicit permissions block (pull-requests: write, needed for duplicate-comment cleanup). It also gates on contains(labels, 'action-translation'), which forward resync PRs do not carry — they get action-translation-sync + resync — so resync PRs never auto-trigger review even where the workflow exists. That last point is also true of production wiring and is really a #104 follow-up question for the tool: either forward PRs should additionally carry action-translation, or the review template should trigger on the resync labels too; whichever way it lands, the harness should match.

Suggested work

1. Labels: create the production label set in all three repos (done by hand for .fa on 2026-07-18 — verify .zh-cn and the source repo, and check the label set the sync pr-labels default expects: action-translation, automated).
2. Workflows: replace the drifted/absent review workflows in both target repos with the current v0.17.0 template (labeled trigger, concurrency, permissions), and repin the source repo's two sync workflows from ref: v0.16.1 to the pattern production uses (@v0, or a deliberate ref: main in exactly one designated canary repo if we want pre-release coverage — but make it a labeled choice, not drift).
3. Bootstrap: run status --write-state in both target repos and migrate base-state frontmatter from legacy heading-map: to the translation: block, then bake both into the reset procedure.
4. Make resets config-preserving: encode 1–3 in a small bootstrap script or checklist (e.g. scripts/harness/) so "Reset: base state" restores production-equivalent config, not just content. The e2e harness docs should state which action ref each workflow is expected to pin so drift is detectable by inspection.

Why it matters

The harness exists to catch exactly the class of failure #108 shipped fixes for — and the smoke run only found the label gap because the config had drifted from production. Aligned config also means the next e2e wave (and the #107 validation wave, when it runs) tests what target repos will actually execute, including the review trigger path that failed silently for resync PRs in production (#104).

Refs: #108 (smoke run), #104 (review trigger labels), #92 (2026-07-15 e2e harness run), QuantEcon/test-translation-sync.fa#78 (the smoke PR, closed after passing).

🤖 Generated with Claude Code

[mmcky]

Reframing this as standing maintenance rather than a one-off alignment task (Matt, 2026-07-21), alongside a policy decision that raises its priority: engine changes are validated on the harness before being deployed to production editions.

That makes harness drift a gate on shipping, not a tidiness concern. And it cuts both ways — a harness that has drifted from production gives false confidence, which is worse than having no harness, because the run goes green and the deployment proceeds on it.

So the working rule going forward: when production workflow shape changes — a new pin convention, a template change, a new mode or trigger — the harness gets the same change as part of that work, not as a follow-up that competes for priority later.

Today's instance, which is a clean example of the gap

v0.18.1 shipped the #115 fix (rebase now covers resync/* branches). Five production targets have staged PRs for the workflow half, deliberately unmerged pending harness validation. That validation cannot currently run:

Gap	Consequence for this specific test
Harness targets have no rebase-translations.yml at all	nothing to broaden; the path under test does not exist there
Harness workflows use uses: ./action (local build)	a green run would not exercise @v0 resolution — and every deployed repo pins @v0
Targets do not mirror production shape (no job-level concurrency, no permissions, no label filter)	the interaction between the label filter and the rebase trigger is unrepresented

So the first concrete piece of this issue is: add rebase-translations.yml to a harness target, pinned @v0 rather than ./action, then drive two resync/* PRs through it and merge one to confirm the sibling actually rebases.

The ./action → @v0 change is the more consequential of the two. It is also the one flagged in QuantEcon/project-translation#9's Task 1 — "it does not test published-tag resolution … our whole upgrade is a tag-resolution change" — which has now bitten twice on different work, so it is the gap most worth closing first.

Why this test in particular needs a harness rather than production

The rebase defect's failure mode is silence. A skipped job and a job with nothing to do are indistinguishable in the Actions UI. That is precisely how the original bug survived a 69-PR wave, and why "deploy and watch for errors" would produce no signal here.

[mmcky]

Target-side alignment done (2026-07-21, committed directly to the harness repos' main). Both targets now mirror production; the source repo still has drift, described below.

What changed

Repo	Before	After
test-translation-sync.fa	no .github/workflows at all	review-translations.yml + rebase-translations.yml
test-translation-sync.zh-cn	review-translations.yml only, off-template	both, matching the upstream templates

rebase-translations.yml did not exist in any harness target, so the rebase path had never been exercised there — including the #115 resync-branch fix released in v0.18.1. Both new copies carry the widened filter covering translation-sync-* and resync/*.

The review workflow gained what the template specifies and the harness lacked: the labeled trigger and its label filter, job-level concurrency, and the permissions block v0.17.0's review dedupe needs to delete superseded comments. Those are exactly the config surfaces the harness exists to protect, and none of them were represented.

The pin change is the substantive one

Both target workflows now use uses: QuantEcon/action-translation@v0 rather than checking out the action and building it with uses: ./action. That closes gap 1 in this issue and the identical finding in QuantEcon/project-translation#9 Task 1 — "it does not test published-tag resolution … our whole upgrade is a tag-resolution change" — which has now blocked verification twice on different work.

Worth recording: the previous review-translations.yml checked out the action with no ref: at all, so it silently tested whatever last landed on main. The sync workflow's own comment warns against exactly that ("tracking the default branch makes a run unreproducible"), so the harness was inconsistent with itself.

Also dropped: a hardcoded claude-model: claude-sonnet-4-5-20250929, and the node-version: '20' setup that predated the action declaring node24.

Trade-off, taken deliberately

uses: cannot take an expression, so a workflow cannot switch action versions dynamically — there is no way to have one workflow test both a published tag and an arbitrary ref. Pinning @v0 therefore means the target workflows validate released code only. Validating unreleased code means driving the CLI locally against the harness with a built bundle, as the v0.18.0 smoke test did. Both files say so in a comment.

If pre-release validation through the workflow layer turns out to matter, the way to get it is a second, separate workflow keeping the checkout-and-build form — not by reverting these.

Still drifted: the source repo

test-translation-sync is unchanged, and both its workflows (translation-sync.yml, sync-translations-fa.yml) still carry:

• ref: v0.16.1 — three releases stale, so a sync run here tests code no production repo has run since;
• node-version: '20' against an action that now declares node24 and engines.node >= 24;
• actions/checkout@v4;
• the uses: ./action local-build form.

I have deliberately not changed these yet, because the source sync workflow is the only remaining place that can test an arbitrary ref, and that capability is worth a decision rather than a silent removal. The stale v0.16.1 pin should move regardless — it makes the harness's sync coverage meaningless in its current form.

Labels

Both targets already carry action-translation, automated, action-translation-sync and resync, so the label drift recorded in this issue is resolved.

[mmcky]

Accuracy update (2026-07-21, end of day): target-side alignment is done and battle-tested; the source repo's staleness has widened.

Since the alignment comment earlier today, the harness targets validated two releases end-to-end (v0.19.0 pre-release via a pinned PR head; the #125 PAT A/B), took the PAT + same-repo-head-guard configuration into steady state, and both now mirror the post-#130 template exactly. The label-drift item this issue originally carried is resolved — verified present on both targets.

Two corrections to keep this issue honest: the source repo's ref: v0.16.1 (re-verified live today) is now four releases stale, not three — v0.17.0, v0.18.x, v0.19.0 and v0.20.0 have all shipped past it — and note v0.20.0 matters more than most for sync coverage, since it added the structural parity guard to the sync path this workflow exercises. The open decision is unchanged: the checkout-and-build form is the only remaining way to run an arbitrary ref through the workflow layer, so modernising it (fresh pin, node24, checkout bump) should keep that capability rather than blindly converting to @v0.

[mmcky]

Done — all four suggested items landed on 2026-07-23, and the harness has been re-validated end-to-end since.

1. Labels. The production set (action-translation, automated, action-translation-sync, resync) is present in all three repos, alongside the Stage 1 review taxonomy (editor, audit, auto-merged, spot-check, divergence). Verified live. The source repo still carries the old translate:* labels, but nothing reads them and they gate nothing — left alone rather than churn.

2. Workflows. Both source sync workflows moved off ref: v0.16.1 + npm ci + uses: ./action to uses: QuantEcon/action-translation@v0, applying the same reasoning this issue applied to the targets. Two further gaps closed while there: they never checked out the source repo at all (production does actions/checkout@v7 with fetch-depth: 2), and they had no issue_comment / \translate-resync trigger, so that production lever was untestable. Two harness-only divergences are kept and documented in the file headers — the test-translation label trigger with test-mode, and target-repo derived from ${{ github.repository }}. No stale claude-sonnet-4-5-20250929 model pin survives anywhere; all six workflows are on @v0.

Also fixed in passing: the fa target's rebase workflow still pushed with GITHUB_TOKEN, giving rebased branches run-less heads — #125, whose both-ways validation was run on this harness on 2026-07-21 and whose fix never reached fa.

3. Bootstrap and frontmatter. Both targets' .translate/ was regenerated with translate status --write-state, and all four fixture lectures migrated from legacy heading-map: to translation.title + translation.headings.

The state finding was worse than "absent". Both targets had hand-written state using key names the engine does not recognise — source_commit/translated_at against source-sha/synced-at, with model, mode and section-count missing. readFileState() validates all five keys and returns undefined on any mismatch, silently, so both targets read as untranslated. Every state-dependent path had been running against a target the engine believed had never been translated.

The frontmatter finding had a sharper consequence than cosmetics. Production is entirely on the new schema — 0 legacy files across 196 in the three zh-cn editions — so the harness was the only place the old key survived. A failed headingMapCorrect routes to editor on its own, which meant the harness could never produce an auto-merge recommendation, whatever the translation quality. It would have silently invalidated the auto-merge base-rate measurement that Stage 4 of the human-review program depends on. After migration the same fixture re-ran to recommendation: auto-merge, no reasons, all four diff checks true — the first auto-merge the program has seen. Detail in #147.

4. Config-preserving resets. Encoded as bin/harness-reset and bin/harness-baseline in QuantEcon/project-translation, with make harness / make harness-reset. The baseline is a harness-baseline tag rather than an ad-hoc "Reset: base state for testing" commit.

The rule this issue asked for is explicit in the implementation: reset restores content and .translate/, and never touches .github/. Workflows must stay current, not baseline — a reset that rolled them back to whatever they were when the tag was cut would quietly re-introduce exactly the drift this issue is about. Verified with a preservation probe: a workflow edit survives a reset while content edits revert.

Two things this issue did not anticipate, found while doing it:

The fixture suite was spent. A fixture fires when test-translation is added, and GitHub emits no labeled event for a label already present. No fixture had ever had its label stripped, so all 26 were inert: gh pr edit --add-label reported success, emitted nothing, started no workflow. The first attempt at a verdict-v2 test looked like it worked and did nothing — the same success-shaped-failure class as #131. Re-arming is now part of every reset.

The fa target could not run review at all — ANTHROPIC_API_KEY was not org-visible to it (QUANTECON_SERVICES_PAT was), failing with Input required and not supplied: anthropic-api-key. Granted since; fa now posts a valid block and reaches auto-merge too.

Also cleared: 685 accumulated translation-sync-* / resync/* branches and 26 stale March-2026 translation PRs on fa.

Documentation, which this issue asked for in its last line: infrastructure/harness.md in project-translation states which ref each workflow is expected to pin, so drift is detectable by inspection. The harness repo also has a README for the first time — recording that the 26 open PRs are a standing fixture suite rather than a backlog, and that the @v0 pins are load-bearing.

The general lesson, worth keeping: a test environment that has drifted does not fail loudly, it just stops testing what you think it tests. Three defects here, all silent, all found only by going looking.