Sync overwrites localised _toc.yml part captions with English, and nothing detects it

[mmcky]

A lecture-adding sync regenerates the target's lectures/_toc.yml as a full mirror of the source, and part captions are carried over untranslated. Any localised caption in the target is silently overwritten with English, and nothing in the pipeline notices.

The field instance

QuantEcon/lecture-python.zh-cn#202 merged 2026-07-24 and replaced 14 hand-set Chinese captions with the English source's. The published Chinese site served 19 English section captions and zero Chinese ones for 17 days before anyone noticed. Repaired by hand in QuantEcon/lecture-python.zh-cn#245, which will itself be reverted by the next lecture-adding sync.

A second instance is open right now: QuantEcon/lecture-intro.zh-cn#278 and #279 both revert 14 Chinese captions (导言→Introduction, 经济数据→Economic Data, and twelve more) and are sitting in the merge queue.

Why nothing catches it

Three independent gaps, each verified:

The engine never translates a caption. _toc.yml is parsed only for lecture discovery — src/cli/commands/init.ts:68 declares caption?: string; and parseTocLectures ignores it. No mode emits a translated caption, so there is no localisation to preserve in the first place.

Review mode cannot see it. reviewer.ts:688-691 filters the diff to docsFolder + .md, so _toc.yml is not reviewed at all. The verdict on #202 reads PASS, scores 9/9/9/9, all four diffChecks true, recommendationReasons: [] — a clean bill on a PR that de-localised the entire published navigation. The gating terminology category never sees a caption.

The build cannot see it. English captions are valid MyST; the strict -n -W build passes. There is no reader-facing failure signal at all — the page simply renders in the wrong language.

That combination is the reason dwell time was 17 days rather than one CI cycle, and it is why this class deserves a deterministic check rather than reviewer judgement.

What would fix it

Roughly in increasing order of cost:

1. A deterministic assertion — for a target whose language is non-Latin-script, fail when a _toc.yml caption loses target-script characters relative to the target's current file. This is a script, not a model, and it catches the whole class. It is also the cheapest of the three and worth doing regardless of the others.
2. Preserve target captions on regeneration — when mirroring _toc.yml, carry the target's existing caption for any part whose membership is unchanged, rather than overwriting from source.
3. Translate new captions — a genuinely new part (as Reinforcement Learning was in E2E harness: a full run should test a single release, not sync-at-ref plus review-at-@v0 #202) has no target caption to preserve, so it needs one produced. dev: record the v0.25.0 release and shadow-gate validation #245 shows what good looks like here: every new caption was already attested in the edition's own prose, so the material to do this well exists in-repo.

Option 2 alone would have prevented both field instances. Option 3 is what makes the result complete rather than half-English.

Relationship to existing issues

Adjacent to #178 (sync applies no localisation rules), but distinct: #178 concerns figure labels and font config inside .md lectures, whereas this is a non-.md file that no mode reads for content. The mechanism, the detection gap and the fix are all different, which is why it is filed separately rather than folded in.

Also adjacent to #156 (sync opens a PR whose _toc.yml references a lecture the run failed to translate) — same file, same full-mirror regeneration, different failure.

Programme-side context and the defect-class framing: QuantEcon/project-translation#28, which proposes certifying the reviewer by injected-defect catch rate. De-localisation is one of its M0 classes, and this issue is why it is expected to score near zero there and to need a shape gate rather than a tuned floor.

[mmcky]

Related surface, and worth deciding together before this one is built: #255 finds 85 of 222 mystnb figure captions on lecture-python.zh-cn are byte-identical to the English source — the same localisation hole as the _toc.yml captions here, on a different YAML surface.

The predicate is identical in both cases: a target-language file should not contain a caption byte-identical to its source. So the deterministic assertion proposed as option 1 above could cover _toc.yml part captions and {figure} / code-cell caption: metadata in one check, rather than being written twice.

Byte-comparison against the source is also the more reliable test than target-script detection. lecture-intro.zh-cn carries 19 captions written as double-quoted \u escapes that decode and render as Chinese correctly — a script-detection check flags them as English; a source-comparison check does not.

[mmcky]

Ruled by @mmcky, 2026-08-18: one shared deterministic check covers both surfaces — this issue's _toc.yml part-caption reverts and #255's untranslated mystnb figure captions share the predicate a target-language file should not carry a value byte-identical to its source, and it will be built once, not twice. Design draft lands on these threads next session (project-translation#36 item 3), harness-first — which requires a caption-bearing fixture, since the current harness fixtures carry no mystnb captions at all (same gap-shape as the #117 citation fixture). Sits under W2 (#260) in the 2026-08-10 plan structure.

[mmcky]

Live instance during the 2026-08-19 backlog drain, caught by the manual byte-comparison audit (invisible to CI and review mode as predicted): lecture-intro.zh-cn#296 (lecture-adding sync, engine v0.25.0, review verdict PASS 9.3) reverts every translated part caption in _toc.yml — 导言→Introduction, 经济数据→Economic Data, 基础工具→Foundations, and eleven more — and delivers ~17 mystnb figure captions in the new lecture byte-identical to source English (the #255 class). The PR is held unmerged pending an on-branch fix. This is exactly the two-surface case the 2026-08-18 shared-check ruling covers: one deterministic predicate (target file must not carry a value byte-identical to source) would have flagged both surfaces on one PR.

[mmcky]

Field data for the shared check, 2026-08-20 — a defect in the predicate, and a surface four editions wide

Measured against lecture-python.zh-cn@origin/main (1965827) and the four programming editions while re-verifying project-translation#37. Three things bear on the design.

1. A line-based caption: predicate is wrong in both directions

The 2026-08-18 ruling here reads "a target-language file should not carry a value byte-identical to its source"; #260's box narrows that to "must not carry a caption byte-identical to its source (covers _toc.yml part captions and mystnb caption: metadata)". Implemented the obvious way — line-wise over caption: — it breaks on YAML block scalars.

Scanning python.zh-cn's 237 caption: values gives 83 byte-identical matches across 20 lectures naively, but 74 across 18 once block scalars are resolved. The nine-caption difference is pure false positive: merging_of_opinions.md matches on | eight times and misspecified_recovery.md once, and in both cases the caption body on the following lines is fully Chinese. The check would flag correctly-translated lectures.

The converse also happens. lq_robust_bewley.md (1) and lq_robust_smoothing.md (4) also use block scalars, and there a naive scan returns the right verdict for the wrong reason — it compares the indicator, never the body, and the bodies happen to be English. Fourteen of the 237 captions are block scalars in total, so this is not an edge case.

Recommendation: specify the check over parsed values, not source lines. The same applies to _toc.yml, where captions are plain scalars today but need not stay that way.

2. The class is actively recurring — the 2026-08-19 drain shipped two new instances

Not a static backlog. Two lectures gained their first English caption in sync PRs merged during the drain:

Lecture	Merged by	New English caption
imp_sample	zh-cn#232 (e7e518c)	caption: 'Monte Carlo and importance sampling estimates'
util_rand_resp	zh-cn#237 (b88dbb9)	caption: Iso-variance curves

Both had zero English captions before those commits. That bounds the growth rate, not just the stock, and it is why the held set moved 16 → 18. Detail on #255.

3. A second surface, four editions wide, that no issue covers

All four editions of the programming series carry _toc.yml part captions byte-identical to the English source — six each, and the caption block is md5-identical across all four and the source:

Edition	Part captions	First _toc.yml
lecture-python-programming.zh-cn	6 of 6 English	2026-03-20, translate init seed
lecture-python-programming.fa	6 of 6 English	2025-04-30, jupyter-book scaffold
lecture-python-programming.fr	6 of 6 English	2026-07-15, seed
lecture-python-programming.ml	6 of 6 English	2026-08-06, scaffold (partial seed — 1 lecture on main)

They are reader-visible: curl of the published intro.html on the zh-cn and fr sites serves "Introduction to Python", "Foundations of Scientific Computing", "High Performance Computing" and "Working with Data" as navigation.

This is a different state from the one this issue describes. It is not a reversion — the captions have been English from each edition's first _toc.yml, by four different routes and on four different dates. They were never localised at all. So the shared check's blast radius is six editions, not two, and this half is a static state rather than a race, which makes it a far cleaner test case than lecture-intro.zh-cn#296.

4. The fixture gap is on both surfaces

git grep over the 27-scenario fixture set returns zero caption: strings, and the four ToC fixtures are flat chapters: lists with no parts: and no captions. A harness-first implementation needs a fixture on both surfaces.

Scheduling note

The "after D1" wording on #260 does not gate this work: D1 is the editions-level font-mechanism decision, and #260 states the byte-identity predicate unconditionally, deferring only caption translation to D1's outcome. #259 (W1, P0, v0.27.0) separately carries the _toc.yml structured merge "plus #254's deterministic caption-script assertion", gated on #169 — whose slice 1 landed 2026-08-19. So the assertion is dual-homed across W1 and W2 and neither home waits on D1; worth settling that split explicitly when the design is drafted. Detail on #260.

[mmcky]

Design draft — the shared deterministic localisation check (2026-08-21)

This is the draft the 2026-08-18 ruling asked for, covering #254 and #255 together. Program tracker: QuantEcon/project-translation#40. Everything below was read off main today; file references are to the current tree.

1. The predicate

A target file must not carry a localisable value whose parsed value is byte-identical to the source's. Two surfaces, one function:

Surface	Where the value lives	How to extract it
ToC part captions	_toc.yml → parts[i].caption	js-yaml load of source and target (already a runtime dep, init.ts:18); walk parts by index
Figure captions	mystnb cell metadata mystnb: figure: caption: inside the {code-cell} option header; {figure}/{glue:figure} directive bodies and :alt:;	parse the option header of each fenced directive as YAML (the header is YAML — that is why `

Comparing parsed values, not source lines, is what fixes both failure directions measured on 2026-08-20: block-scalar indicators stop matching (the nine merging_of_opinions/misspecified_recovery false positives), block-scalar bodies start being compared (the lq_robust_* blind spot), and \u escapes decode before comparison so the 19 escaped-but-Chinese captions on intro.zh-cn are clean.

Exemption rule, so the check is not noisy: an identical value is a violation only if it contains a Latin word of three or more letters. That passes empty captions, pure maths ($\alpha$), numbers, and proper nouns like NumPy as the whole caption, and still catches every caption in the 74-caption held set (all are English sentences). No per-edition allowlist in the first cut; if one is needed it goes in .translate/ beside the state files, never in the lecture.

2. Where it runs — every write path, one module

The engine already has exactly one shared output guard: checkStructuralParity in src/structural-parity.ts:173, fail-closed, Jest-loadable, called from four write sites. The new check is a sibling in the same module (checkLocalisationParity(sourceText, targetText, kind)) with the same call shape (if (!ok) throw). Call sites:

Path	Today	Change
sync .md	sync-orchestrator.ts:529 parity guard	add the caption check beside it
sync _toc.yml	processTocFile (:661) copies source bytes with no guard — the #254 mechanism	replaced by W1's structured merge (- file: add/remove only, captions never taken from source) plus the caption check on the merged result
rebase	replays metadata.files through the same orchestrator, so it re-copies English _toc.yml today and inherits both fixes for free	none beyond the above
init .md	init.ts:298 parity guard	add the caption check
init non-.md copy	copyNonMarkdownFiles (:159) copies _toc.yml verbatim, unguarded — this is how the six programming part captions were never localised	run the ToC check after the copy; on first seed (no existing target ToC) the check necessarily fails, so init must localise part captions as a translation step, or refuse with an actionable message
forward	forward.ts:405 parity guard	add the caption check
headingmap	rewrites target .md at headingmap.ts:237 with no parity guard at all	out of scope here but worth a W1 box — it is a fifth .md write path
review (reviewPR)	filters to .md at reviewer.ts:688 before any model call, so _toc.yml is invisible; verified again by the M0 certification	feed _toc.yml to the deterministic checks only (not the model), and emit a blocker finding in a gating category on violation — the unexpectedDeletionFindings pattern at reviewer.ts:1082, which does not rescale diffScore the way a fifth DIFF_CHECK_NAMES entry would

Failure semantics. On the write paths the check is a refusal: nothing is written, the run fails loudly and, on sync, files the translation-sync-failure issue — consistent with W1's "make partial delivery loud". The 2026-08-20 field data says caption localisation is unreliable rather than absent (mobility 0/4 against prob_dist 19/19 at the same engine), so one bounded retry of the translation call before refusing is worth having on the LLM surfaces; the ToC surface is deterministic and gets no retry. On the review side it is a gate, not a refusal: the PR is created, annotated, and ineligible for auto-merge — which is also Decision 2's eligible-class gate (QuantEcon/project-translation#42) arriving through the same door.

3. The W1 / W2 split

All of the detection goes in W1 (#259, v0.27.0). It is one function with two callers, cheap, and already named in #259's ToC box ("plus #254's deterministic caption-script assertion"); splitting the caption half into W2 would ship a guard that sees half the damage on intro.zh-cn#296. W2 (#260) keeps what actually waits on D1: producing localised captions and the font block deterministically (applyLocalizationRules reaching the Action bundle, #178/#141/#182). The detection does not need D1 and D1 should be recorded as deferred with an owner in .dev/decisions/ (D-2026-08-21-d1-font-mechanism-deferred.md, context / decision / consequences shape) so the next reader stops re-deriving that it does not gate this.

4. Harness-first

Nothing in either fixture set exercises the predicate today: the seven ToC fixtures in tool-test-action-on-github/ are flat chapters: lists, and there is no caption: string anywhere under the test data. Needed, each failing before the check and passing after:

• a parts:-bearing base-toc.yml with the three target bases carrying localised captions, so every lecture-adding scenario (17, 18, 20) exercises preservation as a side effect;
• a caption-bearing base lecture with both a plain caption: and a | block scalar, plus one new scenario (28) that edits the source caption and expects a localised one back;
• Jest unit cases for the predicate built from the real false-positive and blind-spot data on Sync overwrites localised _toc.yml part captions with English, and nothing detects it #254, and the \u-escape case from intro.zh-cn;
• the injection harness already has the ToC case — deloc-toc-captions-english in tool-review-injection/fixtures/fixtures.json, predicted catch rate zero by construction — so it becomes the acceptance test for the review-side gate: it must flip from auto-merge to ineligible.

5. Rollout after release

Edition repairs wait for the tagged release so they are not re-delivered broken: the 18-lecture / 74-caption held set on python.zh-cn; bivariate_dist init -f into intro.zh-cn; the six never-localised programming part captions across zh-cn/fa/fr/ml. Then the exit criterion is observational: the next lecture-adding sync on a zh-cn edition preserves its captions.

Objections and amendments on this thread; the W1 reconciliation is noted on #259 and the split on #260.

[mmcky]

PR #291 status 2026-09-01: reviewed in full and upgraded on the contributor's branch (head bb7cc4b, CI green, mergeable, awaiting @kp992's look). Its Fixes #254 was changed to a plain reference — this is the interim preservation step, not the structured merge the 2026-08-21 design specifies. The review found a blocker in the original: matching parts by identical file set lost the caption of exactly the part a lecture-adding sync touches (measured on the real python/zh-cn pair — append one lecture and 基础工具 reverted to English), i.e. this issue's field case recurring one part per sync. It now matches by membership overlap; a part with no shared files keeps the source caption and is named in a warning, with no positional guessing — English is the honest state the shared deterministic check will gate on. Output is the source text with only caption lines substituted (the original re-dumped 201 of 204 lines); the rebase replay site is covered; the target fetch swallows 404 only. On the real pair: 14 captions preserved, 14 lines changed, all caption lines. Details in the review comment on #291. Remaining for W1 (#259 box 3): captions never taken from source at all, plus the shared byte-identical-caption check with #255.