Measure glossary adherence across the estate — delivery is fixed, compliance has never been measured

[mmcky]

v0.23.0 fixed glossary delivery on both paths (#149, #146). What has never been measured, anywhere in the estate, is glossary adherence: given that the glossary reaches the model, how often does the output actually use it?

There is no general glossary audit issue in the estate. QuantEcon/lecture-python.zh-cn#201 audits one term in one edition; #133 covers coined terms that are not in the glossary, which is the opposite case; #136 is the audit workflow mechanism. Nothing measures compliance with the 357-term glossaries we ship.

Why it matters now rather than later

terminology is a gating category in verdict v2: a single minor terminology finding blocks auto-merge. So adherence failures do not merely produce slightly-off prose — they route correct-looking PRs to a human, and in Stage 4's shadow data they are indistinguishable from translation defects. The base rate that Stage 4 calibrates its floors against therefore includes an unmeasured adherence term.

This is the same shape as #148, one layer up: there, model error was laundered into a field treated as ground truth. Here, adherence failure is laundered into the terminology score.

Evidence that adherence is not guaranteed

From the v0.23.0 validation run (2026-07-23, harness, live translate forward with the fix in place):

The glossary was demonstrably delivered — ✓ Loaded built-in glossary for zh-cn — 357 terms — and the newly generated Chinese honoured 4 of 5 glossary terms present in the added source paragraph:

en	glossary zh-cn	model output
Matrix multiplication	矩阵乘法	矩阵乘法	✅
Dominant eigenvalue	主特征值	主特征值	✅
OLS	普通最小二乘法	普通最小二乘法	✅
Kernel	核	核	✅
Leontief Inverse	列昂惕夫逆矩阵	里昂惕夫逆矩阵	❌

The miss is a variant transliteration — both forms appear in Chinese economics writing, but the glossary specifies one and the model produced the other. Mild in isolation; the point is that it happened at all, on a five-term sample, with the glossary in the prompt.

For contrast, the init path honoured 边缘分布 for Marginal distribution in 4 of 4 term-bearing lectures among the 37 seeded by lecture-python.zh-cn#196. So adherence is usually good — which is exactly what makes an unmeasured failure rate dangerous rather than obvious.

What an audit would establish

1. A fleet-wide adherence rate, per edition and per language: for each glossary term appearing in the English source, does the translation use the specified rendering?
2. Which terms fail, and whether failures cluster — transliterations of proper nouns (the Leontief case), terms with context-dependent senses (the 边缘/边际 case), or short common words where the glossary rendering is unnatural in context.
3. A pre/post baseline, since everything translated before v0.23.0 may have been produced with no glossary at all on the forward path. Separating "never had the glossary" from "had it and ignored it" is the whole point, and only possible now that delivery is fixed.
4. Whether any glossary entries are simply wrong or over-broad — a term the model consistently declines to use may be a glossary defect, not a translation defect.

Notes on approach

This is largely deterministic and cheap: intersect each glossary's en terms against the English source text, then check the aligned target for the specified rendering. No LLM calls needed for the measurement itself — the same technique that found the 邊际/边缘 fleet-wide counts and the 248-of-249 heading-map figure. An LLM pass is only needed to judge legitimate deviations (a term inside a code block, a quoted English phrase, a genuinely better contextual rendering).

Worth being careful about false positives: glossary terms that are substrings of other words, terms appearing in code or math, and context fields (economics vs probability) that make a single English term legitimately map to different renderings.

Related

• translate forward loads its glossary CWD-relative and fails silently — resyncs can run with no glossary and nothing says so #149 / Review mode silently ignores glossary-path — verdicts judged against the wrong terminology #146 — delivery, fixed in v0.23.0; this issue is what remains once delivery is reliable
• Feedback: add a first-class consistency check for recurring document-local phrases (same source phrase → one target rendering) #133 — document-local consistency for phrases not in the glossary; complementary, not overlapping
• feat: whole-lecture human-audit workflow — labels, templates, paced triggering, feedback mining (#103 Proposal B) #136 — the human-audit workflow this could be the first automated input to
• Verdict v2 diffChecks are model output but gate absolutely — a hallucinated boolean routes a correct PR to editor, and biases Stage 4 calibration #148 — the same "unmeasured error term inside a gating signal" shape
• Audit: 边际分布 → 边缘分布 for "marginal distribution" across 8 remaining lectures (54 occurrences) lecture-python.zh-cn#201 — the first instance of this class, scoped to one term
• QuantEcon/project-translation#15 — Stage 4 calibration, which this de-risks

[mmcky]

First fleet-wide measurement exists. Tool: bin/glossary-audit in project-translation, re-runnable, with full findings and source context written as JSON. Write-up: reports/2026-07-24-glossary-adherence-baseline.md.

Edition	Lang	Pairs	Observations	Adhered	Mixed	Unexplained misses
lecture-python.zh-cn	zh-cn	121	1157	83.8%	88	90
lecture-intro.zh-cn	zh-cn	49	673	75.9%	63	93
lecture-python-programming.zh-cn	zh-cn	26	109	91.7%	5	3
lecture-python-programming.fa	fa	26	109	76.1%	n/a	26
lecture-python-programming.fr	fr	26	136	78.7%	n/a	29

These are not adherence rates and should not be quoted as such — they are a denominator and a first pass. False positives dominate the top of every failure list: the largest cluster in three editions is Lecture → 讲座, firing because English prose says "this lecture" constantly while the target says 本讲, and Support → 支持 is a straight context collision (the entry is the probability sense; the source text mostly means "support for X"). On a hand-checked sample, 3 of 3 initial misses were artefacts — 外生的 (the citation form carries an adjectival 的 that Chinese drops before a noun: 外生冲击) and FRED (a database name that must stay English).

Two matcher defects, both of which had been inflating the failure count

• \b is defined against \w, which includes CJK. So \bOLS\b never matched inside 逐方程的OLS估计 — precisely where a retained English acronym appears in a Chinese target. Now bounded against ASCII only.
• A presence test is not an adherence test. prob_matrix.md contains 边缘分布 and 25 occurrences of 边际分布, and scored clean. Rival detection was added for this class; a first attempt that held the head noun constant made 正态分布 a rival of 边缘分布 and collapsed the apparent rate to 38.5%, so it is now same-length/one-character-substitution. That still emits fragments where a 4-character window slides across a 5-character term (理性预期 / 应性预期, from 适应性预期) — roughly half the mixed findings inspected.

Validation against the one case with ground truth

zh-cn#201 — 边际分布 where the glossary says 边缘分布, ground truth 49 occurrences across 8 files. The checker flags 5 (aiyagari, ak_aiyagari, finite_markov, mle, prob_matrix) with zero false positives, and misses 3. The misses share a structural limit rather than a bug: the tool only inspects targets whose English source carries the term, and in ar1_turningpts, lake_model and multi_hyper the English says something else while the Chinese reached for 边际分布 anyway. Precision 5/5, recall 5/8. Catching the rest needs a target-driven sweep for known-wrong renderings, which is a different query.

The finding that bears on Stage 4

The terminology category is not currently measuring glossary adherence. On zh-cn#202 the review raised four terminology/consistency findings — 逃离/逃逸, 运气不好/运气变化, 自然率假说, 卢卡斯批判 — and none of those terms is in the 357-term glossary. The gating category was measuring intra-document consistency of un-glossaried terms. That is a legitimate signal (#133 proposes exactly it) but a different quantity, and shadow data should carry the two separately rather than treating the terminology score as an adherence proxy.

New work looks materially better than the corpus baseline. The ten Phillips lectures — organic, post-v0.23.0, glossary confirmed delivered — score 91/100 with 6 acronym retentions and 3 unexplained misses, of which hand-checking found 2 genuine (Discounting → 贴现 rendered 折扣, the retail sense rather than the economics one) and 1 arguable (Volatility → 波动率 rendered 波动性). That is roughly 97% effective adherence on new work, against a lower corpus baseline that largely predates the delivery fix. If it survives triage, the conclusion is that adherence on new translations is high enough not to swamp Stage 4's terminology signal, while the existing corpus carries a backlog whose size is not yet known to better than a factor of two.

Next

1. Triage a stratified sample of 40–60 findings to convert the raw rate into a corrected one with a stated false-positive rate — the step this issue anticipated needing eyes or an LLM pass.
2. Add a per-term audit: skip flag for entries like Lecture and Support that are too common or too polysemous to measure this way. Arguably a glossary-quality finding in its own right.
3. Add the target-driven sweep for the recall gap above.
4. Then decide hand-fix vs resync for fix(harness): say which version the E2E run is testing, and stop testing v0.16.1 #201 and whatever else survives, as one wave.

[mmcky]

Re-run complete on the v0.25.0 estate (2026-08-07) — full report: reports/2026-08-07-glossary-adherence-rerun.md, per-edition JSON alongside it. Headline: flat everywhere (python.zh-cn 83.8→83.5%, intro 75.9→75.6%, programming.zh-cn 91.7→90.4%, fa 76.1→76.5%, fr 78.7→77.2%) through v0.24.0, v0.25.0, 17 new audited pairs on python.zh-cn and a 41-PR sync backlog forming — glossary delivery is stable and the residual is standing corpus + the characterised matcher false-positive floor (tool byte-identical to the baseline, deliberately). The known-true 边缘分布 cluster reproduces on exactly the same 8 lectures, so the zh-cn#201 wave is confirmed ready — with one new sequencing constraint: zh-cn#230 in the open backlog touches kalman_filter_var.md (a cluster lecture), so it merges first. Also recorded: hand fixes on sync-path lectures survive only until the next re-delivery, so the durable adherence carrier is the engine's localisation/glossary path (#178-adjacent), with hand waves as bridge repairs. What stays open here: the matcher precision/recall pass before any per-edition rate is quoted as an adherence figure.