Field report from the QuantEcon/project-translation#14 mini-wave (six resync PRs on QuantEcon/lecture-python.zh-cn, all reviewed by @v0 = v0.20.0, model claude-sonnet-5, same day, same workflow).
Observation
Five of the six reviews wrote the Translation Quality Summary and Suggestions in English. One — QuantEcon/lecture-python.zh-cn#191 (comment) — wrote both entirely in Chinese. Within that same review, the Diff Quality summary stayed English, which isolates the drift to the quality-evaluation LLM call; the diff-position call is a separate prompt and did not drift.
| PR |
Quality summary language |
| #188 career |
English |
| #189 jv |
English |
| #190 kesten_processes |
English |
| #191 lqcontrol |
Chinese |
| #192 mix_model |
English |
| #193 wealth_dynamics |
English |
Cause
The quality prompt (src/reviewer.ts, You are a professional translator and quality evaluator specializing in … translation from English to ${targetLangName}) never specifies the language the review itself should be written in. The model reads a mostly-target-language document and sometimes follows the content language instead of the prompt language — nondeterministically, 1-in-6 here under identical conditions.
Suggested fix
Pin the report language with one prompt line (e.g. "Write the summary and all suggestions in English") in both review prompts. Worth considering a review-language input rather than hard-coding English: under the #103 editor-routing design, target-language suggestions may actually be preferable for native editors — but that should be a configured choice, not model whim. Either way the defect is the nondeterminism: downstream consumers (verdict parsing today, the #66 metadata contract and #103 verdict v2 tomorrow) need a predictable report language.
Field report from the QuantEcon/project-translation#14 mini-wave (six resync PRs on QuantEcon/lecture-python.zh-cn, all reviewed by
@v0= v0.20.0, model claude-sonnet-5, same day, same workflow).Observation
Five of the six reviews wrote the Translation Quality Summary and Suggestions in English. One — QuantEcon/lecture-python.zh-cn#191 (comment) — wrote both entirely in Chinese. Within that same review, the Diff Quality summary stayed English, which isolates the drift to the quality-evaluation LLM call; the diff-position call is a separate prompt and did not drift.
Cause
The quality prompt (src/reviewer.ts,
You are a professional translator and quality evaluator specializing in … translation from English to ${targetLangName}) never specifies the language the review itself should be written in. The model reads a mostly-target-language document and sometimes follows the content language instead of the prompt language — nondeterministically, 1-in-6 here under identical conditions.Suggested fix
Pin the report language with one prompt line (e.g. "Write the summary and all suggestions in English") in both review prompts. Worth considering a
review-languageinput rather than hard-coding English: under the #103 editor-routing design, target-language suggestions may actually be preferable for native editors — but that should be a configured choice, not model whim. Either way the defect is the nondeterminism: downstream consumers (verdict parsing today, the #66 metadata contract and #103 verdict v2 tomorrow) need a predictable report language.