Malayalam (ml) test project: benchmark vs native-speaker reference, calibration with Adisankar, go/no-go

[mmcky]

Why Malayalam needs its own test project

Malayalam is the hardest target language to date, for a structural reason: the policy is inverted relative to every language we ship. For zh-cn/fa/fr the failure mode is under-translation and the tool's job is to translate everything; for ml (#70, #71) the failure mode is over-translation — the tool must keep all technical terms in English and generate Malayalam only as grammatical connective tissue, attaching case-suffixes and auxiliaries directly to English roots (economy-യിലെ, process ചെയ്ത്). None of our existing evaluation habits, review-judge expectations, or intuitions transfer directly, so ml gets a dedicated test project with detailed assessment rounds with Adisankar (native-speaker reviewer, #70) before we seed a real repo.

Tracking issue for that test project. Prerequisite: #71 (ml config, glossary seed, review-judge policy injection) merges first. — #71 merged 2026-07-24 and #70 closed 2026-07-26: Malayalam has shipped in v0.24.0, the policy is recorded as .dev/decisions/D-2026-06-01-malayalam-keep-english-policy.md, and test-translation-sync.ml is a first-class harness language at 26/26 green (#206, #209). So this issue is no longer gated on anything in the engine — it is gated on native-speaker review capacity, which is #207 and now #228.

Assets in hand

• Strategy doc (Malayalam Translation Strategy: Technical Term Handling, Adisankar) — the policy plus 16 worked examples across economics, finance, statistics, and programming.
• Reference translation — Adisankar's hand-crafted Malayalam getting_started.md (source: lecture-python-programming). This is our ground truth for style: keep-English density, morphology attachment, heading treatment. Note it contains a few typos ("eentire", "substanial") and some fully-English sentences, so divergences from it are questions for Adisankar, not automatic errors.
• PR feat: add Malayalam (ml) language support — keep-English-dominant policy #71 — ml language config (7 rules), 52-term glossary seed (47 keep-English + 5 everyday), review-mode policy injection so the judge knows keep-English is deliberate.

The ml advantage: the policy is deterministically checkable

Because the policy is "keep these exact English strings", much of the assessment can be scripted rather than judged by an LLM — and our experience so far is that deterministic checks beat the LLM judge (#157 lineage). Planned metrics, implemented in experiments/ml-benchmark/scripts/ and candidates for graduating into diff-checks.ts as ml-specific gates later:

Metric	Check	Target
Pinned-term retention	each of the 47 glossary keep-English terms appears in English, never in Malayalam script	100%
Beyond-glossary retention	list of ~30 terms the reference keeps English (notebook, cell, kernel, terminal, browser, dashboard, …) stays English	100%
Transliteration detection	no Malayalam-script phonetic renderings of English technical terms	0 instances
Heading fidelity	every heading identical to the English source	100%
Consistency	every repeated term handled the same way (same script, same root) on all occurrences	100%
Morphology pattern	hyphenated suffix attachment (X-യിലെ, X-ന്റെ) present and well-formed	qualitative
Script ratio	Latin vs Malayalam codepoints per paragraph, compared against the reference's band	within band

The script-ratio band is the interesting one: too much Malayalam signals over-translation, too much Latin signals the tool lazily leaving prose untranslated — a single metric that catches both failure directions.

Phases

Phase 0 — prerequisites

• Merge feat: add Malayalam (ml) language support — keep-English-dominant policy #71 — merged 2026-07-24 (c2c52fd). The four open questions (headings, whole-English sentences, proper names, relationship optionality) were NOT yet confirmed by Adisankar; they carry forward into Phase 1's report and Phase 2's flagging protocol.
• Commit Adisankar's reference translation and PLAN.md — merged 2026-07-24 (dev(ml): scaffold the ml benchmark experiment — plan, deterministic metrics, reference slot #191, 1b2e2ea) (reference fetched byte-exact from adisankarmt/quantecon-malayalam@c30578f and verified: 6706 Malayalam codepoints, ZWJ/ZWNJ intact; metrics script validated against it — the reference passes every gate, band mean 0.485 / p10 0.132 / p90 0.707 — corrected 2026-07-28 from 0.127, which was a transcription error unique to this issue; experiments/ml-benchmark/PLAN.md and Phase 1 run-book (Mon 2026-07-28): ml benchmark — Sonnet 5 vs Opus 4.8 vs native reference #194 both record 0.132, and PLAN.md is authoritative as the record committed alongside the calibration)
• Create test-translation-sync.ml — done 2026-07-24 via translate setup (repo + scaffolding + labels bootstrapped). Source-repo ml sync workflow in ci: add Malayalam (ml) sync workflow mirroring the fa harness test-translation-sync#671. Org-level secret access granted to the new repo 2026-07-24 (confirmed by @mmcky)

Phase 1 — re-scoped to the reviewer packet: #228

Re-scoped 2026-07-28. Phase 1 was designed as a two-arm model benchmark that would also produce a report. Both halves were wrong. The benchmark half is n=1 per arm on one file, which #227 puts at ~13% power — the Opus-vs-Sonnet contrast it was meant to settle remains unresolved at p = 0.192 after 26 invocations, so model selection moves to bench/ (#227). The report half is the part that matters, and it is re-scoped from "a report" to one packet that converts what we already have into the smallest set of questions only a native speaker can answer — because Adisankar's availability, not our tooling, is what paces Phases 2–4.

Full plan in #228, which also folds in #207's seed review so he is not asked the same four open #71 questions twice. The old run-book #194 is closed, with a section-by-section disposition recorded there.

• Stage 1 — divergence inventory: translate getting_started.md once (claude-opus-5, --localize none), score it, fix what the FAIL gates catch, reduce to ~30 ranked adjudications (term treatment → morphology → script-ratio outliers)
• Stage 2 — content-mix catalog: ~10 passages from the two harness seed documents, covering the content situations the reference contains none of (display-math prose, admonitions, captions, code-cell prose, nested ####)
• Stage 3 — English-knowns checklist: glossary-review skill over ~5 domain-dense lectures, seed + probe, ranked by whether the two renderings split on keeping a term English
• Deliver REPORT.md (for us) and PACKET.md (numbered questions only) into experiments/ml-benchmark/ — dev(ml): the Adisankar review packet — Stages 1-3 of #228 #231, 2026-07-28
• Get the packet to Adisankar — sent 2026-07-29, as an email attachment rather than GitHub issues. Task-list checkboxes are only editable by users with write access and he is not a collaborator, so the issue-and-checkbox design would have asked him to tick boxes he cannot tick — the failure mode already live on fr#3, 26 boxes never ticked since 2026-07-15. Delivered as experiments/ml-benchmark/malayalam-review-questions.md, 23 answer boxes, with scripts/parse_responses.py to extract the reply

Phase 2 — Adisankar deep assessment (calibration batch)

Phase 1 is complete and closed (#228, delivered by #231 and #233). His reply arrived 2026-08-03 — 23/23 answered (#236) — and everything it decided has shipped: register rules + glossary v0.2.0 (#237, regeneration-verified), the batch translated and delivered, and the review relocated to the real edition lecture-python-programming.ml (PRs 1–5) so his fixes merge into the site (see 2026-08-03 comments). Phase 2 now waits on his PR review comments; the follow-up round (F1–F3, the A5–A7 source sentences) returned 2026-08-06 with all three resolved — A5 a live rule gap queued for v0.3, A6/A7 accepted-as-is (#252).

• Select 3–5 diverse lectures from lecture-python-programming: one prose-heavy, one economics-heavy, one math-heavy, one code-heavy. Phase 1 re-scoped: build the Adisankar packet (divergence inventory, content-mix catalog, English-knowns checklist) #228 section D asks Adisankar to nominate these in the Phase 1 packet, which saves a round trip — our diversity requirement is the constraint, his preference picks inside it — nominated 2026-08-03 (packet D1): python_by_example, numpy, pandas, matplotlib, functions; accepted as-is (dev(ml): Adisankar's packet reply, dispositions, and the follow-up round #236)
• Translate and deliver via test-translation-sync.ml PRs so review happens in the real workflow. There is no "Phase-1-winning model" any more (see Phase 1): use the current default unless bench/: a replicate-based reliability instrument for the document write paths (models × settings × paths) #227 has concluded by then, and record which model produced the batch so the flags stay attributable
• Structured flagging protocol: Adisankar flags each over-translation / awkward rendering as a PR review comment; each flag must resolve to exactly one of (a) glossary entry, (b) language-config rule change, (c) accepted-as-is with rationale — in flight 2026-08-03: instructions emailed (no-deadline framing), he holds triage on the target repo; flags land on lecture-python-programming.ml PRs 1–5
• Fold flags into glossary and rule updates; re-run the affected lectures to confirm the fix generalizes — note: v0.2.0 already shipped from the packet reply (dev(ml): encode the packet rulings — register rules rewrite + glossary v0.2.0 #237), so PR-review flags fold into v0.3

Phase 3 — review-mode calibration

• Run review mode (post-feat: add Malayalam (ml) language support — keep-English-dominant policy #71 judge with policy injection) on all Phase 1–2 outputs and score the judge against Adisankar's flags: does it produce false "untranslated content" findings on compliant text, and does it catch seeded over-translations?
• Adversarial check: hand-inject 5–10 deliberate violations (a translated technical term, a transliterated term, a Malayalam heading) and confirm the judge flags every one as a terminology finding
• Decide which deterministic metrics graduate into diff-checks.ts as ml gates

Phase 4 — decisions and go/no-go

• Model default for ml (Opus 4.8 vs Sonnet 5) from Phase 1–2 quality + cost data — moved to bench/: a replicate-based reliability instrument for the document write paths (models × settings × paths) #227 (2026-07-28). This box assumed Phase 1's two arms would supply the data; they cannot. A per-language default needs replicate-based rates from bench/, not a one-file comparison. Nothing in Phases 1–3 may be read as evidence for it
• Decide the deferred per-term treatment field (feat: add Malayalam (ml) as a target language — research, glossary strategy & model recommendation #70): only if calibration showed ml == en pinning is insufficient — also the last of feat: add Malayalam (ml) language support — keep-English-dominant policy #71's four open questions: relationship parenthetical optionality (rule 7) was NOT in the packet (A12/A13/A10 resolved the other three); observe it in his PR review round
• Glossary v0.2 and rules finalized; docs pass (docs/user/language-config.md, docs/user/glossary.md, glossary/README.md)
• Go/no-go on completing lecture-python-programming.ml — the repo exists with a 5-lecture partial seed since 2026-08-03 (see comments); go = remaining 16 lectures + _static assets, source-repo sync workflow + secrets, .translate/state bootstrap (check --write-state after the five PRs merge), a release containing the register rules with the v0 alias advanced, and deploy

Success criteria for go

All deterministic metrics at target on the calibration batch, zero unresolved Adisankar flags, the review judge produces no false positives on compliant output and catches all seeded violations, and per-file cost is measured and acceptable (fr ran ~$0.21/file real cost; ml should be the same order).

Risks

• The reference is a single lecture from one (programming-flavored) series; economics-heavy prose may behave differently — that's what the Phase 2 diversity requirement is for.
• Adisankar's availability paces Phases 2–4; Phase 1 is fully automatable and should be ready-to-review the day feat: add Malayalam (ml) language support — keep-English-dominant policy #71 merges. This risk is what drove the 2026-07-28 re-scope — it was being made worse, not managed, by Native-speaker review: the ml reference translations seeding the test harness #207 and Phase 1 preparing to put the same four open feat: add Malayalam (ml) language support — keep-English-dominant policy #71 questions to him independently. Phase 1 re-scoped: build the Adisankar packet (divergence inventory, content-mix catalog, English-knowns checklist) #228 is one packet.
• Consistency (the stated primary ml risk, feat: add Malayalam (ml) as a target language — research, glossary strategy & model recommendation #70) is only observable across long documents and multiple documents — the calibration batch must include at least one long lecture.
• A second ml corpus now exists and is not the reference. The two harness seed documents (base-minimal-ml.md, base-lecture-ml.md) are frozen claude-sonnet-5 output at v0.24.0 covering MyST directives, display math and nested #### — constructs the reference contains none of. They are richer, and they are unreviewed machine drafts. Do not pool findings across the two corpora as evidence about one model's behaviour: different documents, different model, no replicates.

Refs #70 (closed), #71. Phase 1 → #228 · model selection → #227 · seed review → #207 · superseded run-book → #194.

[mmcky]

Infrastructure update from 2026-07-25: test-translation-sync.ml is now a fully working, harness-driven target repo.

Before today it was created by translate setup but had no base content, no fixtures, and its sync workflow — added by hand, outside the harness — failed on all 26 PRs of every run with Unsupported target language, because ml was absent from v0.23.0's LANGUAGE_CONFIGS.

After #206/#209 it is a first-class language in the harness:

• three base fixtures seeded by translate init (which applied the ml glossary and passed the tech-debt B: translate init writes model output to disk with no structural-parity guard #159 structural parity guard on write)
• a sync workflow rendered from the shared template, plus review and rebase in the target
• 26/26 sync runs green in the full E2E, with review verdicts posting

So the calibration batch this issue plans has live infrastructure to run on. The two seed documents are also a richer ml sample than the current reference — 17 headings, MyST directives, display math — and are awaiting native review under #207.

Phase 1 (#194) remains unrun.

[mmcky]

Phase 1 is re-scoped and now executes from #228 (added as a sub-issue). The phase structure above changes in three places.

Phase 1's two model arms are retired. #227 — filed off the 2026-07-27 session — measured the Opus-vs-Sonnet contrast at p = 0.192 after 26 invocations and put a post-hoc power calculation on a design of this size at roughly 13%. A single-file, n=1-per-arm comparison cannot decide a model default, and reporting one as if it could is the specific error #227 exists to prevent. Phase 1 runs one arm on claude-opus-5.

Phase 4's first box — "model default for ml" — moves to bench/ (#227) rather than being decided from Phase 1–2 quality data. The other three Phase 4 boxes are unaffected.

Phase 1's deliverable is re-framed from a report to a packet. The binding constraint on Phases 2–4 was always Adisankar's availability, never our tooling, so the output is the smallest set of questions only a native speaker can answer — a ranked, capped divergence inventory adjudicated against his own translation, a content-mix catalog drawn from the harness seeds, and an English-knowns checklist mined from lecture-python-programming. #207's seed review folds into the same packet instead of being a second independent ask, which collapses two round trips into one.

Two smaller corrections while updating the tracker.

Phase 1's first checkbox is stale in a useful way: the harness smoke it implies already happened. #206/#209 took ml from 0/26 to 26/26 through the real workflow with review verdicts posting, so the first live sync-translations-ml.yml run is history rather than pending work — and those verdicts are free Phase 3 evidence, harvested rather than re-generated.

The script-ratio band's p10 disagrees between records. This issue body says 0.127; experiments/ml-benchmark/PLAN.md says 0.132. PLAN.md was committed alongside the calibration run and should be treated as authoritative — worth reconciling before any report quotes the number.

Everything else on this tracker stands: the deterministic-checks-beat-the-judge premise, the keep-English policy, the Phase 2 diversity requirement, and the go/no-go criteria.

[mmcky]

Adisankar's packet reply arrived 2026-08-03 — 23 of 23 questions answered, none skipped. The reply, parsed answers, and a full disposition report are in #236; the rule/glossary changes are in #237 (draft until regeneration evidence lands). Summary for this tracker:

Phase 2 is unblocked. He nominated the calibration batch himself (D1, box ticked above): python_by_example, numpy, pandas, matplotlib, functions — accepted as nominated; numpy + pandas cover the long-document consistency requirement, and about_py holds for the next round as the prose-heavy pick.

The central bet paid off. A9 confirms the light-verb pattern (click ചെയ്യുക, enable ആകും) as the deliberate default for software actions — the single construction behind the one-directional divergence REPORT.md measured. A1 confirms keep-English reaches into ordinary vocabulary (hopefully, example, best…) as a register choice, encoded as one categorial rule rather than 32 glossary entries. A12/A13 formally close the two open #71 questions (headings and proper names stay English).

Glossary v0.2.0 pins exactly five terms (click, option, set, type, press) — the interface verbs machine output transliterated (A14, which he rates a serious defect, pre-authorising the Phase 3 gate graduation of transliteration_check.py). C1 keeps the other 34 frequent terms deliberately unpinned; the policy measured 203/203 holding without pins.

Two of our three "suspected errors" were ours, not the tool's. He declined to rule on A5–A7 without the source sentences, and with the sentences in hand: A6's source line is genuinely comparative ("more work than a cloud option like Colab"), so the tool's -നെക്കാൾ is defensible; A7's source contains two list-like nouns and the packet likely paired different grammatical slots. A follow-up round (malayalam-review-followup.md, in #236) re-asks all three with full sentences. Process rule recorded in DISPOSITIONS.md: case-grammar questions ship with their sentences, and any machine output a packet quotes gets archived — the Stage 1 arm output was not, and its full sentences are unrecoverable.

One new work item from D2: he removed a Nicki Minaj reference in his hand translation and recommends Western cultural references be flagged for human review rather than translated literally — filed separately as a cross-language mechanism question.

Phase 3 inherits: his B1/B3 improved renderings as judge-calibration ground truth for display-math prose, the confirmed A11 defects as seeded-violation test cases, and the A14 severity ruling.