feat(ml): encode native review round 2 — glossary v0.4.0, rules 18 → ~25, first deterministic ml lints

[mmcky]

Round 2 of the Malayalam calibration programme (Phase 2 of #189) is reviewed: Adisankar left 118 committable suggestion blocks on QuantEcon/lecture-python-programming.ml#7 (functions, regenerated at 8625221 with glossary v0.3.0 + the 18-rule set from #272). Per-flag dispositions, class analysis and the evidence behind every item below are in QuantEcon/project-translation reports/2026-09-01-ml-functions-review-disposition.md. Headline: the round-1 rules held — no recurrence of below-deixis, ordinal, light-verb, -യുടെ or suffixed-English flags across the lecture — and the new classes are mostly deterministic. Proposed dispositions: 75 rule / 19 glossary / 24 accepted-as-is. This issue also absorbs the v0.3.1 candidates parked in #273's body and QuantEcon/project-translation#44: the majority-technical-bullet rule and the line/lines pin are both field-corroborated; the In fact calque had no instance to test.

Rules (src/language-config.ts, ml.additionalRules)

1. Terminal punctuation (26 flags): every prose paragraph ends with a full stop; a sentence introducing a following code cell, list or displayed equation ends with a colon — regardless of the English source's punctuation; a fronted ശ്രദ്ധിക്കുക, (note that) or ഓർക്കുക: (recall that) carries its own punctuation.
2. Sentence-initial capitalisation (14): a retained-English word opening a Malayalam sentence or list item — including inside {ref}/{doc} link text — is capitalised (Functions-നെ, Built-in, Return statement, Order, User-defined).
3. Hortative teacher voice (12): we will / we'll / let's / we now → നമുക്ക് … -ആം (കാണാം, നോക്കാം, എഴുതാം, ചെയ്യാം), never നമ്മൾ … -ും; discuss / see / show as lecture verbs → നോക്കാം / കാണാം; consider the following … → താഴെ … നോക്കാം; a lecture-overview list is in the present (പഠിക്കുന്നു). Extends rule 11 with ~10 before/after pairs from the round.
4. Accusative consistency (15): the object of any transitive light verb takes -നെ, plurals included (functions-നെ define, code-നെ simplify, conditionals-നെ ഒഴിവാക്കാം); no additive -ഉം on bullet-list items; coordinated code names → `a`, `b` എന്നിവ; a call to X → X-നെ call ചെയ്യുമ്പോൾ. Tightens rule 3.
5. Fixed pairs (10): a given N → തന്നിരിക്കുന്ന N, never ഒരു നൽകിയ N; consider X (invitation to look) → X നോക്കാം / consider ചെയ്യുക, with കണക്കിലെടുക്കുക reserved for "take into account"; in one line → ഒറ്റ line-ൽ; spelling കുറച്ചുകൂടി.
6. Sandwich sentences (3): when a source sentence is interrupted by code cells ("these definitions [cell] and [cell] are equivalent"), state the whole claim before the first cell and emit nothing between or after the cells — a verb-final fragment after the second cell is unnatural in SOV order.
7. Discourse connectives, extension (7): sentence-initial For example, ⏳ (Q1), In particular,, Obviously,, Basically, stay English, alongside the existing In fact / On the other hand.
8. Exercise-statement scope, extension of D-2026-08-17 (12) — ruled and accepted by @mmcky 2026-09-01, decision record D-2026-09-01-ml-exercise-statements-stay-english (rides the encoding PR): exercise statements whose content is probabilistic or mathematical reasoning stay English; a mixed sentence keeps its mathematical clause English and translates its programming-instruction clause; pointer sentences (Here's a function for the first random device.) stay English. Also reinforce rule 6 with the three misses as worked examples (`plot(x, 'b-')` differs from `plot('b-', x)`, the two pointer sentences).

Glossary (glossary/ml.json → v0.4.0)

• Keep-English pins: useful (×4), already (×4 ⏳ Q3), name — the noun for a bound identifier; പേര് remains for the act of naming (×5 ⏳ Q5), example / examples (×3 ⏳ Q2), provide, exist, work (⏳ Q4 — reviewer kept പ്രവർത്തിക്കുന്നു 2:1), automatically, create, improve, full, stage, answer, successive, and the corroborated line / lines.
• Avoid-renderings (a new optional avoid field, or context text the lint reads): ഉപയോഗപ്രദമായ, ഇതിനകം, നിലവിലുണ്ട്, സൃഷ്ടിക്കുക, ഒരു നൽകിയ, കണക്കിലെടുക്കുക (invitation sense), കുറച്ചുകൂടെ, plus round 1's ആവശ്യപ്പെടുന്നു and കൈകാര്യം.
• Everyday-Malayalam entries (reverse direction, rule-5 class): straightforward → എളുപ്പം; contrived → കൃത്രിമം; limited → പരിമിതി; facilitate → സഹായിക്കുക; hit / reach (a return statement) → എത്തിച്ചേരുക; a bad idea → അത്ര നല്ലതല്ല.

Deterministic checks (prototype in ml_metrics, candidates for diff-checks.ts graduation under #189 Phase 3)

• (i) terminal punctuation on prose paragraphs adjacent to fenced cells / lists;
• (ii) a prose line or list item beginning with a lowercase Latin letter;
• (iii) banned-rendering substrings driven from the glossary's avoid entries;
• (iv) the നമ്മൾ … -ും future-hortative pattern — prototype only, expected to be noisy.

These are string checks, not ratios, so they are unaffected by #240 (homograph retention) and #229 (script-ratio band) and can run on every ml sync PR as soon as they exist.

Verification

• Harness-first per standing policy (test-translation-sync.ml).
• Regeneration check as dev(ml): archive the round-1 regeneration arm — the rules generalise (#189 Phase 2 close) #273 did for round 1: translate functions from lecture-python-programming@81d9734 with the new payload and diff against the reviewed text on ml#7 after the batch-apply; report class-signature hit rate (round 1 hit 20/23).
• Retention checks and the script-ratio band must treat an all-English exercise statement as compliant, not under-translation (same consequence D-2026-08-17 recorded for hints/solutions).

Also in scope

• Decision record for the codification hold ("do not make ഇത് a rule yet") — QuantEcon/project-translation#44 box 2, still missing from .dev/decisions/.
• After merge: cut the release so the rules reach production before round 3 (matplotlib) is generated.

Refs: #189, #272, #273, #240, #229, QuantEcon/lecture-python-programming.ml#7, QuantEcon/project-translation#44, QuantEcon/project-translation reports/2026-09-01-ml-functions-review-disposition.md.

[mmcky]

Encoded and regeneration-checked on branch ml-round2-v04 (local for now, PR to follow). 1251b7b: rules 18 → 24, glossary v0.4.0 (+16 terms, 77 total), the four round-2 lints in ml_metrics.py (LINT bucket, never FAIL), docs and CHANGELOG; the ruling-A decision record rides as 878bde8. Full suite 1538 green, lint and format clean, dist-action rebuilt.

Lint calibration on real text: the pre-review seed (5ffae4e) scores 23 bare endings / 17 lowercase-initial / 15 banned renderings / 6 hortative-watch; the editor's reviewed text (ml main@c1200fa) scores 1 / 0 / 2 / 1, and every residual is a form he himself kept — so the lints measure the round's classes and do not flag his register.

Regeneration (same recipe as #273: functions from source 81d9734, claude-sonnet-5, --localize none; 25.8k tokens, 3.4 min; archived under experiments/ml-benchmark/arms/2026-09-01-round2-rules-sonnet5/): 17 of 26 signatures land at or past the reviewed form on one fresh pass — every fixed pair (തന്നിരിക്കുന്ന 4, ഒരു നൽകിയ 0, കണക്കിലെടുക്ക 0, ഉപയോഗപ്രദ 0, കുറച്ചുകൂടെ 0), no additive -ഉം on list items, the sandwich sentence, a, b എന്നിവ, the three connectives, all four exercise-scope retentions (ex3 device, ex2 sentence, ex1 hybrid, math hint) and both pointer sentences. Direction right but half-magnitude on the hortative (നമുക്ക് 9 → 14 vs reviewed 21).

Where the prompt under-delivers — the two mechanical classes. Terminal punctuation went 23 → 16 bare endings, not → 1; capitalisation 17 → 5 (all five on bullet items). Every one of the 16 is a Malayalam sentence directly before a code cell, and in the reviewed text every such line ends with a colon or full stop. That is the strongest case yet for a deterministic post-processing step (bare Malayalam ending before a {code-cell} → :; lowercase Latin word at the head of a prose line or list item → capitalised) rather than more prompt — W2-shaped (#260); the lints already detect both on every PR so nothing ships silently.

Logged, one instance each: the ex5 statement ("Rewrite the function factorial() … using recursion" — pure programming) stayed English, an over-read of the new scope rule that needs the negative example named in the rule before release; set was transliterated (സെറ്റ് ചെയ്ത്) — a real pinned-term FAIL the gate caught, not the #240 homograph class; refer calqued once (round-1 pin). Also a source-side typo surfaced: "in from" at lecture-python-programming functions.md:672.

Held for ml#12 answers before encoding: "For example," (the arm chose English 3×, matching our guess), and pins for already / name / example(s) / work.

[mmcky]

Shipped in v0.27.0 (#297 merged as 58b9c69; release #298 125801c; §4a gate 81/81 sync runs, 27/27 delivery + 27/27 stamped verdicts per lane; @v0 advanced and smoke-verified on all three lanes). Production now carries rules 18 → 24, glossary v0.4.0 and the round-2 lints; round 3 (matplotlib) can be generated at @v0. Still open from this issue, deliberately: the items held for the editor's answers on QuantEcon/lecture-python-programming.ml#12 ("For example" rule; pins for already / name / example(s) / work), and the deterministic post-processing step the regeneration arm argues for (bare Malayalam ending before a code cell → :; lowercase Latin word opening a line → capitalised) — W2-shaped, #260.

[mmcky]

Round-3 generation at v0.27.0 (matplotlib, QuantEcon/lecture-python-programming.ml#13) — one engine miss worth logging alongside the held items: the heading ### Matplotlib's Split Personality was rendered ### Matplotlib-യുടെ Split Personality (and the heading-map entry with it), i.e. an English possessive inside a heading attracted the genitive suffix despite the keep-headings-English rule. Restored by hand in the seed. One instance; candidate rule sharpening: a heading is kept byte-identical including possessives and punctuation, and the heading-fidelity FAIL gate already catches it (as it did here). Everything else on that seed was clean: zero transliterations, casing 0, round-2 lints 0/1/0/1, the only pinned-term flag the #240 returns verb false positive.

[mmcky]

The ml#12 answers are in (2026-09-01) and encoded on #303. Five of six word choices matched our guesses; work did not — he keeps the light-verb form work ചെയ്യുന്നുണ്ടോ, so the pin is English, not പ്രവർത്തിക്കുന്നു. Line 159 is confirmed a typo. The font needs no change.

The exercise answer reverses rule 8 above. He has not decided the exercise boundary and asks that, for now, all exercise-related content stay in English byte-for-byte, to be decided per exercise later. So D-2026-09-01-ml-exercise-statements-stay-english is superseded by D-2026-09-03-ml-all-exercise-content-stays-english, and the ex5 "over-read" logged in my first comment was correct behaviour and is withdrawn. Because the new rule has no judgement boundary it is enforced in code rather than prompt: verbatim-directives.ts restores every {exercise} / {exercise-start}…{exercise-end} / {hint} / {solution} / {solution-start}…{solution-end} block from the source on all three write paths before the parity guard, and a verbatimDirectives diff-check gates the review as a blocker finding (no schema change). First fully mechanical ml scope rule — the #260 shape.

Also on #303: For example joins the connectives; glossary v0.5.0 pins already / example(s) / name / work; the refer calque is named in the light-verb rule; the heading rule says byte-identical including possessives (the Matplotlib-യുടെ miss); rules 24 → 23. E2E scenario 28 adds the suite's first exercise-family fixture — nothing in the harness could exercise this before. QuantEcon/lecture-python-programming.ml#14 restores functions and python_by_example (4/10 and 7/10 blocks carried Malayalam); matplotlib (ml#13) was already compliant.

Still open here after #303: the deterministic post-processing for bare endings and sentence-initial capitalisation (#260).

[mmcky]

Shipped in v0.28.0 (#303 merged as 07e7c64; release #304 9284fbc; §4a gate 84/84 sync runs, 28/28 delivery + 28/28 stamped verdicts per lane; @v0 advanced and smoke-verified on all three lanes). Production now carries the verbatim exercise-family policy, glossary v0.5.0 and the 23-rule set; round 4 can be generated at @v0. What remains open here is only the deterministic post-processing for bare endings and sentence-initial capitalisation, tracked on #260.