Skip to content

fix(i18n): teach the wrong-language detector what it can actually detect - #387

Merged
github-actions[bot] merged 1 commit into
mainfrom
fix/i18n-detector
Aug 26, 2026
Merged

fix(i18n): teach the wrong-language detector what it can actually detect#387
github-actions[bot] merged 1 commit into
mainfrom
fix/i18n-detector

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

Third time in three PRs that this gate flagged correct Italian and Spanish — "Trovare un tecnico", "Encontrar a quien lo repare", "LO QUE HACEMOS" and seven more, none of them defects. Twice I verified them by hand and bumped the baseline. The rule says the third time you stop fixing instances and close the class.

Why it was wrong

Not a threshold problem — holes in the stop-word lists.

un was listed only under fr. Italian had una/uno but not the masculine un, so "Trovare un tecnico" scored Italian 0.00, French 0.33 and got flagged. Spanish was missing lo and a while Italian had lo, which is why "Encontrar a quien lo repare" looked Italian. A hole in a list makes correct text look foreign.

Filling the holes then made it worse in the other direction: teaching es the word su immediately made correct Italian ("Linux su vecchi ThinkPad") look Spanish, and e, non, nos did the same across the other pairs. Sibling Romance languages share their function words. A bag-of-stop-words comparison cannot separate them, and no amount of dictionary work will change that.

What it does now

It detects what it can detect: untranslated English or German sitting in a translated file. Those function words barely overlap with Romance or CJK ones, and that's the leak that actually happens here — DE is the source language and EN the pivot. Sibling-language comparison is dropped, and the limitation is documented in the code rather than left as a surprise.

Three guards, each one earned:

  • LEAK_SOURCES = ['en','de'] — only judge against languages it can tell apart.
  • MIN_TOKENS_FOR_LANGUAGE_GUESS = 6 — the heuristic's resolution is 1/N; on a three-word string one shared article moves the ratio by 0.33, clearing the threshold on its own.
  • A script guard — because \w is ASCII-only in JavaScript even under /u, so [^\W\d_] counted Hangul, kana and Cyrillic as zero letters and my first attempt at this guard silently did nothing. It uses \p{L} now. Without it, a Korean sentence containing "evig" and "Revamp-IT" scores as English on those crumbs alone.

Proved by mutation, not asserted: 16 cases — eleven correct-language strings (it/es/fr/ko/ja/ru, including CJK carrying Latin brand names) all clean, five real English/German leaks all still caught.

What the fix immediately found

Real untranslated English shipped to five locales. es, fr, it, ja and ko were all rendering English inside the IT-Hilfe hub:

  • "Describe your issue and receive offers from technicians"
  • "Browse profiles and contact someone directly"
  • "Help as a technician — requests near you"
  • "Register as a technician and help people nearby."

plus the privacy page's data-export sentence in ja and ko. The keys existed, so the missing-key audit passed — only the values were English, and the old detector couldn't see them. All translated here.

This is the same class as itHelp.hub.subtitle in #382, which I found by reading a rendered page. The gate finds them now.

And a dead storefront

shop.* held 140 keys; three loading/error boundaries use six. The other 134 describe a storefront whose pages are pure redirect() calls to /marketplace — product grids, cart labels, category breadcrumbs, opening hours — shipped in every visitor's page payload in eight languages.

Pruned: ~54 KB of JSON, 6511 deleted lines. Liveness checked by namespace prefix, not just call sites (the lesson from home.newsletter in #383), then proved by typecheck.

Baseline lowered in the same commit per the ratchet rule: 706 message findings → 84, with 632 resolved and 0 new.

Folded in from #386

That PR deadlocked against this one — its new Italian and Spanish headlines were flagged by the very detector this repairs, so neither could go green alone.

  • /it-hilfe still showed "Computer reparieren, OS neu installieren, IT-Probleme lösen" as its H1 — a keyword list. feat(brand): evig repairs becomes evig technicians, and evig stops asking for printers #382 reframed the metadata and subtitle and missed the line a visitor reads first. It now matches the homepage pillar's link text exactly, so clicking "Jemanden finden, der es repariert" lands on a page with that headline.
  • "CHF 300.00" wrapped onto two lines on a 390px pool card. A price must not wrap; the title beside it gives way instead.

Verification

npm run verify green: lint, umlauts, chrome, docs, compliance, typecheck, 7746 tests, production build.

🤖 Generated with Claude Code

Third time in three PRs that this gate flagged correct Italian and Spanish —
"Trovare un tecnico", "Encontrar a quien lo repare", "LO QUE HACEMOS" and
seven more, none of them defects. Twice I verified them by hand and bumped
the baseline. The rule says the third time you stop fixing instances and
close the class.

## Why it was wrong

Not a threshold problem — HOLES in the stop-word lists. `un` was listed only
under fr; it had `una`/`uno` but not the masculine `un`. So "Trovare un
tecnico" scored Italian 0.00, French 0.33 and got flagged. Spanish was
missing `lo` and `a` while Italian had `lo`, which is why "Encontrar a quien
lo repare" looked Italian. A hole in a list makes CORRECT text look foreign.

Filling the holes then made it worse in the other direction: teaching es the
word `su` immediately made correct Italian ("Linux su vecchi ThinkPad") look
Spanish, and `e`, `non`, `nos` did the same across the other pairs. Sibling
Romance languages share their function words. A bag-of-stop-words comparison
cannot separate them, and no amount of dictionary work will change that.

## What it does now

It detects what it CAN detect: untranslated **English or German** sitting in
a translated file. Those function words barely overlap with Romance or CJK
ones, and that is the leak that actually happens here — DE is the source
language and EN the pivot. Sibling-language comparison is dropped.

Three guards, each earned:
  - LEAK_SOURCES = ['en','de'] — only judge against languages it can tell apart.
  - MIN_TOKENS_FOR_LANGUAGE_GUESS = 6 — the heuristic's resolution is 1/N; on
    a three-word string one shared article moves the ratio by 0.33, clearing
    the threshold on its own.
  - A script guard, because \w is ASCII-ONLY in JavaScript even under /u, so
    `[^\W\d_]` counted Hangul, kana and Cyrillic as ZERO letters and my first
    attempt at this guard silently did nothing. It uses \p{L} now. Without it
    a Korean sentence containing "evig" and "Revamp-IT" scores as English on
    those crumbs.

Proved by mutation, not by assertion: 16 cases, eleven correct-language
strings (it/es/fr/ko/ja/ru, including CJK carrying Latin brand names) all
clean, five real English/German leaks all still caught.

## What the fix immediately found

**Real untranslated English shipped to five locales.** es, fr, it, ja and ko
were all rendering English inside the IT-Hilfe hub — "Describe your issue and
receive offers from technicians", "Browse profiles and contact someone
directly", "Help as a technician — requests near you", "Register as a
technician and help people nearby." — plus the privacy page's data-export
sentence in ja and ko. The KEYS existed, so the missing-key audit passed;
only the values were English, and the old detector could not see them. All
translated here.

This is the same class as itHelp.hub.subtitle in #382, which I found by
reading a rendered page. The gate finds them now.

## And a dead storefront

`shop.*` held 140 keys; three loading/error boundaries use SIX. The other 134
describe a storefront whose pages are pure `redirect()` calls to /marketplace
— product grids, cart labels, category breadcrumbs, opening hours — shipped
in every visitor's page payload in eight languages. Pruned: ~54KB of JSON,
6511 deleted lines. Liveness checked by namespace prefix, not just call
sites (the lesson from home.newsletter), then proved by typecheck.

Baseline lowered in the same commit, per the ratchet rule: 706 message
findings → 84, with 632 resolved and 0 new.

Also folded in (was #386, which deadlocked against this: its new Italian and
Spanish headlines were flagged by the very detector this fixes):

  - /it-hilfe still showed "Computer reparieren, OS neu installieren,
    IT-Probleme lösen" as its H1 — a keyword list. #382 reframed the metadata
    and subtitle and missed the line a visitor reads first. It now matches the
    homepage pillar's link text exactly, so clicking "Jemanden finden, der es
    repariert" lands on a page with that headline.
  - "CHF 300.00" wrapped onto two lines on a 390px pool card. A price must not
    wrap; the title beside it gives way instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit 3d3bd8e into main Aug 26, 2026
9 checks passed
@github-actions
github-actions Bot deleted the fix/i18n-detector branch August 26, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant