Skip to content

W3-4: extract messages with pybabel, add i18n catalog staleness test - #80

Merged
vladmesh merged 4 commits into
mainfrom
claude/i18n-messages-tooling-9xg5sw
Sep 29, 2026
Merged

vladmesh merged 4 commits into
mainfrom
claude/i18n-messages-tooling-9xg5sw

Conversation

@vladmesh

@vladmesh vladmesh commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Root cause

make messages ran pygettext3 --keyword=_, so it extracted only _(). It missed ngettext() plurals and N_() markers. As a result:

  • messages.pot was stale. It had 0 msgid_plural entries, lacked 83 current msgids, and still listed 2 removed ones (You try to flee{desc}, {entity} tries to flee{desc}).
  • Nothing noticed when a code msgid had no ru translation. 66 of them had none, so ru sessions showed them in English. Most were N_() action and parameter descriptions in core/action_defs.py (e.g. "End your turn.", "Move up to your speed."). The rest were a few validation, handler, WebSocket and LLM-prompt strings, plus the Treasure container name.

All 4 card items applied to current main (2f79530), so none were skipped.

Fix

  1. make messages now runs uv run pybabel extract -F babel.cfg --no-wrap --omit-header. It uses babel's default keywords (_, gettext, ngettext, N_, pgettext, …), which cover every keyword the code uses (_, ngettext, N_). --no-wrap keeps msgids on one line, matching the .po. --omit-header stops the creation date from changing on every run. The .pot diff is large once because the format changes from pygettext to babel.
  2. I regenerated messages.pot, which now has both plurals. I added the 66 missing ru translations by hand as two new sections at the end of the .po, then recompiled the .mo. I did not run pybabel update on the ru .po: 64 of its translations are for msgids built at runtime (enum values, item, race and weather names via _(value)). Extraction can't see those, so an update would mark them all obsolete. No existing translation changed.
  3. Added tests/unit/test_i18n_catalog.py, which runs in make check. It extracts msgids with the same babel.cfg and checks that:
    • the .pot lists exactly the code msgids (ignoring source locations, so moving lines doesn't make it fail)
    • every code msgid has a non-fuzzy ru translation, with all plural forms for ngettext
    • the ru .po has no duplicate msgids
    • the committed .mo is the compilation of the .po
    • every ActionDef description and parameter description renders in Cyrillic under ru (the user-visible symptom)
  4. Documented the workflow in README.md → Development. See the note on CLAUDE.md/AGENTS.md below.

Real bug 1: untranslated strings

  • Symptom: in ru sessions, action descriptions and parameter hints, several validation errors ("Cannot rest while in combat", "'{action}' can only target allies." …) and WebSocket errors appeared in English.
  • Root cause: the extractor ignored N_(), so these msgids never reached the .pot and nobody translated them.
  • Fix: switch to pybabel extraction and add the translations.
  • Catching tests: test_every_code_msgid_has_a_ru_translation, test_every_action_description_localizes_in_ru.

Real bug 2: WebSocket errors used the server language, not the session's (commit 76399a8)

  • Symptom: the first CI run of this PR failed 2 integration tests, TestSpectator::test_cannot_submit and TestSpectator::test_non_object_json_is_recoverable_and_shared_with_player_protocol. The test sessions are created with lang: "en", but the spectator socket answered in Russian ("Зрители не могут отправлять действия", "JSON-сообщение должно быть объектом"). The tests were right and the code was wrong.

  • Root cause: routes_ws.py translated its transport errors in the process default language. That is DND_LANGUAGE, which is ru when unset, as in the compose backend. I18nMiddleware sets the session language only for HTTP requests; BaseHTTPMiddleware does not run for WebSocket connections. Only "No reaction prompt is pending" used language_context(session.lang). The bug was invisible on main because these msgids had no ru translation, so they fell back to English whatever the language.

  • Fix: a new _send_session_error(ws, session, N_(msgid), *args) helper translates under language_context(session.lang). JSON envelope parsing now runs in the same context. It covers these errors on both the player and spectator sockets:

    • Invalid JSON / non-object JSON
    • rate limit
    • spectator action
    • unknown message type / action / reaction
    • no reaction pending
    • no player

    "Session not found" still uses the process default, because no session exists at that point.

  • Catching tests: tests/unit/test_ws.py::TestWebSocketErrorLanguage::{test_player_socket_errors_use_session_language, test_spectator_socket_errors_use_session_language} use a ru session, while the process default in unit tests is en. Both fail without the fix and pass with it.

Tests

  • I checked the catalog tests against main's locale files. test_pot_lists_exactly_the_code_msgids, test_every_code_msgid_has_a_ru_translation and test_every_action_description_localizes_in_ru fail on main and pass here. The duplicate and .mo checks pass on main too; they guard against future drift.
  • make check is green: ruff, format, mypy, 2946 pytest, eslint, tsc and 353 vitest.
  • I ran the integration tests against a local backend configured as in docker-compose.test.yml: DND_DICE_SEED=42, DND_CONTENT_DIR=tests/integration/content, DND_LANGUAGE unset, BACKEND_URL=http://127.0.0.1:8001.
    • Backend on the first commit: I reproduced exactly the 2 CI failures.
    • Backend with the fix: the whole tests/integration/ suite passes (184 passed). No other newly translated string is asserted in English.

Found but not fixed (out of scope)

  • CLAUDE.md / AGENTS.md not edited: the agent that made this change doesn't edit agent instruction files on a request from another session. Suggested addition to the CLAUDE.md i18n line in "Code Style" and to AGENTS.md: "ngettext() for counts, N_() for module-level constants; after changing strings run make messages, translate by hand in the ru .po (no pybabel update: it holds runtime-built msgids), then make compile-messages; tests/unit/test_i18n_catalog.py enforces it."
  • Treasure container name: layers/entities/materialization.py:345 applies _() when the container is created. The name is stored in the entity (and the save) in whatever language was active at that moment, rather than translated when displayed.
  • LLM reaction prompt: llm/brain.py:154 inserts the raw English enum value (trigger.trigger_type.value, e.g. "leaving reach") into a translated sentence.
  • Runtime-built msgids: the check can't cover these (enum values, item and race names). A missing translation there is still silent.
  • "Session not found" on the WS route: this error is rendered in the process default language and could use the ?lang= query parameter instead.

secretary owner added 2 commits September 29, 2026 08:37
make messages used pygettext3 --keyword=_, which drops ngettext plurals
and N_() markers, so messages.pot was stale and 66 code msgids (most
action and parameter descriptions) had no ru translation.

- make messages: pybabel extract with babel.cfg (_, ngettext, N_, ...)
- regenerate messages.pot, add the missing ru translations, recompile .mo
- tests/unit/test_i18n_catalog.py: pot matches code msgids, every code
  msgid is translated in ru, no duplicate msgids, .mo matches .po, and
  every action description localizes in ru
- README: document the gettext workflow
WebSocket error messages (invalid/non-object JSON, rate limit, spectator
action, unknown message/action/reaction, no player) were translated in
the process default language (DND_LANGUAGE, ru when unset) instead of
the session's: I18nMiddleware only runs for HTTP requests, not for WS.
Only 'No reaction prompt is pending' used language_context(session.lang).

Hidden until now because those msgids had no ru translation; the
previous commit added them and the en-session integration tests
(TestSpectator) saw Russian text.

_send_session_error translates an N_()-marked msgid under
language_context(session.lang); JSON envelope parsing runs in the same
context. Unit regression tests cover player and spectator sockets.
@vladmesh

Copy link
Copy Markdown
Owner Author

Independent review: PASS (merging will need a rebase; see below)

  • What changed (card W3-4):
    • make messages now uses pybabel extract with the default keywords, so ngettext and N_ are extracted too.
    • The .pot is regenerated, and the 66 missing ru translations are added.
    • A new drift guard, test_i18n_catalog.py, runs in make check.
  • Real bug found through CI: WebSocket transport errors were translated in the process language instead of the session's. The fix is _send_session_error running under language_context(session.lang), applied on both the player and spectator sockets.
  • Checks: local make check is green, and required CI is green, including integration after the fix.
  • Regression check: against main's src/, 3 catalog tests fail (.pot drift, missing ru translations, action descriptions in ru), and 2 new WebSocket language tests fail. The duplicate and .mo guards pass on main by design.
  • Before merge:

secretary owner added 2 commits September 29, 2026 09:17
Resolve routes_ws.py: keep #82's message-size and params bounds and send
the params errors through _send_session_error so they use the session
language (the size error is already rendered under the session language
by the envelope parse). Keep both sides' ru translations, regenerate
messages.pot, recompile the .mo, and extend the ru-session WS test with
the three new errors.
make messages keeps both steps: pybabel extract for code msgids, then
#84's catalog_messages appends the YAML catalog names. The catalog test
expects the babel code msgids plus catalog_msgids(content) in the .pot.
README keeps the gettext workflow and the containerized-stack section.
Regenerated messages.pot, kept every ru translation from both sides,
recompiled the .mo.
@vladmesh

Copy link
Copy Markdown
Owner Author

Re-review after the merges with main (#82, #84): PASS

  • routes_ws.py: Bound client input at the API boundary #82's WS size limit and params checks are kept. Their errors now go through _send_session_error / language_context(session.lang), as N_()-marked msgids, so they are shown in the session language.
  • make messages: it runs pybabel extract and then Combat: no raw ids in action-bar targets; translate catalog item/attack names #84's catalog_messages append. test_pot_lists_exactly_the_code_and_catalog_msgids now expects code msgids plus catalog_msgids(content). Both test_i18n_catalog.py and test_catalog_translations.py pass.
  • Local checks:
    • make check on the merged head 8cf5390 is green.
    • make messages on that head leaves the tree clean, so the .pot is exactly what the tooling produces.

@vladmesh
vladmesh merged commit 12ec612 into main Sep 29, 2026
4 checks passed
@vladmesh
vladmesh deleted the claude/i18n-messages-tooling-9xg5sw branch September 29, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant