Skip to content

fix: Data quality filters, keyboards, and dead code cleanup#119

Merged
Hugo0 merged 1 commit into
feat/frequency-word-listsfrom
fix/data-quality-keyboards
Feb 21, 2026
Merged

fix: Data quality filters, keyboards, and dead code cleanup#119
Hugo0 merged 1 commit into
feat/frequency-word-listsfrom
fix/data-quality-keyboards

Conversation

@Hugo0

@Hugo0 Hugo0 commented Feb 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • Pipeline filters: Blocklist, Roman numeral, and proper-noun filtering for daily word candidates — prevents blocklisted words (Hebrew had 88% overlap!), Roman numerals (Breton had 107), and common names (aaron, david, alice...) from appearing as daily answers
  • Greek sigma normalization: Fix 15 supplement words with wrong final sigma (σ→ς) + pipeline prevention
  • German blocklist: Add 3 offensive/inappropriate terms (arsch, fotze, nsdap)
  • Keyboard layouts: Add proper layouts for 6 languages (mk, hy, hyw, eo, eu, br) — replacing auto-generated alphabetical fallbacks
  • Tests: New TestDailyWordQuality class checking blocklist overlap and Roman numerals
  • Dead code cleanup: Remove commented-out code in app.py
  • Regenerated all 38 language daily words and supplements with new filters

Test plan

  • uv run pytest tests/ -x — 1908 passed, 363 skipped, 24 xfailed
  • pnpm test — 54 passed (3 suites)
  • uv run black --check — all formatted
  • Hebrew: 0 blocklist overlap (was 1,763)
  • German: no arsch/fotze/nsdap in daily words
  • Breton: 0 Roman numerals (was 107)
  • Greek: 0 supplement words ending in σ
  • All 6 new keyboards pass character coverage tests

Pipeline improvements:
- Filter blocklisted words from daily word rotation (Hebrew: 88% overlap → 0%)
- Filter Roman numerals from daily words and supplements (Breton: 107 → 0)
- Filter common proper nouns (387 international names) from daily words
- Normalize Greek final sigma (σ→ς) in supplements with de-dup

Data fixes:
- German blocklist: add arsch, fotze, nsdap
- Greek supplement: fix 15 words with wrong final sigma

Keyboard layouts for 6 languages:
- Macedonian (mk): Cyrillic layout based on Serbian
- Armenian (hy, hyw): typewriter + phonetic layouts
- Esperanto (eo): QWERTY with accented letters
- Basque (eu): QWERTY with ñ, ç
- Breton (br): AZERTY with ê, ñ, ù

Tests: add TestDailyWordQuality (blocklist overlap + Roman numeral checks)
Cleanup: remove dead code in app.py
Regenerate daily words and supplements for all 38 languages
@coderabbitai

coderabbitai Bot commented Feb 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/data-quality-keyboards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@Hugo0

Hugo0 commented Feb 21, 2026

Copy link
Copy Markdown
Owner Author

Squashed into PR #118

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant