fix(retrieval): map Arabic-Indic and other non-ASCII digits to ASCII

[wikithoughts]

What changed

Follow-up to #33 (review F10 + the minor fidelity number item, task T8). #33 already shipped the Unicode-aware tokenizer (casefold, NFKD mark-folding, CJK bigrams) and thousands-separator joining ('12,500' == '12500', including the Arabic separator U+066C). Its own follow-up list noted one gap left open: Arabic-Indic digits were not mapped to ASCII.

Reproduced on origin/main (f497835):

• grounding_score('بلغت الإيرادات 12,500 دولار [S1].', 'بلغت الإيرادات ١٢٬٥٠٠ دولار') → missing_numbers: ['12500'], num_cov: 0.0, content_cov: 0.75. A correct claim is flagged as a fabricated statistic and fed to self-repair.
• tokenize('عام ٢٠٢٤') → ['عام', '٢٠٢٤'], so a BM25 query for 2024 can't find an Arabic document that writes the year in Arabic-Indic digits.

Fix: retrieval._fold (the non-ASCII path only) now maps every Unicode decimal digit (category Nd: Arabic-Indic U+0660–0669, Persian U+06F0–06F9, Devanagari, …) to its ASCII digit via unicodedata.decimal. The ASCII fast path is untouched. fidelity.tokenize already routes through it, so the grouped-number join plus the digit map makes ١٢٬٥٠٠ ≡ 12,500 ≡ 12500. The only fidelity.py change is the module docstring.

Files: passiveworkers/retrieval.py, passiveworkers/fidelity.py (docstring), tests/test_fidelity.py, tests/test_retrieval.py.

Tests (fail on old code)

• test_fidelity.py::test_arabic_indic_digits_match_ascii_digits
• test_retrieval.py::test_tokenize_maps_non_ascii_digits_to_ascii
• test_retrieval.py::test_bm25_finds_arabic_indic_year_from_ascii_query

I checked this by running them with retrieval.py/fidelity.py reverted to origin/main: 3 failed, 41 passed. With the fix: 44 passed.

Verification (venv with .[all,dev], Python 3.12)

• python -m py_compile $(git ls-files '*.py') → exit 0
• ruff check . && pytest tests/ -q --cov=passiveworkers --cov-fail-under=75 → exit 0. Output: All checks passed!, 743 passed, 1 skipped, Total coverage: 83.39%.
• Frontend guard and browser check: not applicable (no served HTML/JS touched).

Not verified / known limits

• CI on this PR was not observed (lane rule: no CI polling).
• Pre-existing behaviour from fix(retrieval): Unicode-aware tokenizer and grouped-number fidelity #33, unchanged here: mark-folding drops Devanagari virama/matras (वर्ष → वरष). Query and document fold the same way, so matching still works, but tokens aren't real words. Arabic stop-words are still not filtered. Thai and other unspaced non-CJK scripts are still not segmented.
• Arabic decimal separator U+066B (٤٫٢) splits like ASCII 4.2 does. That is consistent with the existing decimal policy (significant_numbers ignores decimals).

🤖 Generated with Claude Code