Summary
clean() should be idempotent — running it twice on any input must produce the same result. The pipeline has a "re-NFKC" stage specifically for this, but only one test (test_homoglyph_combining_mark_idempotent) validates it.
What to do
Add a TestIdempotency class to tests/test_clean.py with 6-8 test cases covering combinations of pipeline stages:
- Homoglyph + invisible character combo (e.g., Cyrillic
а inside zero-width joiners)
- Fullwidth ASCII + zero-width spaces
- Cyrillic + Unicode Tag block characters
- Greek uppercase + combining marks + bidi overrides
- Clean ASCII (no-op sanity check)
- Multilingual legitimate text (CJK + Latin + emoji)
Each test should assert:
assert clean(clean(text)) == clean(text)
For 2-3 cases, also verify stability through 3 iterations.
Files
tests/test_clean.py — add new test class
Context
The existing test at line ~117 covers one specific Greek combining mark case. This issue broadens coverage to catch any future regression where adding new homoglyph pairs or invisible char ranges could break idempotency.
Hints
- Look at
src/navi_sanitize/_homoglyphs.py for confusable characters to use as inputs
- Look at
src/navi_sanitize/_invisible.py for invisible character ranges
- Run tests with:
uv run pytest tests/test_clean.py -v --benchmark-disable
Summary
clean()should be idempotent — running it twice on any input must produce the same result. The pipeline has a "re-NFKC" stage specifically for this, but only one test (test_homoglyph_combining_mark_idempotent) validates it.What to do
Add a
TestIdempotencyclass totests/test_clean.pywith 6-8 test cases covering combinations of pipeline stages:аinside zero-width joiners)Each test should assert:
For 2-3 cases, also verify stability through 3 iterations.
Files
tests/test_clean.py— add new test classContext
The existing test at line ~117 covers one specific Greek combining mark case. This issue broadens coverage to catch any future regression where adding new homoglyph pairs or invisible char ranges could break idempotency.
Hints
src/navi_sanitize/_homoglyphs.pyfor confusable characters to use as inputssrc/navi_sanitize/_invisible.pyfor invisible character rangesuv run pytest tests/test_clean.py -v --benchmark-disable