Skip to content

test: add idempotency property tests for full pipeline #17

Description

Summary

clean() should be idempotent — running it twice on any input must produce the same result. The pipeline has a "re-NFKC" stage specifically for this, but only one test (test_homoglyph_combining_mark_idempotent) validates it.

What to do

Add a TestIdempotency class to tests/test_clean.py with 6-8 test cases covering combinations of pipeline stages:

  • Homoglyph + invisible character combo (e.g., Cyrillic а inside zero-width joiners)
  • Fullwidth ASCII + zero-width spaces
  • Cyrillic + Unicode Tag block characters
  • Greek uppercase + combining marks + bidi overrides
  • Clean ASCII (no-op sanity check)
  • Multilingual legitimate text (CJK + Latin + emoji)

Each test should assert:

assert clean(clean(text)) == clean(text)

For 2-3 cases, also verify stability through 3 iterations.

Files

  • tests/test_clean.py — add new test class

Context

The existing test at line ~117 covers one specific Greek combining mark case. This issue broadens coverage to catch any future regression where adding new homoglyph pairs or invisible char ranges could break idempotency.

Hints

  • Look at src/navi_sanitize/_homoglyphs.py for confusable characters to use as inputs
  • Look at src/navi_sanitize/_invisible.py for invisible character ranges
  • Run tests with: uv run pytest tests/test_clean.py -v --benchmark-disable

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions