fix: an empty mapping is a decision, not a failure, so Preserve honours it (#602)

[raeq]

Closes #602.

The confusion

A character the table maps to the empty string is not unknown. It is a decision the table already made: "this has no ASCII form, drop it."

ErrorMode governs what happens to characters the table has nothing to say about. But the mapped/unmapped test excepted Preserve from empty mappings, reading an empty mapping as a kind of failure the caller had asked to keep:

Some(_) => error_mode != ErrorMode::Preserve,  // empty → preserve keeps original

So the three presets that pass Preserve — search_key, catalog_key, sort_key — kept 134 code points that transliterate() deletes. TextPipeline(transliterate=True) passes Ignore and dropped them correctly, and that divergence is what made the bug visible.

catalog_key made it worse

It runs the confusable fold after transliteration, so a leaked Cyrillic soft sign was folded onto Latin b:

catalog_key("Пьеса")  ->  'pbesa'      before
catalog_key("Пьеса")  ->  'pesa'       after

pbesa contains a letter that appears in neither the input nor its romanisation. For a deduplication key, that is the sharpest form of the problem.

The fix

One expression: let is_mapped = mapped.is_some();

A genuinely unmapped code point still reaches handle_unmapped, so Preserve keeps doing its job for the case it exists for — U+3400 still survives, and there is a test asserting it rather than leaving it implied.

find_untranslatable_impl passes Ignore and already treated these as translatable, so the two consumers of this predicate now agree instead of disagreeing.

Verified by a full-range scan reproducing the issue's own predicate over U+0020–U+10FFFF: zero leaks.

A property test had to be restated, not just updated

transliterate_preserve_nonempty asserted that Preserve never returns empty output. That held only because of the exception removed here — and it could not have been true in general anyway: a string of nothing but empty-mapped characters legitimately transliterates to nothing. Its generator already excluded \p{M} "which legitimately map to empty", which was the same problem showing through in a narrower form.

It is now stated as the invariant that actually defines the mode:

Ignore drops both the unmapped and the empty-mapped. Preserve differs only in keeping the unmapped verbatim, so its output can never be shorter.

That needs no generator exclusions at all, and says something stronger than the original.

Verification

Full local gate: 4,709 pytest, 24 Rust test targets, cargo fmt --check, clippy clean on both feature sets, scripts/perf_lint.sh clean, mypy clean, ruff clean, language-consistency audit clean.

Blast radius is four call sites: catalog_key, search_key, transliterate_preserving_latin_into (used by sort_key), and public transliterate(on_unknown="preserve"). The last is a documented behaviour change for 134 characters — a fix with a CHANGELOG note at 0.x, not a break.

[Copilot]

🟡 Changes recommended

The new full-range Python regression test performs ~1.1M iterations with repeated per-iteration work and should be refactored to avoid unnecessary overhead to keep pytest runtime under control.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes a transliteration/preset-policy mismatch where characters with an explicit empty table mapping ("" = “drop”) were incorrectly treated as “unmapped” under ErrorMode::Preserve, causing search_key/catalog_key/sort_key to leak 134 code points that transliterate() deletes.

Changes:

• Treat “present but empty” mappings as mapped (mapped.is_some()), so all error modes honor deliberate empty mappings.
• Restate the Rust property test to assert the defining invariant of Preserve vs Ignore (output length never shorter), instead of incorrectly asserting non-empty output.
• Add Python regression coverage for #602, including a targeted soft-sign case and a full-range scan, plus a CHANGELOG entry documenting the behavior fix.

File summaries

File	Description
src/transliterate.rs	Fixes mapped/unmapped classification for empty mappings and updates the relevant property test invariant.
tests/test_presets.py	Adds regression tests ensuring presets don’t leak empty-mapped characters and that genuinely unmapped code points are still preserved.
CHANGELOG.md	Documents the behavioral fix and its impact on preset key functions.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.