fix: close the two CVE rows behind "no single call" (#614, #615)

[raeq]

Closes #614. Closes #615.

These had to land together. Each closes one of the exactly two vectors behind the published claim that no entry point clears everything, and tests/test_cve_vectors.py was written to fail loudly when either closed. Fixing one alone would have left the other failing and forced the guidance to be rewritten twice.

#614 — strip_obfuscation named confusables instead of folding them

49 code points appear in both emoji_single.tsv and confusables_to_latin.tsv, and most are not emoji at all: typographic punctuation, currency, math operators, CJK brackets. They reach the emoji table from CLDR annotationsDerived, which names non-emoji characters.

strip_obfuscation("€xample.com")   before: 'euro xample.com'    after: 'example.com'

The spoof and the genuine host stopped being equal rather than becoming equal — CVE-2017-5383 surviving a preset documented as maximum-strength deobfuscation.

Not the fix the issue proposed

The issue suggested making the TR39 fold take precedence over the naming. Measurement says that breaks idempotency: punctuation inside emoji names (the ’ in "woman’s hat") has to be folded by the confusable pass. The order is documented three times and pinned by tests/test_presets.py:434.

Instead the overlap is derived at build time as an intersection of the two tables, so it cannot drift the way a curated override file would, and demojize skips those rows inside comparison presets only:

strip_obfuscation("€xample.com")   # 'example.com'      — fold wins
demojize("I ❤ €5")                 # 'I red heart euro 5' — naming unchanged
strip_obfuscation("👒")             # "woman's hat"      — name punctuation still folded

build.rs asserts the count is 49, so a table refresh that claims another confusable source fails the build with a message rather than widening the gap silently. That is the drift gate the issue asked for.

#615 — canonicalize cannot cap its way out of an eclipsing mark

The anti-zalgo step is a count. By count, one Arabic shadda is indistinguishable from one acute accent, so no threshold removes CVE-2017-7833's spoof and keeps café.

The discriminator was already in disarm's script data: strip a combining mark whose own Script is a specific script differing from its base's, and keep Inherited marks, which attach to anything. That is UTS #39's mixed-script reasoning applied per grapheme rather than per string.

canonicalize_strict only. The rule is destructive for scholarly transliteration, IPA and linguistic transcription, where marks from one script legitimately sit on bases of another — the corpus least able to notice. canonicalize stays one short deliberately, with a test asserting that rather than leaving it implied.

Nine legitimate samples in five scripts pass through completely unchanged, including Arabic with its own vowel marks, which is the likeliest false positive:

'café' → 'café'      'مُحَمَّد' → 'مُحَمَّد'      'ภาษาไทย' → 'ภาษาไทย'

The ordering is load-bearing, and a property test proved it

I first placed the step before the confusable fold. canonicalize_strict_idempotent failed on "а҉" — Cyrillic base plus a Cyrillic mark. They agree on the first pass; then the fold rewrites the base to Latin a, and the next pass strips the mark. f(f(x)) != f(x).

Deciding against the final base script is the only stable point, so the step now runs after the fold. Worth recording, because the bug is invisible to any example that does not involve a base the fold rewrites.

The guard is inverted, not deleted

test_no_single_entry_point_clears_everything asserted assert missed, f"{name} now clears everything — update the guidance". It did exactly what it was written to do. It now asserts that the two sufficient entry points stay sufficient and every other one stays short, so a regression in either direction fails.

Measured over the full 24-row matrix: canonicalize_strict and strip_obfuscation each clear everything. catalog_key misses only the Tags block; canonicalize misses only the eclipsing mark.

The published advice is unchanged; its reason moved. From "nothing suffices" to "the two that suffice are the two most destructive ones" — the same conclusion for a caller who has to forward the text they cleaned.

Verification

4,818 pytest, 24 Rust test targets (674 unit tests), cargo fmt --check, clippy clean on both feature sets, scripts/perf_lint.sh clean, mypy clean, ruff clean, language-consistency audit clean. docs/ fences execute under Sybil, so the rewritten guidance is checked rather than asserted — 43 blocks pass.

[github-actions]

📄 Docs preview: https://eea3075e.disarm-docs.pages.dev

[Copilot]

🟡 Changes recommended

The new skip_tr39_claimed demojize path can unintentionally concatenate emoji-name output with later-folded letters (e.g. 👒€ → woman's hate), which is a correctness regression for strip_obfuscation output.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR closes the two CVE vectors (#614, #615) that previously supported the documented claim that no single entry point neutralizes every vector, by adjusting comparison-preset behavior in strip_obfuscation and strengthening canonicalize_strict against cross-script combining-mark eclipsing.

Changes:

• Derive (at build time) the 49-codepoint overlap between emoji_single.tsv and TR39 confusables, and teach comparison presets to skip naming those rows so TR39 folding wins (#614).
• Add a canonicalize_strict-only step to drop combining marks whose Script is a specific script different from their base character’s Script (#615).
• Update CVE-matrix tests and security docs to reflect that strip_obfuscation and canonicalize_strict are now sufficient, while other entry points intentionally remain short.

File summaries

File	Description
tests/test_cve_vectors.py	Updates CVE characterization tests and inverts the “no single call” guard into “two sufficient entry points remain sufficient”.
src/zalgo.rs	Adds strip_cross_script_marks_into to remove cross-script, non-Inherited combining marks (used only by canonicalize_strict).
src/tables/mod.rs	Adds lookup helper for the derived emoji/TR39 overlap set.
src/tables/emoji_data.rs	Includes the generated PHF set for the overlap codepoints.
src/presets.rs	Adds the new preset step and extends demojize step config with skip_tr39_claimed; wires both into canonicalize_strict / strip_obfuscation.
src/pipeline.rs	Updates the pipeline demojize call for the new function signature (passes skip_tr39_claimed=false).
src/emoji.rs	Extends demojize_rust_into to optionally skip naming the TR39-overlap codepoints so later confusable folding can act on them.
proptest-regressions/presets.txt	Records a new proptest regression seed related to idempotency ordering changes.
docs/security/cve-validation.md	Updates security guidance and examples reflecting the closed mirror pair and the new “two calls suffice” framing.
CHANGELOG.md	Documents the behavioral/security fixes and the rationale/order constraints.
build.rs	Generates the overlap set as an intersection and asserts the overlap size is exactly 49 as a drift gate.

Review details

• Files reviewed: 11/11 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.