docs: say what these three functions do not check, and fix the preset mirror that drifted (#599, #600, #608)

CHANGELOG.md

@@ -493,6 +493,42 @@ compatibility (see [RELEASING.md](RELEASING.md)).
 
 ### Fixed
 
+- **`PRESETS["ml_normalize"]` was missing two of the nine steps it claims to describe
+  (#600).** `PRESETS` is a hand-maintained Python mirror of the `const STEPS` arrays in
+  `src/presets.rs`; nothing executes it, and it had drifted. The mirror listed seven
+  steps, omitting the `transliterate` step and the second `demojize` that #498 added
+  after `strip_accents`. `test_preset_steps_exact` did not catch it because it compares
+  the mirror against a literal in the test file, and both were written from the same
+  wrong reading — so the test pinned the drift instead of detecting it. Mirror and test
+  are now correct against Rust.
+
+  `list_profiles()` also said presets are "step-lists defined in Python". They are
+  defined in Rust. That sentence is how the drift went unnoticed, so it is corrected
+  too, and a comment on `PRESETS` now states what the dict is and what it is not.
+
+  A structural gate that parses the Rust arrays is **not** part of this change: the step
+  lists use composite variants (`FixedPoint`, `ConfusablesNfcFixedPoint`) that the
+  mirror flattens, so a real gate needs per-preset expansion rules and deserves its own
+  issue rather than a fragile parser bolted on here.
+
+- **Documentation: three functions whose names promise more than they check.**
+  - `has_bidi_conflict` now says plainly that it is **not** the RLO check (#599). It
+    reads letters, so `"invoice\u202Egpj.exe"` returns `False` — the two conditions are
+    disjoint and a string can satisfy either, both or neither. The docstrings route to
+    `inspect_anomalies` (kind `bidi`) for detection and `strip_bidi` for removal, and
+    note that `strip_bidi` does *not* close the real-letter case, because there is no
+    format character to remove. `docs/concepts/which-function.md` gains the two bidi rows
+    its threat-model table lacked; its only previous mention of bidi was in a *cost*
+    column, so the page could not answer "how do I detect a bidi attack".
+  - `get_pipeline()` now states that profile names and `PRESETS` keys are disjoint
+    namespaces, so `get_pipeline("canonicalize")` raising is expected rather than a bug
+    (#600).
+  - `ml_normalize`'s documented limits stopped at homoglyphs. They now cover the other
+    two: all twelve bidi controls and every PUA code point pass through unchanged (#608).
+    `strip_control` handles `Cc`; bidi controls are `Cf`. No behaviour change — the
+    preset is tokenizer hygiene, and `llm_guardrail` / `rag_ingest` already exist for
+    untrusted input.
+
 - **Python can now call `strip_control_chars` and `strip_zero_width_chars` directly
   (#616).** They already existed in the Rust core (`disarm::api`) and in the C ABI,
   Java/Kotlin, Node and Ruby bindings. Python was the only surface without them, so

docs/concepts/which-function.md

@@ -44,6 +44,8 @@ chose, not of confusable mapping.
 |---|---|---|
 | Homoglyph spoofing in a name / address / prose | `normalize_confusables` | Nothing beyond the fold |
 | Homoglyph spoofing in an identifier or hostname | `is_suspicious_hostname` (`analyzeHostname` in Node/Ruby/Java) | Nothing — these report, they do not transform |
+| A bidi attack — detecting one | `inspect_anomalies` — kind `bidi` is a `U+202x` override, kind `bidi_mixed` is a real-letter direction conflict | Nothing — it reports, it does not transform |
+| A bidi attack — removing one | `strip_bidi` for the override; there is no removal for a real-letter conflict, because there is no format character to remove | The `U+202x` characters |
 | Untrusted input into a store or a key | `canonicalize_strict` | Invisibles, bidi, zalgo |
 | Maximum deobfuscation of adversarial text | `strip_obfuscation` | **Accents** |
 | Feeding an uncased model or tokenizer | `ml_normalize` | **Accents and case** |

docs/user-guide/llm-pipelines.md

@@ -270,6 +270,51 @@ when a cased model needs both. The full threat-model-to-entry-point table, with
 each choice costs, is in
 [what each entry point costs you](../security/adversarial-defense.md#what-each-entry-point-costs-you).
 
+### Two more gaps behind the same name
+
+Homoglyphs are the best-known of `ml_normalize`'s blind spots, not the only ones. Its
+pipeline is NFKC, emoji, transliterate, strip_accents, emoji, fold_case, strip_control,
+strip_zero_width, collapse_whitespace. `strip_control` covers the C0 and C1 controls,
+which are category `Cc`. Bidi controls are `Cf`, so they fall straight through, and
+nothing in the list touches the Private Use Area.
+
+```python
+BIDI = "".join(chr(c) for c in
+               (0x202A, 0x202B, 0x202C, 0x202D, 0x202E,
+                0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F, 0x061C))
+
+# All twelve bidi controls survive.
+assert [c for c in BIDI if c in ml_normalize(f"a{c}b")] == list(BIDI)
+
+# So does a PUA code point.
+assert ml_normalize("Summarize.\U000f0000") == "summarize.\U000f0000"
+```
+
+It does remove zero-width fragmentation, and most of the Tags block — but not all of it.
+`U+E0061`–`U+E007A` and the cancel tag go, because the emoji step consumes tag sequences;
+`U+E0001` LANGUAGE TAG survives. Partial coverage of a class is what makes the whole look
+more complete than it is.
+
+```python
+assert ml_normalize("Summarize.\U000e0061") == "summarize."      # tag letter: removed
+assert ml_normalize("Summarize.\U000e0001") != "summarize."      # LANGUAGE TAG: survives
+```
+
+None of this makes `ml_normalize` broken. It is a tokenizer-hygiene preset, and
+`THREAT_MODEL.md` never lists it as a security mechanism. But the name reads as "the
+preset for ML input", which is exactly the pipeline position where a surviving bidi
+control or PUA code point matters. Reach for a profile when the text is untrusted.
+
+```python
+from disarm import get_pipeline
+
+guardrail = get_pipeline("llm_guardrail")
+assert not any(c in guardrail(f"a{c}b") for c in BIDI)   # bidi handled
+assert guardrail("Summarize.\U000f0000") == "summarize.\U000f0000"  # PUA still not
+
+assert get_pipeline("rag_ingest")("Summarize.\U000f0000") == "Summarize."  # PUA handled
+```
+
 ## Which path, and when NOT to use disarm
 
 Being explicit about the path is what earns credibility with this audience —

python/disarm/_api.py

@@ -1756,6 +1756,18 @@ def has_bidi_conflict(text: str) -> bool:
 
     A ``False`` result is **not** a safety guarantee.
 
+    .. warning::
+       **This is not the RLO check.** Because it reads *letters*, it is
+       structurally blind to the ``U+202x`` overrides — the classic extension
+       spoof ``"invoice\\u202Egpj.exe"`` returns ``False`` here. The two
+       conditions are disjoint; a string can satisfy either, both, or neither.
+
+       To cover an override instead, use :func:`inspect_anomalies` (kind
+       ``bidi``) to detect and :func:`strip_bidi` to remove. Note
+       :func:`strip_bidi` does *not* close this function's case: on a real-letter
+       conflict it returns the input unchanged, because there is no format
+       character to remove.
+
     Args:
         text: Input string.
 
@@ -1767,6 +1779,10 @@ def has_bidi_conflict(text: str) -> bool:
         False
         >>> has_bidi_conflict("helloא")  # Latin + Hebrew
         True
+        >>> has_bidi_conflict("invoice\\u202Egpj.exe")  # RLO override, not letters
+        False
+        >>> inspect_anomalies("invoice\\u202Egpj.exe").kinds  # this is the check
+        ['bidi']
     """
     return _has_bidi_conflict(text)
 

python/disarm/_presets.py

@@ -525,6 +525,17 @@ def strip_zalgo(text: str, *, max_marks: int = 2) -> str:
 
 
 # --- Preset pipeline metadata ---
+#
+# The step recipes behind the top-level functions: `PRESETS["canonicalize"]` is what
+# `canonicalize()` runs. These keys are NOT policy-profile names — the two namespaces
+# are disjoint, so `get_pipeline("canonicalize")` raises and `PRESETS["rag_ingest"]`
+# is a KeyError. Profiles live behind `get_pipeline()` / `list_profiles()` (#600).
+#
+# This dict is a hand-maintained MIRROR of the `const STEPS` arrays in
+# `src/presets.rs`. Nothing executes it — it exists for introspection and docs. It has
+# drifted before (`ml_normalize` was missing `transliterate` and the #498 second
+# `demojize`), so when you change a step list in Rust, change it here in the same
+# commit and check `tests/test_mutant_killers.py::test_preset_steps_exact`.
 
 PRESETS: dict[str, list[tuple[str, str | None]]] = {
     "canonicalize": [
@@ -555,7 +566,16 @@ def strip_zalgo(text: str, *, max_marks: int = 2) -> str:
     "ml_normalize": [
         ("normalize", "NFKC"),
         ("demojize", "cldr"),
+        # Only when a `lang` is set, and in Ignore mode: ML pipelines want clean
+        # ASCII-ish output, so an unmapped character is dropped, not preserved.
+        ("transliterate", None),
         ("strip_accents", None),
+        # #498: a second demojize AFTER strip_accents. A negated-relation symbol
+        # (`≇` U+2247) is not in the CLDR name table, so the first pass leaves it;
+        # strip_accents drops the overlay and exposes the bare base (`≅`), which IS
+        # named. Without this pass that base is only named on the following call —
+        # non-idempotent.
+        ("demojize", "cldr"),
         ("fold_case", None),
         # #433: explicit strip steps (was fused into collapse_whitespace).
         ("strip_control", None),
@@ -676,7 +696,9 @@ def strip_zalgo(text: str, *, max_marks: int = 2) -> str:
 
 * ``PRESETS`` (this dict) — *preset* pipelines: fixed, ordered sequences of
   cleaning/normalization steps exposed as the ``canonicalize``,
-  ``ml_normalize``, ``canonicalize_strict`` … helpers. Defined here, in Python.
+  ``ml_normalize``, ``canonicalize_strict`` … helpers. Defined in the Rust core
+  (``src/presets.rs``); this dict is a hand-maintained **mirror** of those step
+  lists for introspection, and nothing executes it.
 * Policy *profiles* (see :func:`list_profiles` / :func:`get_pipeline`) —
   parameter sets for transliteration workflows (e.g.
   ``scholarly_cyrillic_iso9``). Defined in the Rust core (``src/pipeline.rs``).
@@ -710,6 +732,13 @@ def get_pipeline(profile: str) -> TextPipeline:
     and application workflows.  Each call returns a fresh ``TextPipeline``
     instance.
 
+    .. note::
+       A *profile* name is not a :data:`PRESETS` key, and the two sets are
+       disjoint — ``get_pipeline("canonicalize")`` raises. :data:`PRESETS` holds
+       the step recipes behind the top-level functions (``canonicalize``,
+       ``search_key``, …); profiles are ready-made policy pipelines named for a
+       workflow. Call :func:`list_profiles` for the valid values here.
+
     Args:
         profile: Profile name (see :func:`list_profiles`).
 
@@ -732,9 +761,14 @@ def list_profiles() -> list[str]:
 
     Policy profiles (consumed by :func:`get_pipeline`) are distinct from the
     *preset* pipelines in :data:`PRESETS`: profiles are transliteration
-    parameter sets defined in the Rust core, whereas presets are fixed cleaning
-    step-lists defined in Python. A profile name is not a valid preset name and
-    vice versa.
+    parameter sets, whereas presets are the fixed cleaning step-lists behind the
+    top-level functions. A profile name is not a valid preset name and vice
+    versa.
+
+    Both are defined in the Rust core. :data:`PRESETS` is a hand-maintained
+    Python *mirror* of those step lists for introspection — nothing executes it,
+    so treat it as documentation of what Rust runs rather than as the source of
+    truth.
 
     Returns:
         Sorted list of profile name strings.

src/api/safety.rs

@@ -357,6 +357,26 @@ pub fn is_mixed_script(text: &str) -> bool {
 /// CJK/… are left-to-right; Hebrew/Arabic/Syriac/Thaana/N'Ko are right-to-left;
 /// digits, punctuation and combining marks are neutral and never create a
 /// conflict on their own. A `false` result is **not** a safety guarantee.
+///
+/// # This is not the RLO check (#599)
+///
+/// Because it reads letters, it is structurally blind to the `U+202x` overrides.
+/// `"invoice\u{202E}gpj.exe"` — the classic extension spoof — returns `false`
+/// here. The two conditions are **disjoint**: a string can satisfy either, both,
+/// or neither.
+///
+/// | input | `has_bidi_conflict` | [`inspect_anomalies`] kind |
+/// |---|---|---|
+/// | `"invoice\u{202E}gpj.exe"` | `false` | `bidi` |
+/// | `"varonis.com.\u{05D5}"` | `true` | `bidi_mixed` |
+///
+/// To cover an override instead: detect it with [`inspect_anomalies`] (kind
+/// `bidi`), and remove it with [`strip_bidi`]. Note [`strip_bidi`] does **not**
+/// close this function's case — on a real-letter conflict it returns the input
+/// unchanged, because there is no format character to remove.
+///
+/// [`inspect_anomalies`]: crate::api::inspect_anomalies
+/// [`strip_bidi`]: crate::api::strip_bidi
 #[must_use]
 pub fn has_bidi_conflict(text: &str) -> bool {
     crate::scripts::has_bidi_conflict(text)

src/scripts.rs

@@ -350,6 +350,18 @@ pub(crate) fn strong_dir(ch: char) -> Option<StrongDir> {
 /// involved, so [`crate::whitespace`]-style override stripping is a no-op on it.
 /// A `false` result is not a safety guarantee — it only means no strong-LTR and
 /// strong-RTL letters coexist in this string.
+///
+/// # This is not the RLO check
+///
+/// It reads **letters**, so it is structurally blind to the U+202x overrides —
+/// `"invoice\u{202E}gpj.exe"`, the best-known bidi spoof, returns `false` here
+/// (#599). The two conditions are disjoint and a string can satisfy either, both
+/// or neither. For an override:
+///
+/// - detect with [`has_bidi_control`], or [`crate::anomalies`] kind `bidi`;
+/// - remove with `strip_bidi`.
+///
+/// [`has_bidi_control`]: crate::scripts::has_bidi_control
 pub(crate) fn has_bidi_conflict(text: &str) -> bool {
     let mut ltr = false;
     let mut rtl = false;

tests/test_mutant_killers.py

@@ -324,7 +324,9 @@ def test_deprecated_preset_keys_alias_canonical(self):
                 [
                     ("normalize", "NFKC"),
                     ("demojize", "cldr"),
+                    ("transliterate", None),  # only_if_lang, Ignore mode
                     ("strip_accents", None),
+                    ("demojize", "cldr"),  # #498: names the base strip_accents exposes
                     ("fold_case", None),
                     ("strip_control", None),
                     ("strip_zero_width", None),