docs: say what these three functions do not check, and fix the preset mirror that drifted (#599, #600, #608)

[raeq]

Closes #599. Closes #600. Closes #608.

Three issues with one shape: the name promises more than the function checks, and the documentation stops short of the boundary. Grouping them costs one gate run instead of three and no compiled surface changes shape.

#599 — has_bidi_conflict is not the RLO check

It reads letters, so it is structurally blind to the U+202x overrides. "invoice\u202Egpj.exe" — the best-known bidi spoof there is — returns False.

The docstrings now say that directly and route the reader rather than offering a bare "see also":

input	has_bidi_conflict	inspect_anomalies kind
"invoice\u202Egpj.exe"	False	bidi
"varonis.com.ו"	True	bidi_mixed

Two things the original issue got wrong, now recorded: the conditions are disjoint, not alternatives — a string can satisfy either, both or neither, so a has_bidi_override() alias would be a false promise. And strip_bidi does not close this function's case: on a real-letter conflict it returns the input unchanged, because there is no format character to remove.

docs/concepts/which-function.md gains the two bidi rows its threat-model table lacked. Its only previous mention of bidi was in a cost column, so the page a new user reaches first could not answer "how do I detect a bidi attack".

#600 — and a preset mirror that had drifted

get_pipeline() now states that profile names and PRESETS keys are disjoint namespaces, so get_pipeline("canonicalize") raising is the design rather than a bug.

Checking that claim turned up something worse. PRESETS["ml_normalize"] listed seven steps; src/presets.rs:822 declares [Step; 9]. The mirror was missing the transliterate step and the second demojize that #498 added after strip_accents.

test_preset_steps_exact did not catch it because it compares the mirror against a literal in the test file, and both were written from the same wrong reading. The test pinned the drift instead of detecting it. Both are corrected against Rust here.

The cause is also fixed: list_profiles() said presets are "step-lists defined in Python". They are defined in Rust — PRESETS is a hand-maintained mirror that nothing executes. That sentence is how the drift went unnoticed.

A structural gate is deliberately not in this PR. The step lists use composite variants (FixedPoint, ConfusablesNfcFixedPoint) that the mirror flattens, so a gate that parses the Rust arrays needs per-preset expansion rules. That deserves its own issue rather than a fragile parser bolted onto a documentation change.

#608 — ml_normalize's other two blind spots

Its documented limits stopped at homoglyphs. All twelve bidi controls and every PUA code point also pass through unchanged — strip_control covers Cc, and bidi controls are Cf.

Added to docs/user-guide/llm-pipelines.md as executable assertions, with the contrast that makes them actionable: llm_guardrail clears bidi but not PUA, rag_ingest clears PUA. Doc-only, which is what the issue itself recommended — the preset is tokenizer hygiene and THREAT_MODEL.md never lists it as a security mechanism.

Verification

docs/ fences are executed by Sybil, so the new assertions are live gates rather than prose: pytest docs/user-guide/llm-pipelines.md → 12 passed.

Full local gate: 4,705 pytest, 24 Rust test targets, cargo fmt --check, clippy clean on both feature sets, scripts/perf_lint.sh clean, mypy clean, ruff clean, language-consistency audit clean.

[Copilot]

🟡 Changes recommended

The docs currently overstate ml_normalize behavior (Unicode Tags stripping) and there’s still an internal inconsistency in PRESETS documentation about where preset definitions live.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR clarifies the documented security/coverage boundaries of several bidi- and pipeline-related APIs, and fixes drift in the Python PRESETS["ml_normalize"] mirror so introspection/docs match what the Rust core actually runs.

Changes:

• Document has_bidi_conflict as a letter-direction conflict check (not a bidi-override/RLO check) and route users to the correct detection/removal APIs.
• Clarify the namespace split between policy profiles (get_pipeline / list_profiles) and preset step recipes (PRESETS), and correct PRESETS["ml_normalize"] to match Rust.
• Extend ml_normalize documentation to explicitly call out additional blind spots (bidi controls and PUA).

File summaries

File	Description
tests/test_mutant_killers.py	Updates the pinned expected step list for ml_normalize to include the previously-missing steps.
src/scripts.rs	Expands internal docs for bidi detection, clarifying what has_bidi_conflict does/doesn’t detect.
src/api/safety.rs	Updates public Rust API docs for has_bidi_conflict with explicit examples and guidance.
python/disarm/_presets.py	Clarifies PRESETS vs profiles, fixes ml_normalize preset mirror steps, and updates related docstrings/notes.
python/disarm/_api.py	Updates Python API docs for has_bidi_conflict to warn about RLO/override characters and show the correct check.
docs/user-guide/llm-pipelines.md	Adds executable doc-assertions documenting ml_normalize gaps and contrasts with safer profiles.
docs/concepts/which-function.md	Adds bidi detection/removal rows to the “which function” threat-model table.
CHANGELOG.md	Records the documentation clarifications and the PRESETS["ml_normalize"] mirror drift fix.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📄 Docs preview: https://081f869f.disarm-docs.pages.dev