Nothing detects a compatibility fold: fullwidth and friends canonicalize to ASCII and report clean

[raeq]

Nothing reports a compatibility fold. canonicalize performs one as its first step, so every one of these is neutralized and none is detected — the same asymmetry as #603, #605, #610 and #612, and the last of the rows docs/security/cve-validation.md lists as reported by nothing.

Measured on main at da61f25.

disarm version

0.13.0 + the 0.14.x milestone work.

Minimal reproduction

import disarm

assert disarm.canonicalize("＜script＞") == "<script>"     # neutralized
assert disarm.inspect_anomalies("＜script＞").kinds == []  # and reported clean

Actual output

Every case reports clean:

input	NFKC	kinds
＜script＞	<script>	[]
．．／．．／etc	../../etc	[]
＇ OR 1=1--	' OR 1=1--	[]
ａdmin	admin	[]
ｅxample.com	example.com	[]
ｐａｙｐａｌ	paypal	[]
file	file	[]

The page already says this, in prose

docs/security/cve-validation.md:538, on CVE-2019-9636:

# And there is no detector for it — compatibility-fold unmasking is not one of
# the anomaly kinds, so this input is reported clean.
assert has_anomalies(masked) is False

It is also the mechanism behind the CVE-2024-3098 note on the same page: a safe_eval denylist checked before canonicalization, which the fullwidth spelling walks straight past.

A correction to something #612 asserted. That issue's closing text said the remaining undetected rows each need "a comparison rather than the presence of a character, so no further character class will close them". That is right about the other two and wrong about this one. A compatibility fold is checkable per token — compare the token to its NFKC form — and cheaply. What makes it hard is not detection but false positives.

Scale

Full-range scan, U+0020–U+10FFFF, surrogates skipped:

	count
code points NFKC changes	4,964
of those, fullwidth U+FF01–U+FF5E	94
halfwidth forms U+FF61–U+FFDC	115
fold to an ASCII metacharacter	46

That last row is the security-relevant subset, and it is wider than the fullwidth block: U+037E GREEK QUESTION MARK folds to ;, U+1FEF to a backtick, U+207D/U+208D to (.

The false positives are the whole problem

Fullwidth Latin is ordinary Japanese typography, not an attack. A naive "differs under NFKC" rule is unusable:

rule	catches	false positives
A. token differs under NFKC	9/9	7/11 — Ｑ＆Ａ, ＮＨＫ, 全角１２３, 日本語ＩＴ, ＣＤ－ＲＯＭ, １９９５年, Ｔシャツ
B. differs, and no CJK in the token	9/9	3/11 — Ｑ＆Ａ, ＮＨＫ, ＣＤ－ＲＯＭ
C. differs, and the token also has ASCII alphanumerics	6/9	0/11
E. the fold target contains an ASCII metacharacter	3/9	1/11 — Ｑ＆Ａ, via ＆ → &
F. C or E	6/9	1/11

Legitimate corpus: こんにちは, Ｑ＆Ａ, ＮＨＫ, 全角１２３, 日本語ＩＴ, ＣＤ－ＲＯＭ, １９９５年, Ｔシャツ, plus ASCII, French and Cyrillic controls.

This is not one attack, it is three shapes

The measurement separates them, and they are not equally tractable:

1. Mixed fullwidth and ASCII in one token — ａdmin, ｅxample.com. Nobody types half a word in fullwidth. Rule C catches these with zero false positives, and it is the same style of context gate mixed_script already uses to exempt CJK and is_majority_latin uses for bidi isolates.
2. Folds to a metacharacter — ＜script＞, ．．／．．／etc, ＇ OR 1=1--. Sharper in consequence, because the fold produces a character with syntactic meaning at a sink. The one false positive is ＆ → &, which is arguably not a structural metacharacter and could be excluded.
3. A whole word in fullwidth — ｐａｙｐａｌ, Ｈｅｌｌｏ, １２３. By character class alone this is indistinguishable from ＮＨＫ or １９９５年. Nothing in the measurement separates them, and I do not think anything can without corpus or locale context the library does not have.

Suggested shape

Take (1) first: it is exact on this corpus, needs no new tables, and covers the spoofing case a hostname or identifier screen cares about. Consider (2) as a second kind rather than folding it into the first — the two have different consequences and different false-positive profiles, so one flag would blur them.

Do not take (3) without a decision that a CJK false-positive rate is acceptable, and preferably not at all: a detector that fires on ＮＨＫ is one that CJK-facing callers will turn off entirely, which costs the other two shapes as well.

Worth deciding alongside this: whether the rule belongs in inspect_anomalies (which documents itself as flagging characters "disguising a real word" — a fullwidth ａ disguising a fits that reading) or in is_suspicious_hostname, where shape (1) is most valuable and IDNA already constrains the input.