test: add ten encoding CVEs, and record how rows are found - #621
Merged
Conversation
Registry 36 -> 46. Found by sweeping NVD across the operations disarm actually performs rather than by recalling known attacks, then verifying each ID against the REST API before writing anything down. The byte-level rows are the first on this page whose input is not a str. Every other vector arrives as text that already decoded; these are attacks on the decoder itself. CVE-2024-46954 Ghostscript, overlong UTF-8 -> ../ traversal CVE-2026-44288 protobufjs, overlong accepted and decoded CVE-2009-4142 PHP htmlspecialchars, overlong + invalid Shift_JIS/EUC-JP All three are not-affected, measured rather than asserted. decode_to_utf8 replaces overlong sequences instead of decoding them, so the traversal never materializes, and strict=True refuses outright rather than returning a lossy string. CVE-2026-44288's description names the correct behaviour in so many words -- protobufjs decoded them "to canonical characters instead of replacing them" -- which is what makes the claim checkable. Verified the gate bites by making the decoder accept overlong and watching the suite go red. CVE-2009-4142's trick is the other half: an invalid multibyte lead byte placed before a special character, so the escaper downstream never sees the character. The payload survives here rather than being swallowed, which is the property worth asserting. Lone surrogates: CVE-2022-31116 UltraJSON, key confusion in dictionaries CVE-2025-64439 LangGraph, illegal surrogates -> fallback to insecure deserialization. A Unicode edge case reaching RCE through an error handler, scored under CVSS v4.0. CVE-2008-4066 Firefox, HTML-escaped low surrogate -- out of scope, because � is ordinary ASCII until an HTML parser decodes it Substituting rather than dropping is what prevents the CVE-2022-31116 shape: a dropped surrogate would make key<U+DC00>value collide with keyvalue, which is the bug rather than the fix. CVE-2007-2688 is the Threat Model's ordering rule eighteen years early. Cisco IPS, Check Point (CVE-2007-2689) and IBM ISS Proventia (CVE-2007-2690) shipped the same missing normalization step in the same month -- three vendors matching before they normalized. It is also the same fold TestFullwidthUnmaskingHazard pins as a hazard, and both readings are correct: folding before a detector is the fix, folding before an output sink is the hazard. Pipeline position decides. CVE-2001-0669 (%u encoding) and CVE-2022-3782 (double URL encoding) are out of scope for the same reason: there is no Unicode there yet. disarm exposes percent_encode and no decoder at all, deliberately -- how many times to decode is a property of the protocol stack. CVE-2006-2753 is out of scope because escaping is not disarm's job and never was. The whole encoding class is undetected, like the terminal class before it. Every row is neutralized or not-affected and none is reported; decode_to_utf8 returns had_errors, which is a return value rather than a panel predicate. The page now records how rows are found: the sweeps by mechanism rather than by product, and the per-ID verification that caught CVE-2017-20190 having no CVSS score at all. Also the non-CVE research that informed rows -- Paul Butler on variation-selector smuggling, which is the Tags-block channel's sibling that no CVE covers, and the CoreText Telugu crash, whose trigger was a zero-width non-joiner in the ordinary word for "knowledge". Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude Code:claude-opus-5
Owner
Author
|
Closing and reopening to re-trigger CI. Every workflow on this branch was queued during a runner-allocation outage at 15:39 UTC — CI and DCO ended in |
|
📄 Docs preview: https://a6cb6e09.disarm-docs.pages.dev |
Self-review, because Copilot errored on this PR and reviewed nothing. It found the same kind of gap the self-review on #609 did. The CVE is about escaping failing, not decoding. PHP decoded and escaped in one pass, so an invalid Shift_JIS lead byte consumed the following "<" and htmlspecialchars never saw a character to escape. My test asserted only that the decode was lossy and that "<script>" survived it, which is the first half of the story and does not reach the thing the CVE is about. The claim now runs to the sink: the decoder substitutes and keeps the "<", and escape_html then escapes it. That two-stage separation is the actual reason disarm does not reproduce this, so it is what the test should say. Parametrized over both byte shapes the CVE names, invalid Shift_JIS and overlong UTF-8. Verified the new assertion bites by neutering escape_html. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude Code:claude-opus-5
raeq
added a commit
that referenced
this pull request
Aug 26, 2026
#621 added ten encoding CVEs to the same registry this branch re-pins, so the conflict was real rather than textual. Resolving it mechanically would have been wrong: git put #621's nine new UNDETECTED_IN_SCOPE rows inside CLOSED_BY_THE_CONTROL_KIND, which claims the opposite of what they are. Resolved by structure, then verified by measurement rather than by reasoning — and the measurement found one more row than expected. CVE-2009-4142's probe is '�\x00<script>', a NUL-byte injection, so the control branch reports it: eight rows closed, not seven, and has_anomalies covers 19 rather than 18. Its detectors, disposition and rendered docs row moved with it. The other two byte-level rows carry no control and stay silent, which keeps the section's point intact: everything still undetected needs a comparison — a fold collision, a length budget, a decode result — rather than the presence of a character. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude Code:claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Registry 36 → 46 rows, covering encoding rather than code points, plus the survey method that found them.
The byte-level rows are new in kind
Every other vector on the page arrives as text that already decoded. These are attacks on the decoder itself, and their input cannot be written as a Python
strat all.../traversalhtmlspecialchars— overlong + invalid Shift_JIS/EUC-JPAll three are not affected, measured rather than asserted:
CVE-2026-44288's description names the correct behaviour in so many words — protobufjs decoded overlong sequences "to canonical characters instead of replacing them". That is what makes the claim checkable rather than a shrug. Verified the gate bites by making the decoder accept overlong and watching the suite go red.
CVE-2009-4142's trick is the other half: an invalid multibyte lead byte placed before a special character, so the escaper downstream never sees it. The payload survives here rather than being swallowed, which is the property worth asserting.
Lone surrogates, including an RCE through an error path
CVE-2025-64439 is the one worth reading: illegal surrogates made msgpack serialization fail in LangGraph, and the fallback was JSON deserialization of untrusted data. A Unicode edge case reaching RCE through an error handler. Scored under CVSS v4.0.
Substituting rather than dropping is what prevents CVE-2022-31116's shape — a dropped surrogate would make
key<U+DC00>valuecollide withkeyvalue, which is the bug rather than the fix.CVE-2008-4066 is out of scope:
jav�ascriptis ordinary ASCII until an HTML parser decodes it.CVE-2007-2688 is the ordering rule, eighteen years early
Cisco IPS, Check Point (CVE-2007-2689) and IBM ISS Proventia (CVE-2007-2690) shipped the same missing normalization step in the same month — three vendors matching before they normalized.
It is also the same fold
TestFullwidthUnmaskingHazardpins as a hazard, and both readings are correct. Folding before a detector is the fix; folding before an output sink is the hazard. Pipeline position decides.CVE-2001-0669 (
%uencoding) and CVE-2022-3782 (double URL encoding) are out of scope for the same reason: there is no Unicode there yet. disarm exposespercent_encodeand no decoder at all, deliberately — how many times to decode is a property of the protocol stack.The whole encoding class is undetected
Like the terminal class before it. Every row is neutralized or not-affected and none is reported —
decode_to_utf8returnshad_errors, which is a return value rather than a panel predicate.The survey method is now on the page
Rows are found by sweeping NVD by mechanism rather than by product, then verifying each ID individually. The page records the sweeps, and two habits worth copying:
ANSI escape sequencereturned twenty CVEs across terminals, loggers, VCS clients and an LLM agent. No list of products would have found that set.It also credits the non-CVE research that informed rows: Paul Butler on variation-selector smuggling (the Tags-block channel's sibling, which no CVE covers) and the CoreText Telugu crash, whose trigger was a ZWNJ in the ordinary word for "knowledge".
Verification
Mutation-checked, including a new case for this class: making
decode_to_utf8accept overlong sequences turns the suite red.No Rust file is touched.
🤖 Generated with Claude Code