Motivation
Downstream consumers (notably nboot audit in navi-bootstrap) need to locate and classify hostile-character findings in source files, not just produce a sanitized string. The current public API is:
clean(text: str, escaper: Callable | None = None) -> str
walk(data: Any, escaper: Callable | None = None, max_depth: int = 128) -> Any
detect_scripts(text: str) -> dict[str, int]
is_mixed_script(text: str) -> bool
decode_evasion(text: str) -> str
clean() gives "the safe version" but throws away everything we need for actionable reporting:
- Where in the input the problem lives (
line:col / char offset / byte range).
- What class of problem it is (zero-width / homoglyph / bidi-override / C0-C1 control / NFKC-bypass / mixed-script / evasion-encoded).
- Whether a given file even needs action (currently you have to call
clean(text) != text and then you've still lost the classification).
The internal pipeline already has this information — each of the six stages (null-byte strip, invisible strip, NFKC, homoglyph, NFKC re-pass, escaper) classifies what it's stripping/replacing. It's just not surfaced.
Requested API
from navi_sanitize import scan, Finding, Vector
findings: list[Finding] = scan(text)
# Finding is a small dataclass, e.g.:
# offset: int # char offset in the ORIGINAL (pre-NFD) text
# length: int # char length of the offending run
# vector: Vector # enum: ZERO_WIDTH, HOMOGLYPH, BIDI_OVERRIDE,
# # CONTROL_CHAR, NFKC_BYPASS, MIXED_SCRIPT,
# # NULL_BYTE, EVASION_ENCODED
# codepoints: tuple[int, ...] # the actual code points involved
# replacement: str | None # what clean() would substitute (or None for strips)
An equivalent file-level helper would also be useful:
scan_file(path: pathlib.Path, encoding: str = "utf-8") -> list[Finding]
…which preserves the offset but also adds line / column derived from the file content.
Why it can't be built externally today
A third-party "invisible char scanner" library can't just call clean(text) != text — that tells us there's something wrong, not what. Reimplementing the sanitizer's classification tables in a parallel detector would drift from navi-sanitize's 492-char invisible table, 66 homoglyph pairs, NFD-first ordering, etc. The whole value proposition of this library is "one audited, deterministic pipeline" — we should ship the classification as a first-class output instead of hiding it.
Suggested implementation sketch
- Refactor the internal pipeline to yield findings alongside its transformations (functional-core variant), then have
clean() collapse findings into the transformed string as today.
- Keep
clean()'s semantics identical (idempotent, deterministic, legitimate Unicode preserved).
- Add
scan() as a thin wrapper that returns findings without the final string collapse.
Downstream use case
nboot audit <target> in navi-bootstrap will walk text files under a target repo, call scan() on each, and emit SARIF 2.1.0 so findings appear in the GitHub Security tab alongside Semgrep/CodeQL alerts. Per-finding line:col + vector-class are required for SARIF to be useful (GitHub dedupes results by rule + location fingerprint).
Constraints
- No new runtime deps (keep the zero-dep promise).
clean() byte-for-byte unchanged on the existing test corpus.
- Additive-only API — no breaking changes to existing surface.
Happy to send a draft PR if the design shape is agreed. CC Nelson Spence (@Fieldnote-Echo).
Motivation
Downstream consumers (notably
nboot auditinnavi-bootstrap) need to locate and classify hostile-character findings in source files, not just produce a sanitized string. The current public API is:clean()gives "the safe version" but throws away everything we need for actionable reporting:line:col/ char offset / byte range).clean(text) != textand then you've still lost the classification).The internal pipeline already has this information — each of the six stages (null-byte strip, invisible strip, NFKC, homoglyph, NFKC re-pass, escaper) classifies what it's stripping/replacing. It's just not surfaced.
Requested API
An equivalent file-level helper would also be useful:
…which preserves the
offsetbut also addsline/columnderived from the file content.Why it can't be built externally today
A third-party "invisible char scanner" library can't just call
clean(text) != text— that tells us there's something wrong, not what. Reimplementing the sanitizer's classification tables in a parallel detector would drift fromnavi-sanitize's 492-char invisible table, 66 homoglyph pairs, NFD-first ordering, etc. The whole value proposition of this library is "one audited, deterministic pipeline" — we should ship the classification as a first-class output instead of hiding it.Suggested implementation sketch
clean()collapse findings into the transformed string as today.clean()'s semantics identical (idempotent, deterministic, legitimate Unicode preserved).scan()as a thin wrapper that returns findings without the final string collapse.Downstream use case
nboot audit <target>innavi-bootstrapwill walk text files under a target repo, callscan()on each, and emit SARIF 2.1.0 so findings appear in the GitHub Security tab alongside Semgrep/CodeQL alerts. Per-findingline:col+ vector-class are required for SARIF to be useful (GitHub dedupes results by rule + location fingerprint).Constraints
clean()byte-for-byte unchanged on the existing test corpus.Happy to send a draft PR if the design shape is agreed. CC Nelson Spence (@Fieldnote-Echo).