Skip to content

Add scan() API: return structured findings with offsets + vector class #38

Description

Motivation

Downstream consumers (notably nboot audit in navi-bootstrap) need to locate and classify hostile-character findings in source files, not just produce a sanitized string. The current public API is:

clean(text: str, escaper: Callable | None = None) -> str
walk(data: Any, escaper: Callable | None = None, max_depth: int = 128) -> Any
detect_scripts(text: str) -> dict[str, int]
is_mixed_script(text: str) -> bool
decode_evasion(text: str) -> str

clean() gives "the safe version" but throws away everything we need for actionable reporting:

  • Where in the input the problem lives (line:col / char offset / byte range).
  • What class of problem it is (zero-width / homoglyph / bidi-override / C0-C1 control / NFKC-bypass / mixed-script / evasion-encoded).
  • Whether a given file even needs action (currently you have to call clean(text) != text and then you've still lost the classification).

The internal pipeline already has this information — each of the six stages (null-byte strip, invisible strip, NFKC, homoglyph, NFKC re-pass, escaper) classifies what it's stripping/replacing. It's just not surfaced.

Requested API

from navi_sanitize import scan, Finding, Vector

findings: list[Finding] = scan(text)
# Finding is a small dataclass, e.g.:
#   offset: int              # char offset in the ORIGINAL (pre-NFD) text
#   length: int              # char length of the offending run
#   vector: Vector           # enum: ZERO_WIDTH, HOMOGLYPH, BIDI_OVERRIDE,
#                            #       CONTROL_CHAR, NFKC_BYPASS, MIXED_SCRIPT,
#                            #       NULL_BYTE, EVASION_ENCODED
#   codepoints: tuple[int, ...]   # the actual code points involved
#   replacement: str | None  # what clean() would substitute (or None for strips)

An equivalent file-level helper would also be useful:

scan_file(path: pathlib.Path, encoding: str = "utf-8") -> list[Finding]

…which preserves the offset but also adds line / column derived from the file content.

Why it can't be built externally today

A third-party "invisible char scanner" library can't just call clean(text) != text — that tells us there's something wrong, not what. Reimplementing the sanitizer's classification tables in a parallel detector would drift from navi-sanitize's 492-char invisible table, 66 homoglyph pairs, NFD-first ordering, etc. The whole value proposition of this library is "one audited, deterministic pipeline" — we should ship the classification as a first-class output instead of hiding it.

Suggested implementation sketch

  • Refactor the internal pipeline to yield findings alongside its transformations (functional-core variant), then have clean() collapse findings into the transformed string as today.
  • Keep clean()'s semantics identical (idempotent, deterministic, legitimate Unicode preserved).
  • Add scan() as a thin wrapper that returns findings without the final string collapse.

Downstream use case

nboot audit <target> in navi-bootstrap will walk text files under a target repo, call scan() on each, and emit SARIF 2.1.0 so findings appear in the GitHub Security tab alongside Semgrep/CodeQL alerts. Per-finding line:col + vector-class are required for SARIF to be useful (GitHub dedupes results by rule + location fingerprint).

Constraints

  • No new runtime deps (keep the zero-dep promise).
  • clean() byte-for-byte unchanged on the existing test corpus.
  • Additive-only API — no breaking changes to existing surface.

Happy to send a draft PR if the design shape is agreed. CC Nelson Spence (@Fieldnote-Echo).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions