fix(scanner): 5 correctness and FP improvements#1
Merged
Conversation
- regex: compile custom patterns once before scan loop, not per line - regex: return all matches per line instead of first-match-wins - entropy: expand FP filter with UUIDs, all hash lengths, npm integrity hashes, version strings - ast: collect all sensitive assignments per file instead of stopping at first - mod: fan out L3 semantic inference to top-3 ambiguous candidates in parallel
…rets The CI scan checks the whole file, so test assertion strings that embed realistic-looking fake keys (AWS keys, OpenAI keys, private key headers) were tripping provn's own scanner. All affected lines now carry // provn:allow so the bootstrap problem is suppressed.
parse_file (used by `provn check`) was treating every line as a candidate without applying the provn:allow / provn:skip-file filters that parse_diff_text already had. This meant the annotation only worked in pre-commit scan mode, not when scanning a file directly — the path taken by the CI scan script. Both filters are now applied consistently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
regex_scan.rs): user-defined patterns were compiled fresh on every line of the diff. Now compiled once per scan viacompile_custom_patterns().regex_scan.rs):scan_linepreviously returned on the first match. A line with both an AWS key and an OpenAI key would silently drop one. Now returns all matches sorted by confidence, with two new tests covering this.entropy.rs): added UUID detection, all standard hash lengths (MD5/SHA1/SHA512), npm/yarn/pip integrity hash prefixes, and dotted version strings — the most common sources of day-to-day entropy noise.ast.rs):scan_nodepreviously returned on the first sensitive assignment found during the tree walk. A file with bothsystem_promptandapi_keyassignments would only report one. Now accumulates all matches via&mut Vec. Also replaced binary confidence scoring (length-only) with entropy-aware scoring.mod.rs): the ambiguous pool wasOption<ScanResult>— only the single best candidate reached semantic inference. A diff with 3 ambiguous lines would send 1 to L3 and silently drop 2. Now aVeccapped at 3, fanned out to L3 in parallel viastd::thread::spawn.Test plan
cargo build --release)cargo test)detects_all_sensitive_assignments(new) confirms AST multi-matchreturns_all_matches_on_multi_secret_line(new) confirms regex multi-matchskips_uuid/skips_npm_integrity_hash(new) confirm entropy FP filter