Repository navigation
Feat/m5 corpus - #9
Merged
Merged
Conversation
A scanner's worth is precision and recall on real input, not that it compiles, and for Brazilian data precision is the hard one: the upstream secret rules score candidates against an English dictionary, so "senhapadrao" reads as random and gets reported. That cost is invisible without labeled data. This is the data, and the test that gates on it. Format is one directive per line: "expect <type> <value>" for a true positive, "reject <value> # reason" for a false positive. The reason is required on a reject, because a false-positive fixture nobody can explain is one nobody can safely remove. The parser rejects a malformed line rather than skipping it — a corpus that silently drops a case reports a precision it did not measure. The gate is strict: every expect found, every reject quiet. That is right for hand-labeled data where each line is a deliberate claim; a threshold is for scraped corpora. Precision and recall are printed regardless, because the numbers are the point and should be seen to move. Writing the fixtures already earned its keep: two were mislabeled and the harness caught both. 00.000.000/0001-91 is the Uniao's real, valid CNPJ, not the reject I wrote; and a +55 phone is correctly a pix-key now, not the card-pan it used to be read as. Both are the kind of error that only shows up against labeled data. cases/false-positives.txt is the running record of what scanning the real world produced — the Go pseudo-version timestamp, the UUID, the phone — each one a finding that was once wrong. 31 cases today, precision 1.000, recall 1.000.
… newlines Two divergences in how document types tolerated separators, resolved into one rule: a group boundary accepts the document's own punctuation, a space, or a tab, and never a newline. The first divergence was a latent bug. CNS and título used "[\s.]", and \s matches a line break, so either could match a value split across two lines — the exact failure the card-number pattern was already written to avoid. Now none of them can. The second was an inconsistency the architecture notes had flagged and deferred: CPF, CNPJ and PIS rejected a space outright, so "529 982 247 25" was not caught while a space-separated CNS was. The worry about admitting a space was that it matches any three-column table of three-digit numbers. That worry is now measured instead of guessed — corpus/cases/cpf.txt has the table-column case, and the check digit a column does not satisfy keeps it quiet. Precision stays 1.000 across 36 cases, recall 1.000. The deferred note in ARCHITECTURE.md is replaced with what was decided and why.
…s misses betterleaks reports nothing for an AWS access key id — not bare, not with an "AWS_ACCESS_KEY_ID=" context, not for ASIA. Measured, not assumed: a probe against it returned zero for every shape, including the 40-char secret key. An access key id in a log or an agent transcript is exactly what vedei is meant to keep out of a model's context, so this is the one detection rule vedei owns outside the Brazilian core. It is structural, like a CPF whose check digits close: a credential-bearing prefix (AKIA long-term, ASIA temporary) followed by sixteen base32 characters. base32 has no 0, 1, 8 or 9, so a 20-char run with one of those in the body is ruled out for free — that is a real check, not just a shape. Reported High confidence, because the prefix-plus-base32 combination is specific, and Unknown validity, because confirming a live key means calling AWS (M9). AIDA, AROA and the other prefixes are IAM resource ids, not secrets, so they are left out rather than add noise. This is a deliberate exception to ADR-001. betterleaks stays the only *imported* corpus — the detector is vedei's own, pure, std-lib only — and the exception is there because the gap is real and corpus/cases/secrets.txt proves it: those expect-cases failed before this and pass after. The account-id checksum an access key id carries is noted for a later revision rather than invented now. The 40-char secret access key has no structure to key on and is left to context-based rules, a gap stated in SECURITY.md. .vedeiignore gains the corpus and test fixtures so the self-scan stays clean. Corpus: 40 cases, precision and recall 1.000.
The structure block had two report/ lines and no detect/aws/; fixed, and corpus/ now describes what it is — a labeled precision/recall harness with its own make target — rather than an aspiration. M5 moves to in-progress in both READMEs: the corpus and the first detection fixes have landed, the pt-BR dictionary calibration that is the heart of M5 has not.
mkmuniz
added
bug
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Lands the first third of M5: a labeled precision/recall corpus with its own test gate, a standardized mask separator across every document type, and vedei's first secret rule of its own — AWS access key ids, which the upstream corpus misses entirely.
Why
A scanner's worth is precision and recall on real input, and until now neither was measured. Three things follow from being able to measure them:
The corpus. For Brazilian data, precision is the hard axis: the upstream secret rules score candidates against an English dictionary, so
senhapadraoreads as random and gets reported. That cost is invisible without labeled data, so there was no way to tell whether a change made the tool noisier. Now there is.The masks. Two divergences in how document types tolerated separators. CNS and título used
[\s.], and\smatches a newline, so either matched a value split across two lines — the exact bug the card pattern was already written to avoid. And CPF, CNPJ and PIS rejected a space outright, so529 982 247 25was missed while a space-separated CNS was caught.AWS. Measured, not assumed: betterleaks reports nothing for an AWS access key id — not bare, not with an
AWS_ACCESS_KEY_ID=context, not for ASIA, not even the 40-char secret key. An access key id in a log or an agent transcript is exactly what vedei exists to keep out of a model's context.Closes #
Checklist
make testpasses, including-racemake lintreports zero issues (plusgosecclean on the newdetect/awsandcorpuspackages)detect/awsandcorpusarrive with their own suitesIf this adds or changes a detector
This touches two: the mask change in
detect/br, and the newdetect/aws.detect/awshas the same zero-dependency shape asdetect/brFuzzCPFCheckDigitsand the rest), which generate valid values and corrupt them. The space/tab masks are asserted equivalent inTestExtract_SpaceAndTabAreUniformSeparators. AWS has no check digit to fuzz, so it is covered by example from both sides — valid prefixes and bodies pass, non-AWS prefixes and non-base32 bodies failcorpus/entries, both kinds — this is the PR that makes that checklist line real. Space-separated CPF as a true positive, a three-digit table column as the false positive it must not become; two AWS keys as true positives, a non-AWS-prefix lookalike as the rejectThe edge cases that matter:
100 200 300 40). The check digit a column does not satisfy is what keeps it quiet —corpus/cases/cpf.txthas it.0,1,8or9, soAKIA1KJ7WQZR2LMNXBTQis rejected — a real structural check, not just a 20-char shape.If this touches anything a detected value passes through
detect/awsproduces findings that flow through the same report, redaction and fingerprint paths as every other.