Repository navigation
Feat/m5 calibration - #11
Merged
Merged
Conversation
"No maintained open-source tool detects CPF, CNPJ, CNH, Pix keys, card PANs and E2EIDs. Not underserved — empty." is not true: at least one maintained project detects and validates Brazilian identifiers in free text. A README that makes a checkable claim and gets it wrong costs more credibility than the claim was ever worth. The paragraph now says what vedei does rather than what others do not: it validates Brazilian data by check digit instead of matching by shape, and it enforces at the boundary where data leaves — agent context, CI, logs — in one Go binary with no runtime and no network.
…07-06 The Receita Federal generated the first alphanumeric CNPJ on 2026-07-31; the 2026-07-06 date in seven places came from an early timeline, not from the authority. Corrected everywhere it appeared — README, the package doc in cnpj.go, test comments, the corpus header — with the gov.br announcement linked from both READMEs. The corpus gains that first CNPJ itself: 00.000.000/E08G-12, a Banco do Brasil branch, published by the Receita. Every other CNPJ vector in the suite is generated, which only proves the algorithm agrees with itself; this one proves it agrees with the issuer. It validates — the check digits computed from 00000000E08G are 1 and 2. Its fingerprint joins .vedeiignore, so the self-scan stays clean. Corpus: 41 cases, precision and recall 1.000.
The "The name" section still explained Надзор, Russian for oversight, with "(vedei)" pasted into it by the rename — an etymology for a word the project no longer uses. Vedei is Portuguese: the first person past tense of vedar, to seal or keep from passing. "Eu vedei" — I sealed it. It is what the person running the tool says once it has done its job: the CPF did not reach the model, the key did not reach the log. Written in both READMEs, and no reference to the old name remains anywhere in the repository.
The overall figure averages problems away. A run can sit at 0.99 precision while one type sits at 0.80, because the noisy type is outnumbered by clean ones. Reporting per type is what turns "precision of at least 0.99 for every type with a check digit" into a claim the test log can show. Report.Record charges a false positive to the type that was wrongly reported — the one whose precision it damages — and a miss to the type that was expected. A reject the engine left alone counts overall and belongs to no type. That attribution is unit-tested, because getting it wrong would let a noisy type hide behind a clean one. The gate stays strict (every case passes), which is a stronger target than the 0.99 floor; the table makes a type drifting toward the floor visible before it crosses it. Today every type reads 1.000 / 1.000 across 41 cases.
…ey names
M5 assumed the Portuguese problem was precision: that the upstream dictionary
filter, trained on English, would report a Portuguese placeholder like
"senhapadrao" as a secret. Measured on betterleaks v1.8.1 it does not — the
full filter chain rejects prose placeholders in English, Portuguese and Spanish
alike, 0 of 30 each.
The real problem is the opposite one, and it is worse. The rules key on English
names, so a real credential under a Portuguese or Spanish name is missed:
real secrets found English 100% Spanish 57% Portuguese 29%
senha, segredo and senha_banco were never recognized at all. A leak under the
most natural variable name a Brazilian developer would choose walked past.
The fix is a second scan of the same text with key names translated — senha to
password, segredo to secret, chave to key, and their Spanish equivalents —
rather than a parallel rule set. The upstream filters have English baked into
them too (generic-password suppresses "password = variable" in code by matching
the word), so copied regexes with translated keywords would recover recall and
lose that precision. Translating the name and reusing the rules applies every
filter exactly as in English.
Only names being assigned to are translated, and values never are. The engine
already locates each secret in the original text by substring search, so a
finding from the translated pass lands on the right bytes, and a value both
passes report collapses into one finding by fingerprint.
One more step was needed. Portuguese and Spanish names are head-initial
(senha_banco), English ones head-final (db_password), and the upstream password
rule requires a word boundary right after "password" — so password_db is missed
even in English. A translated leading "password" is moved to the end, giving the
rule the name an English speaker would have written.
Result, identical values under equivalent keys:
recall English 88% Portuguese 100% Spanish 100%
English trails only on password_db, the upstream gap the reordering closes for
the other two. Placeholder false positives are the same per key in every
language; they come from generic-password reporting any prose password at low
confidence, which is upstream behaviour in English too and not a language bias.
Six of seven test cases fail with the pass disabled.
…ames
78 cases that pin the localization pass: one made-up credential under seven key
names per language, in three syntaxes each — quoted assignment, YAML, and an
upper-case environment variable. The value is identical across languages, so a
difference in what is found can only come from the name.
Each language also carries the references that must stay quiet — ${VAR},
os.getenv("VAR"), <placeholder> — and a prose sentence using the word. Those
guard the other half of parity: translating a key name must not stop the
upstream filters from recognizing a value that is not a value.
Corpus: 119 cases, precision 1.000, recall 1.000. Without the localization pass,
15 of the Portuguese and 9 of the Spanish expect-cases here fail — measured by
disabling it and running the corpus.
…ay so Five places — both READMEs, MILESTONES, SECURITY.md and the corpus package doc — stated that betterleaks reports Portuguese placeholders like "senhapadrao" as secrets. That conclusion came from the token ratio measured in isolation. Measured end to end on v1.8.1, the full filter chain rejects prose placeholders equally in English, Portuguese and Spanish. The real bias ran the other way and was larger: a credential under a Portuguese key name was found 29% of the time, under a Spanish one 57%, under an English one 100%. That is what the earlier commits in this branch fix, and what every one of these documents now reports. MILESTONES records both the original hypothesis and why it was wrong, rather than rewriting history: the pt-BR dictionary the plan called for was not built because the measurement showed it would fix nothing. It also records a side finding — the upstream password rule reports instructional placeholders at low confidence in every language — and marks M5's exit criteria against what was measured: the corpus gates CI and the numbers are published; 500 cases and an upstream issue are still open.
.vedeiignore path patterns were applied by "vedei scan" and nowhere else. A
directory silenced there was still reported by "vedei diff" — the mode CI runs
on every pull request — and by "vedei git". Nothing noticed, because until now
every ignore entry in this repository was a fingerprint, not a path.
The first version of the fix introduced a worse bug, caught by its own test.
History scans identical content once, by blob hash. Checking the path after
the blob was marked as seen meant an ignored copy could use up that one scan,
so the same bytes committed outside the ignored directory were never scanned:
a-fixtures/case.txt (ignored) \ same blob
z-app/config.txt (real leak) / -> Total() = 0
A leak hidden by whichever path happens to sort first. The path is now checked
before the blob is marked, so an ignored path never consumes it. Two tests pin
both behaviours, and the second fails with the check moved back.
The per-commit loop moves into scanCommit, which also brings History back
under the complexity limit it had already crossed.
…tory Every ignore entry in this repository has been a fingerprint, on purpose: silencing *_test.go wholesale would also silence a credential genuinely committed into a test. The corpus is the exception that rule cannot carry. corpus/cases/ exists to hold labeled example values, and naming each one meant a fingerprint per case — 10 already, hundreds at the size M5 aims for, each one easy to forget when a case is added. A value there is a labeled case by construction: the corpus test fails on any line that is not an "expect" or a "reject". That is what makes a path rule safe for this directory and nowhere else. The ten fingerprints that only covered corpus files are removed (39 to 29). One is kept that the tree no longer needs: an earlier version of .vedeiignore quoted the official alphanumeric CNPJ in a comment, so git history still holds it, and the weekly history scan would report it. Found by running "vedei git" after the change, not before — the tree scan alone said it was safe to drop. Both READMEs and the corpus README describe the exception; the claim "nothing is silenced by path" is gone. Tree and history scans are both clean.
mkmuniz
added
documentation
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
| 37845344 | Triggered | Generic High Entropy Secret | 275487e | corpus/cases/secrets-i18n.txt | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Credentials assigned to Portuguese and Spanish key names are now found: recall goes from 29% (Portuguese) and 57% (Spanish) to 100% in both. The corpus gains per-type metrics and 78 multilingual cases,
.vedeiignorepath rules now hold invedei diffandvedei git, and the M5 documentation is corrected to match what was measured.Why
M5 set out to fix a precision bias: the assumption that betterleaks' English-dictionary filter would report Portuguese placeholders like
senhapadraoas secrets. Measured end to end on betterleaks v1.8.1, it does not. Onlygeneric-api-keyuses that filter, and the full chain rejects prose placeholders equally in all three languages — 0 of 30 each. The earlier conclusion came from looking at the token ratio outside the filter chain.The real bias ran the other way and was larger. The upstream rules recognize English key names, so a real credential under a Portuguese or Spanish name was missed:
senha,segredoandsenha_bancowere never recognized. The Portuguese keys that did work (chave_api,token_acesso) worked only because they containapiandtoken. A credential under the most natural variable name a Brazilian developer would choose was walking past.Closes #
Checklist
make testpasses, including-racemake lintreports zero issues;make sastcleanlocalize.go,Reportand the git path-rule fix each arrive with their own testsIf this adds or changes a detector
It changes how the secrets engine reaches the upstream rules, not a Brazilian validator.
engine/secrets/localize.go, standard library only, and adds no rules — it feeds the existing upstream rules a translated copy of the textcorpus/entries, both kinds — 63 credentials under English, Portuguese and Spanish names (expect), and per language the values that must stay quiet:${VAR},os.getenv(...),<placeholder>, and a prose sentence using the word (reject)The edge cases that matter:
senha_banco), English ones head-final (db_password), and the upstream password rule requires a word boundary right afterpassword— sopassword_dbis missed even in English. A straight translation lands on exactly the order the rule cannot see, so a leadingpasswordis moved to the end:senha_banco→banco_password.token = "minhasenha123"keeps its value untouched — that is what lets the engine locate a secret from the translated pass in the original text by substring search.${DB_PASSWORD}by matching English words; the corpus pins that they still suppress${SENHA_BANCO}once the name is translated.If this touches anything a detected value passes through
The translated text is a second input to the same engine, so every finding from it travels the existing path.
Finding.Rawout of the process — nothing new is emitted; findings from both passes go through the same constructionValidityInvalidstill never leave the engineTrade-offs and limitations
This is not the fix M5 planned. The pt-BR dictionary for the token-efficiency filter was not built, because the measurement showed it would fix nothing. MILESTONES records the original hypothesis and why it was wrong rather than rewriting it.
Why a second pass instead of new rules. The upstream filters have English baked into them too —
generic-passwordsuppresses an unquotedpassword = variablein code by matching the wordpassword. Copying the regexes with translated keywords would recover recall and lose that precision. Translating the name and rerunning the same rules applies every filter exactly as in English. The cost is a secondDetectStringcall, paid only when the text contains one of the translated stems.Parity, not improvement. The target is that a Portuguese name is treated like its English equivalent, no better and no worse. One consequence is visible in the numbers: with identical values, Portuguese and Spanish now reach 100% recall while English sits at 88%, because English
password_dbis still missed upstream and the reordering only runs on translated names. Applying it to English too would be a separate change.Upstream placeholder noise is unchanged, in every language.
generic-passwordreports instructional placeholders such as"your-password-here","replace_me"and"changeme"at low confidence — 5 of 10 in English, and the same 5 of 10 for the Portuguese and Spanish equivalents. That is a language-neutral weakness of the upstream rule, recorded in MILESTONES as a side finding and a candidate upstream issue, not fixed here.The vocabulary is small and hand-picked. 16 stems:
senha,segredo,chave,credencial,acesso,autenticaçãoand the Spanish equivalents. A name outside that list is not translated. It covers the common cases measured here, not the language.M5 is still open. The corpus is at 119 cases against a target of 500, and the upstream issue with the measurement has not been filed.
corpus/cases/is now silenced by path — the one path rule in the repository. Every line there must be a labeledexpectorreject, which is what makes it safe; a fingerprint per case would not scale to 500. Everything else stays fingerprint-only.A bug fixed on the way, and one nearly introduced.
.vedeiignorepath rules were applied byvedei scanonly, sovedei diff— what CI runs on every pull request — ignored them. The first version of that fix let an ignored path consume a blob thatvedei gitscans once by hash, hiding the same content committed elsewhere. Its own test caught it (Total() = 0); the path is now checked before the blob is marked as seen.How to verify