From f1067906d63779021ed63a01e0d5ef5958ed5610 Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 02:24:11 -0300 Subject: [PATCH 1/9] docs: drop the claim that no maintained tool detects Brazilian data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "No maintained open-source tool detects CPF, CNPJ, CNH, Pix keys, card PANs and E2EIDs. Not underserved — empty." is not true: at least one maintained project detects and validates Brazilian identifiers in free text. A README that makes a checkable claim and gets it wrong costs more credibility than the claim was ever worth. The paragraph now says what vedei does rather than what others do not: it validates Brazilian data by check digit instead of matching by shape, and it enforces at the boundary where data leaves — agent context, CI, logs — in one Go binary with no runtime and no network. --- README.md | 2 +- README.pt-BR.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e78446e..3c3c90d 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ ISPB registry from [`guibranco/BancosBrasileiros`](https://github.com/guibranco/ ## Why it exists -Secret scanning is solved and crowded — betterleaks ships 463 rules, kingfisher 1,051. Rebuilding that corpus loses. So vedei **imports betterleaks** (MIT) and spends everything on what nobody built: **Brazilian sensitive data**. No maintained open-source tool detects CPF, CNPJ, CNH, Pix keys, card PANs and E2EIDs. Not underserved — empty. +Secret scanning is solved and crowded — betterleaks ships 463 rules, kingfisher 1,051. Rebuilding that corpus loses. So vedei **imports betterleaks** (MIT) and spends its effort where global tools are weakest: **Brazilian sensitive data**, validated by check digit rather than matched by shape — CPF, CNPJ including the alphanumeric format, CNH, Pix keys and E2EIDs, card PANs with Brazilian brands. And it enforces at the boundary where data leaves: the agent's context, the CI run, the log line. One Go binary, no runtime, no network. The second reason is architectural. betterleaks validates a secret by asking the provider over HTTP, which needs a rate limiter, a timeout budget, and care about where those requests go. Brazilian personal data doesn't work that way: a CPF is a mod-11 check digit, a card is Luhn, a Pix key is its type. **All offline.** No network, no rate limits, no SSRF surface. Simpler and safer, not just different. diff --git a/README.pt-BR.md b/README.pt-BR.md index 76a2879..2435d61 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -87,7 +87,7 @@ Base de ISPB do [`guibranco/BancosBrasileiros`](https://github.com/guibranco/Ban ## Por que existe -Detecção de segredo é problema resolvido e concorrido — betterleaks tem 463 regras, kingfisher 1.051. Reconstruir esse corpus é perder. Então o vedei **importa o betterleaks** (MIT) e gasta tudo no que ninguém construiu: **dado sensível brasileiro**. Nenhuma ferramenta open source mantida detecta CPF, CNPJ, CNH, chave Pix, PAN e E2EID. Não é mal servido — é vazio. +Detecção de segredo é problema resolvido e concorrido — betterleaks tem 463 regras, kingfisher 1.051. Reconstruir esse corpus é perder. Então o vedei **importa o betterleaks** (MIT) e concentra o esforço onde as ferramentas globais são mais fracas: **dado sensível brasileiro**, validado por dígito verificador em vez de casado por formato — CPF, CNPJ inclusive o alfanumérico, CNH, chave Pix e E2EID, PAN de cartão com bandeiras brasileiras. E aplica o controle na fronteira por onde o dado sai: o contexto do agente, a execução de CI, a linha de log. Um binário Go, sem runtime, sem rede. A segunda razão é arquitetural. O betterleaks valida segredo perguntando ao provedor por HTTP, o que exige rate limiter, orçamento de timeout e cuidado com o destino das requisições. Dado pessoal brasileiro não funciona assim: CPF é módulo 11, cartão é Luhn, chave Pix é o tipo dela. **Tudo offline.** Sem rede, sem limite de vazão, sem superfície de SSRF. Mais simples e mais seguro, não só diferente. From 9ad892ef01c9ace605cddcdc031d0066b659cc8f Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 02:25:14 -0300 Subject: [PATCH 2/9] fix(docs): the alphanumeric CNPJ was first issued on 2026-07-31, not 07-06 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Receita Federal generated the first alphanumeric CNPJ on 2026-07-31; the 2026-07-06 date in seven places came from an early timeline, not from the authority. Corrected everywhere it appeared — README, the package doc in cnpj.go, test comments, the corpus header — with the gov.br announcement linked from both READMEs. The corpus gains that first CNPJ itself: 00.000.000/E08G-12, a Banco do Brasil branch, published by the Receita. Every other CNPJ vector in the suite is generated, which only proves the algorithm agrees with itself; this one proves it agrees with the issuer. It validates — the check digits computed from 00000000E08G are 1 and 2. Its fingerprint joins .vedeiignore, so the self-scan stays clean. Corpus: 41 cases, precision and recall 1.000. --- .vedeiignore | 4 +++- README.md | 2 +- README.pt-BR.md | 2 +- corpus/cases/cnpj.txt | 8 +++++++- detect/br/cnpj.go | 2 +- detect/br/cnpj_test.go | 2 +- detect/br/fuzz_test.go | 2 +- 7 files changed, 15 insertions(+), 7 deletions(-) diff --git a/.vedeiignore b/.vedeiignore index e1bebc6..cdbccaa 100644 --- a/.vedeiignore +++ b/.vedeiignore @@ -26,7 +26,7 @@ aede264bea2e590f2e60467ab2ae74b8 # *************31 - 2x, detect/br/others_test. cf4011fdac9970fb933927c3c2dc2e79 # *********80 - 1x, detect/br/others_test.go # --- cnpj --- -# CNPJ test vectors, numeric and alphanumeric. The alphanumeric form takes effect 2026-07-06 and needs a fixture before then, not after. +# CNPJ test vectors, numeric and alphanumeric. The alphanumeric form was first issued on 2026-07-31. e901dbbb99a1ba293aee61bf533bb180 # **.***.***/****-35 - 7x, detect/br/cnpj_test.go and 5 more 3e210b2f70e511a859c255d442cc346a # **.***.***/****-81 - 12x, detect/br/cnpj_test.go and 6 more e186e56ca5d2b8fda5aa3be015e79407 # **.***.***/****-91 - 1x, detect/br/cnpj_test.go @@ -88,6 +88,8 @@ f60e47e7fe914c3e5d10a9b2a99d8b96 # generated titulo vector ec2a7642f9821664294f09e54573ca66 # generated CNS vector 83f7388d806b21f4c0c4ea6e6a651937 # Pix email key fixture +3d916d891e1aab79bc68592c2d7cd3a5 # 00.000.000/E08G-12, the first alphanumeric CNPJ the Receita issued (2026-07-31), a public record used as a corpus case + # Nothing is ignored by path, on purpose. Silencing *_test.go wholesale would # also silence a credential genuinely committed into a test, which is a place # credentials genuinely end up. Every entry above is one value, named. diff --git a/README.md b/README.md index 3c3c90d..4038bcd 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ Wire the hook into your agent: [`hooks/`](hooks/). Two binaries, because the spl | **CPF** | ✅ | Mod 11, two check digits; repeated-sequence rejection | | **CIN** | ✅ | Uses the CPF as the national number — same validation | | **CNPJ** numeric | ✅ | Mod 11, weights `5,4,3,2,9,8,7,6,5,4,3,2` | -| **CNPJ** alphanumeric | ✅ | Mod 11 with `ASCII − 48` for letters — **in force since 2026-07-06** | +| **CNPJ** alphanumeric | ✅ | Mod 11 with `ASCII − 48` for letters — **first issued on 2026-07-31** ([Receita Federal](https://www.gov.br/receitafederal/pt-br/assuntos/noticias/2026/julho/receita-federal-gera-o-primeiro-cnpj-em-formato-alfanumerico)) | | **CNH** | ✅ | Mod 11, 11-digit variant | | **PIS / NIS / NIT** | ✅ | Mod 11, weights `3,2,9,8,7,6,5,4,3,2` | | **Título de eleitor** | ✅ | Two check digits, mod 11, embedded state code | diff --git a/README.pt-BR.md b/README.pt-BR.md index 2435d61..21d0b22 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -59,7 +59,7 @@ Depois ligue o hook no seu agente: [`hooks/`](hooks/). São dois binários, e é | **CPF** | ✅ | Módulo 11, dois DVs; rejeita sequências repetidas | | **CIN** | ✅ | Usa o CPF como número nacional — mesma validação | | **CNPJ** numérico | ✅ | Módulo 11, pesos `5,4,3,2,9,8,7,6,5,4,3,2` | -| **CNPJ** alfanumérico | ✅ | Módulo 11 com `ASCII − 48` nas letras — **vigente desde 06/07/2026** | +| **CNPJ** alfanumérico | ✅ | Módulo 11 com `ASCII − 48` nas letras — **primeiro emitido em 31/07/2026** ([Receita Federal](https://www.gov.br/receitafederal/pt-br/assuntos/noticias/2026/julho/receita-federal-gera-o-primeiro-cnpj-em-formato-alfanumerico)) | | **CNH** | ✅ | Módulo 11, variante de 11 dígitos | | **PIS / NIS / NIT** | ✅ | Módulo 11, pesos `3,2,9,8,7,6,5,4,3,2` | | **Título de eleitor** | ✅ | Dois DVs, módulo 11, com código de UF | diff --git a/corpus/cases/cnpj.txt b/corpus/cases/cnpj.txt index 546e000..ffaf6d7 100644 --- a/corpus/cases/cnpj.txt +++ b/corpus/cases/cnpj.txt @@ -1,8 +1,14 @@ -# CNPJ — numeric and the alphanumeric format in force from 2026-07-06. +# CNPJ — numeric and the alphanumeric format first issued on 2026-07-31. expect cnpj 11.222.333/0001-81 expect cnpj 11222333000181 +# The first alphanumeric CNPJ the Receita Federal ever issued, on 2026-07-31: a +# Banco do Brasil branch. Not a generated vector — the real one, published by +# the Receita, so it proves the algorithm against the authority, not against +# itself. +expect cnpj 00.000.000/E08G-12 + expect cnpj 00.000.000/0001-91 # the Uniao's CNPJ: body all zeros, but check digits close reject 11.222.333/0001-82 # check digit wrong diff --git a/detect/br/cnpj.go b/detect/br/cnpj.go index b0018e1..9239add 100644 --- a/detect/br/cnpj.go +++ b/detect/br/cnpj.go @@ -12,7 +12,7 @@ var cnpjWeights = [12]int{5, 4, 3, 2, 9, 8, 7, 6, 5, 4, 3, 2} // ValidateCNPJ reports whether s is a structurally valid CNPJ, in either the // numeric or the alphanumeric format. // -// Since 2026-07-06 the Receita Federal issues alphanumeric CNPJs: the first +// Since 2026-07-31 the Receita Federal issues alphanumeric CNPJs: the first // twelve positions may hold 0-9 and A-Z, while the last two remain numeric // check digits. Letters enter the mod-11 sum as their ASCII value minus 48, // so 'A' (65) contributes 17. Existing numeric CNPJs are unchanged, and a diff --git a/detect/br/cnpj_test.go b/detect/br/cnpj_test.go index 0b7ca3f..32b4a48 100644 --- a/detect/br/cnpj_test.go +++ b/detect/br/cnpj_test.go @@ -17,7 +17,7 @@ func TestCNPJ_GeneratedNumericAreValid(t *testing.T) { } } -// The alphanumeric format has been in force since 2026-07-06. +// The alphanumeric format has been issued since 2026-07-31. func TestCNPJ_GeneratedAlphanumericAreValid(t *testing.T) { r := rand.New(rand.NewSource(11)) for i := 0; i < genCases; i++ { diff --git a/detect/br/fuzz_test.go b/detect/br/fuzz_test.go index 02b636b..d5b7304 100644 --- a/detect/br/fuzz_test.go +++ b/detect/br/fuzz_test.go @@ -190,7 +190,7 @@ func FuzzCPFCheckDigits(f *testing.F) { } // FuzzCNPJCheckDigits does the same for both CNPJ formats. The alphanumeric one -// takes effect on 2026-07-06 and its weighting reads a letter as its ASCII code +// was first issued on 2026-07-31 and its weighting reads a letter as its ASCII code // minus 48, which is the part most likely to be got wrong. func FuzzCNPJCheckDigits(f *testing.F) { f.Add("112223330001", false) From f3481d0f6d26ab278527ea439b87497a3899ad46 Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 02:36:40 -0300 Subject: [PATCH 3/9] docs: explain the name vedei, replacing the leftover nadzor etymology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "The name" section still explained Надзор, Russian for oversight, with "(vedei)" pasted into it by the rename — an etymology for a word the project no longer uses. Vedei is Portuguese: the first person past tense of vedar, to seal or keep from passing. "Eu vedei" — I sealed it. It is what the person running the tool says once it has done its job: the CPF did not reach the model, the key did not reach the log. Written in both READMEs, and no reference to the old name remains anywhere in the repository. --- README.md | 4 ++-- README.pt-BR.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4038bcd..85d2b25 100644 --- a/README.md +++ b/README.md @@ -163,9 +163,9 @@ corpus/ # labeled precision/recall harness (make corpus) ## The name -**Надзор** (*vedei*) is Russian for *oversight* — the word used for regulatory supervision. It is `dozor` (the watch) with the prefix `nad-` (over, above). +**Vedei** is Portuguese: the first person past tense of *vedar* — to seal, to block, to keep from passing. *Eu vedei*: "I sealed it." -A tool that inspects what passes a boundary and enforces a rule about it is not a guard. It is oversight. +It is what the person running the tool says once it has done its job. The CPF did not reach the model, the key did not reach the log, the card number did not reach the commit. Sealed. ## Contributing · Security · License diff --git a/README.pt-BR.md b/README.pt-BR.md index 21d0b22..6ad8c00 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -162,9 +162,9 @@ corpus/ # harness rotulado de precision/recall (make corpus) ## O nome -**Надзор** (*vedei*) é "supervisão" ou "fiscalização" em russo — a palavra usada para supervisão regulatória. É `dozor` (a ronda) com o prefixo `nad-` (sobre, acima). +**Vedei** é o verbo *vedar* na primeira pessoa do pretérito perfeito — vedar como quem impede a passagem, como quem tampa uma fresta. -Uma ferramenta que inspeciona o que passa por uma fronteira e aplica uma regra sobre isso não é um guarda. É fiscalização. +É o que diz quem usa a ferramenta depois que ela fez o trabalho dela. O CPF não chegou ao modelo, a chave não chegou ao log, o número do cartão não chegou ao commit. Vedei. ## Contribuir · Segurança · Licença From a52bdb0423588f1402807f1b9874c46cfd5480e4 Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 14:54:05 -0300 Subject: [PATCH 4/9] feat(corpus): report precision and recall per finding type MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The overall figure averages problems away. A run can sit at 0.99 precision while one type sits at 0.80, because the noisy type is outnumbered by clean ones. Reporting per type is what turns "precision of at least 0.99 for every type with a check digit" into a claim the test log can show. Report.Record charges a false positive to the type that was wrongly reported — the one whose precision it damages — and a miss to the type that was expected. A reject the engine left alone counts overall and belongs to no type. That attribution is unit-tested, because getting it wrong would let a noisy type hide behind a clean one. The gate stays strict (every case passes), which is a stronger target than the 0.99 floor; the table makes a type drifting toward the floor visible before it crosses it. Today every type reads 1.000 / 1.000 across 41 cases. --- corpus/corpus.go | 91 ++++++++++++++++++++++++++++++++++++ corpus/corpus_test.go | 104 +++++++++++++++++++++++++++++++++++------- 2 files changed, 178 insertions(+), 17 deletions(-) diff --git a/corpus/corpus.go b/corpus/corpus.go index ddf45cd..0c5b439 100644 --- a/corpus/corpus.go +++ b/corpus/corpus.go @@ -195,3 +195,94 @@ func (m Metrics) String() string { m.Precision(), m.TruePositives+m.FalsePositives, m.FalsePositives, m.Recall(), m.TruePositives, m.TruePositives+m.FalseNegatives) } + +// Report accumulates metrics for a run, overall and per finding type. +// +// The per-type split is what the overall number hides. A corpus can sit at 0.99 +// precision while one type sits at 0.80, because the noisy type is outnumbered by +// clean ones; the overall figure averages the problem away. Reporting per type +// is what makes "precision of at least 0.99 for every type with a check digit" a +// claim that can be checked rather than one that is implied. +type Report struct { + Overall Metrics + byType map[string]*Metrics +} + +// NewReport returns an empty report. +func NewReport() *Report { + return &Report{byType: map[string]*Metrics{}} +} + +// Record scores one case against the set of types the engine reported for it, +// and reports whether the case passed. +// +// A false positive is charged to every type that was reported on a Reject case, +// because that is the type whose precision it damages. A Reject the engine left +// alone is a true negative for the run but belongs to no type, so it counts only +// overall. +func (r *Report) Record(c Case, found map[string]bool) bool { + switch c.Kind { + case Expect: + m := r.metricsFor(c.Type) + if found[c.Type] { + r.Overall.TruePositives++ + m.TruePositives++ + return true + } + r.Overall.FalseNegatives++ + m.FalseNegatives++ + return false + case Reject: + if len(found) == 0 { + r.Overall.TrueNegatives++ + return true + } + r.Overall.FalsePositives++ + for t := range found { + r.metricsFor(t).FalsePositives++ + } + return false + default: + return false + } +} + +// Type returns the metrics recorded for one finding type. +func (r *Report) Type(t string) Metrics { + if m, ok := r.byType[t]; ok { + return *m + } + return Metrics{} +} + +// Types returns every type the report has seen, sorted, so two runs print in +// the same order. +func (r *Report) Types() []string { + out := make([]string, 0, len(r.byType)) + for t := range r.byType { + out = append(out, t) + } + sort.Strings(out) + return out +} + +// Table renders one line per type, for a test log. +func (r *Report) Table() string { + var b strings.Builder + fmt.Fprintf(&b, "%-20s %9s %7s %6s %6s %6s\n", "TYPE", "PRECISION", "RECALL", "TP", "FP", "FN") + for _, t := range r.Types() { + m := r.byType[t] + fmt.Fprintf(&b, "%-20s %9.3f %7.3f %6d %6d %6d\n", + t, m.Precision(), m.Recall(), m.TruePositives, m.FalsePositives, m.FalseNegatives) + } + return b.String() +} + +func (r *Report) metricsFor(t string) *Metrics { + m, ok := r.byType[t] + if !ok { + m = &Metrics{} + r.byType[t] = m + } + return m +} diff --git a/corpus/corpus_test.go b/corpus/corpus_test.go index 51033fe..7b800f0 100644 --- a/corpus/corpus_test.go +++ b/corpus/corpus_test.go @@ -32,32 +32,28 @@ func TestCorpus(t *testing.T) { } eng := engine.All() - var m corpus.Metrics + report := corpus.NewReport() for _, c := range cases { types := typesFound(t, eng, c.Text) - + if report.Record(c, types) { + continue + } switch c.Kind { case corpus.Expect: - if types[c.Type] { - m.TruePositives++ - } else { - m.FalseNegatives++ - t.Errorf("%s:%d: expected %s in %q, found %v", - c.Source, c.Line, c.Type, c.Text, keys(types)) - } + t.Errorf("%s:%d: expected %s in %q, found %v", + c.Source, c.Line, c.Type, c.Text, keys(types)) case corpus.Reject: - if len(types) == 0 { - m.TrueNegatives++ - } else { - m.FalsePositives++ - t.Errorf("%s:%d: %q should be quiet (%s), but was reported as %v", - c.Source, c.Line, c.Text, c.Reason, keys(types)) - } + t.Errorf("%s:%d: %q should be quiet (%s), but was reported as %v", + c.Source, c.Line, c.Text, c.Reason, keys(types)) } } - t.Logf("%d cases: %s", len(cases), m) + // The strict gate above is the explicit target: precision and recall of + // 1.000 for every type, which is stricter than the 0.99 floor the project + // sets for types with a check digit. The table shows where each type stands, + // so a type drifting toward that floor is visible before it crosses it. + t.Logf("%d cases: %s\n%s", len(cases), report.Overall, report.Table()) } // typesFound returns the set of finding types the engine reports for text. @@ -104,6 +100,80 @@ func TestCorpus_MetricsMath(t *testing.T) { } } +// A false positive is charged to the type that was wrongly reported, and a miss +// to the type that was expected. That attribution is what makes the per-type +// table mean something; get it wrong and a noisy type hides behind a clean one. +func TestReport_Record(t *testing.T) { + tests := []struct { + name string + c corpus.Case + found map[string]bool + wantPass bool + check func(t *testing.T, r *corpus.Report) + }{ + { + name: "expect found", + c: corpus.Case{Kind: corpus.Expect, Type: "cpf"}, + found: map[string]bool{"cpf": true}, + wantPass: true, + check: func(t *testing.T, r *corpus.Report) { + if got := r.Type("cpf").TruePositives; got != 1 { + t.Errorf("cpf TruePositives = %d, want 1", got) + } + }, + }, + { + name: "expect found under the wrong type is a miss", + c: corpus.Case{Kind: corpus.Expect, Type: "cpf"}, + found: map[string]bool{"cnh": true}, + wantPass: false, + check: func(t *testing.T, r *corpus.Report) { + if got := r.Type("cpf").FalseNegatives; got != 1 { + t.Errorf("cpf FalseNegatives = %d, want 1", got) + } + }, + }, + { + name: "reject reported is charged to the reported type", + c: corpus.Case{Kind: corpus.Reject}, + found: map[string]bool{"card-pan": true}, + wantPass: false, + check: func(t *testing.T, r *corpus.Report) { + if got := r.Type("card-pan").FalsePositives; got != 1 { + t.Errorf("card-pan FalsePositives = %d, want 1", got) + } + if got := r.Overall.FalsePositives; got != 1 { + t.Errorf("overall FalsePositives = %d, want 1", got) + } + }, + }, + { + name: "reject left alone belongs to no type", + c: corpus.Case{Kind: corpus.Reject}, + found: map[string]bool{}, + wantPass: true, + check: func(t *testing.T, r *corpus.Report) { + if len(r.Types()) != 0 { + t.Errorf("types = %v, want none", r.Types()) + } + if r.Overall.TrueNegatives != 1 { + t.Errorf("overall TrueNegatives = %d, want 1", r.Overall.TrueNegatives) + } + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + r := corpus.NewReport() + if got := r.Record(tt.c, tt.found); got != tt.wantPass { + t.Errorf("Record() = %v, want %v", got, tt.wantPass) + } + tt.check(t, r) + }) + } +} + func approx(a, b float64) bool { d := a - b return d < 1e-9 && d > -1e-9 From 60de769e161ff1d506d9c454504f80d7bdc7bfbe Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 15:00:25 -0300 Subject: [PATCH 5/9] feat(engine/secrets): find credentials under Portuguese and Spanish key names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M5 assumed the Portuguese problem was precision: that the upstream dictionary filter, trained on English, would report a Portuguese placeholder like "senhapadrao" as a secret. Measured on betterleaks v1.8.1 it does not — the full filter chain rejects prose placeholders in English, Portuguese and Spanish alike, 0 of 30 each. The real problem is the opposite one, and it is worse. The rules key on English names, so a real credential under a Portuguese or Spanish name is missed: real secrets found English 100% Spanish 57% Portuguese 29% senha, segredo and senha_banco were never recognized at all. A leak under the most natural variable name a Brazilian developer would choose walked past. The fix is a second scan of the same text with key names translated — senha to password, segredo to secret, chave to key, and their Spanish equivalents — rather than a parallel rule set. The upstream filters have English baked into them too (generic-password suppresses "password = variable" in code by matching the word), so copied regexes with translated keywords would recover recall and lose that precision. Translating the name and reusing the rules applies every filter exactly as in English. Only names being assigned to are translated, and values never are. The engine already locates each secret in the original text by substring search, so a finding from the translated pass lands on the right bytes, and a value both passes report collapses into one finding by fingerprint. One more step was needed. Portuguese and Spanish names are head-initial (senha_banco), English ones head-final (db_password), and the upstream password rule requires a word boundary right after "password" — so password_db is missed even in English. A translated leading "password" is moved to the end, giving the rule the name an English speaker would have written. Result, identical values under equivalent keys: recall English 88% Portuguese 100% Spanish 100% English trails only on password_db, the upstream gap the reordering closes for the other two. Placeholder false positives are the same per key in every language; they come from generic-password reporting any prose password at low confidence, which is upstream behaviour in English too and not a language bias. Six of seven test cases fail with the pass disabled. --- engine/secrets/engine.go | 10 +++ engine/secrets/localize.go | 153 ++++++++++++++++++++++++++++++++ engine/secrets/localize_test.go | 115 ++++++++++++++++++++++++ 3 files changed, 278 insertions(+) create mode 100644 engine/secrets/localize.go create mode 100644 engine/secrets/localize_test.go diff --git a/engine/secrets/engine.go b/engine/secrets/engine.go index 41555c5..b0b2a2f 100644 --- a/engine/secrets/engine.go +++ b/engine/secrets/engine.go @@ -110,6 +110,16 @@ func (e *Engine) Scan(ctx context.Context, content []byte, meta detect.Metadata) text := string(content) raw := e.detector.DetectString(text) + // A second pass over the same text with Portuguese and Spanish key names + // translated, so a credential in "senha" or "segredo" meets the same rules + // and filters as one in "password" or "secret". Values are never rewritten, + // so locationsOf below finds every secret from this pass in the original + // text, and a value both passes report collapses into one finding by + // fingerprint. See localize.go for the measurement behind it. + if localized, ok := localizeKeys(text); ok { + raw = append(raw, e.detector.DetectString(localized)...) + } + // One credential found in several places is one finding with several // locations, matching how the Brazilian engine reports. byPrint := make(map[string]*detect.Finding, len(raw)) diff --git a/engine/secrets/localize.go b/engine/secrets/localize.go new file mode 100644 index 0000000..785e6e4 --- /dev/null +++ b/engine/secrets/localize.go @@ -0,0 +1,153 @@ +package secrets + +import ( + "regexp" + "strings" +) + +// The upstream secret rules key on English names: password, passwd, key, +// secret, token, credential, access, auth. A credential assigned to a variable +// named in Portuguese or Spanish never reaches them. Measured on real-looking +// secrets under equivalent key names, betterleaks finds 100% in English, 57% in +// Spanish and 29% in Portuguese: senha, segredo and senha_banco are never +// recognized at all. +// +// The fix is not a parallel rule set. The upstream rules carry filters with +// English baked into them too — generic-password suppresses an unquoted +// "password = variable" in code by matching the word "password" — so a copied +// regex with translated keywords would recover recall and lose that precision. +// Instead, key names are translated and the text is scanned again by the same +// rules, so every filter applies exactly as it does in English. Parity is the +// target: a Portuguese key name is treated like its English equivalent, no +// better and no worse. +// +// Only key names are rewritten, never values. That is what makes the second pass +// safe to merge: the engine locates each secret in the original text by +// substring search, and a value the translation never touched is found exactly +// where it was. + +// keyTranslations maps a Portuguese or Spanish stem to the English word the +// upstream rules recognize. Longest first, so "credenciais" is not consumed as +// "credencial" plus a stray "s". +var keyTranslations = []struct{ from, to string }{ + {"credenciales", "credentials"}, + {"credenciais", "credentials"}, + {"autenticação", "auth"}, + {"autenticacao", "auth"}, + {"autenticación", "auth"}, + {"autenticacion", "auth"}, + {"contraseña", "password"}, + {"contrasena", "password"}, + {"credencial", "credential"}, + {"segredo", "secret"}, + {"secreto", "secret"}, + {"acesso", "access"}, + {"acceso", "access"}, + {"senha", "password"}, + {"chave", "key"}, + {"clave", "key"}, +} + +// stemRe is the cheap gate: most text holds none of these words, and it should +// cost one regexp scan to find that out, not an identifier walk. +var stemRe = func() *regexp.Regexp { + parts := make([]string, 0, len(keyTranslations)) + for _, t := range keyTranslations { + parts = append(parts, regexp.QuoteMeta(t.from)) + } + return regexp.MustCompile(`(?i)` + strings.Join(parts, "|")) +}() + +// identRe matches a key name: letters in any script, digits, and the +// separators key names are written with. +var identRe = regexp.MustCompile(`[\p{L}\p{N}_.\-]+`) + +// assignRe matches what follows a key name in an assignment. It mirrors the +// upstream generic rules, which allow up to three quote, space or backslash +// characters between the name and the operator — enough for "senha =", +// `"senha":`, `'senha' =>` and `SENHA=`. +var assignRe = regexp.MustCompile(`^[ \t'"\\]{0,3}(?:=>|:=|=|:)`) + +// localizeKeys returns text with Portuguese and Spanish key names translated to +// English, and whether anything changed. +// +// A name is translated only where it is being assigned to. "A senha expirou" is +// prose and stays as it is; the upstream rules would ignore it either way, but +// rewriting prose would also rewrite the value side of a line, and the value is +// the one thing this pass must never touch. +func localizeKeys(text string) (string, bool) { + if !stemRe.MatchString(text) { + return text, false + } + + var b strings.Builder + last, changed := 0, false + + for _, loc := range identRe.FindAllStringIndex(text, -1) { + start, end := loc[0], loc[1] + if !assignRe.MatchString(text[end:]) { + continue + } + ident := text[start:end] + translated, ok := translateIdent(ident) + if !ok { + continue + } + if !changed { + b.Grow(len(text) + 64) + } + b.WriteString(text[last:start]) + b.WriteString(translated) + last, changed = end, true + } + + if !changed { + return text, false + } + b.WriteString(text[last:]) + return b.String(), true +} + +// translateIdent rewrites the stems inside one key name. +// +// The result is lower case unless the name was entirely upper case, which keeps +// environment variables recognizable: SENHA_BANCO becomes PASSWORD_BANCO. Case +// elsewhere does not matter to the rules this feeds, which match key names +// case-insensitively — the one exception, "api", is not a stem translated here. +func translateIdent(ident string) (string, bool) { + if !stemRe.MatchString(ident) { + return ident, false + } + out := strings.ToLower(ident) + for _, t := range keyTranslations { + out = strings.ReplaceAll(out, t.from, t.to) + } + out = passwordLast(out) + if ident == strings.ToUpper(ident) { + out = strings.ToUpper(out) + } + return out, true +} + +// passwordLast moves a leading "password" segment to the end of a key name: +// password_banco becomes banco_password. +// +// Word order is the reason. English names are head-final — db_password — and +// the upstream password rule depends on it: it requires a word boundary right +// after "password", so password_db is missed even in English. Portuguese and +// Spanish are head-initial — senha_banco, contraseña_bd — so a straight +// translation lands on exactly the order the rule cannot see. Moving the word +// gives the rule the name an English speaker would have written. The other +// stems need no such step, because the rules they feed allow text after the +// keyword. +func passwordLast(name string) string { + const word = "password" + if !strings.HasPrefix(name, word) || len(name) <= len(word)+1 { + return name + } + sep := name[len(word)] + if sep != '_' && sep != '-' && sep != '.' { + return name + } + return name[len(word)+1:] + string(sep) + word +} diff --git a/engine/secrets/localize_test.go b/engine/secrets/localize_test.go new file mode 100644 index 0000000..586e620 --- /dev/null +++ b/engine/secrets/localize_test.go @@ -0,0 +1,115 @@ +package secrets + +import ( + "context" + "strings" + "testing" + + "github.com/mkmuniz/vedei/detect" +) + +// realSecret is shaped like a credential: mixed case and digits, no words. It +// is assembled so this file carries no single literal another scanner flags. +var realSecret = "8f3kq9zLw2" + "Xc7vB1nM4p" + +func TestLocalizeKeys(t *testing.T) { + tests := []struct { + name string + in string + want string + wantChanged bool + }{ + {"quoted assignment", `senha = "v"`, `password = "v"`, true}, + {"env var keeps upper case and reorders", `SENHA_BANCO=v`, `BANCO_PASSWORD=v`, true}, + {"json key", `"segredo": "v"`, `"secret": "v"`, true}, + {"php array key", `'contraseña' => 'v'`, `'password' => 'v'`, true}, + {"go short assignment", `senha := valor`, `password := valor`, true}, + {"yaml", `chave_api: v`, `key_api: v`, true}, + {"longest stem wins", `credenciais: v`, `credentials: v`, true}, + {"accented upper case", `AUTENTICAÇÃO=v`, `AUTH=v`, true}, + {"spanish reorders too", `contrasena_bd = "v"`, `bd_password = "v"`, true}, + + // Prose is not an assignment, so it is left alone. + {"prose", `a senha expirou ontem`, `a senha expirou ontem`, false}, + // The value side is never rewritten, even when it holds a stem: the engine + // finds secrets from this pass by searching the original text for them. + {"value holding a stem is untouched", `token = "minhasenha123"`, `token = "minhasenha123"`, false}, + {"nothing to translate", `password = "v"`, `password = "v"`, false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, changed := localizeKeys(tt.in) + if got != tt.want || changed != tt.wantChanged { + t.Errorf("localizeKeys(%q) = (%q, %v), want (%q, %v)", + tt.in, got, changed, tt.want, tt.wantChanged) + } + }) + } +} + +// The measurement behind the localization pass, pinned. Before it, a real +// credential under a Portuguese name was found 29% of the time and under a +// Spanish one 57%, against 100% in English; senha and segredo were never +// recognized at all. +func TestScan_FindsSecretsUnderPortugueseAndSpanishKeys(t *testing.T) { + tests := []struct { + name string + text string + wantType string + }{ + {"senha", `senha = "` + realSecret + `"`, "generic-password"}, + {"senha, head-initial", `senha_banco = "` + realSecret + `"`, "generic-password"}, + {"senha, env var", `SENHA_BANCO=` + realSecret, "generic-password"}, + {"segredo", `segredo: ` + realSecret, "generic-api-key"}, + {"chave", `chave_autenticacao = "` + realSecret + `"`, "generic-api-key"}, + {"contrasena, head-initial", `contrasena_bd = "` + realSecret + `"`, "generic-password"}, + {"secreto", `secreto_cliente: ` + realSecret, "generic-api-key"}, + } + + e := New() + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + fs, err := e.Scan(context.Background(), []byte(tt.text), detect.Metadata{Path: "config.yaml"}) + if err != nil { + t.Fatalf("Scan: %v", err) + } + var got *detect.Finding + for i := range fs { + if fs[i].Type == tt.wantType { + got = &fs[i] + } + } + if got == nil { + t.Fatalf("no %s finding in %q; got %d findings", tt.wantType, tt.text, len(fs)) + } + + // The offsets must point at the value in the original text, which is + // what lets the redaction path mask it even though it was found in a + // translated copy. + loc := got.Locations[0] + if span := tt.text[loc.ByteStart:loc.ByteEnd]; span != realSecret { + t.Errorf("location selects %q, want the secret", span) + } + }) + } +} + +// A value both passes report is one finding, not two: same rule, same value, +// same fingerprint. +func TestScan_LocalizedPassDoesNotDuplicate(t *testing.T) { + text := "password = \"" + realSecret + "\"\nsenha = \"" + realSecret + "\"\n" + fs, err := New().Scan(context.Background(), []byte(text), detect.Metadata{Path: "config.yaml"}) + if err != nil { + t.Fatalf("Scan: %v", err) + } + n := 0 + for _, f := range fs { + if strings.Contains(f.Type, "password") { + n++ + } + } + if n != 1 { + t.Errorf("got %d generic-password findings, want 1 deduplicated", n) + } +} From 275487e412a7dc8c1e2989567d3e94267308bdf8 Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 15:01:51 -0300 Subject: [PATCH 6/9] test(corpus): credentials under English, Portuguese and Spanish key names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 78 cases that pin the localization pass: one made-up credential under seven key names per language, in three syntaxes each — quoted assignment, YAML, and an upper-case environment variable. The value is identical across languages, so a difference in what is found can only come from the name. Each language also carries the references that must stay quiet — ${VAR}, os.getenv("VAR"), — and a prose sentence using the word. Those guard the other half of parity: translating a key name must not stop the upstream filters from recognizing a value that is not a value. Corpus: 119 cases, precision 1.000, recall 1.000. Without the localization pass, 15 of the Portuguese and 9 of the Spanish expect-cases here fail — measured by disabling it and running the corpus. --- .vedeiignore | 6 ++ corpus/cases/secrets-i18n.txt | 103 ++++++++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100644 corpus/cases/secrets-i18n.txt diff --git a/.vedeiignore b/.vedeiignore index cdbccaa..c5591ec 100644 --- a/.vedeiignore +++ b/.vedeiignore @@ -90,6 +90,12 @@ ec2a7642f9821664294f09e54573ca66 # generated CNS vector 3d916d891e1aab79bc68592c2d7cd3a5 # 00.000.000/E08G-12, the first alphanumeric CNPJ the Receita issued (2026-07-31), a public record used as a corpus case +# --- multilingual credential fixtures (corpus/cases/secrets-i18n.txt) --- +# One made-up credential-shaped value under key names in three languages. The +# same value is reported by two rules, so it has two fingerprints. +bd02d2bf6ebb75a6c085a966323c06f1 # fake credential under password/senha/contrasena keys +c3fca3e7a0151668ec85c974b0f4f57f # fake credential under api_key/chave/clave keys + # Nothing is ignored by path, on purpose. Silencing *_test.go wholesale would # also silence a credential genuinely committed into a test, which is a place # credentials genuinely end up. Every entry above is one value, named. diff --git a/corpus/cases/secrets-i18n.txt b/corpus/cases/secrets-i18n.txt new file mode 100644 index 0000000..9ed5321 --- /dev/null +++ b/corpus/cases/secrets-i18n.txt @@ -0,0 +1,103 @@ +# Credentials under key names in English, Portuguese and Spanish. +# +# Measured before the localization pass in engine/secrets, a real credential was +# found 100% of the time under an English name, 57% under a Spanish one and 29% +# under a Portuguese one: senha and segredo were never recognized. These cases +# pin the fix. The same value appears under every name, so a difference between +# languages can only come from the name. +# +# Every secret value here is a made-up string shaped like a credential. + +# --- English --- +expect generic-password password = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password password: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password PASSWORD=8f3kq9zLw2Xc7vB1nM4p +expect generic-password db_password = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password db_password: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password DB_PASSWORD=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key api_key = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key api_key: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key API_KEY=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key secret = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key secret: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key SECRET=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key access_token = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key access_token: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key ACCESS_TOKEN=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key client_secret = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key client_secret: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key CLIENT_SECRET=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key auth_key = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key auth_key: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key AUTH_KEY=8f3kq9zLw2Xc7vB1nM4p + +# A reference to a value is not a value. Suppressed in every language, +# including after the key name is translated. +reject password = "${DB_PASSWORD}" # environment variable reference +reject password = os.getenv("DB_PASSWORD") # read at runtime, no literal +reject password = "" # template marker +reject api_key = "${API_KEY}" # environment variable reference +reject the password expired yesterday # prose, not an assignment + +# --- Portuguese --- +expect generic-password senha = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password senha: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password SENHA=8f3kq9zLw2Xc7vB1nM4p +expect generic-password senha_banco = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password senha_banco: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password SENHA_BANCO=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key chave_api = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key chave_api: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key CHAVE_API=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key segredo = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key segredo: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key SEGREDO=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key token_acesso = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key token_acesso: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key TOKEN_ACESSO=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key segredo_cliente = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key segredo_cliente: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key SEGREDO_CLIENTE=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key chave_autenticacao = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key chave_autenticacao: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key CHAVE_AUTENTICACAO=8f3kq9zLw2Xc7vB1nM4p + +# A reference to a value is not a value. Suppressed in every language, +# including after the key name is translated. +reject senha = "${SENHA_BANCO}" # environment variable reference +reject senha = os.getenv("SENHA_BANCO") # read at runtime, no literal +reject senha = "" # template marker +reject chave_api = "${CHAVE_API}" # environment variable reference +reject a senha expirou ontem # prose, not an assignment + +# --- Spanish --- +expect generic-password contrasena = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password contrasena: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password CONTRASENA=8f3kq9zLw2Xc7vB1nM4p +expect generic-password contrasena_bd = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-password contrasena_bd: 8f3kq9zLw2Xc7vB1nM4p +expect generic-password CONTRASENA_BD=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key clave_api = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key clave_api: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key CLAVE_API=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key secreto = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key secreto: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key SECRETO=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key token_acceso = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key token_acceso: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key TOKEN_ACCESO=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key secreto_cliente = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key secreto_cliente: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key SECRETO_CLIENTE=8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key clave_autenticacion = "8f3kq9zLw2Xc7vB1nM4p" +expect generic-api-key clave_autenticacion: 8f3kq9zLw2Xc7vB1nM4p +expect generic-api-key CLAVE_AUTENTICACION=8f3kq9zLw2Xc7vB1nM4p + +# A reference to a value is not a value. Suppressed in every language, +# including after the key name is translated. +reject contrasena = "${CONTRASENA_BD}" # environment variable reference +reject contrasena = os.getenv("CONTRASENA_BD") # read at runtime, no literal +reject contrasena = "" # template marker +reject clave_api = "${CLAVE_API}" # environment variable reference +reject la contrasena vencio ayer # prose, not an assignment + From 9b1f6fe12a95a351b0f593e84a20294512c3120e Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 15:03:53 -0300 Subject: [PATCH 7/9] docs: M5 measured the opposite of what it assumed, and the docs now say so MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five places — both READMEs, MILESTONES, SECURITY.md and the corpus package doc — stated that betterleaks reports Portuguese placeholders like "senhapadrao" as secrets. That conclusion came from the token ratio measured in isolation. Measured end to end on v1.8.1, the full filter chain rejects prose placeholders equally in English, Portuguese and Spanish. The real bias ran the other way and was larger: a credential under a Portuguese key name was found 29% of the time, under a Spanish one 57%, under an English one 100%. That is what the earlier commits in this branch fix, and what every one of these documents now reports. MILESTONES records both the original hypothesis and why it was wrong, rather than rewriting history: the pt-BR dictionary the plan called for was not built because the measurement showed it would fix nothing. It also records a side finding — the upstream password rule reports instructional placeholders at low confidence in every language — and marks M5's exit criteria against what was measured: the corpus gates CI and the numbers are published; 500 cases and an upstream issue are still open. --- MILESTONES.md | 49 ++++++++++++++++++++++++++++-------------------- README.md | 2 +- README.pt-BR.md | 2 +- SECURITY.md | 15 +++++++++------ corpus/README.md | 10 ++++++---- corpus/corpus.go | 11 ++++++----- 6 files changed, 52 insertions(+), 37 deletions(-) diff --git a/MILESTONES.md b/MILESTONES.md index 9e71291..cab8808 100644 --- a/MILESTONES.md +++ b/MILESTONES.md @@ -424,37 +424,46 @@ pendência. **Objetivo.** Transformar falso positivo de opinião em métrica, e corrigir um viés medido. -### O problema, com número +### A hipótese original — e o que a medição mostrou -O filtro de token efficiency do betterleaks usa dicionário de 33.775 palavras **em inglês** e limiar fixo de 2,5 caracteres por token. Medido com o tokenizador que ele mesmo embute: +**A hipótese era de precision.** O filtro de token efficiency do betterleaks descarta um candidato como prosa quando ele contém uma palavra de um dicionário de 33.775 palavras **em inglês**; senão compara a razão caracteres/token com 2,5. Medida isolada, essa razão dava `senhapadrao` = 2,20 e `defaultpassword` = 7,50, e daí se concluiu que placeholder em português seria reportado como segredo. -| String | Razão | Resultado | -|---|---:|---| -| `defaultpassword` | 7,50 | descartado, correto | -| `senhapadrao` | **2,20** | **reportado como segredo** | -| `cobrancaboleto` | **2,33** | **reportado como segredo** | -| segredo real | 1,0–1,6 | reportado, correto | +**Medido de ponta a ponta (betterleaks v1.8.1), não é.** Só a regra `generic-api-key` usa esse filtro, e a cadeia completa rejeita placeholders em prosa nos três idiomas igualmente — 0 de 30 em inglês, português e espanhol. A conclusão vinha de olhar a razão fora da cadeia de filtros. -Em inglês a separação entre prosa e segredo é confortável. Em português quase some. Ninguém mediu isso. +**O viés real era de recall, e era maior.** As regras reconhecem nomes de chave em inglês. Com o mesmo segredo real sob nomes equivalentes: -### O que implementar +| idioma da chave | segredos reais achados | +|---|---:| +| inglês | 100% | +| espanhol | 57% | +| português | **29%** | + +`senha`, `segredo` e `senha_banco` nunca eram reconhecidos. As chaves em português que funcionavam (`chave_api`, `token_acesso`) funcionavam só por conter `api` e `token`. + +### O que foi implementado + +1. **`corpus/`** com formato rotulado (`expect` / `reject`), gate estrito e métrica **por tipo** — ✅ +2. **Passada de localização** em `engine/secrets/localize.go`: nomes de chave em português e espanhol que recebem atribuição são traduzidos (`senha`→`password`, `segredo`→`secret`, `chave`→`key`…) e o texto é escaneado de novo pelas **mesmas regras**. Valores nunca são reescritos, então o segredo é localizado no texto original. Os filtros do upstream — que também têm inglês embutido — se aplicam inteiros, o que dá paridade em vez de uma regra paralela com precision própria — ✅ +3. **Reordenação head-final**: português e espanhol nomeiam `senha_banco`; inglês nomeia `db_password`, e a regra do upstream exige fronteira de palavra logo depois de `password`, então `password_db` falha até em inglês. A tradução move `password` para o fim — ✅ + +Resultado, mesmos valores sob chaves equivalentes: recall **100%** em português e espanhol (inglês 88%, por causa do `password_db` que o próprio upstream perde). Falsos positivos com placeholders são iguais por chave nos três idiomas. + +**Não implementado, e por quê:** o dicionário pt-BR para o filtro de token efficiency (itens 1–2 do plano original). A medição mostrou que ele não corrigiria nada. + +### Achado lateral — fora do escopo do M5 -1. Dicionário pt-BR embutido, mesmo mecanismo `go:embed` + gzip -2. `--locale pt-BR`, com detecção automática do idioma predominante do repositório -3. **`corpus/`** — strings reais de código em pt, en, es, com rótulo esperado -4. `make fp-report` — imprime precisão e recall do conjunto; **falha o CI se regredir** -5. Publicar a medição como issue no betterleaks +A `generic-password` do upstream reporta placeholders instrucionais com confiança baixa **em qualquer idioma**: `"your-password-here"`, `"replace_me"`, `"changeme"` e seus equivalentes `"sua-senha-aqui"`, `"troque_isto"` — 5 de 10 em cada idioma. Referências (`${VAR}`, `os.getenv`, ``) são suprimidas corretamente nos três. É fraqueza de precision do upstream, neutra de idioma; candidata a issue no betterleaks ou a um filtro do vedei num marco próprio. ### Como testar -O próprio `make fp-report` é o teste. Roda no CI, com limiar mínimo travado. +`make corpus` (também roda em `go test ./...`, portanto no CI). A tabela por tipo aparece no log. ### Critério de saída -- [ ] `corpus/` com ≥ 500 casos rotulados, em 3 idiomas -- [ ] `make fp-report` no CI, falhando em regressão -- [ ] Número de precisão publicado no README -- [ ] Issue aberta no betterleaks com a medição +- [ ] `corpus/` com ≥ 500 casos rotulados, em 3 idiomas — **119 hoje, nos 3 idiomas** +- [x] Corpus no CI, falhando em regressão — `corpus_test.go` roda dentro de `go test ./...` +- [x] Número publicado no README — recall por idioma e precision/recall por tipo +- [ ] Issue aberta no betterleaks com a medição (recall por idioma + `password_db`) **Risco:** nenhum técnico. É curadoria — chato, e é por isso que ninguém fez. diff --git a/README.md b/README.md index 85d2b25..e91df3c 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ Full records in [`docs/adr/`](docs/adr/). Features, implementation steps, test strategy and exit criteria per stage: [`MILESTONES.md`](MILESTONES.md). Component design and performance budgets: [`ARCHITECTURE.md`](ARCHITECTURE.md). -**On M5.** betterleaks filters candidates with a BPE token-efficiency ratio against a 33,775-word **English** dictionary and a fixed 2.5 threshold. Measured with its own embedded tokenizer: `defaultpassword` scores 7.50 and is correctly discarded; `senhapadrao` scores 2.20 and is reported as a secret. Real secrets sit at 1.0–1.6. In English the gap is comfortable; in Portuguese it nearly vanishes. M5 turns that into a versioned corpus and a CI-enforced metric. +**On M5.** The original hypothesis was a precision bias: betterleaks discards a candidate as prose when it contains a word from a 33,775-word **English** dictionary, so `defaultpassword` is dropped and `senhapadrao` falls through to a ratio threshold. Measured end to end on v1.8.1, the full filter chain rejects prose placeholders equally in English, Portuguese and Spanish. The real bias was in **recall**: the rules key on English names, so a real credential was found 100% of the time under an English key, 57% under a Spanish one and **29% under a Portuguese one** — `senha` and `segredo` were never recognized. vedei now translates key names before a second pass, reaching 100% in all three. The labeled corpus behind these numbers is in [`corpus/`](corpus/): 119 cases, precision and recall 1.000 per type, enforced in CI. ## Project structure diff --git a/README.pt-BR.md b/README.pt-BR.md index 6ad8c00..44f2aba 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -141,7 +141,7 @@ Registros completos em [`docs/adr/`](docs/adr/). Features, passo a passo, estratégia de teste e critério de saída por etapa: [`MILESTONES.md`](MILESTONES.md). Desenho de componentes e orçamentos: [`ARCHITECTURE.md`](ARCHITECTURE.md). -**Sobre o M5.** O betterleaks filtra candidatos por razão de token BPE contra um dicionário de 33.775 palavras **em inglês** e limiar fixo de 2,5. Medido com o tokenizador que ele mesmo embute: `defaultpassword` dá 7,50 e é corretamente descartado; `senhapadrao` dá 2,20 e é reportado como segredo. Segredos reais ficam em 1,0–1,6. Em inglês a separação é confortável; em português quase some. O M5 transforma isso em corpus versionado e métrica cobrada no CI. +**Sobre o M5.** A hipótese original era um viés de precision: o betterleaks descarta um candidato como prosa quando ele contém uma palavra de um dicionário de 33.775 palavras **em inglês**, então `defaultpassword` cai e `senhapadrao` segue para um limiar de razão. Medido de ponta a ponta na v1.8.1, a cadeia completa de filtros rejeita placeholders em prosa igualmente em inglês, português e espanhol. O viés real estava no **recall**: as regras reconhecem nomes de chave em inglês, então uma credencial real era achada 100% das vezes sob uma chave em inglês, 57% em espanhol e **29% em português** — `senha` e `segredo` nunca eram reconhecidos. O vedei agora traduz nomes de chave antes de uma segunda passada e chega a 100% nos três. O corpus rotulado por trás desses números está em [`corpus/`](corpus/): 119 casos, precision e recall 1.000 por tipo, cobrados no CI. ## Estrutura do projeto diff --git a/SECURITY.md b/SECURITY.md index 48339bf..c6c5ddf 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -50,12 +50,15 @@ peak use is roughly twice the input. A 1 GB file needs about 2 GB. Streaming arrives with the CLI in M4; until then, do not point the library at arbitrarily large input. -**The upstream corpus brings its own false positives.** betterleaks filters -candidates with a token-efficiency ratio calibrated on a 33,775-word English -dictionary. Portuguese text sits far closer to the threshold than English -does — `senhapadrao` scores 2.20 and is reported, `defaultpassword` scores -7.50 and is correctly discarded — so a Portuguese codebase sees more noise -from the secrets engine. M5 measures and corrects this. +**The upstream secret rules were built around English key names.** Measured +on betterleaks v1.8.1, a real credential was found 100% of the time under an +English name but 57% under a Spanish one and 29% under a Portuguese one: +`senha` and `segredo` were never recognized. vedei translates key names and +scans again, reaching 100% in all three; the corpus in `corpus/` pins it. +The upstream rules also report instructional placeholders such as +`"your-password-here"` at low confidence — in every language equally, so it is +noise, not a language bias. An earlier note here claimed Portuguese placeholders +were the problem; measured end to end, they are not. **124 transitive dependencies.** Two direct ones, of which betterleaks brings the rest, including cloud SDKs it uses for credential validation that diff --git a/corpus/README.md b/corpus/README.md index 7f008ae..59e8b5a 100644 --- a/corpus/README.md +++ b/corpus/README.md @@ -3,10 +3,12 @@ Labeled examples that measure what vedei catches and what it cries wolf over. A scanner's worth is not that it compiles. It is precision and recall on real -input, and for Brazilian data the hard one is precision: the upstream secret -rules score candidates against an English dictionary, so a Portuguese word like -`senhapadrao` reads as random and gets reported. That cost is invisible until -you measure it against labeled data. This directory is that data. +input, and neither can be settled by reading the rules. M5 assumed Portuguese +placeholders like `senhapadrao` would be reported as secrets; measured here, +they were not. The real gap ran the other way: a credential under a Portuguese +key name was found 29% of the time, against 100% under an English one. That is +what labeled data is for — it tells you which way a bias runs. This directory +is that data. ## Format diff --git a/corpus/corpus.go b/corpus/corpus.go index 0c5b439..58b355e 100644 --- a/corpus/corpus.go +++ b/corpus/corpus.go @@ -1,11 +1,12 @@ // Package corpus loads the labeled examples that measure detection quality. // // It is the measurement tool the milestones call for: a scanner's worth is not -// "does it compile" but "what does it catch and what does it cry wolf over", -// and in Portuguese the second question is the hard one. The upstream secret -// rules score candidates against an English dictionary, so a Portuguese word -// like "senhapadrao" reads as random and is reported. That cost is only visible -// against labeled data, which is what this package is. +// "does it compile" but "what does it catch and what does it cry wolf over". +// Neither question can be answered by reasoning about the rules. M5 set out +// expecting Portuguese placeholders to be reported as secrets; measured against +// labeled cases, they were not — the real gap was the reverse, credentials +// under Portuguese key names going unfound. Only labeled data shows which +// way a bias runs, which is what this package is. // // A case file is plain text, one directive per line: // From bda9c790d9b6043f3f18e8c0e078e4df02182308 Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 15:10:34 -0300 Subject: [PATCH 8/9] fix(scan): honour .vedeiignore path rules in history and diff scans MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit .vedeiignore path patterns were applied by "vedei scan" and nowhere else. A directory silenced there was still reported by "vedei diff" — the mode CI runs on every pull request — and by "vedei git". Nothing noticed, because until now every ignore entry in this repository was a fingerprint, not a path. The first version of the fix introduced a worse bug, caught by its own test. History scans identical content once, by blob hash. Checking the path after the blob was marked as seen meant an ignored copy could use up that one scan, so the same bytes committed outside the ignored directory were never scanned: a-fixtures/case.txt (ignored) \ same blob z-app/config.txt (real leak) / -> Total() = 0 A leak hidden by whichever path happens to sort first. The path is now checked before the blob is marked, so an ignored path never consumes it. Two tests pin both behaviours, and the second fails with the check moved back. The per-commit loop moves into scanCommit, which also brings History back under the complexity limit it had already crossed. --- scan/git.go | 60 ++++++++++++++++++++++++++++++--------------- scan/git_test.go | 63 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 20 deletions(-) diff --git a/scan/git.go b/scan/git.go index fdaab94..62d8592 100644 --- a/scan/git.go +++ b/scan/git.go @@ -170,31 +170,43 @@ func (g *GitScanner) History(ctx context.Context, revs ...string) (Result, error if err := ctx.Err(); err != nil { return res, err } - entries, err := g.run(ctx, "ls-tree", "-r", "-z", "--long", commit) - if err != nil { - res.Errs = append(res.Errs, err) + g.scanCommit(ctx, &res, commit, ignores, seen) + } + return res, nil +} + +// scanCommit scans the blobs in one commit that have not been scanned yet. +func (g *GitScanner) scanCommit(ctx context.Context, res *Result, commit string, ignores Stack, seen map[string]bool) { + entries, err := g.run(ctx, "ls-tree", "-r", "-z", "--long", commit) + if err != nil { + res.Errs = append(res.Errs, err) + return + } + for _, e := range splitNUL(entries) { + blob, path, size, ok := parseLsTree(e) + if !ok || seen[blob] { + continue + } + // An ignored path must not consume the blob. Identical content is + // scanned once, so if the ignored copy marked it as seen, the same + // bytes committed somewhere that is not ignored would never be + // scanned — a leak hidden by whichever path sorted first. + if ignores.Ignored(path, false) { + continue + } + seen[blob] = true + if g.scanner.maxFileSize > 0 && size > g.scanner.maxFileSize { + res.Skipped++ continue } - for _, e := range splitNUL(entries) { - blob, path, size, ok := parseLsTree(e) - if !ok || seen[blob] { - continue - } - seen[blob] = true - if g.scanner.maxFileSize > 0 && size > g.scanner.maxFileSize { - res.Skipped++ - continue - } - content, err := g.run(ctx, "cat-file", "blob", blob) - if err != nil { - res.Errs = append(res.Errs, err) - continue - } - g.appendFindings(ctx, &res, GitRef{Commit: commit, Path: path}, content, ignores) + content, err := g.run(ctx, "cat-file", "blob", blob) + if err != nil { + res.Errs = append(res.Errs, err) + continue } + g.appendFindings(ctx, res, GitRef{Commit: commit, Path: path}, content, ignores) } - return res, nil } // scanIndexPaths scans the staged version of each path, reading from the index @@ -219,6 +231,14 @@ func (g *GitScanner) scanIndexPaths(ctx context.Context, paths []string) (Result // appendFindings scans one blob and records what it held. func (g *GitScanner) appendFindings(ctx context.Context, res *Result, ref GitRef, content []byte, ignores Stack) { + // Path rules in .vedeiignore apply to history and diffs as they do to a tree + // scan. Without this, a path rule silenced a file in "vedei scan" and the + // same file was still reported by "vedei diff" — the mode CI runs on every + // pull request. + if ignores.Ignored(ref.Path, false) { + res.Skipped++ + return + } if g.scanner.maxFileSize > 0 && int64(len(content)) > g.scanner.maxFileSize { res.Skipped++ return diff --git a/scan/git_test.go b/scan/git_test.go index 0d2394f..2cab993 100644 --- a/scan/git_test.go +++ b/scan/git_test.go @@ -265,6 +265,69 @@ func TestHistory_VedeiignoreComesFromTheWorkingTree(t *testing.T) { } } +// Path rules in .vedeiignore must hold in history and diff scans too. They used +// to apply only to "vedei scan", so a directory silenced there was still +// reported by "vedei diff" — which is what CI runs on every pull request. +func TestGitScanner_HonoursVedeiignorePathRules(t *testing.T) { + repo := gitRepo(t) + writeFile(t, repo, ".vedeiignore", "fixtures/\n") + writeFile(t, repo, "fixtures/cases.txt", "cpf "+testCPF) + writeFile(t, repo, "app/config.txt", "cpf "+testCPF) + git(t, repo, "add", ".") + git(t, repo, "commit", "-qm", "first") + base := git(t, repo, "rev-parse", "HEAD~0") + + writeFile(t, repo, "fixtures/more.txt", "cnpj 11.222.333/0001-81") + git(t, repo, "add", ".") + git(t, repo, "commit", "-qm", "second") + + g := gitScanner(t, repo) + hist, err := g.History(context.Background()) + if err != nil { + t.Fatalf("History: %v", err) + } + diff, err := g.Diff(context.Background(), base, "HEAD") + if err != nil { + t.Fatalf("Diff: %v", err) + } + + for name, res := range map[string]scan.Result{"history": hist, "diff": diff} { + for _, f := range res.Files { + if strings.Contains(f.Path, "fixtures/") { + t.Errorf("%s reported %s, which .vedeiignore silences by path", name, f.Path) + } + } + } + // The rule must not silence everything: the file outside it is still found. + if hist.Total() != 1 { + t.Errorf("history Total() = %d, want 1 (app/config.txt only)", hist.Total()) + } +} + +// History scans identical content once. An ignored path must not use up that +// one scan: here the ignored copy sorts first, and the same bytes committed +// outside the ignored directory still have to be reported. +func TestHistory_IgnoredPathDoesNotHideTheSameBlobElsewhere(t *testing.T) { + repo := gitRepo(t) + writeFile(t, repo, ".vedeiignore", "a-fixtures/\n") + // Identical content, so one blob. "a-fixtures" sorts before "z-app". + writeFile(t, repo, "a-fixtures/case.txt", "cpf "+testCPF) + writeFile(t, repo, "z-app/config.txt", "cpf "+testCPF) + git(t, repo, "add", ".") + git(t, repo, "commit", "-qm", "first") + + res, err := gitScanner(t, repo).History(context.Background()) + if err != nil { + t.Fatalf("History: %v", err) + } + if res.Total() != 1 { + t.Fatalf("Total() = %d, want 1 — the copy outside the ignored directory was hidden", res.Total()) + } + if got := res.Files[0].Path; !strings.Contains(got, "z-app/config.txt") { + t.Errorf("reported %q, want the z-app copy", got) + } +} + func TestNewGitScanner_NotARepository(t *testing.T) { if _, err := exec.LookPath("git"); err != nil { t.Skip("git not available") From 647fdf5c407c0b2b52defba600e2c7697671f46d Mon Sep 17 00:00:00 2001 From: Mkmuniz Date: Sat, 3 Oct 2026 15:13:03 -0300 Subject: [PATCH 9/9] chore: silence corpus/cases/ by path, the one path rule in the repository MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every ignore entry in this repository has been a fingerprint, on purpose: silencing *_test.go wholesale would also silence a credential genuinely committed into a test. The corpus is the exception that rule cannot carry. corpus/cases/ exists to hold labeled example values, and naming each one meant a fingerprint per case — 10 already, hundreds at the size M5 aims for, each one easy to forget when a case is added. A value there is a labeled case by construction: the corpus test fails on any line that is not an "expect" or a "reject". That is what makes a path rule safe for this directory and nowhere else. The ten fingerprints that only covered corpus files are removed (39 to 29). One is kept that the tree no longer needs: an earlier version of .vedeiignore quoted the official alphanumeric CNPJ in a comment, so git history still holds it, and the weekly history scan would report it. Found by running "vedei git" after the change, not before — the tree scan alone said it was safe to drop. Both READMEs and the corpus README describe the exception; the claim "nothing is silenced by path" is gone. Tree and history scans are both clean. --- .vedeiignore | 45 +++++++++++++++++++-------------------------- README.md | 2 +- README.pt-BR.md | 2 +- corpus/README.md | 6 ++++++ 4 files changed, 27 insertions(+), 28 deletions(-) diff --git a/.vedeiignore b/.vedeiignore index c5591ec..81a635a 100644 --- a/.vedeiignore +++ b/.vedeiignore @@ -72,30 +72,23 @@ a83871a7916f6b9c44907082aa289cfc # +***********21 - a second phone in the same d554cea51a6155fd70c74664b448aff2 # +***********94 - a third 9ee2bb2c7f2a591ab50bb3070faa1a6d # a UUID v4 Pix key, in the overlap test -# --- M5 corpus and test fixtures --- -# The labeled corpus and the mask/AWS tests carry example values on purpose. -# Fingerprinted, not path-ignored, so the principle holds: every value is named, -# and a real secret committed into these files would still be reported. -d0dead6eceff137cafd15c0029bf1214 # AKIA...LE example key in corpus -a8cadd7b8a5caff89d90408d8a6c31bc # AKIA...TQ assembled test key -e58b0b5d6b990a005bb44e5d490a851e # ASIA...TQ assembled test key -2ffebaa6e81961e4cef8beeb61c5f8d1 # generated CNH vector -648782aa9ca9e35c33bddea812bb255f # generated CNH vector -2040ad664a71a504a6ff863b0b78a4f0 # generated PIS vector, space-separated -4d4eb30077a44f6a1e088b2c4cff6f1d # generated PIS vector -62949e15935851bd24d617292e7230f8 # generated titulo vector, space-separated -f60e47e7fe914c3e5d10a9b2a99d8b96 # generated titulo vector -ec2a7642f9821664294f09e54573ca66 # generated CNS vector -83f7388d806b21f4c0c4ea6e6a651937 # Pix email key fixture +# --- test fixtures outside the corpus --- +# Values the mask and hook tests use. The same values also sit in corpus/cases/, +# which the path rule below covers; these lines cover the test files. +d0dead6eceff137cafd15c0029bf1214 # AKIA...LE example key, cmd/vedei-hook/main_test.go +2040ad664a71a504a6ff863b0b78a4f0 # generated PIS vector, space-separated, detect/br/extract_test.go +62949e15935851bd24d617292e7230f8 # generated titulo vector, space-separated, detect/br/extract_test.go +ec2a7642f9821664294f09e54573ca66 # generated CNS vector, detect/br/extract_test.go +3d916d891e1aab79bc68592c2d7cd3a5 # the first alphanumeric CNPJ the Receita issued; an earlier version of this file quoted it in a comment, so git history still holds it -3d916d891e1aab79bc68592c2d7cd3a5 # 00.000.000/E08G-12, the first alphanumeric CNPJ the Receita issued (2026-07-31), a public record used as a corpus case - -# --- multilingual credential fixtures (corpus/cases/secrets-i18n.txt) --- -# One made-up credential-shaped value under key names in three languages. The -# same value is reported by two rules, so it has two fingerprints. -bd02d2bf6ebb75a6c085a966323c06f1 # fake credential under password/senha/contrasena keys -c3fca3e7a0151668ec85c974b0f4f57f # fake credential under api_key/chave/clave keys - -# Nothing is ignored by path, on purpose. Silencing *_test.go wholesale would -# also silence a credential genuinely committed into a test, which is a place -# credentials genuinely end up. Every entry above is one value, named. +# --- the one path rule: the labeled corpus --- +# Everything else in this file is a single value, named, and that is the rule: +# silencing *_test.go wholesale would also silence a credential genuinely +# committed into a test, which is a place credentials genuinely end up. +# +# corpus/cases/ is the one exception. It exists to hold labeled example values, +# and the corpus test fails on any line that is not an "expect" or a "reject", +# so a value there is a labeled case by construction. Naming each one here +# would mean a fingerprint per case — hundreds, at the size M5 aims for — each +# one easy to forget. History and diff scans honour this rule too. +corpus/cases/ diff --git a/README.md b/README.md index e91df3c..fd0eb64 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Wire the hook into your agent: [`hooks/`](hooks/). Two binaries, because the spl ## Silencing what is deliberate -`.vedeiignore` takes a finding fingerprint, derived from the type and the normalized value and **never from the location** — so an entry survives the file being moved or renamed. This repository's own file is [`.vedeiignore`](.vedeiignore): 71 findings, 21 fingerprints, each with the reason written next to it. Nothing is silenced by path, on purpose. Silencing `*_test.go` wholesale would also silence a credential genuinely committed into a test, which is a place credentials genuinely end up. +`.vedeiignore` takes a finding fingerprint, derived from the type and the normalized value and **never from the location** — so an entry survives the file being moved or renamed. This repository's own file is [`.vedeiignore`](.vedeiignore): one fingerprint per deliberate test value, each with the reason written next to it. Paths are almost never silenced: ignoring `*_test.go` wholesale would also silence a credential genuinely committed into a test, which is a place credentials genuinely end up. The single exception is `corpus/cases/`, the labeled corpus — every line there is a labeled example by construction, and a fingerprint per case would not scale. ## What it detects diff --git a/README.pt-BR.md b/README.pt-BR.md index 44f2aba..8030306 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -48,7 +48,7 @@ No CI: ## Silenciando o que é deliberado -O `.vedeiignore` aceita o fingerprint do achado, derivado do tipo e do valor normalizado e **nunca da localização** — então a entrada sobrevive ao arquivo ser movido ou renomeado. O arquivo deste repositório é o [`.vedeiignore`](.vedeiignore): 71 achados, 21 fingerprints, cada um com o motivo escrito ao lado. Nada é silenciado por caminho, de propósito. Silenciar `*_test.go` inteiro também silenciaria uma credencial de verdade commitada num teste, que é um lugar onde credencial de verdade acaba indo. +O `.vedeiignore` aceita o fingerprint do achado, derivado do tipo e do valor normalizado e **nunca da localização** — então a entrada sobrevive ao arquivo ser movido ou renomeado. O arquivo deste repositório é o [`.vedeiignore`](.vedeiignore): um fingerprint por valor de teste deliberado, cada um com o motivo escrito ao lado. Caminhos quase nunca são silenciados: ignorar `*_test.go` inteiro também silenciaria uma credencial de verdade commitada num teste, que é um lugar onde credencial de verdade acaba indo. A única exceção é `corpus/cases/`, o corpus rotulado — toda linha ali é um exemplo rotulado por construção, e um fingerprint por caso não escala. Depois ligue o hook no seu agente: [`hooks/`](hooks/). São dois binários, e é essa separação que deixa o hook rápido — o `vedei` carrega 417 regras de segredo, o `vedei-hook` tem 5 MB e só conversa com o daemon. diff --git a/corpus/README.md b/corpus/README.md index 59e8b5a..1adbb8b 100644 --- a/corpus/README.md +++ b/corpus/README.md @@ -48,3 +48,9 @@ reason, then fix the detector until this test passes again. When a detector gains a document type, add `expect` cases for it. The file `cases/false-positives.txt` is the running record of what scanning the real world actually produced — every entry in it was once a wrong finding. + +No `.vedeiignore` entry is needed. `corpus/cases/` is silenced by path — the one +path rule in this repository — so the self-scan does not report the values that +live here. That rule also means a value pasted into these files is never +reported by `vedei scan`, which is safe only because every line must be a +labeled `expect` or `reject`: the corpus test fails on anything else.