Repository navigation
Rozpoznawanie oparte na pewności + pakiety identyfikatorów PL/UE/US (wzorowane na Presidio) - #22
Merged
Merged
Conversation
Patterns were binary: a regex either matched or it did not, and which rule claimed a span came down to the order rules happened to run in. That forced a choice between recall and precision — visible in this repo's own history, where the PESEL rule was reduced to eleven bare digits with the checksum removed to stop missing real ones. Confidence replaces the choice, following Presidio's model: - a rule holds several scored variants instead of one regex - context words near a match raise its score (+0.35, floor 0.4) - a passing checksum raises it to at least 0.85 - `--sensitivity` picks the threshold: balanced 0.5, strict 0.35, paranoid 0.1 - candidates from every rule are gathered, overlaps resolved by score, and substitutions applied by offset Checksums now say what a failure means. For a card number, failing Luhn means it is not a card number, so the candidate goes. For a PESEL it means a typo or a fictitious number — still eleven digits that must not reach the cloud — so the checksum only adds confidence. That is the new `checksumMode: 'boost'`. Language rule sets are derived from the registry by locale, so a new rule file is picked up by adding it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…zers Covers the identifiers that appear next to a PESEL in Polish paperwork and were previously invisible to the masker: - REGON — Mod-11 over 9 or 14 digits, remainder 10 folded to 0 - dowód osobisty — weights 7,3,1 with the check digit at position four - paszport — weights 7,3,9,1,7,3,1,7,3, valid when divisible by 10 - księga wieczysta — official character table, weights cycling 1,3,7 - KRS — no check digit exists, so it rides entirely on its label Nine-digit REGONs and ten-digit KRS numbers overlap with phone numbers and order numbers, so both start below the masking threshold and are lifted by the checksum or the surrounding words rather than firing on shape alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The IBAN rule matched any two letters, two digits and a run of alphanumerics — no length check, no checksum — so reference numbers of that shape were masked as bank accounts and nothing verified the ones that were. - ISO 7064 mod-97 with the ISO 13616 length table for 76 countries, computed piecewise to stay inside safe integer range - bare 26-digit Polish NRB numbers validated by restoring the PL prefix - EU VAT: formats for all 27 member states and XI, with national check digits for PL (NIP), IT (Luhn), NL (elfproef), SI and LU Both use checksumMode 'boost': a valid checksum promotes the match to near-certainty, while a mistyped account number stays a candidate rather than being dropped, since that is precisely the case where a miss leaks data. The PL-prefixed VAT variant moves out of the NIP rule so one string cannot surface as NIP in one document and VAT_ID in another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten locales' national identifiers, each validated by its own checksum: Steuer-ID (ISO 7064 Mod 11,10 plus the digit-frequency rule), codice fiscale (odd/even substitution tables), NIF and NIE (modulo 23), NIR (modulo 97 with Corsica mapped), BSN (elfproef), rodné číslo (divisible by 11), personnummer (Luhn), hetu (modulo 31) and the NHS number (modulo 11). NINO and the UK postcode have no checksum, so they rest on shape and context alone. Locale packs used to be unreachable unless the document language matched, and an unknown language quietly fell back to the English rules — which is how a French NIR came out tagged as a credit card. Rule sets are now selected from the registry by locale, with `--extra-locales` to enable further packs beside the main language. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also splits SSN into the dashed form, which is unmistakable on its own, and the dotted, spaced and bare-nine-digit spellings, which fire only with the label nearby. Two calibration fixes came out of it. The SSN validator checks area, group and serial ranges — it rules out impossible numbers but does not identify an SSN, so promoting a match to checksum-grade confidence let "Passport no 123456789" come back tagged SSN. That is the new 'gate' mode: reject on failure, no promotion on success. And a context word now lifts any match to the balanced threshold. A rule scoring 0.1 plus the old +0.35 landed at 0.45, just under the bar, so "Driver license D1234567" stayed unmasked even though the document said what it was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Bitcoin addresses verified for real: Base58Check double-SHA256 and the bech32/bech32m polymod, not just the address shape - Ethereum addresses by shape, since a lowercase address carries no checksum - IMEI (Luhn) and VIN (ISO 3779 mod-11) with their check digits - MAC addresses and UUIDs - dates, deliberately below the default threshold: a deadline is not personal data, so they mask when the text reads as a date of birth, or at higher sensitivity Documents the whole recognizer set in the README, including how a match is scored, and adds a NOTICE crediting the MIT-licensed Presidio work the pattern model and several check-digit algorithms are adapted from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ±120-character window reached into the next field. In a contract reading "Umowa z dnia 2026-03-14. … data urodzenia 1944-05-14", the phrase "data urodzenia" sat inside the contract date's window and masked it as a birth date. Documents label a value before printing it, so the window is now eight words back and four forward. Adds the labels an address is written under to the postal code rule, and a full-document regression test covering which values must be masked and — just as important — which must not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--sensitivity(balanced/strict/paranoid).filter— nieudana suma znaczy „to nie ta encja" (karta bez Luhna).boost— znaczy literówkę, kandydat zostaje, bo przy narzędziu do maskowania fałszywie ujemny to wyciek (PESEL, IBAN).gate— waliduje, ale nie promuje (zakresy SSN). Bez tego „Passport no 123456789" wracało jako[SSN:1].Zmiany zachowania
--langnie wpada już cicho w reguły angielskie (tak francuski NIR wychodził jakoCREDIT_CARD)Test plan
npm run lint,npm run test:run(331 testów),npm run build,npm run format:check— zieloneRSSMRA85T10A562S,86095742719,021000021,1HGBH41JXMN109186NOTICEz atrybucją MIT dla Presidio, dołączony dofilesw paczce🤖 Generated with Claude Code