Redaction kills context, what do you use instead?

[danielbrianjohnson]

I've been using Presidio for PII detection in an AI support ticket pipeline and it's been solid. Detection is accurate, the recognizer registry is fairly easy to extend, and for detect-and-redact it just works OOTB.

It stopped fitting my use case when the pipeline grew past single-ticket processing. Aggressive redaction is great for compliance but it flattens a lot of downstream signal/context. Once every customer is <PERSON> and every number is <PHONE_NUMBER>, the model (or a human) can't tell tickets apart, can't correlate a repeat caller, and can't spot patterns across someone's history.

What's been working for me is format-preserving, deterministic replacement where the same input always maps to the same token (so Dan Johnson becomes a stable <PERSON>Pf8 kLXJbDp</PERSON> everywhere it shows up). That way systems can still correlate entities without ever seeing raw values, and downstream parsers don't choke since the tokens stay structurally valid (a name-shaped string, a valid-looking phone number).

The catch is that determinism keys on the exact string, so the same real entity written differently still tokenizes differently. Dan johnson vs Dan Johnson, or danjohnson@gmail.com vs dan.johnson@gmail.com, break the consistency unless you normalize/canonicalize before tokenizing. Curious how people are handling that part too.

I know Presidio is extensible here (custom operators, plus encrypt/hash), so you can reach toward some of this with enough custom code. Where it got harder was consistency across documents and reversibility gated by role/RBAC, since that's not really there OOTB and ends up being something you build/maintain yourself.

So how are you all handling this?

• Consistent pseudonymization across a whole corpus, not just within a single document
• Reversible replacement where only authorized roles can recover the originals
• Whether you do this inside Presidio with custom operators or layer something on top

Disclosure: I work at Protegrity and we build tooling that uses Presidio for detection and adds tokenization (and a few other things) on top.