QualiDeID

Local de-identification and consistent pseudonymization for qualitative research data: interview transcripts, focus groups, and open ended responses.

QualiDeID finds the identifiers in a transcript (names, emails, phone numbers, organizations, places, and more), gives each person, organization, and place a consistent pseudonym, lets you review and correct the proposals, and writes de-identified copies plus a pseudonym map and an audit log. The same identifier becomes the same pseudonym across every file and every mention, which is the part that hand editing in a word processor always gets wrong.

It runs fully offline. The default detection engine is pure Python (no model download, no network, no API key). Optional backends (Microsoft Presidio, or a local LLM via Ollama for quasi identifiers) can be switched on, but are never required.

The gap it fills

General PII tools exist (Presidio is the obvious one), but they are frameworks aimed at structured or enterprise data and one shot redaction. None of them provides the qualitative research workflow: consistent pseudonyms across a whole corpus, a human review and merge step, transcript formats (DOCX, VTT, SRT), and an audit log fit for an ethics board. Searches for a research specific tool turn up only manually anonymized thesis data, not software. QualiDeID is that missing workflow layer. See ../OPPORTUNITIES.md for the evidence.

Full scope (what this project ships)

Core:

• Ingest transcripts in plain text (.txt), Word (.docx), WebVTT (.vtt), and SubRip (.srt). For captions, only the spoken text is changed; timing lines and cue identifiers are preserved. For Word, paragraph structure is preserved.
• A built in detection engine covering emails, phone numbers, URLs, IP addresses, SSNs, credit cards (Luhn checked), IBANs, dates, persons (titles, self introductions, and a bundled given name gazetteer), organizations, locations and addresses, and transcript speaker labels.
• A persistent SQLite project store that makes pseudonyms consistent: an entity is keyed by type and normalized surface, so the same name maps to the same pseudonym across files and runs.
• Per type de-identification actions: pseudonym (P01, ORG1, LOC1), redact to a generic tag ([EMAIL]), or keep. Defaults are sensible and configurable.
• A review workflow: list detected entities, change any decision, reject false positives, and merge entities (so "Jane" and "Jane Doe" share one pseudonym).
• De-identified output files, a pseudonym map (CSV), and an audit log (CSV).

Optional and opt in:

• A Presidio backend for spaCy based NER.
• A local LLM pass (via Ollama) that proposes quasi identifiers (employer, unique role, rare place).
• An interactive terminal review command.

Install

Requires Python 3.10 or newer (developed and tested on 3.14).

python -m venv .venv
# Windows PowerShell:  .venv\Scripts\Activate.ps1
# Git Bash / Linux / macOS:  source .venv/bin/activate
pip install -e .

Exact pinned environment:

pip install -r requirements-lock.txt
pip install -e . --no-deps

Optional extras:

pip install -e ".[presidio]"  # spaCy based NER backend (also needs a spaCy model)
pip install -e ".[tui]"       # interactive Textual reviewer
pip install -e ".[dev]"       # pytest and hypothesis for the test suite

Usage

One step de-identification

qualideid deid interview.txt -o out/

This scans the file, applies the default decisions, and writes out/interview.deid.txt, out/pseudonym_map.csv, out/audit_log.csv, and a project store out/qualideid.sqlite. Example output:

INTERVIEWER: Could you introduce yourself?
P: My name is P01. You can reach me at [EMAIL] or [PHONE].
I work at ORG01 and I live in LOC01. My colleague P02 joined in 2019.

The full review workflow

# 1. Scan one or more transcripts into a project store
qualideid scan interview1.txt interview2.vtt --project study.sqlite

# 2. Review what was found and how it will be replaced
qualideid entities --project study.sqlite
qualideid entities --project study.sqlite --csv review.csv   # or export to CSV

# 3. Correct decisions: reject a false positive, change an action, or merge
qualideid set --project study.sqlite --id 7 --status rejected
qualideid set --project study.sqlite --id 3 --action redact
qualideid merge --project study.sqlite --into 2 --from 5     # "Jane" and "Jane Doe"

# 4. Apply to produce de-identified copies plus the map and audit log
qualideid apply interview1.txt interview2.vtt --project study.sqlite --out out/

# Inspect the mapping and the trail at any time
qualideid map --project study.sqlite
qualideid audit --project study.sqlite

Run python -m qualideid ... if you prefer not to install the console script.

Configuration

Settings come from built in defaults, an optional TOML file (--config), and two environment variables. A TOML file can change the action per entity type and the pseudonym label prefixes:

[qualideid]
backend = "builtin"   # or "presidio"

[qualideid.actions]
DATE = "redact"       # redact dates instead of keeping them
LOCATION = "pseudonym"

[qualideid.labels]
PERSON = "Participant"

Environment variables: QUALIDEID_BACKEND and QUALIDEID_OLLAMA_HOST (used only by the optional LLM pass). No secrets are read or stored.

Detection notes and limits

The built in engine is deliberately cautious and is not a substitute for a human reviewer; that is exactly why the review and merge steps exist. Names are found by titles (Dr., Ms.), self introductions ("my name is ..."), a bundled given name list, and capitalized name sequences. Co-reference (linking "Jane" to "Jane Doe") is resolved by the merge command. Enable the Presidio or LLM backends for stronger name, organization, and quasi identifier coverage.

How to run the tests

pip install -e ".[dev]"
pytest

The suite (42 tests) builds its own transcripts (including DOCX, VTT, and SRT) in temporary directories and runs fully offline. It covers the recognizers, overlap resolution, the store and pseudonym consistency, the redactor, every transcript format, the end to end de-identification property, the CLI, configuration, and the security properties below.

Architecture notes

• models.py entity types, actions, and the entity record.
• config.py per type actions, labels, and detection settings.
• recognizers.py the built in detectors (structured and heuristic).
• names.py the bundled given name gazetteer and stop word list.
• detect.py runs recognizers and optional backends, then de-overlaps spans.
• pseudonyms.py pseudonym formatting and surface normalization.
• store.py the SQLite project store (parameterized SQL only).
• transcripts.py format specific reading and writing.
• engine.py scanning, the redactor, and applying decisions.
• pii_backends.py the optional Presidio and Ollama backends.
• cli.py the Typer command line interface.

Security notes

QualiDeID handles the most sensitive data a researcher holds, so safety is a first class concern.

• No SQL injection. Every database statement uses bound parameters, never string formatting, so transcript text (which becomes entity surfaces) can never be interpreted as SQL. There is a test that a malicious surface containing DROP TABLE is stored as data and the database is unharmed.
• Output stays in the output directory. Output file names are sanitized and the resolved path is confirmed to be inside the chosen output directory, so a crafted input file name cannot write elsewhere.
• Bounded input. Files larger than a configurable limit (default 25 MB) are refused, and every detection regex is linear and length bounded to avoid catastrophic backtracking (there is a test on adversarial input).
• Offline by default, opt in network. The default engine makes no network calls. The only outbound traffic is the optional LLM pass, and it goes only to the local Ollama host you configure, with a timeout. The host is read from the environment and never logged.
• No unsafe execution. There is no eval, no exec, no pickle, and no untrusted deserialization. Configuration is read with the standard library tomllib, which cannot execute code.
• Sensitive artifacts. By design the project store and the pseudonym map contain the link between real identifiers and pseudonyms (this is what makes pseudonyms consistent). Treat the .sqlite store and pseudonym_map.csv as sensitive as the source data: keep them out of shared folders and version control, and delete them when the de-identified outputs are finalized.

De-identification is never perfect. Always have a human review the output before sharing or archiving data, especially for quasi identifiers (unique roles, rare combinations of attributes) that no automatic tool can catch reliably.

License

Apache-2.0. See LICENSE.