Turn the same healthcare professionals — as they appear, messily, across a CRM, an offline-engagement system and web analytics — into de-duplicated golden records, and measure the match quality against ground truth.
The same HCP shows up in every source system with a different spelling: Dr. Fatima Khan in the CRM, Khan, F. offline, just an email on the web, an NPI present in
one system and missing in another. Until those are resolved to one identity, every
duplicate inflates reach, wastes field visits on "new" doctors who aren't new, and
splits a customer's history across records. The hard part is doing it without
merging two different people who happen to share a name.
The standard master-data pipeline, measured honestly:
- Profile each source (row counts, how often the NPI is populated).
- Cleanse / standardise — strip titles, split names, normalise organisations, reduce NPIs to digits, lowercase emails.
- Block on a typo-robust key (sorted surname characters) so near-duplicates land together without comparing every pair.
- Match — deterministic on NPI or a shared email; otherwise probabilistic: surname agreement, first-name agreement (initial-aware), and a corroborating field (city or organisation). Surname alone is never enough.
- Cluster matched pairs with union-find, then survive the best value per attribute by source trust (CRM > offline > web) and completeness into a golden record with a trust score and its member records.
On the seeded data (220 true entities, ~435 fragmented records):
| Metric | Value |
|---|---|
| Pairwise precision | ~0.93 |
| Pairwise recall | ~0.84 |
| Pairwise F1 | ~0.88 |
Precision is deliberately favoured over recall — over-merging two real HCPs is a worse error in commercial operations than leaving one HCP briefly split, and the numbers say so plainly.
This is a demonstration on synthetic data; the pipeline and the scoring are the point, not the absolute figures.
pip install -r requirements.txt # rapidfuzz
bash run.sh # generate fragmented sources -> resolve -> score
python tests/test_resolve.py # quality regression- Synthetic, seeded data; a real estate needs address parsing, nickname dictionaries and a human-review queue for the borderline band.
- One fuzzy threshold set is used; production would tune per field and add active learning on reviewer decisions.
MIT