Skip to content

prov-fhir: import FHIR (Bulk ndjson) into a medical workspace - #28

Open
adammharris wants to merge 1 commit into
medical-presetfrom
fhir-import
Open

adammharris wants to merge 1 commit into
medical-presetfrom
fhir-import

Conversation

@adammharris

Copy link
Copy Markdown
Member

Stacked on #27 (the medical preset). Adds prov-fhir, a new workspace crate (publish = false) that imports a FHIR export into a workspace kept with that preset, through the prov library: every write is a Workspace call (create_with_fields, file, attach_manifest_claiming, save_document, retitle), with no shelling out to the CLI.

prov-fhir patients <export>                          # who is in it
prov-fhir import <export> --into <root> [--patient <id>]
prov-fhir import <export> --each <dir>                # one workspace per patient

<export> is a Bulk Data directory of *.ndjson, one ndjson file, or a JSON Bundle. One workspace holds one patient's record, so --into takes one patient (--patient may be a unique id prefix, and can be left out when the export holds only one). A --into root that is missing or empty is made the way --each makes one: prov init + the medical preset (built into the binary with include_str!; --preset DIR swaps it), then the indexes. A workspace that already holds a different patient is refused.

What it does

  1. The export is kept as evidence. The patient's share of the export (their resources plus the practitioners, roles, organizations, locations and medications those refer to) is written under imports/<date>-<digest>/, each line exactly as the export wrote it, and covered by a verbatim manifest node (kind: import, fhir: {export: sha256:…, patient: …}, generated.how: imported). Other patients' data never enters the workspace. A Bundle is kept whole.
  2. Every imported document carries fhir: {resourceType, id}, derived_from: the import node, generated: {by: process:prov-fhir, at, how: imported} and an audience.
  3. Idempotent. Documents are found again by fhir (medicines by fhir.medication), and an import node by its digest. Running the same export twice writes nothing (tested by comparing every byte). A later export rewrites only what the importer owns (kind, date_of_service, status, clinician, codes, other_codes, fhir, and the body), points derived_from at the new import, and leaves generated alone. audience belongs to the patient: the importer moves it only when it set it and the patient hasn't changed it since (for example, a medicine that stops drops out of emergency).
  4. Closed vocabularies stay closed. A coding's system URI maps to a codes[].system term by the term's uri:, read from the workspace's own vocabulary file, so adding a system there is enough. A coding from any other system is kept under other_codes with its URI and counted in the summary. Statuses map through the vocabularies' fhir: hints, plus a small fallback table (recurrence/relapse → active, remission → inactive, completed/cancelled/not-taken → stopped). draft, intended and unknown medication statuses get no status at all, rather than a guess.

Mapping

FHIR Becomes Where Notes
Patient the root's title, birth_date, gender, deceased, fhir root retitled on the first import only
Practitioner, Organization one document each that something names Care team roles (PractitionerRole) give specialty and clinic; no kind (none fits)
Encounter kind: encounter, date_of_service = period.start's date, codes from type, clinician = primary performer Encounters, filed by year through filing.encounters body: class, time, clinician, clinic, reason (linked to the Condition when it's a reasonReference), a table of the observations taken, and links to what it produced
Condition kind: problem, status from clinicalStatus Problems refuted / entered-in-error are skipped
MedicationRequest, MedicationStatement kind: medication, one per medicine, status from the most recent Medications each prescription is a table row (date, status, prescriber, encounter, reason, dosage; newest 50)
AllergyIntolerance kind: allergy Allergies (new index) clinical status kept in fhir.clinicalStatus, since status isn't declared outside Problems/Medications
Immunization kind: immunization, codes cvx Immunizations (new index) not-done is skipped
Procedure kind: procedure Procedures (new index)
DiagnosticReport (laboratory) kind: lab, one per report, codes = the panel's and every result's LOINC Labs, filed by year through filing.labs results as a table (value, unit, range and flag when present)
Observation (laboratory, in no report) kind: lab, one each Labs, by year
Observation (vital signs, survey, social history, exam…) a row in its encounter's table none survey panels get one row per answer; blood pressure reads 113/84 mm[Hg]

Decisions that differ from the brief, or that the data forced

  • Indexes are folder notes (encounters/index.md, labs/index.md, …). The preset's own comment says prov new Encounters --in <root>, which writes /encounters.md. Both filing entries then make their year index at /<year>/index.md, so the second record filed into a year that already holds the other kind is refused (route segment "2024" would create 2024/index.md, but that file is already a node). I reproduced this with the CLI on Add a medical preset; fix presets declaring path-named fields #27's branch. Folder notes avoid it. The comment in presets/medical/prov.yaml should probably say --as encounters/index.md (or similar); I left it alone here, since it's Add a medical preset; fix presets declaring path-named fields #27's file.
  • Labs are one document per report, not one per Observation. Synthea's results all belong to panels: one patient's 5,469 lab observations become 621 documents, and each one reads like a lab report.
  • Medicines are one document per drug, not one per MedicationRequest. Synthea re-issues chronic prescriptions: one patient has 1,525 requests for 17 medicines. A medication list is a list of medicines, and current-medications only works if each one appears once.
  • emergency audience. Without it, prov exports emergency is empty. The importer puts active allergies, active/on-hold medicines, and active problems that SNOMED CT tags (disorder) in emergency. Synthea records employment, education, stress and "medication review due" as active (finding)s and (situation)s, which don't belong in a stranger's summary. --no-emergency leaves everything at self.
  • Clinical-note DiagnosticReports (Synthea's History & Physical notes, base64 in presentedForm) are not imported. They are counted and kept in the import.

Left out on purpose

These are kept in the import only, and the summary lists them by type: Claim, ExplanationOfBenefit, CarePlan, CareTeam, Device, SupplyDelivery, MedicationAdministration, Provenance, DocumentReference / clinical notes, and ImagingStudy (kind: imaging exists, but Synthea's studies are thin). Locations become an address on the clinic, not documents. Names are kept exactly as Synthea writes them, digit suffixes included (Sharyn437).

Testing

  • Fixture: prov-fhir/tests/fixtures/synthea, 423 KB: two patients from the Synthea run below, cut down to the mapped types and the shared resources they reference. Synthea data is entirely synthetic.
  • Unit tests cover titles (SNOMED semantic tags), dates, values and units, Markdown escaping, the emergency rule, vocabulary URIs and status fallbacks.
  • tests/mapping.rs: the patient slice (only their lines, byte for byte), kinds, code-system mapping, medicine grouping, problem statuses, vitals as rows, the emergency list, and a Bundle with an unknown code system and an entered-in-error condition.
  • tests/import.rs: import into a scratch workspace, then check (the library's, plus check_about, as prov check runs it) with no findings; re-import is byte-identical; a changed export updates in place (status, derived_from, audience follows only where the importer set it); a second patient is refused; a workspace without the preset is refused; a Bundle imports and is kept whole.
  • ship ci: all 7 jobs pass locally (fmt, clippy -D warnings, cargo test --workspace, guide transcripts, prov check of the repo, package isolation, MSRV 1.88). prov-fhir is publish = false, so per xtask's rule it isn't on the isolation list.

Synthea, 25-person Texas run, release build:

patient resources documents made import re-import prov check
46976cf7 597 183 0.2 s 0.0 s, all unchanged clean
c053e996 (2 allergies) 520 197 0.2 s 0.0 s clean
e74367bd 2,372 654 0.8 s 0.2 s clean
922fb35e 2,546 653 0.7 s 0.1 s clean
5e688e99 14,923 2,207 3.4 s 0.6 s clean (0.33 s)

--each over the whole export (26 patients) took 16 s, made 10,855 documents, and every one of the 26 workspaces checks clean. No coding in Synthea's output falls outside the six vocabulary systems. prov views timeline, active-problems, current-medications and prov exports emergency give sensible results; for example, e74367bd's emergency export is 5 current medicines and 9 active disorders (diabetes, breast cancer, ischemic heart disease…), with 678 documents held back.

Reproduce

gh release download master-branch-latest --repo synthetichealth/synthea -p synthea-with-dependencies.jar
nix shell nixpkgs#jdk17_headless -c java -jar synthea-with-dependencies.jar -s 42 -cs 42 -p 25 Texas \
  --exporter.fhir.bulk_data=true --exporter.baseDirectory=/tmp/synthea-tx
cargo run --release -p prov-fhir -- patients /tmp/synthea-tx/fhir
cargo run --release -p prov-fhir -- import /tmp/synthea-tx/fhir --into /tmp/rec --patient e74367bd
cd /tmp/rec && prov check && prov views active-problems && prov exports emergency

An imported encounter

encounters/2021/2021-06-12-emergency-room-admission.md, from patient e74367bd:

---
title: Emergency room admission
part_of: '[2021](/encounters/2021/index.md)'
kind: encounter
date_of_service: 2021-06-12
clinician: '[Dr. Sofia418 Zepeda235](/care-team/dr-sofia418-zepeda235.md)'
codes:
- system: snomed-ct
  code: '50849002'
  display: Emergency room admission (procedure)
fhir:
  resourceType: Encounter
  id: e74367bd-7385-97dc-cfca-35c5651bfd1e
  class: EMER
audience: self
derived_from: '[FHIR export, 2026-11-18](/imports/2026-11-18-ba92ec47.yaml)'
generated:
  by: process:prov-fhir
  at: 2026-11-18T23:44:57.526729Z
  how: imported
---

# Emergency room admission

Emergency visit, 2021-06-12 22:13–23:33 with [Dr. Sofia418 Zepeda235](/care-team/dr-sofia418-zepeda235.md) at [HEALING EXPERTS HOME HEALTH LLC](/care-team/healing-experts-home-health-llc.md).

Reason: Fracture of bone.

## Observations

| Observation | Value |
| --- | --- |
| DXA Radius and Ulna \[T-score\] Bone density | 0.1701 |

## Problems

- [Fracture of bone](/problems/2021-06-12-fracture-of-bone.md) — resolved, onset 2021-06-12
- [Fracture of clavicle](/problems/2021-06-12-fracture-of-clavicle.md) — resolved, onset 2021-06-12

## Medications

- [Meperidine Hydrochloride 50 MG Oral Tablet](/medications/meperidine-hydrochloride-50-mg-oral-tablet.md) — prescribed (completed)
- [Acetaminophen 325 MG Oral Tablet](/medications/acetaminophen-325-mg-oral-tablet.md) — prescribed (completed)

## Procedures

- [Plain X-ray of clavicle](/procedures/2021-06-12-plain-x-ray-of-clavicle.md)
- [Admission to orthopedic department](/procedures/2021-06-12-admission-to-orthopedic-department.md)
- [Bone density scan](/procedures/2021-06-12-bone-density-scan.md)

…al workspace

`prov-fhir import <export> --into <root> [--patient <id>]` (or `--each <dir>`,
one workspace per patient) reads a Bulk Data ndjson export or a Bundle, cuts
out one patient's share, keeps it whole under Imports as a verbatim manifest
(`kind: import`), and writes the record as documents through the prov
library: encounters and lab reports filed by year, problems, medicines (one
per drug, its prescriptions a table), allergies, immunizations, procedures,
and the care team. Every document carries `fhir: {resourceType, id}`,
`derived_from` the import and `generated.how: imported`; vital signs and
surveys become rows of their encounter's table rather than documents.

Codings map to the workspace's `codes[].system` terms by their `uri:`; a
coding from any other system is kept in `other_codes` and counted. Statuses
map through the vocabularies' `fhir:` hints. Re-running an import finds every
document by its `fhir` id and rewrites only what changed, so the same export
twice writes nothing.

Tested on a seeded two-patient Synthea fixture (tests/fixtures/synthea).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant