Repository navigation
prov-fhir: import FHIR (Bulk ndjson) into a medical workspace - #28
Open
adammharris wants to merge 1 commit into
Open
adammharris wants to merge 1 commit into
adammharris wants to merge 1 commit into
Conversation
…al workspace
`prov-fhir import <export> --into <root> [--patient <id>]` (or `--each <dir>`,
one workspace per patient) reads a Bulk Data ndjson export or a Bundle, cuts
out one patient's share, keeps it whole under Imports as a verbatim manifest
(`kind: import`), and writes the record as documents through the prov
library: encounters and lab reports filed by year, problems, medicines (one
per drug, its prescriptions a table), allergies, immunizations, procedures,
and the care team. Every document carries `fhir: {resourceType, id}`,
`derived_from` the import and `generated.how: imported`; vital signs and
surveys become rows of their encounter's table rather than documents.
Codings map to the workspace's `codes[].system` terms by their `uri:`; a
coding from any other system is kept in `other_codes` and counted. Statuses
map through the vocabularies' `fhir:` hints. Re-running an import finds every
document by its `fhir` id and rewrites only what changed, so the same export
twice writes nothing.
Tested on a seeded two-patient Synthea fixture (tests/fixtures/synthea).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #27 (the
medicalpreset). Addsprov-fhir, a new workspace crate (publish = false) that imports a FHIR export into a workspace kept with that preset, through the prov library: every write is aWorkspacecall (create_with_fields,file,attach_manifest_claiming,save_document,retitle), with no shelling out to the CLI.<export>is a Bulk Data directory of*.ndjson, one ndjson file, or a JSON Bundle. One workspace holds one patient's record, so--intotakes one patient (--patientmay be a unique id prefix, and can be left out when the export holds only one). A--intoroot that is missing or empty is made the way--eachmakes one:prov init+ the medical preset (built into the binary withinclude_str!;--preset DIRswaps it), then the indexes. A workspace that already holds a different patient is refused.What it does
imports/<date>-<digest>/, each line exactly as the export wrote it, and covered by a verbatim manifest node (kind: import,fhir: {export: sha256:…, patient: …},generated.how: imported). Other patients' data never enters the workspace. A Bundle is kept whole.fhir: {resourceType, id},derived_from:the import node,generated: {by: process:prov-fhir, at, how: imported}and anaudience.fhir(medicines byfhir.medication), and an import node by its digest. Running the same export twice writes nothing (tested by comparing every byte). A later export rewrites only what the importer owns (kind,date_of_service,status,clinician,codes,other_codes,fhir, and the body), pointsderived_fromat the new import, and leavesgeneratedalone.audiencebelongs to the patient: the importer moves it only when it set it and the patient hasn't changed it since (for example, a medicine that stops drops out ofemergency).systemURI maps to acodes[].systemterm by the term'suri:, read from the workspace's own vocabulary file, so adding a system there is enough. A coding from any other system is kept underother_codeswith its URI and counted in the summary. Statuses map through the vocabularies'fhir:hints, plus a small fallback table (recurrence/relapse→ active,remission→ inactive,completed/cancelled/not-taken→ stopped).draft,intendedandunknownmedication statuses get nostatusat all, rather than a guess.Mapping
title,birth_date,gender,deceased,fhirkind(none fits)kind: encounter,date_of_service=period.start's date,codesfromtype,clinician= primary performerfiling.encountersreasonReference), a table of the observations taken, and links to what it producedkind: problem,statusfromclinicalStatusrefuted/entered-in-errorare skippedkind: medication, one per medicine,statusfrom the most recentkind: allergyfhir.clinicalStatus, sincestatusisn't declared outside Problems/Medicationskind: immunization,codescvxnot-doneis skippedkind: procedurekind: lab, one per report,codes= the panel's and every result's LOINCfiling.labskind: lab, one each113/84 mm[Hg]Decisions that differ from the brief, or that the data forced
encounters/index.md,labs/index.md, …). The preset's own comment saysprov new Encounters --in <root>, which writes/encounters.md. Both filing entries then make their year index at/<year>/index.md, so the second record filed into a year that already holds the other kind is refused (route segment "2024" would create 2024/index.md, but that file is already a node). I reproduced this with the CLI on Add a medical preset; fix presets declaring path-named fields #27's branch. Folder notes avoid it. The comment inpresets/medical/prov.yamlshould probably say--as encounters/index.md(or similar); I left it alone here, since it's Add a medical preset; fix presets declaring path-named fields #27's file.current-medicationsonly works if each one appears once.emergencyaudience. Without it,prov exports emergencyis empty. The importer puts active allergies,active/on-holdmedicines, and active problems that SNOMED CT tags(disorder)inemergency. Synthea records employment, education, stress and "medication review due" as active(finding)s and(situation)s, which don't belong in a stranger's summary.--no-emergencyleaves everything atself.presentedForm) are not imported. They are counted and kept in the import.Left out on purpose
These are kept in the import only, and the summary lists them by type: Claim, ExplanationOfBenefit, CarePlan, CareTeam, Device, SupplyDelivery, MedicationAdministration, Provenance, DocumentReference / clinical notes, and ImagingStudy (
kind: imagingexists, but Synthea's studies are thin). Locations become an address on the clinic, not documents. Names are kept exactly as Synthea writes them, digit suffixes included (Sharyn437).Testing
prov-fhir/tests/fixtures/synthea, 423 KB: two patients from the Synthea run below, cut down to the mapped types and the shared resources they reference. Synthea data is entirely synthetic.tests/mapping.rs: the patient slice (only their lines, byte for byte), kinds, code-system mapping, medicine grouping, problem statuses, vitals as rows, the emergency list, and a Bundle with an unknown code system and an entered-in-error condition.tests/import.rs: import into a scratch workspace, thencheck(the library's, pluscheck_about, asprov checkruns it) with no findings; re-import is byte-identical; a changed export updates in place (status,derived_from, audience follows only where the importer set it); a second patient is refused; a workspace without the preset is refused; a Bundle imports and is kept whole.ship ci: all 7 jobs pass locally (fmt, clippy-D warnings,cargo test --workspace, guide transcripts,prov checkof the repo, package isolation, MSRV 1.88).prov-fhirispublish = false, so per xtask's rule it isn't on the isolation list.Synthea, 25-person Texas run, release build:
prov check--eachover the whole export (26 patients) took 16 s, made 10,855 documents, and every one of the 26 workspaces checks clean. No coding in Synthea's output falls outside the six vocabulary systems.prov views timeline,active-problems,current-medicationsandprov exports emergencygive sensible results; for example, e74367bd's emergency export is 5 current medicines and 9 active disorders (diabetes, breast cancer, ischemic heart disease…), with 678 documents held back.Reproduce
An imported encounter
encounters/2021/2021-06-12-emergency-room-admission.md, from patient e74367bd: