Template to transform US Birth data to OMOP - #43
Open
csafreen wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a flow template that transforms 2022 U.S. birth data into OMOP CDM tables.
Changes:
- Parses fixed-width natality data into staged CSV chunks.
- Maps records into OMOP clinical tables.
- Adds staging cleanup and provider loading nodes.
Suppressed comments (1)
flows/ETL_US_BirthData_to_OMOP.json:90
- This placeholder does not explain that the CSV supplies the concept mapping consumed by
transform_facts, so the node does not satisfy the checklist requirement for a helpful short description.
"description": "Describe the task of node csv_node_0"
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,176 @@ | |||
| { | |||
| "id": "59745eff-9bb9-486a-ab64-04433cb586d5", | |||
| "name": "ETL US BirthData to OMOP", | |||
| "data": { | ||
| "name": "transform_facts", | ||
| "description": "Gold: build PERSON, VISIT_OCCURRENCE, OBSERVATION, MEASUREMENT, CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE, and PAYER_PLAN_PERIOD in a single per-record pass (merged from 7 originally separate nodes; see node_backups/ for the originals).", | ||
| "python_code": "# Deployment-topology constants (deliberately NOT pipeline variables) -- see\n# node-load-bronze for the full rationale: fixed by the platform's container\n# mounts; the two paths are the flow container's and trex's views of the same\n# shared staging directory on the trex data volume.\nNAT2022_DATA_DIR = \"/app/data_load\"\nSTAGING_DIR_FLOW = \"/app/duckdb_data/flow_staging\"\nSTAGING_DIR_TREX = \"/usr/src/data/flow_staging\"\n\nMAPPING_CSV_NODE = \"csv_node_0\"\n\n_PREFIX_GROUP_RE = re.compile(r'^(all )?variables beginning with\\s+(\\w+?)_?$', re.I)\n\n# Resolved relative to this file's own directory, not the caller's cwd --\n# matters both for local runs and once this module is volume-mounted into a\n# flow-run container alongside the csv (see flowNodes/ directory).\n_TABLE_NORMALIZE = {\n \"PPROCEDURE\": \"PROCEDURE\",\n}\n\n_FIELD_NAME_NORMALIZE = {\n \"observation_date\": \"observation_concept_id\", # mislabeled in spreadsheet; see module docstring\n \"unit_as_concept_id\": \"unit_concept_id\", # spreadsheet typo -- no \"_as_\" in the real OMOP column name\n}\n\n# (person_role, source_field, raw Values string) -> corrected Values string.\n# precare's Child row has Values=\"1\" -- a single literal value -- even though\n# its own NBER_Concept_Name says \"Month Prenatal Care Began 01 - 10\" and\n# UserGuide2022.pdf documents PRECARE's real range as 01-10. fagecomb's Child\n# row has Values=\"9\" even though its own NBER_Concept_Name says \"09-98\" and\n# UserGuide2022.pdf documents FAGECOMB's real range as 09-98. Applied\n# narrowly (exact role+field+raw-value match) so it can't accidentally\n# affect any other row. Interpretation flagged for spreadsheet-owner\n# confirmation in questions_for_spreadsheet_owner.md -- not a settled fact,\n# best current guess.\n_VALUES_NORMALIZE = {\n (\"Child\", \"precare\", \"1\"): \"1-10\",\n (\"Child\", \"fagecomb\", \"9\"): \"9-98\",\n # previs's Values=\"12\" only matched a raw visit count of exactly 12, leaving\n # value_as_concept_id null for every other real count. UserGuide2022.pdf documents\n # PREVIS's valid range as 00-98, and previs=0 already has its own dedicated row\n # (\"No prenatal care\"), so widened to 1-98 -- matching the Mother-role sibling row's\n # own already-correct \"1 to 98\" range for this same concept.\n (\"Child\", \"previs\", \"12\"): \"1-98\",\n # Mother's precare row (observation_date -> observation_concept_id 40771565) only\n # matched literal \"2\", even though its own Notes describe a general subtract-months\n # formula (see _observation_date_for) meant to apply to any value in precare's real\n # 1-10 range. 0 already has its own dedicated \"No prenatal care\" row.\n (\"Mother\", \"precare\", \"2\"): \"1-10\",\n}\n\n_REAL_CDM_COLUMNS = {\n \"PERSON\": {\"person_id\", \"gender_concept_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\",\n \"birth_datetime\", \"race_concept_id\", \"ethnicity_concept_id\", \"location_id\", \"provider_id\",\n \"care_site_id\", \"person_source_value\", \"gender_source_value\", \"gender_source_concept_id\",\n \"race_source_value\", \"race_source_concept_id\", \"ethnicity_source_value\",\n \"ethnicity_source_concept_id\"},\n \"VISIT_OCCURRENCE\": {\"visit_occurrence_id\", \"person_id\", \"visit_concept_id\", \"visit_start_date\",\n \"visit_start_datetime\", \"visit_end_date\", \"visit_end_datetime\",\n \"visit_type_concept_id\", \"provider_id\", \"care_site_id\", \"visit_source_value\",\n \"visit_source_concept_id\", \"admitted_from_concept_id\", \"admitted_from_source_value\",\n \"discharged_to_concept_id\", \"discharged_to_source_value\",\n \"preceding_visit_occurrence_id\"},\n \"OBSERVATION\": {\"observation_id\", \"person_id\", \"observation_concept_id\", \"observation_date\",\n \"observation_datetime\", \"observation_type_concept_id\", \"value_as_number\",\n \"value_as_string\", \"value_as_concept_id\", \"qualifier_concept_id\", \"unit_concept_id\",\n \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\", \"observation_source_value\",\n \"observation_source_concept_id\", \"unit_source_value\", \"qualifier_source_value\",\n \"value_source_value\", \"observation_event_id\", \"obs_event_field_concept_id\"},\n \"MEASUREMENT\": {\"measurement_id\", \"person_id\", \"measurement_concept_id\", \"measurement_date\",\n \"measurement_datetime\", \"measurement_time\", \"measurement_type_concept_id\",\n \"operator_concept_id\", \"value_as_number\", \"value_as_concept_id\", \"unit_concept_id\",\n \"range_low\", \"range_high\", \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\",\n \"measurement_source_value\", \"measurement_source_concept_id\", \"unit_source_value\",\n \"unit_source_concept_id\", \"value_source_value\", \"measurement_event_id\",\n \"meas_event_field_concept_id\"},\n \"CONDITION\": {\"condition_occurrence_id\", \"person_id\", \"condition_concept_id\", \"condition_start_date\",\n \"condition_start_datetime\", \"condition_end_date\", \"condition_end_datetime\",\n \"condition_type_concept_id\", \"condition_status_concept_id\", \"stop_reason\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"condition_source_value\",\n \"condition_source_concept_id\", \"condition_status_source_value\"},\n \"PROCEDURE\": {\"procedure_occurrence_id\", \"person_id\", \"procedure_concept_id\", \"procedure_date\",\n \"procedure_datetime\", \"procedure_end_date\", \"procedure_end_datetime\",\n \"procedure_type_concept_id\", \"modifier_concept_id\", \"quantity\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"procedure_source_value\",\n \"procedure_source_concept_id\", \"modifier_source_value\"},\n \"PAYER_PLAN_PERIOD\": {\"payer_plan_period_id\", \"person_id\", \"payer_plan_period_start_date\",\n \"payer_plan_period_end_date\", \"payer_concept_id\", \"payer_source_value\",\n \"payer_source_concept_id\", \"plan_concept_id\", \"plan_source_value\",\n \"plan_source_concept_id\", \"sponsor_concept_id\", \"sponsor_source_value\",\n \"sponsor_source_concept_id\", \"family_source_value\", \"stop_reason_concept_id\",\n \"stop_reason_source_value\", \"stop_reason_source_concept_id\"},\n \"PROVIDER\": {\"provider_id\", \"provider_name\", \"npi\", \"dea\", \"specialty_concept_id\", \"care_site_id\",\n \"year_of_birth\", \"gender_concept_id\", \"provider_source_value\", \"specialty_source_value\",\n \"specialty_source_concept_id\", \"gender_source_value\", \"gender_source_concept_id\"},\n}\n\n# (Target_Table, Field_Name) pairs already confirmed as spreadsheet-side issues --\n# tracked in questions_for_spreadsheet_owner.md with an existing code-level\n# workaround. Logged as a warning but doesn't halt the pipeline. Anything NOT\n# in this set is treated as a new, unreviewed problem and raises immediately.\n_KNOWN_MAPPING_ISSUES = {\n (\"MEASUREMENT\", \"qualifier_concept_id\"), # questions doc #10 -- dropped from MEASUREMENT's own TABLE_COLUMNS\n (\"PROVIDER\", \"provider_concept_id\"), # questions doc #14 -- PROVIDER_DIMENSION hardcodes specialty_concept_id instead\n}\n\n\ndef _validate_mapping_against_cdm(rows: \"tuple[MappingRow, ...]\", logger) -> None:\n \"\"\"\n Checks every spreadsheet row's (Target_Table, Field_Name) against the real\n OMOP CDM 5.4 column list for that table, so a spreadsheet edit (typo,\n mislabeled column, wrong table name) is caught once, up front, before it\n silently reaches 3M+ records -- rather than requiring the same manual\n cross-referencing against the DDL that found the issues already tracked\n in questions_for_spreadsheet_owner.md.\n \"\"\"\n unexpected = []\n for row in rows:\n valid_columns = _REAL_CDM_COLUMNS.get(row.target_table)\n key = (row.target_table, row.field_name)\n if valid_columns is None:\n if key not in _KNOWN_MAPPING_ISSUES:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: unrecognized Target_Table {row.target_table!r}\"\n )\n continue\n if row.field_name.lower() not in valid_columns:\n if key in _KNOWN_MAPPING_ISSUES:\n logger.warning(\n \"Known spreadsheet issue (see questions_for_spreadsheet_owner.md): \"\n f\"{row.source_field}/{row.person_role} sets Field_Name={row.field_name!r} on \"\n f\"{row.target_table}, which isn't a real column there.\"\n )\n else:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: Field_Name {row.field_name!r} is not a real \"\n f\"column on {row.target_table} (spreadsheet concept: {row.concept_name!r})\"\n )\n if unexpected:\n raise ValueError(\n \"Mapping spreadsheet has unrecognized Target_Table/Field_Name combinations not seen before \"\n \"-- check for a new spreadsheet edit or typo before proceeding:\\n\"\n + \"\\n\".join(f\" - {e}\" for e in unexpected)\n )\n\n\n\n@dataclass(frozen=True)\nclass MappingRow:\n source_field: str # lowercase NBER field name, e.g. \"rf_pdiab\"\n person_role: str # \"Child\", \"Mother\", or \"Father\"\n target_table: str # normalized OMOP table name, e.g. \"OBSERVATION\"\n field_name: str # OMOP column this row supplies, e.g. \"value_as_concept_id\"\n concept_id: int\n concept_name: str\n matcher: \"ValueMatcher\"\n\n\ndef _is_number(s: str) -> bool:\n \"\"\"True for anything float() accepts, e.g. \"9\", \"-5\", \"69.9\" -- used\n instead of str.isdigit() so decimal Values (e.g. bmi's \"13-69.9\") parse\n as ranges/exact values instead of falling through to literal-string\n matching, which no real decimal value would ever hit.\"\"\"\n try:\n float(s)\n return True\n except ValueError:\n return False\n\n\nclass ValueMatcher:\n \"\"\"\n Parses the spreadsheet's free-text Values column into something\n callable against an actual field value from the data.\n\n Recognized forms: blank/\"any\" (always matches -- used for field-level\n \"header\" rows that apply regardless of the specific code), exact\n tokens separated by \"or\"/\",\", and numeric ranges (\"0-30\", \"1 to 98\",\n \"13-69.9\" -- bounds/values may be decimal, e.g. bmi).\n \"\"\"\n\n def __init__(self, raw: str):\n raw = (raw or \"\").strip()\n self.raw = raw\n self.kind: str\n if raw == \"\" or raw.lower() == \"any\":\n self.kind = \"always\"\n return\n if raw.startswith(\">\") or raw.startswith(\"<\"):\n bound = raw[1:].strip()\n if _is_number(bound):\n self.kind = \"inequality\"\n self.op = raw[0]\n self.bound = float(bound)\n return\n low = raw.lower().replace(\" to \", \"-\")\n parts = [p.strip() for p in low.split(\"-\")]\n if len(parts) == 2 and all(_is_number(p) for p in parts):\n self.kind = \"range\"\n self.lo = float(parts[0])\n self.hi = float(parts[1])\n return\n self.kind = \"exact\"\n self.tokens = {t.strip().lower() for t in raw.replace(\",\", \" or \").split(\" or \")}\n\n def matches(self, value) -> bool:\n if value is None:\n return False\n value_s = str(value).strip()\n if value_s == \"\":\n return False\n if self.kind == \"always\":\n return True\n if self.kind == \"range\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return self.lo <= v <= self.hi\n if self.kind == \"inequality\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return v > self.bound if self.op == \">\" else v < self.bound\n if value_s.lower() in self.tokens:\n return True\n # Raw fields are read with dtype=str (pd.read_fwf), so numeric codes\n # keep their fixed-width zero-padding (e.g. \"09\") -- compare\n # numerically too so an exact token like \"9\" still matches \"09\",\n # and a decimal token like \"99.9\" still matches a decimal value.\n if _is_number(value_s):\n return any(_is_number(t) and float(t) == float(value_s) for t in self.tokens)\n return False\n\n\ndef _normalize_table(raw: str) -> str:\n t = raw.strip().upper().replace(\" \", \"_\")\n return _TABLE_NORMALIZE.get(t, t)\n\n\n_MAPPING_ROWS: tuple[MappingRow, ...] | None = None\n_PREFIX_GROUP_DEFAULTS: dict[tuple[str, str], int] | None = None\n_FIELD_INDEX: dict[tuple[str, str], list[MappingRow]] | None = None\n_MAPPING_DF_FROM_NODE: \"pd.DataFrame | None\" = None\n\n\ndef _load_mapping_df() -> \"pd.DataFrame\":\n if _MAPPING_DF_FROM_NODE is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n return _MAPPING_DF_FROM_NODE.copy()\n\n\ndef load_mapping_rows() -> tuple[MappingRow, ...]:\n global _MAPPING_ROWS\n if _MAPPING_ROWS is not None:\n return _MAPPING_ROWS\n\n out = []\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n nber_field, values, _nber_name, field_name, target_table, vocab_id, concept_name = r[:7]\n person_role = (r[11] or \"\").strip()\n if not person_role or not target_table or not field_name:\n continue\n vocab_id_s = str(vocab_id).strip() if vocab_id is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue # computed/imputed row, handled separately in omop_transform.py\n if _PREFIX_GROUP_RE.match(str(nber_field).strip()):\n continue # prefix-group default row, handled by _prefix_group_defaults below\n field_name_s = _FIELD_NAME_NORMALIZE.get(str(field_name).strip(), str(field_name).strip())\n source_field_s = str(nber_field).strip().lower()\n values_s = \"\" if values is None else str(values)\n values_s = _VALUES_NORMALIZE.get((person_role, source_field_s, values_s.strip()), values_s)\n out.append(MappingRow(\n source_field=source_field_s,\n person_role=person_role,\n target_table=_normalize_table(str(target_table)),\n field_name=field_name_s,\n concept_id=int(vocab_id_s),\n concept_name=str(concept_name).strip() if concept_name else \"\",\n matcher=ValueMatcher(values_s),\n ))\n _MAPPING_ROWS = tuple(out)\n return _MAPPING_ROWS\n\n\ndef _prefix_group_defaults() -> dict[tuple[str, str], int]:\n \"\"\"(person_role, prefix) -> default observation_concept_id for that field family.\"\"\"\n global _PREFIX_GROUP_DEFAULTS\n if _PREFIX_GROUP_DEFAULTS is not None:\n return _PREFIX_GROUP_DEFAULTS\n\n out = {}\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n m = _PREFIX_GROUP_RE.match(str(r[0]).strip())\n if not m:\n continue\n vocab_id_s = str(r[5]).strip() if r[5] is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue\n person_role = (r[11] or \"\").strip()\n prefix = m.group(2).lower()\n out[(person_role, prefix)] = int(vocab_id_s)\n _PREFIX_GROUP_DEFAULTS = out\n return _PREFIX_GROUP_DEFAULTS\n\n\ndef _index_by_field() -> dict[tuple[str, str], list[MappingRow]]:\n global _FIELD_INDEX\n if _FIELD_INDEX is not None:\n return _FIELD_INDEX\n\n index: dict[tuple[str, str], list[MappingRow]] = {}\n for row in load_mapping_rows():\n index.setdefault((row.person_role, row.source_field), []).append(row)\n _FIELD_INDEX = index\n return _FIELD_INDEX\n\n\n# (source_field, field_name, concept_id) rows that should be treated as an \"Integer\n# example\" placeholder (substitute the real value in) even though their\n# Target_Concept_Name isn't literally \"Integer example\". apgar5's value_as_number row\n# has a real concept id (3004221, \"5 minute Apgar Score\") in that slot instead of the\n# placeholder pattern used elsewhere (oegest_comb/dbwt/combgest), so without this it\n# stores the literal concept id 3004221 instead of the actual apgar score. Flagged in\n# questions_for_spreadsheet_owner.md -- not a settled fact, best current guess.\n_INTEGER_PLACEHOLDER_OVERRIDE = {\n (\"apgar5\", \"value_as_number\", 3004221),\n (\"apgar10\", \"value_as_number\", 3016162),\n}\n\n# no_mmorb (\"no maternal morbidity reported\") answers the same underlying survey\n# question as the mm_* fields (its own answer concept is literally \"None of the\n# above (maternal morbidity)\"), and would pick up the \"Variables beginning with\n# mm_\" header row's observation_concept_id (43533805) via the normal prefix-group\n# match below -- except its own field name starts with \"no_\", not \"mm_\", so the\n# prefix never matches. Applied directly instead. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_OBSERVATION_CONCEPT_OVERRIDE = {\n (\"Mother\", \"no_mmorb\"): 43533805,\n # no_abnorm (\"no abnormal conditions of the newborn\") is the same underlying\n # survey question as the ab_* fields (its own answer concept is literally \"None\n # of the above\"), and would pick up the \"Variables beginning with ab_\" header\n # row's observation_concept_id (43533835) via the normal prefix-group match --\n # except its own field name starts with \"no_\", not \"ab_\", so the prefix never\n # matches. Applied directly instead, same shape as no_mmorb above. Not a settled\n # fact -- see questions_for_spreadsheet_owner.md.\n (\"Child\", \"no_abnorm\"): 43533835,\n # no_congen (\"no congenital anomalies of the newborn checked\") is the same\n # underlying survey question as the ca_* fields, and would pick up the\n # \"Variables beginning with ca_\" header row's observation_concept_id\n # (43533804) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ca_\". Same shape as no_mmorb/no_abnorm above.\n (\"Child\", \"no_congen\"): 43533804,\n # no_lbrdlv (\"no characteristics of labor and delivery reported\") is the\n # same underlying survey question as the ld_* fields, and would pick up the\n # \"Variables beginning with LD_\" header row's observation_concept_id\n # (43533836) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ld_\". Same shape as no_mmorb/no_abnorm/no_congen\n # above.\n (\"Mother\", \"no_lbrdlv\"): 43533836,\n # wtgain's own \"99 = Unknown or not stated\" row supplies only a\n # qualifier_concept_id (4129922, \"unknown\"), unlike its real-value row\n # (00-97), which supplies observation_concept_id (40759199) directly.\n # Applied directly so the unknown case still identifies itself as a\n # pregnancy weight-gain observation, same concept as the real-value case.\n (\"Mother\", \"wtgain\"): 40759199,\n}\n\n# (person_role, source_field, field_name, concept_id from spreadsheet) ->\n# replacement concept_id. fagecomb's Child-role \"99 = Unknown or Not Stated\"\n# row supplies observation_concept_id=0 (\"No matching concept\") -- unlike\n# every other 0-row in the spreadsheet (bfacil/fhispx/mracehisp/dmar/attend\n# etc.), all of which land in an *answer* column (value_as_concept_id,\n# ethnicity_concept_id, provider_concept_id) where 0 legitimately means \"the\n# answer is Other/Unknown,\" this one lands in the *identifying*\n# observation_concept_id column, so it doesn't say what the observation is\n# about at all. Remapped to 4071357 (\"Paternal Age\"), the same concept\n# fagecomb's own Child-role real-value row already uses -- matching how\n# fagecomb's Father-role row already reuses one concept (4265453) for both\n# its real-value and its own 99-unknown case. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_CONCEPT_ID_REMAP = {\n (\"Child\", \"fagecomb\", \"observation_concept_id\", 0): 4071357,\n # itran (\"Infant Transferred\") and ilive (\"Infant Living at Time of Report\") both\n # mapped to observation_concept_id 43533781 in the spreadsheet (questions_for_\n # spreadsheet_owner.md #7) -- confirmed by the spreadsheet owner as genuinely\n # different concepts (an ATHENA hierarchy display issue caused the mix-up).\n # itran's corrected code is 43533782 (\"Infant was transferred within 24 hr of\n # delivery\"); ilive's correct code is still pending confirmation, so ilive is\n # left unchanged at 43533781 for now.\n (\"Child\", \"itran\", \"observation_concept_id\", 43533781): 43533782,\n}\n\n# (source_field, sentinel value) pairs where the raw value is NCHS's own\n# \"unknown or not stated\" placeholder rather than a real measurement --\n# confirmed via each field's own Values=99 row (wtgain: \"Unknown or not\n# stated\"; fagecomb: \"Unknown or Not Stated\" / \"Age unknown\"; previs:\n# \"Unknown or Not Stated\"). Excluded from the value_as_number backfill below\n# so \"unknown\" doesn't get recorded as a literal, implausible fact (e.g.\n# wtgain=99 read as \"gained 99 lbs\", fagecomb=99 read as \"father is 99\").\n# previs=99 found via the same check while fixing wtgain -- not itself\n# reported, flagged here for visibility.\n_UNKNOWN_VALUE_SENTINEL = {\n (\"fagecomb\", 99),\n (\"wtgain\", 99),\n (\"previs\", 99),\n}\n\n\ndef get_records(person_role: str, source_field: str, value):\n \"\"\"\n Returns {target_table: {field_name: concept_id, ...}, ...} for every\n target table that has at least one matching mapping row for this\n (person_role, source_field, value). Rows with no value match (e.g. the\n field is blank/not reported) contribute nothing.\n \"\"\"\n source_field = source_field.lower()\n candidates = _index_by_field().get((person_role, source_field), [])\n out: dict[str, dict[str, int]] = {}\n for row in candidates:\n if row.matcher.matches(value):\n concept_id = row.concept_id\n is_placeholder = (\n row.concept_name.strip().lower() == \"integer example\"\n or (row.source_field, row.field_name, row.concept_id) in _INTEGER_PLACEHOLDER_OVERRIDE\n )\n if is_placeholder:\n try:\n concept_id = int(str(value).strip())\n except ValueError:\n continue\n concept_id = _CONCEPT_ID_REMAP.get(\n (person_role, source_field, row.field_name, concept_id), concept_id\n )\n out.setdefault(row.target_table, {})[row.field_name] = concept_id\n\n if \"OBSERVATION\" in out and \"observation_concept_id\" not in out[\"OBSERVATION\"]:\n override = _OBSERVATION_CONCEPT_OVERRIDE.get((person_role, source_field))\n if override is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = override\n else:\n prefix = source_field.split(\"_\")[0]\n default = _prefix_group_defaults().get((person_role, prefix))\n if default is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = default\n\n return out\n\n\n\n\nDEFAULT_TYPE_CONCEPT_ID = 0 # TODO: confirm correct *_type_concept_id with vocabulary before use\n\nWEEKDAY_CODE_TO_PY = {1: 6, 2: 0, 3: 1, 4: 2, 5: 3, 6: 4, 7: 5} # 1=Sunday..7=Saturday -> Python Monday=0..Sunday=6\n\n\ndef _int_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return int(s)\n except ValueError:\n return None\n\n\ndef _float_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return float(s)\n except ValueError:\n return None\n\n\ndef _impute_child_dob(dob_yy, dob_mm, dob_wk):\n year, month, wk = _int_or_none(dob_yy), _int_or_none(dob_mm), _int_or_none(dob_wk)\n if year is None or month is None:\n return None, None, None\n if wk is None or wk not in WEEKDAY_CODE_TO_PY:\n return year, month, None\n target_py_weekday = WEEKDAY_CODE_TO_PY[wk]\n first_of_month = dt.date(year, month, 1)\n offset = (target_py_weekday - first_of_month.weekday()) % 7\n return year, month, 1 + offset\n\n\ndef _parse_dob_time(dob_tt) -> tuple[int, int] | None:\n \"\"\"DOB_TT is a raw 4-digit HHMM code (e.g. '0830'); '9999' means not stated.\"\"\"\n s = \"\" if dob_tt is None else str(dob_tt).strip()\n if len(s) != 4 or not s.isdigit() or s == \"9999\":\n return None\n hour, minute = int(s[:2]), int(s[2:])\n if not (0 <= hour <= 23 and 0 <= minute <= 59):\n return None\n return hour, minute\n\n\ndef dob_date_for(rec):\n year, month, day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n if year and month and day:\n return dt.date(year, month, day)\n return None\n\n\ndef _subtract_months(base_date, months):\n total = base_date.year * 12 + (base_date.month - 1) - months\n year, month = divmod(total, 12)\n day = min(base_date.day, 28)\n return dt.date(year, month + 1, day)\n\n\nclass IdGenerator:\n # Dense, chunk-safe id allocation: exactly 3 roles per record, so\n # record_idx*3 + offset + 1 produces 1,2,3,4,5,6,... with no gaps,\n # while still being computable independently per record (no shared\n # counter needed across chunks/workers).\n PERSON_ROLE_OFFSET = {\"Child\": 0, \"Mother\": 1, \"Father\": 2}\n\n @staticmethod\n def person_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def visit_occurrence_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def fact_id(record_idx, local_counter):\n # 500-id block per record: the destination CDM's fact id columns\n # (observation_id etc.) are 32-bit INTEGER (max 2,147,483,647) and the\n # full Nat2022 file has 3,676,029 records, so the block size must keep\n # the largest id under 2^31 (500 tops out at ~1.84e9) while exceeding\n # the most facts one record produces in a table (~60 observations).\n return (record_idx + 1) * 500 + local_counter\n\n\nPROVIDER_DIMENSION = [\n {\"provider_concept_id\": 38004446, \"provider_source_value\": \"Physician\"},\n {\"provider_concept_id\": 38003822, \"provider_source_value\": \"Osteopathic Practitioner\"},\n {\"provider_concept_id\": 38004482, \"provider_source_value\": \"CNM/CM\"},\n {\"provider_concept_id\": 38003807, \"provider_source_value\": \"Other midwife\"},\n {\"provider_concept_id\": 0, \"provider_source_value\": \"Other/Unknown\"},\n]\nPROVIDER_ID_BY_CONCEPT = {row[\"provider_concept_id\"]: i + 1 for i, row in enumerate(PROVIDER_DIMENSION)}\n\n\n\n_CHILD_FIELDS = [\n # dmeth_rec deliberately excluded -- spreadsheet owner confirmed it's a collapsed\n # duplicate of rdmeth_rec (dmeth_rec=1 combines rdmeth_rec 1/2/5, dmeth_rec=2\n # combines rdmeth_rec 3/4/6) and to prefer rdmeth_rec, which has more granularity.\n # See questions_for_spreadsheet_owner.md #9.\n # mtran deliberately excluded -- spreadsheet owner confirmed mtran (\"Mother\n # Transferred\") belongs on the mother's own record only, not the child's; itran\n # (\"Infant Transferred\") is the child's own, separate fact. See new_findings.md.\n \"ab_anti\", \"ab_aven1\", \"ab_aven6\", \"ab_nicu\", \"ab_seiz\", \"ab_surf\",\n \"attend\", \"bfed\", \"ca_anen\", \"ca_cchd\", \"ca_cdh\", \"ca_cleft\", \"ca_clpal\",\n \"ca_disor\", \"ca_downs\", \"ca_gast\", \"ca_hypo\", \"ca_limb\", \"ca_mnsb\", \"ca_omph\",\n \"dmar\", \"dplural\", \"fagecomb\", \"ilive\", \"itran\", \"ld_indl\",\n \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\", \"no_abnorm\",\n \"no_congen\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\", \"pay\", \"precare\",\n \"previs\", \"rdmeth_rec\", \"setorder_r\", \"wic\", \"apgar5\", \"apgar10\",\n \"combgest\", \"dbwt\",\n]\n_MOTHER_FIELDS = [\n # dlmp_mm/dlmp_yy deliberately excluded -- combined into one hand-computed\n # OBSERVATION row below instead of two independent per-field rows.\n # dmeth_rec deliberately excluded -- see _CHILD_FIELDS comment above; same\n # rdmeth_rec-preferred resolution applies to Mother's own copy.\n \"attend\", \"cig0_r\", \"cig1_r\", \"cig2_r\", \"cig3_r\",\n \"dmar\", \"illb_r\", \"ilop_r\", \"ip_chlam\", \"ip_gon\", \"ip_hepatb\", \"ip_hepatc\",\n \"ip_syph\", \"ld_anes\", \"ld_antb\", \"ld_augm\", \"ld_chor\", \"ld_indl\", \"ld_ster\",\n \"m_ht_in\", \"mager\", \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\",\n \"mm_aicu\", \"mm_mtr\", \"mm_plac\", \"mm_rupt\", \"mm_uhyst\", \"mrace15\", \"mtran\",\n \"no_lbrdlv\", \"no_mmorb\", \"no_risks\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\",\n \"pay\", \"precare\", \"previs\", \"rdmeth_rec\", \"rf_artec\", \"rf_cesar\",\n \"rf_ehype\", \"rf_fedrg\", \"rf_gdiab\", \"rf_ghype\", \"rf_inftr\", \"rf_pdiab\",\n \"rf_phype\", \"rf_ppterm\", \"sex\", \"setorder_r\", \"wic\", \"wtgain\", \"bmi\",\n \"combgest\", \"dbwt\", \"pwgt_r\", \"dplural\",\n]\n_FATHER_FIELDS = [\"fagecomb\", \"feduc\", \"frace15\"]\n\n\ndef _observation_date_for(field: str, value, dob_date: dt.date | None) -> dt.date | None:\n # illb_r/ilop_r: codes 0-3 = plural delivery (use child's dob); 4-300 = months\n # since last live birth/pregnancy outcome (subtract from child's dob). See\n # module docstring.\n if field in (\"illb_r\", \"ilop_r\") and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 4 <= v <= 300:\n return _subtract_months(dob_date, v)\n # precare (Mother): spreadsheet's own Notes on this row say \"Impute by Subtracting\n # value from birth month in months\" -- same subtract-months formula as illb_r/ilop_r,\n # applied to precare's real 1-10 range (0 = no prenatal care, handled by its own\n # separate row and never reaches here).\n if field == \"precare\" and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 1 <= v <= 10:\n return _subtract_months(dob_date, v)\n return dob_date\n\n\n# Spreadsheet NBER Field names that don't match the raw bronze column name for the\n# same field (source_field.upper() -> real raw column). Confirmed against both\n# node-load-bronze's FIELDS list and UserGuide2022.pdf: the real fields are\n# CA_DOWN/IP_HEPB/IP_HEPC, not CA_DOWNS/IP_HEPATB/IP_HEPATC. Without this, rec.get()\n# always misses and these 3 fields silently never produce any record, on any real\n# data -- verified via _iter_field_records() returning [] for all three. See\n# questions_for_spreadsheet_owner.md.\n_RAW_FIELD_ALIAS = {\n \"ca_downs\": \"CA_DOWN\",\n \"ip_hepatb\": \"IP_HEPB\",\n \"ip_hepatc\": \"IP_HEPC\",\n}\n\n\ndef _iter_field_records(rec: dict, dob_date: dt.date | None):\n \"\"\"\n Yields (target_table, role, fact_dict) for every field/value/role match\n found via the spreadsheet-driven mapping, across all of OBSERVATION,\n MEASUREMENT, CONDITION, PROCEDURE, PAYER_PLAN_PERIOD, and PROVIDER.\n\n Each of the 8 build_<table>() functions below filters this same stream\n down to its own table, rather than duplicating the field-matching logic\n once per table -- the per-table split is purely about which node emits\n which rows, not a second independent implementation of the mapping.\n \"\"\"\n for role, fields in ((\"Child\", _CHILD_FIELDS), (\"Mother\", _MOTHER_FIELDS), (\"Father\", _FATHER_FIELDS)):\n for field in fields:\n value = rec.get(_RAW_FIELD_ALIAS.get(field, field.upper()))\n if value is None or str(value).strip() == \"\":\n continue\n matches = get_records(role, field, value)\n for table, cols in matches.items():\n if table == \"OBSERVATION\":\n if field in (\"mager\", \"fagecomb\", \"previs\", \"precare\", \"wtgain\") and \"value_as_number\" not in cols:\n # spreadsheet gives observation_concept_id (4028487 \"Maternal age\"\n # for mager, 4071357 \"Paternal Age\" for fagecomb/Child, 4265453\n # \"Age\" for fagecomb/Father, 46270506/43533800 for previs, 40771565\n # for precare, 40759199 \"Pregnancy weight.gain.current\" for wtgain)\n # but no paired value row; the field's own numeric value is the\n # natural value_as_number -- except when that value is itself one\n # of NCHS's \"unknown or not stated\" sentinels (see\n # _UNKNOWN_VALUE_SENTINEL), where backfilling would record the\n # sentinel as if it were a real measurement.\n backfill_value = _int_or_none(value)\n if (field, backfill_value) not in _UNKNOWN_VALUE_SENTINEL:\n cols = {**cols, \"value_as_number\": backfill_value}\n if field == \"oegest_comb\" and \"unit_concept_id\" not in cols:\n # spreadsheet's unit row for oegest_comb has Values=\"wk\" (a literal\n # string), which only matches if the raw value were literally \"wk\" --\n # it never matches oegest_comb's real numeric week value, so the\n # unit was never actually being set. oegest_comb's unit is always\n # weeks (concept 8511), so applied unconditionally here as an\n # exception rather than via the normal value-matching path.\n cols = {**cols, \"unit_concept_id\": 8511}\n yield \"OBSERVATION\", role, {\n \"observation_date\": _observation_date_for(field, value, dob_date),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"MEASUREMENT\":\n if field in (\"combgest\", \"dbwt\") and \"unit_concept_id\" not in cols:\n # same issue as oegest_comb's unit (see OBSERVATION branch above):\n # the spreadsheet's unit row has a literal Values (\"wk\"/\"g\") that\n # never matches the field's real numeric value, so the unit was\n # never actually being set. combgest is always weeks (8511), dbwt\n # is always grams (8504) -- applied unconditionally as an exception.\n cols = {**cols, \"unit_concept_id\": 8511 if field == \"combgest\" else 8504}\n if field == \"pwgt_r\" and \"unit_concept_id\" not in cols:\n # pwgt_r has no unit row in the spreadsheet at all -- its own\n # NBER_Concept_Name (\"Pre-pregnancy Weight Recode Weight in pounds\")\n # and UserGuide2022.pdf (position 292-294) both confirm the unit is\n # always pounds, same as wtgain's own unit (8739 \"pounds (US)\").\n # Applied unconditionally as an exception, same treatment as\n # combgest/dbwt/oegest_comb above. Not a settled fact -- see\n # questions_for_spreadsheet_owner.md.\n cols = {**cols, \"unit_concept_id\": 8739}\n if field in (\"m_ht_in\", \"bmi\", \"pwgt_r\", \"wtgain\") and \"value_as_number\" not in cols:\n # same shape as the OBSERVATION-table numeric-value gap above (mager/\n # fagecomb/previs/precare/wtgain): the spreadsheet has a row supplying\n # measurement_concept_id for the field's real-value range, but no row\n # at all supplying value_as_number -- the actual measured number\n # (height in inches, BMI, pre-pregnancy weight in pounds) was never\n # being recorded. wtgain's own value_as_number is already recorded on\n # OBSERVATION (see above); this only covers its separate, still-\n # incomplete MEASUREMENT row (missing measurement_concept_id -- see\n # questions_for_spreadsheet_owner.md).\n cols = {**cols, \"value_as_number\": _float_or_none(value)}\n yield \"MEASUREMENT\", role, {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"CONDITION\":\n yield \"CONDITION\", role, {\n \"condition_start_date\": dob_date,\n \"condition_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PROCEDURE\":\n yield \"PROCEDURE\", role, {\n \"procedure_date\": dob_date,\n \"procedure_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PAYER_PLAN_PERIOD\":\n # end_date left null rather than also set to dob_date -- the source\n # data has no real coverage-period dates at all (only \"which payer\"),\n # so forcing both dates to the birth date would assert a false\n # one-day coverage period. start_date must stay non-null (OMOP\n # requires it) and dob_date is the only date this data has to offer;\n # end_date left open is more honest than a fabricated end. Flagged\n # for confirmation in questions_for_spreadsheet_owner.md -- not a\n # settled fact.\n yield \"PAYER_PLAN_PERIOD\", role, {\n \"payer_plan_period_start_date\": dob_date,\n \"payer_plan_period_end_date\": None,\n **cols,\n }\n elif table == \"PROVIDER\":\n provider_id = PROVIDER_ID_BY_CONCEPT.get(cols.get(\"provider_concept_id\"))\n yield \"PROVIDER_USAGE\", role, {\"provider_id\": provider_id}\n\n # hand-computed OBSERVATION/MEASUREMENT rows: these fields' value_as_number row has a\n # non-numeric \"Integer\" placeholder Target_Vocabulary_Id (not a real vocab_id), so\n # load_mapping_rows() can't load it as a normal MappingRow -- but each field also has its\n # own separate, valid, numeric concept_id row (e.g. priorlive -> 3018989) that\n # get_records() resolves correctly. None of these 4 fields are in _MOTHER_FIELDS, so that\n # concept_id was never being looked up at all, leaving these rows without the concept_id\n # OMOP requires. Merged into one row per field here. previs is NOT handled here -- it's in\n # _CHILD_FIELDS/_MOTHER_FIELDS, so it already gets its concept_id row via the field-family\n # loop above; its value_as_number backfill happens there instead (see the OBSERVATION\n # branch above) to avoid emitting a second, duplicate row for the same fact.\n for field in (\"priorlive\", \"priordead\", \"rf_cesarn\"):\n v = _int_or_none(rec.get(field.upper()))\n if v is not None:\n fact = {\n \"observation_date\": dob_date,\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": v,\n }\n fact.update(get_records(\"Mother\", field, v).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # dlmp_mm/dlmp_yy -> one combined OBSERVATION row (spreadsheet's own Notes on\n # dlmp_mm: \"Combine dlmp_yy and dlmp_mm and assume 1st day of month\"), instead of\n # two independent per-field rows sharing the same concept (3002314). dlmp_yy's own\n # Values (\"2020\") is a single literal example, not a real range, so the concept is\n # looked up via dlmp_mm's row (Values \"1 to 12\") instead. DLMP_MM=99/DLMP_YY=9999\n # are the official \"unknown or not stated\" sentinels -- no date computed for those.\n dlmp_month = _int_or_none(rec.get(\"DLMP_MM\"))\n dlmp_year = _int_or_none(rec.get(\"DLMP_YY\"))\n if dlmp_month is not None and 1 <= dlmp_month <= 12 and dlmp_year is not None and dlmp_year != 9999:\n fact = {\n \"observation_date\": dt.date(dlmp_year, dlmp_month, 1),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n fact.update(get_records(\"Mother\", \"dlmp_mm\", dlmp_month).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # priorterm -> MEASUREMENT (spreadsheet's own table choice, see module docstring)\n priorterm = _int_or_none(rec.get(\"PRIORTERM\"))\n if priorterm is not None:\n fact = {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": priorterm,\n }\n fact.update(get_records(\"Mother\", \"priorterm\", priorterm).get(\"MEASUREMENT\", {}))\n yield \"MEASUREMENT\", \"Mother\", fact\n\n\ndef build_persons(rec: dict, record_idx: int) -> list[dict]:\n persons = []\n\n dob_year, dob_month, dob_day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n birth_datetime = None\n if dob_year is not None and dob_month is not None and dob_day is not None:\n # DOB_TT unknown/unparseable (incl. NCHS's \"9999\" sentinel) leaves\n # birth_datetime null rather than assuming midnight -- birth_datetime is\n # nullable in the destination schema, so no time is a truer statement than\n # an assumed one. See questions_for_spreadsheet_owner.md #15.\n dob_time = _parse_dob_time(rec.get(\"DOB_TT\"))\n if dob_time is not None:\n birth_datetime = dt.datetime(dob_year, dob_month, dob_day, dob_time[0], dob_time[1])\n child = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Child\"),\n \"year_of_birth\": dob_year,\n \"month_of_birth\": dob_month,\n \"day_of_birth\": dob_day,\n \"birth_datetime\": birth_datetime,\n }\n child.update(get_records(\"Child\", \"sex\", rec.get(\"SEX\")).get(\"PERSON\", {}))\n persons.append(child)\n\n mager = _int_or_none(rec.get(\"MAGER\"))\n mother = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Mother\"),\n \"year_of_birth\": (dob_year - mager) if (dob_year is not None and mager is not None) else None,\n }\n mother.update(get_records(\"Mother\", \"sex\", \"F\").get(\"PERSON\", {}))\n for field in (\"mrace6\", \"mhispx\", \"mracehisp\"):\n mother.update(get_records(\"Mother\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(mother)\n\n fagecomb = _int_or_none(rec.get(\"FAGECOMB\"))\n if fagecomb == 99:\n # NCHS's own \"unknown or not stated\" sentinel for FAGECOMB, not a real\n # age -- confirmed via the spreadsheet's own fagecomb=99 rows (\"Unknown\n # or Not Stated\" / \"Age unknown\"). Excluded here so the father's\n # year_of_birth isn't computed as if 99 were a real age.\n fagecomb = None\n father = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Father\"),\n \"year_of_birth\": (dob_year - fagecomb) if (dob_year is not None and fagecomb is not None) else None,\n }\n father.update(get_records(\"Father\", \"sex\", \"M\").get(\"PERSON\", {}))\n for field in (\"frace6\", \"fhispx\", \"fracehisp\"):\n father.update(get_records(\"Father\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(father)\n\n return persons\n\n\ndef build_visits(rec: dict, record_idx: int, dob_date: dt.date | None) -> list[dict]:\n visits = []\n for role in (\"Child\", \"Mother\"):\n match = get_records(role, \"bfacil\", rec.get(\"BFACIL\"))\n if \"VISIT_OCCURRENCE\" not in match:\n continue\n visit = {\n \"visit_occurrence_id\": IdGenerator.visit_occurrence_id(record_idx, role),\n \"person_id\": IdGenerator.person_id(record_idx, role),\n \"visit_start_date\": dob_date,\n \"visit_end_date\": dob_date,\n \"visit_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n # spreadsheet field name is \"Visit_Concept_id\" (case-as-published); normalize here\n for k, v in match[\"VISIT_OCCURRENCE\"].items():\n visit[k.lower()] = v\n visits.append(visit)\n return visits\n\n\n_ID_COLUMN = {\n \"OBSERVATION\": \"observation_id\",\n \"MEASUREMENT\": \"measurement_id\",\n \"CONDITION\": \"condition_occurrence_id\",\n \"PROCEDURE\": \"procedure_occurrence_id\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period_id\",\n}\n\nSQL_TABLE_NAME = {\n \"PERSON\": \"person\",\n \"VISIT_OCCURRENCE\": \"visit_occurrence\",\n \"OBSERVATION\": \"observation\",\n \"MEASUREMENT\": \"measurement\",\n \"CONDITION\": \"condition_occurrence\",\n \"PROCEDURE\": \"procedure_occurrence\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period\",\n}\n\nTABLE_COLUMNS = {\n \"PERSON\": [\"person_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\", \"birth_datetime\",\n \"gender_concept_id\", \"race_concept_id\", \"ethnicity_concept_id\"],\n \"VISIT_OCCURRENCE\": [\"visit_occurrence_id\", \"person_id\", \"visit_start_date\", \"visit_end_date\",\n \"visit_type_concept_id\", \"visit_concept_id\"],\n \"OBSERVATION\": ['person_id', 'observation_id', 'observation_date', 'observation_type_concept_id',\n 'observation_concept_id', 'qualifier_concept_id', 'unit_concept_id',\n 'value_as_concept_id', 'value_as_number'],\n \"MEASUREMENT\": ['person_id', 'measurement_id', 'measurement_date', 'measurement_type_concept_id',\n 'measurement_concept_id', 'unit_concept_id', 'value_as_concept_id', 'value_as_number'],\n \"CONDITION\": ['person_id', 'condition_occurrence_id', 'condition_start_date', 'condition_type_concept_id',\n 'condition_concept_id', 'condition_status_concept_id'],\n \"PROCEDURE\": ['person_id', 'procedure_occurrence_id', 'procedure_date', 'procedure_type_concept_id',\n 'procedure_concept_id'],\n \"PAYER_PLAN_PERIOD\": ['person_id', 'payer_plan_period_id', 'payer_plan_period_start_date',\n 'payer_plan_period_end_date', 'payer_concept_id'],\n}\n\n\ndef build_facts(rec: dict, record_idx: int, dob_date: dt.date | None) -> dict[str, list[dict]]:\n \"\"\"\n One pass over _iter_field_records per record, bucketed by target table --\n _iter_field_records computes matches for every table in a single\n generator pass. Covers OBSERVATION/MEASUREMENT/CONDITION/\n PROCEDURE/PAYER_PLAN_PERIOD only -- PERSON and VISIT_OCCURRENCE are\n built separately (build_persons/build_visits) since they're driven by\n a handful of targeted get_records() calls, not this field-family scan.\n \"\"\"\n counters: dict[str, int] = {}\n out: dict[str, list[dict]] = {table: [] for table in _ID_COLUMN}\n for table, role, fact in _iter_field_records(rec, dob_date):\n if table not in _ID_COLUMN:\n continue # e.g. PROVIDER_USAGE -- not one of this node's tables\n counters[table] = counters.get(table, 0) + 1\n out[table].append({\n \"person_id\": IdGenerator.person_id(record_idx, role),\n _ID_COLUMN[table]: IdGenerator.fact_id(record_idx, counters[table]),\n **fact,\n })\n return out\n\n\ndef _sanitize_cache_id(raw_id: str) -> str:\n \"\"\"\n Mirrors the platform's own Dataset.cacheId derivation\n (portal/src/dataset/entity/dataset.entity.ts: sanitizeIdForCacheId) --\n hyphens aren't valid in a bare SQL/DuckDB identifier, so the Dataset's\n UUID gets converted the same way here before use as a trex cache_id.\n \"\"\"\n cleaned = raw_id.replace(\"-\", \"_\")\n return f\"_{cleaned}\" if cleaned[:1].isdigit() else cleaned\n\n\ndef exec(myinput):\n \"\"\"\n Gold transform: reads the staging CSV written by node-load-bronze in\n chunks, builds the OMOP rows in Python, writes each chunk's rows\n per-table to a CSV in the shared trex-volume staging dir, and loads each\n with one `INSERT INTO <table> (cols) SELECT * FROM read_csv('<trex\n path>')` -- trex's DuckDB reads the file straight from disk at native\n speed. Each load statement is atomic, and row counts are verified per\n table at the end; a mismatch raises.\n \"\"\"\n import csv\n import time\n import uuid\n\n logger = get_run_logger()\n\n global _MAPPING_DF_FROM_NODE\n mapping_input = myinput.get(MAPPING_CSV_NODE)\n if mapping_input is None or mapping_input.result is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n if not isinstance(mapping_input.result, pd.DataFrame):\n raise RuntimeError(\n f\"Upstream mapping input {MAPPING_CSV_NODE!r} must be a pandas DataFrame, got \"\n f\"{type(mapping_input.result).__name__}\"\n )\n _MAPPING_DF_FROM_NODE = mapping_input.result.copy()\n\n _validate_mapping_against_cdm(load_mapping_rows(), logger)\n cache_catalog = _sanitize_cache_id(dataset_id)\n dbdao = DBDao(database_code=destination_database_code, cache_id=cache_catalog, dialect=SupportedDatabaseDialects.TREX)\n dest_schema = f\"{cache_catalog}.{destination_schema_name}\"\n\n # Input: the staging CSV written by node-load-bronze. The .complete marker\n # holds the row count and only exists if bronze finished; csv-without-\n # marker means an interrupted bronze run and is refused.\n src_fname = f\"{cache_catalog}_nat2022_raw.csv\"\n src_csv_path = os.path.join(STAGING_DIR_FLOW, src_fname)\n src_complete_path = src_csv_path + \".complete\"\n if not (os.path.exists(src_csv_path) and os.path.exists(src_complete_path)):\n raise RuntimeError(\n f\"Staging CSV from node-load-bronze not found or incomplete \"\n f\"({src_csv_path}; marker present: {os.path.exists(src_complete_path)}). \"\n f\"Run node-load-bronze first -- a CSV without its .complete marker means \"\n f\"the bronze run was interrupted mid-write.\"\n )\n with open(src_complete_path) as f:\n total = int(f.read().strip())\n\n for sql_table in SQL_TABLE_NAME.values():\n # destination_schema_name.<table> is created by the OMOP CDM plugin, not here --\n # truncate rather than drop so we never touch its schema/table definition.\n dbdao.truncate_table(dest_schema, sql_table)\n\n size = int(chunk_size)\n\n # Self-healing sweep: staging files are deleted right after each bulk\n # load, but a hard-killed run orphans its in-flight file. Current-run\n # filenames get a fresh run_token below, so anything matching these\n # prefixes now is stale by construction; other datasets' files\n # (different prefix) are never touched.\n table_prefixes = tuple(f\"{cache_catalog}_{t}_\" for t in SQL_TABLE_NAME.values())\n removed = 0\n for f in os.listdir(STAGING_DIR_FLOW):\n if f.startswith(table_prefixes) and f.endswith(\".csv\"):\n try:\n os.remove(os.path.join(STAGING_DIR_FLOW, f))\n removed += 1\n except OSError:\n pass\n if removed:\n logger.info(f\"Removed {removed} stale staging file(s) left by a previous interrupted run\")\n\n run_token = uuid.uuid4().hex\n\n def _bulk_load(sql_table, columns, values, tag):\n \"\"\"One CSV in the shared staging dir + one read_csv INSERT for this table-chunk.\"\"\"\n fname = f\"{cache_catalog}_{sql_table}_{tag}_{run_token}.csv\"\n fpath = os.path.join(STAGING_DIR_FLOW, fname)\n with open(fpath, \"w\", newline=\"\") as f:\n w = csv.writer(f)\n w.writerow(columns)\n for row in values:\n w.writerow([\"\" if v is None else v for v in row])\n col_list = \", \".join(f'\"{c}\"' for c in columns)\n try:\n dbdao.execute_sql(\n f\"INSERT INTO {dest_schema}.{sql_table} ({col_list}) \"\n f\"SELECT * FROM read_csv('{STAGING_DIR_TREX}/{fname}', header=true, all_varchar=true)\"\n )\n finally:\n os.remove(fpath)\n\n expected_counts = {table: 0 for table in SQL_TABLE_NAME}\n processed = 0\n # Sequential single-pass read of the staging CSV -- flat cost per chunk,\n # order preserved (ROW_IDX rides along as a column and record_idx comes\n # from its stored value, not from position). dtype=str + keep_default_na\n # =False gives '' for missing values, which the transform treats the same\n # as NULL.\n for chunk_idx, src_chunk in enumerate(pd.read_csv(src_csv_path, dtype=str, keep_default_na=False, chunksize=size)):\n t_chunk = time.time()\n rows_by_table = {table: [] for table in SQL_TABLE_NAME}\n for rec in src_chunk.to_dict(\"records\"):\n record_idx = int(rec[\"ROW_IDX\"])\n dob_date = dob_date_for(rec)\n rows_by_table[\"PERSON\"].extend(build_persons(rec, record_idx))\n rows_by_table[\"VISIT_OCCURRENCE\"].extend(build_visits(rec, record_idx, dob_date))\n built = build_facts(rec, record_idx, dob_date)\n for table, rows in built.items():\n rows_by_table[table].extend(rows)\n t_transform = time.time() - t_chunk\n\n chunk_rows = 0\n for table, sql_table in SQL_TABLE_NAME.items():\n columns = TABLE_COLUMNS[table]\n values = [tuple(row.get(col) for col in columns) for row in rows_by_table[table]]\n if not values:\n continue\n expected_counts[table] += len(values)\n chunk_rows += len(values)\n t_tbl = time.time()\n _bulk_load(sql_table, columns, values, str(chunk_idx))\n logger.info(f\" {sql_table}: {len(values):,} rows bulk-loaded in {time.time() - t_tbl:.1f}s\")\n\n processed += len(src_chunk)\n done = processed\n logger.info(\n f\"{done:,}/{total:,} records -> {chunk_rows:,} OMOP rows \"\n f\"(transform {t_transform:.0f}s, chunk total {time.time() - t_chunk:.0f}s)\"\n )\n\n if processed != total:\n raise RuntimeError(f\"staging CSV row count mismatch: marker says {total:,}, read {processed:,}\")\n\n mismatches = []\n for table, sql_table in SQL_TABLE_NAME.items():\n actual = int(dbdao.execute_sql(f\"SELECT COUNT(*) FROM {dest_schema}.{sql_table}\", fetch=True)[0][0])\n if actual != expected_counts[table]:\n mismatches.append(f\"{sql_table}: expected {expected_counts[table]:,}, table has {actual:,}\")\n if mismatches:\n raise RuntimeError(\"row-count verification failed -- \" + \"; \".join(mismatches))\n logger.info(\"Row-count verification passed for all tables: \"\n + \", \".join(f\"{SQL_TABLE_NAME[t]}={expected_counts[t]:,}\" for t in SQL_TABLE_NAME))\n\n return (f\"Loaded {total} records into person, visit_occurrence, observation, measurement, \"\n f\"condition_occurrence, procedure_occurrence, payer_plan_period\")" |
| "data": { | ||
| "name": "transform_facts", | ||
| "description": "Gold: build PERSON, VISIT_OCCURRENCE, OBSERVATION, MEASUREMENT, CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE, and PAYER_PLAN_PERIOD in a single per-record pass (merged from 7 originally separate nodes; see node_backups/ for the originals).", | ||
| "python_code": "# Deployment-topology constants (deliberately NOT pipeline variables) -- see\n# node-load-bronze for the full rationale: fixed by the platform's container\n# mounts; the two paths are the flow container's and trex's views of the same\n# shared staging directory on the trex data volume.\nNAT2022_DATA_DIR = \"/app/data_load\"\nSTAGING_DIR_FLOW = \"/app/duckdb_data/flow_staging\"\nSTAGING_DIR_TREX = \"/usr/src/data/flow_staging\"\n\nMAPPING_CSV_NODE = \"csv_node_0\"\n\n_PREFIX_GROUP_RE = re.compile(r'^(all )?variables beginning with\\s+(\\w+?)_?$', re.I)\n\n# Resolved relative to this file's own directory, not the caller's cwd --\n# matters both for local runs and once this module is volume-mounted into a\n# flow-run container alongside the csv (see flowNodes/ directory).\n_TABLE_NORMALIZE = {\n \"PPROCEDURE\": \"PROCEDURE\",\n}\n\n_FIELD_NAME_NORMALIZE = {\n \"observation_date\": \"observation_concept_id\", # mislabeled in spreadsheet; see module docstring\n \"unit_as_concept_id\": \"unit_concept_id\", # spreadsheet typo -- no \"_as_\" in the real OMOP column name\n}\n\n# (person_role, source_field, raw Values string) -> corrected Values string.\n# precare's Child row has Values=\"1\" -- a single literal value -- even though\n# its own NBER_Concept_Name says \"Month Prenatal Care Began 01 - 10\" and\n# UserGuide2022.pdf documents PRECARE's real range as 01-10. fagecomb's Child\n# row has Values=\"9\" even though its own NBER_Concept_Name says \"09-98\" and\n# UserGuide2022.pdf documents FAGECOMB's real range as 09-98. Applied\n# narrowly (exact role+field+raw-value match) so it can't accidentally\n# affect any other row. Interpretation flagged for spreadsheet-owner\n# confirmation in questions_for_spreadsheet_owner.md -- not a settled fact,\n# best current guess.\n_VALUES_NORMALIZE = {\n (\"Child\", \"precare\", \"1\"): \"1-10\",\n (\"Child\", \"fagecomb\", \"9\"): \"9-98\",\n # previs's Values=\"12\" only matched a raw visit count of exactly 12, leaving\n # value_as_concept_id null for every other real count. UserGuide2022.pdf documents\n # PREVIS's valid range as 00-98, and previs=0 already has its own dedicated row\n # (\"No prenatal care\"), so widened to 1-98 -- matching the Mother-role sibling row's\n # own already-correct \"1 to 98\" range for this same concept.\n (\"Child\", \"previs\", \"12\"): \"1-98\",\n # Mother's precare row (observation_date -> observation_concept_id 40771565) only\n # matched literal \"2\", even though its own Notes describe a general subtract-months\n # formula (see _observation_date_for) meant to apply to any value in precare's real\n # 1-10 range. 0 already has its own dedicated \"No prenatal care\" row.\n (\"Mother\", \"precare\", \"2\"): \"1-10\",\n}\n\n_REAL_CDM_COLUMNS = {\n \"PERSON\": {\"person_id\", \"gender_concept_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\",\n \"birth_datetime\", \"race_concept_id\", \"ethnicity_concept_id\", \"location_id\", \"provider_id\",\n \"care_site_id\", \"person_source_value\", \"gender_source_value\", \"gender_source_concept_id\",\n \"race_source_value\", \"race_source_concept_id\", \"ethnicity_source_value\",\n \"ethnicity_source_concept_id\"},\n \"VISIT_OCCURRENCE\": {\"visit_occurrence_id\", \"person_id\", \"visit_concept_id\", \"visit_start_date\",\n \"visit_start_datetime\", \"visit_end_date\", \"visit_end_datetime\",\n \"visit_type_concept_id\", \"provider_id\", \"care_site_id\", \"visit_source_value\",\n \"visit_source_concept_id\", \"admitted_from_concept_id\", \"admitted_from_source_value\",\n \"discharged_to_concept_id\", \"discharged_to_source_value\",\n \"preceding_visit_occurrence_id\"},\n \"OBSERVATION\": {\"observation_id\", \"person_id\", \"observation_concept_id\", \"observation_date\",\n \"observation_datetime\", \"observation_type_concept_id\", \"value_as_number\",\n \"value_as_string\", \"value_as_concept_id\", \"qualifier_concept_id\", \"unit_concept_id\",\n \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\", \"observation_source_value\",\n \"observation_source_concept_id\", \"unit_source_value\", \"qualifier_source_value\",\n \"value_source_value\", \"observation_event_id\", \"obs_event_field_concept_id\"},\n \"MEASUREMENT\": {\"measurement_id\", \"person_id\", \"measurement_concept_id\", \"measurement_date\",\n \"measurement_datetime\", \"measurement_time\", \"measurement_type_concept_id\",\n \"operator_concept_id\", \"value_as_number\", \"value_as_concept_id\", \"unit_concept_id\",\n \"range_low\", \"range_high\", \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\",\n \"measurement_source_value\", \"measurement_source_concept_id\", \"unit_source_value\",\n \"unit_source_concept_id\", \"value_source_value\", \"measurement_event_id\",\n \"meas_event_field_concept_id\"},\n \"CONDITION\": {\"condition_occurrence_id\", \"person_id\", \"condition_concept_id\", \"condition_start_date\",\n \"condition_start_datetime\", \"condition_end_date\", \"condition_end_datetime\",\n \"condition_type_concept_id\", \"condition_status_concept_id\", \"stop_reason\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"condition_source_value\",\n \"condition_source_concept_id\", \"condition_status_source_value\"},\n \"PROCEDURE\": {\"procedure_occurrence_id\", \"person_id\", \"procedure_concept_id\", \"procedure_date\",\n \"procedure_datetime\", \"procedure_end_date\", \"procedure_end_datetime\",\n \"procedure_type_concept_id\", \"modifier_concept_id\", \"quantity\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"procedure_source_value\",\n \"procedure_source_concept_id\", \"modifier_source_value\"},\n \"PAYER_PLAN_PERIOD\": {\"payer_plan_period_id\", \"person_id\", \"payer_plan_period_start_date\",\n \"payer_plan_period_end_date\", \"payer_concept_id\", \"payer_source_value\",\n \"payer_source_concept_id\", \"plan_concept_id\", \"plan_source_value\",\n \"plan_source_concept_id\", \"sponsor_concept_id\", \"sponsor_source_value\",\n \"sponsor_source_concept_id\", \"family_source_value\", \"stop_reason_concept_id\",\n \"stop_reason_source_value\", \"stop_reason_source_concept_id\"},\n \"PROVIDER\": {\"provider_id\", \"provider_name\", \"npi\", \"dea\", \"specialty_concept_id\", \"care_site_id\",\n \"year_of_birth\", \"gender_concept_id\", \"provider_source_value\", \"specialty_source_value\",\n \"specialty_source_concept_id\", \"gender_source_value\", \"gender_source_concept_id\"},\n}\n\n# (Target_Table, Field_Name) pairs already confirmed as spreadsheet-side issues --\n# tracked in questions_for_spreadsheet_owner.md with an existing code-level\n# workaround. Logged as a warning but doesn't halt the pipeline. Anything NOT\n# in this set is treated as a new, unreviewed problem and raises immediately.\n_KNOWN_MAPPING_ISSUES = {\n (\"MEASUREMENT\", \"qualifier_concept_id\"), # questions doc #10 -- dropped from MEASUREMENT's own TABLE_COLUMNS\n (\"PROVIDER\", \"provider_concept_id\"), # questions doc #14 -- PROVIDER_DIMENSION hardcodes specialty_concept_id instead\n}\n\n\ndef _validate_mapping_against_cdm(rows: \"tuple[MappingRow, ...]\", logger) -> None:\n \"\"\"\n Checks every spreadsheet row's (Target_Table, Field_Name) against the real\n OMOP CDM 5.4 column list for that table, so a spreadsheet edit (typo,\n mislabeled column, wrong table name) is caught once, up front, before it\n silently reaches 3M+ records -- rather than requiring the same manual\n cross-referencing against the DDL that found the issues already tracked\n in questions_for_spreadsheet_owner.md.\n \"\"\"\n unexpected = []\n for row in rows:\n valid_columns = _REAL_CDM_COLUMNS.get(row.target_table)\n key = (row.target_table, row.field_name)\n if valid_columns is None:\n if key not in _KNOWN_MAPPING_ISSUES:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: unrecognized Target_Table {row.target_table!r}\"\n )\n continue\n if row.field_name.lower() not in valid_columns:\n if key in _KNOWN_MAPPING_ISSUES:\n logger.warning(\n \"Known spreadsheet issue (see questions_for_spreadsheet_owner.md): \"\n f\"{row.source_field}/{row.person_role} sets Field_Name={row.field_name!r} on \"\n f\"{row.target_table}, which isn't a real column there.\"\n )\n else:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: Field_Name {row.field_name!r} is not a real \"\n f\"column on {row.target_table} (spreadsheet concept: {row.concept_name!r})\"\n )\n if unexpected:\n raise ValueError(\n \"Mapping spreadsheet has unrecognized Target_Table/Field_Name combinations not seen before \"\n \"-- check for a new spreadsheet edit or typo before proceeding:\\n\"\n + \"\\n\".join(f\" - {e}\" for e in unexpected)\n )\n\n\n\n@dataclass(frozen=True)\nclass MappingRow:\n source_field: str # lowercase NBER field name, e.g. \"rf_pdiab\"\n person_role: str # \"Child\", \"Mother\", or \"Father\"\n target_table: str # normalized OMOP table name, e.g. \"OBSERVATION\"\n field_name: str # OMOP column this row supplies, e.g. \"value_as_concept_id\"\n concept_id: int\n concept_name: str\n matcher: \"ValueMatcher\"\n\n\ndef _is_number(s: str) -> bool:\n \"\"\"True for anything float() accepts, e.g. \"9\", \"-5\", \"69.9\" -- used\n instead of str.isdigit() so decimal Values (e.g. bmi's \"13-69.9\") parse\n as ranges/exact values instead of falling through to literal-string\n matching, which no real decimal value would ever hit.\"\"\"\n try:\n float(s)\n return True\n except ValueError:\n return False\n\n\nclass ValueMatcher:\n \"\"\"\n Parses the spreadsheet's free-text Values column into something\n callable against an actual field value from the data.\n\n Recognized forms: blank/\"any\" (always matches -- used for field-level\n \"header\" rows that apply regardless of the specific code), exact\n tokens separated by \"or\"/\",\", and numeric ranges (\"0-30\", \"1 to 98\",\n \"13-69.9\" -- bounds/values may be decimal, e.g. bmi).\n \"\"\"\n\n def __init__(self, raw: str):\n raw = (raw or \"\").strip()\n self.raw = raw\n self.kind: str\n if raw == \"\" or raw.lower() == \"any\":\n self.kind = \"always\"\n return\n if raw.startswith(\">\") or raw.startswith(\"<\"):\n bound = raw[1:].strip()\n if _is_number(bound):\n self.kind = \"inequality\"\n self.op = raw[0]\n self.bound = float(bound)\n return\n low = raw.lower().replace(\" to \", \"-\")\n parts = [p.strip() for p in low.split(\"-\")]\n if len(parts) == 2 and all(_is_number(p) for p in parts):\n self.kind = \"range\"\n self.lo = float(parts[0])\n self.hi = float(parts[1])\n return\n self.kind = \"exact\"\n self.tokens = {t.strip().lower() for t in raw.replace(\",\", \" or \").split(\" or \")}\n\n def matches(self, value) -> bool:\n if value is None:\n return False\n value_s = str(value).strip()\n if value_s == \"\":\n return False\n if self.kind == \"always\":\n return True\n if self.kind == \"range\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return self.lo <= v <= self.hi\n if self.kind == \"inequality\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return v > self.bound if self.op == \">\" else v < self.bound\n if value_s.lower() in self.tokens:\n return True\n # Raw fields are read with dtype=str (pd.read_fwf), so numeric codes\n # keep their fixed-width zero-padding (e.g. \"09\") -- compare\n # numerically too so an exact token like \"9\" still matches \"09\",\n # and a decimal token like \"99.9\" still matches a decimal value.\n if _is_number(value_s):\n return any(_is_number(t) and float(t) == float(value_s) for t in self.tokens)\n return False\n\n\ndef _normalize_table(raw: str) -> str:\n t = raw.strip().upper().replace(\" \", \"_\")\n return _TABLE_NORMALIZE.get(t, t)\n\n\n_MAPPING_ROWS: tuple[MappingRow, ...] | None = None\n_PREFIX_GROUP_DEFAULTS: dict[tuple[str, str], int] | None = None\n_FIELD_INDEX: dict[tuple[str, str], list[MappingRow]] | None = None\n_MAPPING_DF_FROM_NODE: \"pd.DataFrame | None\" = None\n\n\ndef _load_mapping_df() -> \"pd.DataFrame\":\n if _MAPPING_DF_FROM_NODE is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n return _MAPPING_DF_FROM_NODE.copy()\n\n\ndef load_mapping_rows() -> tuple[MappingRow, ...]:\n global _MAPPING_ROWS\n if _MAPPING_ROWS is not None:\n return _MAPPING_ROWS\n\n out = []\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n nber_field, values, _nber_name, field_name, target_table, vocab_id, concept_name = r[:7]\n person_role = (r[11] or \"\").strip()\n if not person_role or not target_table or not field_name:\n continue\n vocab_id_s = str(vocab_id).strip() if vocab_id is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue # computed/imputed row, handled separately in omop_transform.py\n if _PREFIX_GROUP_RE.match(str(nber_field).strip()):\n continue # prefix-group default row, handled by _prefix_group_defaults below\n field_name_s = _FIELD_NAME_NORMALIZE.get(str(field_name).strip(), str(field_name).strip())\n source_field_s = str(nber_field).strip().lower()\n values_s = \"\" if values is None else str(values)\n values_s = _VALUES_NORMALIZE.get((person_role, source_field_s, values_s.strip()), values_s)\n out.append(MappingRow(\n source_field=source_field_s,\n person_role=person_role,\n target_table=_normalize_table(str(target_table)),\n field_name=field_name_s,\n concept_id=int(vocab_id_s),\n concept_name=str(concept_name).strip() if concept_name else \"\",\n matcher=ValueMatcher(values_s),\n ))\n _MAPPING_ROWS = tuple(out)\n return _MAPPING_ROWS\n\n\ndef _prefix_group_defaults() -> dict[tuple[str, str], int]:\n \"\"\"(person_role, prefix) -> default observation_concept_id for that field family.\"\"\"\n global _PREFIX_GROUP_DEFAULTS\n if _PREFIX_GROUP_DEFAULTS is not None:\n return _PREFIX_GROUP_DEFAULTS\n\n out = {}\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n m = _PREFIX_GROUP_RE.match(str(r[0]).strip())\n if not m:\n continue\n vocab_id_s = str(r[5]).strip() if r[5] is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue\n person_role = (r[11] or \"\").strip()\n prefix = m.group(2).lower()\n out[(person_role, prefix)] = int(vocab_id_s)\n _PREFIX_GROUP_DEFAULTS = out\n return _PREFIX_GROUP_DEFAULTS\n\n\ndef _index_by_field() -> dict[tuple[str, str], list[MappingRow]]:\n global _FIELD_INDEX\n if _FIELD_INDEX is not None:\n return _FIELD_INDEX\n\n index: dict[tuple[str, str], list[MappingRow]] = {}\n for row in load_mapping_rows():\n index.setdefault((row.person_role, row.source_field), []).append(row)\n _FIELD_INDEX = index\n return _FIELD_INDEX\n\n\n# (source_field, field_name, concept_id) rows that should be treated as an \"Integer\n# example\" placeholder (substitute the real value in) even though their\n# Target_Concept_Name isn't literally \"Integer example\". apgar5's value_as_number row\n# has a real concept id (3004221, \"5 minute Apgar Score\") in that slot instead of the\n# placeholder pattern used elsewhere (oegest_comb/dbwt/combgest), so without this it\n# stores the literal concept id 3004221 instead of the actual apgar score. Flagged in\n# questions_for_spreadsheet_owner.md -- not a settled fact, best current guess.\n_INTEGER_PLACEHOLDER_OVERRIDE = {\n (\"apgar5\", \"value_as_number\", 3004221),\n (\"apgar10\", \"value_as_number\", 3016162),\n}\n\n# no_mmorb (\"no maternal morbidity reported\") answers the same underlying survey\n# question as the mm_* fields (its own answer concept is literally \"None of the\n# above (maternal morbidity)\"), and would pick up the \"Variables beginning with\n# mm_\" header row's observation_concept_id (43533805) via the normal prefix-group\n# match below -- except its own field name starts with \"no_\", not \"mm_\", so the\n# prefix never matches. Applied directly instead. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_OBSERVATION_CONCEPT_OVERRIDE = {\n (\"Mother\", \"no_mmorb\"): 43533805,\n # no_abnorm (\"no abnormal conditions of the newborn\") is the same underlying\n # survey question as the ab_* fields (its own answer concept is literally \"None\n # of the above\"), and would pick up the \"Variables beginning with ab_\" header\n # row's observation_concept_id (43533835) via the normal prefix-group match --\n # except its own field name starts with \"no_\", not \"ab_\", so the prefix never\n # matches. Applied directly instead, same shape as no_mmorb above. Not a settled\n # fact -- see questions_for_spreadsheet_owner.md.\n (\"Child\", \"no_abnorm\"): 43533835,\n # no_congen (\"no congenital anomalies of the newborn checked\") is the same\n # underlying survey question as the ca_* fields, and would pick up the\n # \"Variables beginning with ca_\" header row's observation_concept_id\n # (43533804) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ca_\". Same shape as no_mmorb/no_abnorm above.\n (\"Child\", \"no_congen\"): 43533804,\n # no_lbrdlv (\"no characteristics of labor and delivery reported\") is the\n # same underlying survey question as the ld_* fields, and would pick up the\n # \"Variables beginning with LD_\" header row's observation_concept_id\n # (43533836) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ld_\". Same shape as no_mmorb/no_abnorm/no_congen\n # above.\n (\"Mother\", \"no_lbrdlv\"): 43533836,\n # wtgain's own \"99 = Unknown or not stated\" row supplies only a\n # qualifier_concept_id (4129922, \"unknown\"), unlike its real-value row\n # (00-97), which supplies observation_concept_id (40759199) directly.\n # Applied directly so the unknown case still identifies itself as a\n # pregnancy weight-gain observation, same concept as the real-value case.\n (\"Mother\", \"wtgain\"): 40759199,\n}\n\n# (person_role, source_field, field_name, concept_id from spreadsheet) ->\n# replacement concept_id. fagecomb's Child-role \"99 = Unknown or Not Stated\"\n# row supplies observation_concept_id=0 (\"No matching concept\") -- unlike\n# every other 0-row in the spreadsheet (bfacil/fhispx/mracehisp/dmar/attend\n# etc.), all of which land in an *answer* column (value_as_concept_id,\n# ethnicity_concept_id, provider_concept_id) where 0 legitimately means \"the\n# answer is Other/Unknown,\" this one lands in the *identifying*\n# observation_concept_id column, so it doesn't say what the observation is\n# about at all. Remapped to 4071357 (\"Paternal Age\"), the same concept\n# fagecomb's own Child-role real-value row already uses -- matching how\n# fagecomb's Father-role row already reuses one concept (4265453) for both\n# its real-value and its own 99-unknown case. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_CONCEPT_ID_REMAP = {\n (\"Child\", \"fagecomb\", \"observation_concept_id\", 0): 4071357,\n # itran (\"Infant Transferred\") and ilive (\"Infant Living at Time of Report\") both\n # mapped to observation_concept_id 43533781 in the spreadsheet (questions_for_\n # spreadsheet_owner.md #7) -- confirmed by the spreadsheet owner as genuinely\n # different concepts (an ATHENA hierarchy display issue caused the mix-up).\n # itran's corrected code is 43533782 (\"Infant was transferred within 24 hr of\n # delivery\"); ilive's correct code is still pending confirmation, so ilive is\n # left unchanged at 43533781 for now.\n (\"Child\", \"itran\", \"observation_concept_id\", 43533781): 43533782,\n}\n\n# (source_field, sentinel value) pairs where the raw value is NCHS's own\n# \"unknown or not stated\" placeholder rather than a real measurement --\n# confirmed via each field's own Values=99 row (wtgain: \"Unknown or not\n# stated\"; fagecomb: \"Unknown or Not Stated\" / \"Age unknown\"; previs:\n# \"Unknown or Not Stated\"). Excluded from the value_as_number backfill below\n# so \"unknown\" doesn't get recorded as a literal, implausible fact (e.g.\n# wtgain=99 read as \"gained 99 lbs\", fagecomb=99 read as \"father is 99\").\n# previs=99 found via the same check while fixing wtgain -- not itself\n# reported, flagged here for visibility.\n_UNKNOWN_VALUE_SENTINEL = {\n (\"fagecomb\", 99),\n (\"wtgain\", 99),\n (\"previs\", 99),\n}\n\n\ndef get_records(person_role: str, source_field: str, value):\n \"\"\"\n Returns {target_table: {field_name: concept_id, ...}, ...} for every\n target table that has at least one matching mapping row for this\n (person_role, source_field, value). Rows with no value match (e.g. the\n field is blank/not reported) contribute nothing.\n \"\"\"\n source_field = source_field.lower()\n candidates = _index_by_field().get((person_role, source_field), [])\n out: dict[str, dict[str, int]] = {}\n for row in candidates:\n if row.matcher.matches(value):\n concept_id = row.concept_id\n is_placeholder = (\n row.concept_name.strip().lower() == \"integer example\"\n or (row.source_field, row.field_name, row.concept_id) in _INTEGER_PLACEHOLDER_OVERRIDE\n )\n if is_placeholder:\n try:\n concept_id = int(str(value).strip())\n except ValueError:\n continue\n concept_id = _CONCEPT_ID_REMAP.get(\n (person_role, source_field, row.field_name, concept_id), concept_id\n )\n out.setdefault(row.target_table, {})[row.field_name] = concept_id\n\n if \"OBSERVATION\" in out and \"observation_concept_id\" not in out[\"OBSERVATION\"]:\n override = _OBSERVATION_CONCEPT_OVERRIDE.get((person_role, source_field))\n if override is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = override\n else:\n prefix = source_field.split(\"_\")[0]\n default = _prefix_group_defaults().get((person_role, prefix))\n if default is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = default\n\n return out\n\n\n\n\nDEFAULT_TYPE_CONCEPT_ID = 0 # TODO: confirm correct *_type_concept_id with vocabulary before use\n\nWEEKDAY_CODE_TO_PY = {1: 6, 2: 0, 3: 1, 4: 2, 5: 3, 6: 4, 7: 5} # 1=Sunday..7=Saturday -> Python Monday=0..Sunday=6\n\n\ndef _int_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return int(s)\n except ValueError:\n return None\n\n\ndef _float_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return float(s)\n except ValueError:\n return None\n\n\ndef _impute_child_dob(dob_yy, dob_mm, dob_wk):\n year, month, wk = _int_or_none(dob_yy), _int_or_none(dob_mm), _int_or_none(dob_wk)\n if year is None or month is None:\n return None, None, None\n if wk is None or wk not in WEEKDAY_CODE_TO_PY:\n return year, month, None\n target_py_weekday = WEEKDAY_CODE_TO_PY[wk]\n first_of_month = dt.date(year, month, 1)\n offset = (target_py_weekday - first_of_month.weekday()) % 7\n return year, month, 1 + offset\n\n\ndef _parse_dob_time(dob_tt) -> tuple[int, int] | None:\n \"\"\"DOB_TT is a raw 4-digit HHMM code (e.g. '0830'); '9999' means not stated.\"\"\"\n s = \"\" if dob_tt is None else str(dob_tt).strip()\n if len(s) != 4 or not s.isdigit() or s == \"9999\":\n return None\n hour, minute = int(s[:2]), int(s[2:])\n if not (0 <= hour <= 23 and 0 <= minute <= 59):\n return None\n return hour, minute\n\n\ndef dob_date_for(rec):\n year, month, day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n if year and month and day:\n return dt.date(year, month, day)\n return None\n\n\ndef _subtract_months(base_date, months):\n total = base_date.year * 12 + (base_date.month - 1) - months\n year, month = divmod(total, 12)\n day = min(base_date.day, 28)\n return dt.date(year, month + 1, day)\n\n\nclass IdGenerator:\n # Dense, chunk-safe id allocation: exactly 3 roles per record, so\n # record_idx*3 + offset + 1 produces 1,2,3,4,5,6,... with no gaps,\n # while still being computable independently per record (no shared\n # counter needed across chunks/workers).\n PERSON_ROLE_OFFSET = {\"Child\": 0, \"Mother\": 1, \"Father\": 2}\n\n @staticmethod\n def person_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def visit_occurrence_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def fact_id(record_idx, local_counter):\n # 500-id block per record: the destination CDM's fact id columns\n # (observation_id etc.) are 32-bit INTEGER (max 2,147,483,647) and the\n # full Nat2022 file has 3,676,029 records, so the block size must keep\n # the largest id under 2^31 (500 tops out at ~1.84e9) while exceeding\n # the most facts one record produces in a table (~60 observations).\n return (record_idx + 1) * 500 + local_counter\n\n\nPROVIDER_DIMENSION = [\n {\"provider_concept_id\": 38004446, \"provider_source_value\": \"Physician\"},\n {\"provider_concept_id\": 38003822, \"provider_source_value\": \"Osteopathic Practitioner\"},\n {\"provider_concept_id\": 38004482, \"provider_source_value\": \"CNM/CM\"},\n {\"provider_concept_id\": 38003807, \"provider_source_value\": \"Other midwife\"},\n {\"provider_concept_id\": 0, \"provider_source_value\": \"Other/Unknown\"},\n]\nPROVIDER_ID_BY_CONCEPT = {row[\"provider_concept_id\"]: i + 1 for i, row in enumerate(PROVIDER_DIMENSION)}\n\n\n\n_CHILD_FIELDS = [\n # dmeth_rec deliberately excluded -- spreadsheet owner confirmed it's a collapsed\n # duplicate of rdmeth_rec (dmeth_rec=1 combines rdmeth_rec 1/2/5, dmeth_rec=2\n # combines rdmeth_rec 3/4/6) and to prefer rdmeth_rec, which has more granularity.\n # See questions_for_spreadsheet_owner.md #9.\n # mtran deliberately excluded -- spreadsheet owner confirmed mtran (\"Mother\n # Transferred\") belongs on the mother's own record only, not the child's; itran\n # (\"Infant Transferred\") is the child's own, separate fact. See new_findings.md.\n \"ab_anti\", \"ab_aven1\", \"ab_aven6\", \"ab_nicu\", \"ab_seiz\", \"ab_surf\",\n \"attend\", \"bfed\", \"ca_anen\", \"ca_cchd\", \"ca_cdh\", \"ca_cleft\", \"ca_clpal\",\n \"ca_disor\", \"ca_downs\", \"ca_gast\", \"ca_hypo\", \"ca_limb\", \"ca_mnsb\", \"ca_omph\",\n \"dmar\", \"dplural\", \"fagecomb\", \"ilive\", \"itran\", \"ld_indl\",\n \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\", \"no_abnorm\",\n \"no_congen\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\", \"pay\", \"precare\",\n \"previs\", \"rdmeth_rec\", \"setorder_r\", \"wic\", \"apgar5\", \"apgar10\",\n \"combgest\", \"dbwt\",\n]\n_MOTHER_FIELDS = [\n # dlmp_mm/dlmp_yy deliberately excluded -- combined into one hand-computed\n # OBSERVATION row below instead of two independent per-field rows.\n # dmeth_rec deliberately excluded -- see _CHILD_FIELDS comment above; same\n # rdmeth_rec-preferred resolution applies to Mother's own copy.\n \"attend\", \"cig0_r\", \"cig1_r\", \"cig2_r\", \"cig3_r\",\n \"dmar\", \"illb_r\", \"ilop_r\", \"ip_chlam\", \"ip_gon\", \"ip_hepatb\", \"ip_hepatc\",\n \"ip_syph\", \"ld_anes\", \"ld_antb\", \"ld_augm\", \"ld_chor\", \"ld_indl\", \"ld_ster\",\n \"m_ht_in\", \"mager\", \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\",\n \"mm_aicu\", \"mm_mtr\", \"mm_plac\", \"mm_rupt\", \"mm_uhyst\", \"mrace15\", \"mtran\",\n \"no_lbrdlv\", \"no_mmorb\", \"no_risks\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\",\n \"pay\", \"precare\", \"previs\", \"rdmeth_rec\", \"rf_artec\", \"rf_cesar\",\n \"rf_ehype\", \"rf_fedrg\", \"rf_gdiab\", \"rf_ghype\", \"rf_inftr\", \"rf_pdiab\",\n \"rf_phype\", \"rf_ppterm\", \"sex\", \"setorder_r\", \"wic\", \"wtgain\", \"bmi\",\n \"combgest\", \"dbwt\", \"pwgt_r\", \"dplural\",\n]\n_FATHER_FIELDS = [\"fagecomb\", \"feduc\", \"frace15\"]\n\n\ndef _observation_date_for(field: str, value, dob_date: dt.date | None) -> dt.date | None:\n # illb_r/ilop_r: codes 0-3 = plural delivery (use child's dob); 4-300 = months\n # since last live birth/pregnancy outcome (subtract from child's dob). See\n # module docstring.\n if field in (\"illb_r\", \"ilop_r\") and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 4 <= v <= 300:\n return _subtract_months(dob_date, v)\n # precare (Mother): spreadsheet's own Notes on this row say \"Impute by Subtracting\n # value from birth month in months\" -- same subtract-months formula as illb_r/ilop_r,\n # applied to precare's real 1-10 range (0 = no prenatal care, handled by its own\n # separate row and never reaches here).\n if field == \"precare\" and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 1 <= v <= 10:\n return _subtract_months(dob_date, v)\n return dob_date\n\n\n# Spreadsheet NBER Field names that don't match the raw bronze column name for the\n# same field (source_field.upper() -> real raw column). Confirmed against both\n# node-load-bronze's FIELDS list and UserGuide2022.pdf: the real fields are\n# CA_DOWN/IP_HEPB/IP_HEPC, not CA_DOWNS/IP_HEPATB/IP_HEPATC. Without this, rec.get()\n# always misses and these 3 fields silently never produce any record, on any real\n# data -- verified via _iter_field_records() returning [] for all three. See\n# questions_for_spreadsheet_owner.md.\n_RAW_FIELD_ALIAS = {\n \"ca_downs\": \"CA_DOWN\",\n \"ip_hepatb\": \"IP_HEPB\",\n \"ip_hepatc\": \"IP_HEPC\",\n}\n\n\ndef _iter_field_records(rec: dict, dob_date: dt.date | None):\n \"\"\"\n Yields (target_table, role, fact_dict) for every field/value/role match\n found via the spreadsheet-driven mapping, across all of OBSERVATION,\n MEASUREMENT, CONDITION, PROCEDURE, PAYER_PLAN_PERIOD, and PROVIDER.\n\n Each of the 8 build_<table>() functions below filters this same stream\n down to its own table, rather than duplicating the field-matching logic\n once per table -- the per-table split is purely about which node emits\n which rows, not a second independent implementation of the mapping.\n \"\"\"\n for role, fields in ((\"Child\", _CHILD_FIELDS), (\"Mother\", _MOTHER_FIELDS), (\"Father\", _FATHER_FIELDS)):\n for field in fields:\n value = rec.get(_RAW_FIELD_ALIAS.get(field, field.upper()))\n if value is None or str(value).strip() == \"\":\n continue\n matches = get_records(role, field, value)\n for table, cols in matches.items():\n if table == \"OBSERVATION\":\n if field in (\"mager\", \"fagecomb\", \"previs\", \"precare\", \"wtgain\") and \"value_as_number\" not in cols:\n # spreadsheet gives observation_concept_id (4028487 \"Maternal age\"\n # for mager, 4071357 \"Paternal Age\" for fagecomb/Child, 4265453\n # \"Age\" for fagecomb/Father, 46270506/43533800 for previs, 40771565\n # for precare, 40759199 \"Pregnancy weight.gain.current\" for wtgain)\n # but no paired value row; the field's own numeric value is the\n # natural value_as_number -- except when that value is itself one\n # of NCHS's \"unknown or not stated\" sentinels (see\n # _UNKNOWN_VALUE_SENTINEL), where backfilling would record the\n # sentinel as if it were a real measurement.\n backfill_value = _int_or_none(value)\n if (field, backfill_value) not in _UNKNOWN_VALUE_SENTINEL:\n cols = {**cols, \"value_as_number\": backfill_value}\n if field == \"oegest_comb\" and \"unit_concept_id\" not in cols:\n # spreadsheet's unit row for oegest_comb has Values=\"wk\" (a literal\n # string), which only matches if the raw value were literally \"wk\" --\n # it never matches oegest_comb's real numeric week value, so the\n # unit was never actually being set. oegest_comb's unit is always\n # weeks (concept 8511), so applied unconditionally here as an\n # exception rather than via the normal value-matching path.\n cols = {**cols, \"unit_concept_id\": 8511}\n yield \"OBSERVATION\", role, {\n \"observation_date\": _observation_date_for(field, value, dob_date),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"MEASUREMENT\":\n if field in (\"combgest\", \"dbwt\") and \"unit_concept_id\" not in cols:\n # same issue as oegest_comb's unit (see OBSERVATION branch above):\n # the spreadsheet's unit row has a literal Values (\"wk\"/\"g\") that\n # never matches the field's real numeric value, so the unit was\n # never actually being set. combgest is always weeks (8511), dbwt\n # is always grams (8504) -- applied unconditionally as an exception.\n cols = {**cols, \"unit_concept_id\": 8511 if field == \"combgest\" else 8504}\n if field == \"pwgt_r\" and \"unit_concept_id\" not in cols:\n # pwgt_r has no unit row in the spreadsheet at all -- its own\n # NBER_Concept_Name (\"Pre-pregnancy Weight Recode Weight in pounds\")\n # and UserGuide2022.pdf (position 292-294) both confirm the unit is\n # always pounds, same as wtgain's own unit (8739 \"pounds (US)\").\n # Applied unconditionally as an exception, same treatment as\n # combgest/dbwt/oegest_comb above. Not a settled fact -- see\n # questions_for_spreadsheet_owner.md.\n cols = {**cols, \"unit_concept_id\": 8739}\n if field in (\"m_ht_in\", \"bmi\", \"pwgt_r\", \"wtgain\") and \"value_as_number\" not in cols:\n # same shape as the OBSERVATION-table numeric-value gap above (mager/\n # fagecomb/previs/precare/wtgain): the spreadsheet has a row supplying\n # measurement_concept_id for the field's real-value range, but no row\n # at all supplying value_as_number -- the actual measured number\n # (height in inches, BMI, pre-pregnancy weight in pounds) was never\n # being recorded. wtgain's own value_as_number is already recorded on\n # OBSERVATION (see above); this only covers its separate, still-\n # incomplete MEASUREMENT row (missing measurement_concept_id -- see\n # questions_for_spreadsheet_owner.md).\n cols = {**cols, \"value_as_number\": _float_or_none(value)}\n yield \"MEASUREMENT\", role, {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"CONDITION\":\n yield \"CONDITION\", role, {\n \"condition_start_date\": dob_date,\n \"condition_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PROCEDURE\":\n yield \"PROCEDURE\", role, {\n \"procedure_date\": dob_date,\n \"procedure_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PAYER_PLAN_PERIOD\":\n # end_date left null rather than also set to dob_date -- the source\n # data has no real coverage-period dates at all (only \"which payer\"),\n # so forcing both dates to the birth date would assert a false\n # one-day coverage period. start_date must stay non-null (OMOP\n # requires it) and dob_date is the only date this data has to offer;\n # end_date left open is more honest than a fabricated end. Flagged\n # for confirmation in questions_for_spreadsheet_owner.md -- not a\n # settled fact.\n yield \"PAYER_PLAN_PERIOD\", role, {\n \"payer_plan_period_start_date\": dob_date,\n \"payer_plan_period_end_date\": None,\n **cols,\n }\n elif table == \"PROVIDER\":\n provider_id = PROVIDER_ID_BY_CONCEPT.get(cols.get(\"provider_concept_id\"))\n yield \"PROVIDER_USAGE\", role, {\"provider_id\": provider_id}\n\n # hand-computed OBSERVATION/MEASUREMENT rows: these fields' value_as_number row has a\n # non-numeric \"Integer\" placeholder Target_Vocabulary_Id (not a real vocab_id), so\n # load_mapping_rows() can't load it as a normal MappingRow -- but each field also has its\n # own separate, valid, numeric concept_id row (e.g. priorlive -> 3018989) that\n # get_records() resolves correctly. None of these 4 fields are in _MOTHER_FIELDS, so that\n # concept_id was never being looked up at all, leaving these rows without the concept_id\n # OMOP requires. Merged into one row per field here. previs is NOT handled here -- it's in\n # _CHILD_FIELDS/_MOTHER_FIELDS, so it already gets its concept_id row via the field-family\n # loop above; its value_as_number backfill happens there instead (see the OBSERVATION\n # branch above) to avoid emitting a second, duplicate row for the same fact.\n for field in (\"priorlive\", \"priordead\", \"rf_cesarn\"):\n v = _int_or_none(rec.get(field.upper()))\n if v is not None:\n fact = {\n \"observation_date\": dob_date,\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": v,\n }\n fact.update(get_records(\"Mother\", field, v).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # dlmp_mm/dlmp_yy -> one combined OBSERVATION row (spreadsheet's own Notes on\n # dlmp_mm: \"Combine dlmp_yy and dlmp_mm and assume 1st day of month\"), instead of\n # two independent per-field rows sharing the same concept (3002314). dlmp_yy's own\n # Values (\"2020\") is a single literal example, not a real range, so the concept is\n # looked up via dlmp_mm's row (Values \"1 to 12\") instead. DLMP_MM=99/DLMP_YY=9999\n # are the official \"unknown or not stated\" sentinels -- no date computed for those.\n dlmp_month = _int_or_none(rec.get(\"DLMP_MM\"))\n dlmp_year = _int_or_none(rec.get(\"DLMP_YY\"))\n if dlmp_month is not None and 1 <= dlmp_month <= 12 and dlmp_year is not None and dlmp_year != 9999:\n fact = {\n \"observation_date\": dt.date(dlmp_year, dlmp_month, 1),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n fact.update(get_records(\"Mother\", \"dlmp_mm\", dlmp_month).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # priorterm -> MEASUREMENT (spreadsheet's own table choice, see module docstring)\n priorterm = _int_or_none(rec.get(\"PRIORTERM\"))\n if priorterm is not None:\n fact = {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": priorterm,\n }\n fact.update(get_records(\"Mother\", \"priorterm\", priorterm).get(\"MEASUREMENT\", {}))\n yield \"MEASUREMENT\", \"Mother\", fact\n\n\ndef build_persons(rec: dict, record_idx: int) -> list[dict]:\n persons = []\n\n dob_year, dob_month, dob_day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n birth_datetime = None\n if dob_year is not None and dob_month is not None and dob_day is not None:\n # DOB_TT unknown/unparseable (incl. NCHS's \"9999\" sentinel) leaves\n # birth_datetime null rather than assuming midnight -- birth_datetime is\n # nullable in the destination schema, so no time is a truer statement than\n # an assumed one. See questions_for_spreadsheet_owner.md #15.\n dob_time = _parse_dob_time(rec.get(\"DOB_TT\"))\n if dob_time is not None:\n birth_datetime = dt.datetime(dob_year, dob_month, dob_day, dob_time[0], dob_time[1])\n child = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Child\"),\n \"year_of_birth\": dob_year,\n \"month_of_birth\": dob_month,\n \"day_of_birth\": dob_day,\n \"birth_datetime\": birth_datetime,\n }\n child.update(get_records(\"Child\", \"sex\", rec.get(\"SEX\")).get(\"PERSON\", {}))\n persons.append(child)\n\n mager = _int_or_none(rec.get(\"MAGER\"))\n mother = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Mother\"),\n \"year_of_birth\": (dob_year - mager) if (dob_year is not None and mager is not None) else None,\n }\n mother.update(get_records(\"Mother\", \"sex\", \"F\").get(\"PERSON\", {}))\n for field in (\"mrace6\", \"mhispx\", \"mracehisp\"):\n mother.update(get_records(\"Mother\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(mother)\n\n fagecomb = _int_or_none(rec.get(\"FAGECOMB\"))\n if fagecomb == 99:\n # NCHS's own \"unknown or not stated\" sentinel for FAGECOMB, not a real\n # age -- confirmed via the spreadsheet's own fagecomb=99 rows (\"Unknown\n # or Not Stated\" / \"Age unknown\"). Excluded here so the father's\n # year_of_birth isn't computed as if 99 were a real age.\n fagecomb = None\n father = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Father\"),\n \"year_of_birth\": (dob_year - fagecomb) if (dob_year is not None and fagecomb is not None) else None,\n }\n father.update(get_records(\"Father\", \"sex\", \"M\").get(\"PERSON\", {}))\n for field in (\"frace6\", \"fhispx\", \"fracehisp\"):\n father.update(get_records(\"Father\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(father)\n\n return persons\n\n\ndef build_visits(rec: dict, record_idx: int, dob_date: dt.date | None) -> list[dict]:\n visits = []\n for role in (\"Child\", \"Mother\"):\n match = get_records(role, \"bfacil\", rec.get(\"BFACIL\"))\n if \"VISIT_OCCURRENCE\" not in match:\n continue\n visit = {\n \"visit_occurrence_id\": IdGenerator.visit_occurrence_id(record_idx, role),\n \"person_id\": IdGenerator.person_id(record_idx, role),\n \"visit_start_date\": dob_date,\n \"visit_end_date\": dob_date,\n \"visit_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n # spreadsheet field name is \"Visit_Concept_id\" (case-as-published); normalize here\n for k, v in match[\"VISIT_OCCURRENCE\"].items():\n visit[k.lower()] = v\n visits.append(visit)\n return visits\n\n\n_ID_COLUMN = {\n \"OBSERVATION\": \"observation_id\",\n \"MEASUREMENT\": \"measurement_id\",\n \"CONDITION\": \"condition_occurrence_id\",\n \"PROCEDURE\": \"procedure_occurrence_id\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period_id\",\n}\n\nSQL_TABLE_NAME = {\n \"PERSON\": \"person\",\n \"VISIT_OCCURRENCE\": \"visit_occurrence\",\n \"OBSERVATION\": \"observation\",\n \"MEASUREMENT\": \"measurement\",\n \"CONDITION\": \"condition_occurrence\",\n \"PROCEDURE\": \"procedure_occurrence\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period\",\n}\n\nTABLE_COLUMNS = {\n \"PERSON\": [\"person_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\", \"birth_datetime\",\n \"gender_concept_id\", \"race_concept_id\", \"ethnicity_concept_id\"],\n \"VISIT_OCCURRENCE\": [\"visit_occurrence_id\", \"person_id\", \"visit_start_date\", \"visit_end_date\",\n \"visit_type_concept_id\", \"visit_concept_id\"],\n \"OBSERVATION\": ['person_id', 'observation_id', 'observation_date', 'observation_type_concept_id',\n 'observation_concept_id', 'qualifier_concept_id', 'unit_concept_id',\n 'value_as_concept_id', 'value_as_number'],\n \"MEASUREMENT\": ['person_id', 'measurement_id', 'measurement_date', 'measurement_type_concept_id',\n 'measurement_concept_id', 'unit_concept_id', 'value_as_concept_id', 'value_as_number'],\n \"CONDITION\": ['person_id', 'condition_occurrence_id', 'condition_start_date', 'condition_type_concept_id',\n 'condition_concept_id', 'condition_status_concept_id'],\n \"PROCEDURE\": ['person_id', 'procedure_occurrence_id', 'procedure_date', 'procedure_type_concept_id',\n 'procedure_concept_id'],\n \"PAYER_PLAN_PERIOD\": ['person_id', 'payer_plan_period_id', 'payer_plan_period_start_date',\n 'payer_plan_period_end_date', 'payer_concept_id'],\n}\n\n\ndef build_facts(rec: dict, record_idx: int, dob_date: dt.date | None) -> dict[str, list[dict]]:\n \"\"\"\n One pass over _iter_field_records per record, bucketed by target table --\n _iter_field_records computes matches for every table in a single\n generator pass. Covers OBSERVATION/MEASUREMENT/CONDITION/\n PROCEDURE/PAYER_PLAN_PERIOD only -- PERSON and VISIT_OCCURRENCE are\n built separately (build_persons/build_visits) since they're driven by\n a handful of targeted get_records() calls, not this field-family scan.\n \"\"\"\n counters: dict[str, int] = {}\n out: dict[str, list[dict]] = {table: [] for table in _ID_COLUMN}\n for table, role, fact in _iter_field_records(rec, dob_date):\n if table not in _ID_COLUMN:\n continue # e.g. PROVIDER_USAGE -- not one of this node's tables\n counters[table] = counters.get(table, 0) + 1\n out[table].append({\n \"person_id\": IdGenerator.person_id(record_idx, role),\n _ID_COLUMN[table]: IdGenerator.fact_id(record_idx, counters[table]),\n **fact,\n })\n return out\n\n\ndef _sanitize_cache_id(raw_id: str) -> str:\n \"\"\"\n Mirrors the platform's own Dataset.cacheId derivation\n (portal/src/dataset/entity/dataset.entity.ts: sanitizeIdForCacheId) --\n hyphens aren't valid in a bare SQL/DuckDB identifier, so the Dataset's\n UUID gets converted the same way here before use as a trex cache_id.\n \"\"\"\n cleaned = raw_id.replace(\"-\", \"_\")\n return f\"_{cleaned}\" if cleaned[:1].isdigit() else cleaned\n\n\ndef exec(myinput):\n \"\"\"\n Gold transform: reads the staging CSV written by node-load-bronze in\n chunks, builds the OMOP rows in Python, writes each chunk's rows\n per-table to a CSV in the shared trex-volume staging dir, and loads each\n with one `INSERT INTO <table> (cols) SELECT * FROM read_csv('<trex\n path>')` -- trex's DuckDB reads the file straight from disk at native\n speed. Each load statement is atomic, and row counts are verified per\n table at the end; a mismatch raises.\n \"\"\"\n import csv\n import time\n import uuid\n\n logger = get_run_logger()\n\n global _MAPPING_DF_FROM_NODE\n mapping_input = myinput.get(MAPPING_CSV_NODE)\n if mapping_input is None or mapping_input.result is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n if not isinstance(mapping_input.result, pd.DataFrame):\n raise RuntimeError(\n f\"Upstream mapping input {MAPPING_CSV_NODE!r} must be a pandas DataFrame, got \"\n f\"{type(mapping_input.result).__name__}\"\n )\n _MAPPING_DF_FROM_NODE = mapping_input.result.copy()\n\n _validate_mapping_against_cdm(load_mapping_rows(), logger)\n cache_catalog = _sanitize_cache_id(dataset_id)\n dbdao = DBDao(database_code=destination_database_code, cache_id=cache_catalog, dialect=SupportedDatabaseDialects.TREX)\n dest_schema = f\"{cache_catalog}.{destination_schema_name}\"\n\n # Input: the staging CSV written by node-load-bronze. The .complete marker\n # holds the row count and only exists if bronze finished; csv-without-\n # marker means an interrupted bronze run and is refused.\n src_fname = f\"{cache_catalog}_nat2022_raw.csv\"\n src_csv_path = os.path.join(STAGING_DIR_FLOW, src_fname)\n src_complete_path = src_csv_path + \".complete\"\n if not (os.path.exists(src_csv_path) and os.path.exists(src_complete_path)):\n raise RuntimeError(\n f\"Staging CSV from node-load-bronze not found or incomplete \"\n f\"({src_csv_path}; marker present: {os.path.exists(src_complete_path)}). \"\n f\"Run node-load-bronze first -- a CSV without its .complete marker means \"\n f\"the bronze run was interrupted mid-write.\"\n )\n with open(src_complete_path) as f:\n total = int(f.read().strip())\n\n for sql_table in SQL_TABLE_NAME.values():\n # destination_schema_name.<table> is created by the OMOP CDM plugin, not here --\n # truncate rather than drop so we never touch its schema/table definition.\n dbdao.truncate_table(dest_schema, sql_table)\n\n size = int(chunk_size)\n\n # Self-healing sweep: staging files are deleted right after each bulk\n # load, but a hard-killed run orphans its in-flight file. Current-run\n # filenames get a fresh run_token below, so anything matching these\n # prefixes now is stale by construction; other datasets' files\n # (different prefix) are never touched.\n table_prefixes = tuple(f\"{cache_catalog}_{t}_\" for t in SQL_TABLE_NAME.values())\n removed = 0\n for f in os.listdir(STAGING_DIR_FLOW):\n if f.startswith(table_prefixes) and f.endswith(\".csv\"):\n try:\n os.remove(os.path.join(STAGING_DIR_FLOW, f))\n removed += 1\n except OSError:\n pass\n if removed:\n logger.info(f\"Removed {removed} stale staging file(s) left by a previous interrupted run\")\n\n run_token = uuid.uuid4().hex\n\n def _bulk_load(sql_table, columns, values, tag):\n \"\"\"One CSV in the shared staging dir + one read_csv INSERT for this table-chunk.\"\"\"\n fname = f\"{cache_catalog}_{sql_table}_{tag}_{run_token}.csv\"\n fpath = os.path.join(STAGING_DIR_FLOW, fname)\n with open(fpath, \"w\", newline=\"\") as f:\n w = csv.writer(f)\n w.writerow(columns)\n for row in values:\n w.writerow([\"\" if v is None else v for v in row])\n col_list = \", \".join(f'\"{c}\"' for c in columns)\n try:\n dbdao.execute_sql(\n f\"INSERT INTO {dest_schema}.{sql_table} ({col_list}) \"\n f\"SELECT * FROM read_csv('{STAGING_DIR_TREX}/{fname}', header=true, all_varchar=true)\"\n )\n finally:\n os.remove(fpath)\n\n expected_counts = {table: 0 for table in SQL_TABLE_NAME}\n processed = 0\n # Sequential single-pass read of the staging CSV -- flat cost per chunk,\n # order preserved (ROW_IDX rides along as a column and record_idx comes\n # from its stored value, not from position). dtype=str + keep_default_na\n # =False gives '' for missing values, which the transform treats the same\n # as NULL.\n for chunk_idx, src_chunk in enumerate(pd.read_csv(src_csv_path, dtype=str, keep_default_na=False, chunksize=size)):\n t_chunk = time.time()\n rows_by_table = {table: [] for table in SQL_TABLE_NAME}\n for rec in src_chunk.to_dict(\"records\"):\n record_idx = int(rec[\"ROW_IDX\"])\n dob_date = dob_date_for(rec)\n rows_by_table[\"PERSON\"].extend(build_persons(rec, record_idx))\n rows_by_table[\"VISIT_OCCURRENCE\"].extend(build_visits(rec, record_idx, dob_date))\n built = build_facts(rec, record_idx, dob_date)\n for table, rows in built.items():\n rows_by_table[table].extend(rows)\n t_transform = time.time() - t_chunk\n\n chunk_rows = 0\n for table, sql_table in SQL_TABLE_NAME.items():\n columns = TABLE_COLUMNS[table]\n values = [tuple(row.get(col) for col in columns) for row in rows_by_table[table]]\n if not values:\n continue\n expected_counts[table] += len(values)\n chunk_rows += len(values)\n t_tbl = time.time()\n _bulk_load(sql_table, columns, values, str(chunk_idx))\n logger.info(f\" {sql_table}: {len(values):,} rows bulk-loaded in {time.time() - t_tbl:.1f}s\")\n\n processed += len(src_chunk)\n done = processed\n logger.info(\n f\"{done:,}/{total:,} records -> {chunk_rows:,} OMOP rows \"\n f\"(transform {t_transform:.0f}s, chunk total {time.time() - t_chunk:.0f}s)\"\n )\n\n if processed != total:\n raise RuntimeError(f\"staging CSV row count mismatch: marker says {total:,}, read {processed:,}\")\n\n mismatches = []\n for table, sql_table in SQL_TABLE_NAME.items():\n actual = int(dbdao.execute_sql(f\"SELECT COUNT(*) FROM {dest_schema}.{sql_table}\", fetch=True)[0][0])\n if actual != expected_counts[table]:\n mismatches.append(f\"{sql_table}: expected {expected_counts[table]:,}, table has {actual:,}\")\n if mismatches:\n raise RuntimeError(\"row-count verification failed -- \" + \"; \".join(mismatches))\n logger.info(\"Row-count verification passed for all tables: \"\n + \", \".join(f\"{SQL_TABLE_NAME[t]}={expected_counts[t]:,}\" for t in SQL_TABLE_NAME))\n\n return (f\"Loaded {total} records into person, visit_occurrence, observation, measurement, \"\n f\"condition_occurrence, procedure_occurrence, payer_plan_period\")" |
| "data": { | ||
| "name": "transform_facts", | ||
| "description": "Gold: build PERSON, VISIT_OCCURRENCE, OBSERVATION, MEASUREMENT, CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE, and PAYER_PLAN_PERIOD in a single per-record pass (merged from 7 originally separate nodes; see node_backups/ for the originals).", | ||
| "python_code": "# Deployment-topology constants (deliberately NOT pipeline variables) -- see\n# node-load-bronze for the full rationale: fixed by the platform's container\n# mounts; the two paths are the flow container's and trex's views of the same\n# shared staging directory on the trex data volume.\nNAT2022_DATA_DIR = \"/app/data_load\"\nSTAGING_DIR_FLOW = \"/app/duckdb_data/flow_staging\"\nSTAGING_DIR_TREX = \"/usr/src/data/flow_staging\"\n\nMAPPING_CSV_NODE = \"csv_node_0\"\n\n_PREFIX_GROUP_RE = re.compile(r'^(all )?variables beginning with\\s+(\\w+?)_?$', re.I)\n\n# Resolved relative to this file's own directory, not the caller's cwd --\n# matters both for local runs and once this module is volume-mounted into a\n# flow-run container alongside the csv (see flowNodes/ directory).\n_TABLE_NORMALIZE = {\n \"PPROCEDURE\": \"PROCEDURE\",\n}\n\n_FIELD_NAME_NORMALIZE = {\n \"observation_date\": \"observation_concept_id\", # mislabeled in spreadsheet; see module docstring\n \"unit_as_concept_id\": \"unit_concept_id\", # spreadsheet typo -- no \"_as_\" in the real OMOP column name\n}\n\n# (person_role, source_field, raw Values string) -> corrected Values string.\n# precare's Child row has Values=\"1\" -- a single literal value -- even though\n# its own NBER_Concept_Name says \"Month Prenatal Care Began 01 - 10\" and\n# UserGuide2022.pdf documents PRECARE's real range as 01-10. fagecomb's Child\n# row has Values=\"9\" even though its own NBER_Concept_Name says \"09-98\" and\n# UserGuide2022.pdf documents FAGECOMB's real range as 09-98. Applied\n# narrowly (exact role+field+raw-value match) so it can't accidentally\n# affect any other row. Interpretation flagged for spreadsheet-owner\n# confirmation in questions_for_spreadsheet_owner.md -- not a settled fact,\n# best current guess.\n_VALUES_NORMALIZE = {\n (\"Child\", \"precare\", \"1\"): \"1-10\",\n (\"Child\", \"fagecomb\", \"9\"): \"9-98\",\n # previs's Values=\"12\" only matched a raw visit count of exactly 12, leaving\n # value_as_concept_id null for every other real count. UserGuide2022.pdf documents\n # PREVIS's valid range as 00-98, and previs=0 already has its own dedicated row\n # (\"No prenatal care\"), so widened to 1-98 -- matching the Mother-role sibling row's\n # own already-correct \"1 to 98\" range for this same concept.\n (\"Child\", \"previs\", \"12\"): \"1-98\",\n # Mother's precare row (observation_date -> observation_concept_id 40771565) only\n # matched literal \"2\", even though its own Notes describe a general subtract-months\n # formula (see _observation_date_for) meant to apply to any value in precare's real\n # 1-10 range. 0 already has its own dedicated \"No prenatal care\" row.\n (\"Mother\", \"precare\", \"2\"): \"1-10\",\n}\n\n_REAL_CDM_COLUMNS = {\n \"PERSON\": {\"person_id\", \"gender_concept_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\",\n \"birth_datetime\", \"race_concept_id\", \"ethnicity_concept_id\", \"location_id\", \"provider_id\",\n \"care_site_id\", \"person_source_value\", \"gender_source_value\", \"gender_source_concept_id\",\n \"race_source_value\", \"race_source_concept_id\", \"ethnicity_source_value\",\n \"ethnicity_source_concept_id\"},\n \"VISIT_OCCURRENCE\": {\"visit_occurrence_id\", \"person_id\", \"visit_concept_id\", \"visit_start_date\",\n \"visit_start_datetime\", \"visit_end_date\", \"visit_end_datetime\",\n \"visit_type_concept_id\", \"provider_id\", \"care_site_id\", \"visit_source_value\",\n \"visit_source_concept_id\", \"admitted_from_concept_id\", \"admitted_from_source_value\",\n \"discharged_to_concept_id\", \"discharged_to_source_value\",\n \"preceding_visit_occurrence_id\"},\n \"OBSERVATION\": {\"observation_id\", \"person_id\", \"observation_concept_id\", \"observation_date\",\n \"observation_datetime\", \"observation_type_concept_id\", \"value_as_number\",\n \"value_as_string\", \"value_as_concept_id\", \"qualifier_concept_id\", \"unit_concept_id\",\n \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\", \"observation_source_value\",\n \"observation_source_concept_id\", \"unit_source_value\", \"qualifier_source_value\",\n \"value_source_value\", \"observation_event_id\", \"obs_event_field_concept_id\"},\n \"MEASUREMENT\": {\"measurement_id\", \"person_id\", \"measurement_concept_id\", \"measurement_date\",\n \"measurement_datetime\", \"measurement_time\", \"measurement_type_concept_id\",\n \"operator_concept_id\", \"value_as_number\", \"value_as_concept_id\", \"unit_concept_id\",\n \"range_low\", \"range_high\", \"provider_id\", \"visit_occurrence_id\", \"visit_detail_id\",\n \"measurement_source_value\", \"measurement_source_concept_id\", \"unit_source_value\",\n \"unit_source_concept_id\", \"value_source_value\", \"measurement_event_id\",\n \"meas_event_field_concept_id\"},\n \"CONDITION\": {\"condition_occurrence_id\", \"person_id\", \"condition_concept_id\", \"condition_start_date\",\n \"condition_start_datetime\", \"condition_end_date\", \"condition_end_datetime\",\n \"condition_type_concept_id\", \"condition_status_concept_id\", \"stop_reason\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"condition_source_value\",\n \"condition_source_concept_id\", \"condition_status_source_value\"},\n \"PROCEDURE\": {\"procedure_occurrence_id\", \"person_id\", \"procedure_concept_id\", \"procedure_date\",\n \"procedure_datetime\", \"procedure_end_date\", \"procedure_end_datetime\",\n \"procedure_type_concept_id\", \"modifier_concept_id\", \"quantity\", \"provider_id\",\n \"visit_occurrence_id\", \"visit_detail_id\", \"procedure_source_value\",\n \"procedure_source_concept_id\", \"modifier_source_value\"},\n \"PAYER_PLAN_PERIOD\": {\"payer_plan_period_id\", \"person_id\", \"payer_plan_period_start_date\",\n \"payer_plan_period_end_date\", \"payer_concept_id\", \"payer_source_value\",\n \"payer_source_concept_id\", \"plan_concept_id\", \"plan_source_value\",\n \"plan_source_concept_id\", \"sponsor_concept_id\", \"sponsor_source_value\",\n \"sponsor_source_concept_id\", \"family_source_value\", \"stop_reason_concept_id\",\n \"stop_reason_source_value\", \"stop_reason_source_concept_id\"},\n \"PROVIDER\": {\"provider_id\", \"provider_name\", \"npi\", \"dea\", \"specialty_concept_id\", \"care_site_id\",\n \"year_of_birth\", \"gender_concept_id\", \"provider_source_value\", \"specialty_source_value\",\n \"specialty_source_concept_id\", \"gender_source_value\", \"gender_source_concept_id\"},\n}\n\n# (Target_Table, Field_Name) pairs already confirmed as spreadsheet-side issues --\n# tracked in questions_for_spreadsheet_owner.md with an existing code-level\n# workaround. Logged as a warning but doesn't halt the pipeline. Anything NOT\n# in this set is treated as a new, unreviewed problem and raises immediately.\n_KNOWN_MAPPING_ISSUES = {\n (\"MEASUREMENT\", \"qualifier_concept_id\"), # questions doc #10 -- dropped from MEASUREMENT's own TABLE_COLUMNS\n (\"PROVIDER\", \"provider_concept_id\"), # questions doc #14 -- PROVIDER_DIMENSION hardcodes specialty_concept_id instead\n}\n\n\ndef _validate_mapping_against_cdm(rows: \"tuple[MappingRow, ...]\", logger) -> None:\n \"\"\"\n Checks every spreadsheet row's (Target_Table, Field_Name) against the real\n OMOP CDM 5.4 column list for that table, so a spreadsheet edit (typo,\n mislabeled column, wrong table name) is caught once, up front, before it\n silently reaches 3M+ records -- rather than requiring the same manual\n cross-referencing against the DDL that found the issues already tracked\n in questions_for_spreadsheet_owner.md.\n \"\"\"\n unexpected = []\n for row in rows:\n valid_columns = _REAL_CDM_COLUMNS.get(row.target_table)\n key = (row.target_table, row.field_name)\n if valid_columns is None:\n if key not in _KNOWN_MAPPING_ISSUES:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: unrecognized Target_Table {row.target_table!r}\"\n )\n continue\n if row.field_name.lower() not in valid_columns:\n if key in _KNOWN_MAPPING_ISSUES:\n logger.warning(\n \"Known spreadsheet issue (see questions_for_spreadsheet_owner.md): \"\n f\"{row.source_field}/{row.person_role} sets Field_Name={row.field_name!r} on \"\n f\"{row.target_table}, which isn't a real column there.\"\n )\n else:\n unexpected.append(\n f\"{row.source_field}/{row.person_role}: Field_Name {row.field_name!r} is not a real \"\n f\"column on {row.target_table} (spreadsheet concept: {row.concept_name!r})\"\n )\n if unexpected:\n raise ValueError(\n \"Mapping spreadsheet has unrecognized Target_Table/Field_Name combinations not seen before \"\n \"-- check for a new spreadsheet edit or typo before proceeding:\\n\"\n + \"\\n\".join(f\" - {e}\" for e in unexpected)\n )\n\n\n\n@dataclass(frozen=True)\nclass MappingRow:\n source_field: str # lowercase NBER field name, e.g. \"rf_pdiab\"\n person_role: str # \"Child\", \"Mother\", or \"Father\"\n target_table: str # normalized OMOP table name, e.g. \"OBSERVATION\"\n field_name: str # OMOP column this row supplies, e.g. \"value_as_concept_id\"\n concept_id: int\n concept_name: str\n matcher: \"ValueMatcher\"\n\n\ndef _is_number(s: str) -> bool:\n \"\"\"True for anything float() accepts, e.g. \"9\", \"-5\", \"69.9\" -- used\n instead of str.isdigit() so decimal Values (e.g. bmi's \"13-69.9\") parse\n as ranges/exact values instead of falling through to literal-string\n matching, which no real decimal value would ever hit.\"\"\"\n try:\n float(s)\n return True\n except ValueError:\n return False\n\n\nclass ValueMatcher:\n \"\"\"\n Parses the spreadsheet's free-text Values column into something\n callable against an actual field value from the data.\n\n Recognized forms: blank/\"any\" (always matches -- used for field-level\n \"header\" rows that apply regardless of the specific code), exact\n tokens separated by \"or\"/\",\", and numeric ranges (\"0-30\", \"1 to 98\",\n \"13-69.9\" -- bounds/values may be decimal, e.g. bmi).\n \"\"\"\n\n def __init__(self, raw: str):\n raw = (raw or \"\").strip()\n self.raw = raw\n self.kind: str\n if raw == \"\" or raw.lower() == \"any\":\n self.kind = \"always\"\n return\n if raw.startswith(\">\") or raw.startswith(\"<\"):\n bound = raw[1:].strip()\n if _is_number(bound):\n self.kind = \"inequality\"\n self.op = raw[0]\n self.bound = float(bound)\n return\n low = raw.lower().replace(\" to \", \"-\")\n parts = [p.strip() for p in low.split(\"-\")]\n if len(parts) == 2 and all(_is_number(p) for p in parts):\n self.kind = \"range\"\n self.lo = float(parts[0])\n self.hi = float(parts[1])\n return\n self.kind = \"exact\"\n self.tokens = {t.strip().lower() for t in raw.replace(\",\", \" or \").split(\" or \")}\n\n def matches(self, value) -> bool:\n if value is None:\n return False\n value_s = str(value).strip()\n if value_s == \"\":\n return False\n if self.kind == \"always\":\n return True\n if self.kind == \"range\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return self.lo <= v <= self.hi\n if self.kind == \"inequality\":\n try:\n v = float(value_s)\n except ValueError:\n return False\n return v > self.bound if self.op == \">\" else v < self.bound\n if value_s.lower() in self.tokens:\n return True\n # Raw fields are read with dtype=str (pd.read_fwf), so numeric codes\n # keep their fixed-width zero-padding (e.g. \"09\") -- compare\n # numerically too so an exact token like \"9\" still matches \"09\",\n # and a decimal token like \"99.9\" still matches a decimal value.\n if _is_number(value_s):\n return any(_is_number(t) and float(t) == float(value_s) for t in self.tokens)\n return False\n\n\ndef _normalize_table(raw: str) -> str:\n t = raw.strip().upper().replace(\" \", \"_\")\n return _TABLE_NORMALIZE.get(t, t)\n\n\n_MAPPING_ROWS: tuple[MappingRow, ...] | None = None\n_PREFIX_GROUP_DEFAULTS: dict[tuple[str, str], int] | None = None\n_FIELD_INDEX: dict[tuple[str, str], list[MappingRow]] | None = None\n_MAPPING_DF_FROM_NODE: \"pd.DataFrame | None\" = None\n\n\ndef _load_mapping_df() -> \"pd.DataFrame\":\n if _MAPPING_DF_FROM_NODE is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n return _MAPPING_DF_FROM_NODE.copy()\n\n\ndef load_mapping_rows() -> tuple[MappingRow, ...]:\n global _MAPPING_ROWS\n if _MAPPING_ROWS is not None:\n return _MAPPING_ROWS\n\n out = []\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n nber_field, values, _nber_name, field_name, target_table, vocab_id, concept_name = r[:7]\n person_role = (r[11] or \"\").strip()\n if not person_role or not target_table or not field_name:\n continue\n vocab_id_s = str(vocab_id).strip() if vocab_id is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue # computed/imputed row, handled separately in omop_transform.py\n if _PREFIX_GROUP_RE.match(str(nber_field).strip()):\n continue # prefix-group default row, handled by _prefix_group_defaults below\n field_name_s = _FIELD_NAME_NORMALIZE.get(str(field_name).strip(), str(field_name).strip())\n source_field_s = str(nber_field).strip().lower()\n values_s = \"\" if values is None else str(values)\n values_s = _VALUES_NORMALIZE.get((person_role, source_field_s, values_s.strip()), values_s)\n out.append(MappingRow(\n source_field=source_field_s,\n person_role=person_role,\n target_table=_normalize_table(str(target_table)),\n field_name=field_name_s,\n concept_id=int(vocab_id_s),\n concept_name=str(concept_name).strip() if concept_name else \"\",\n matcher=ValueMatcher(values_s),\n ))\n _MAPPING_ROWS = tuple(out)\n return _MAPPING_ROWS\n\n\ndef _prefix_group_defaults() -> dict[tuple[str, str], int]:\n \"\"\"(person_role, prefix) -> default observation_concept_id for that field family.\"\"\"\n global _PREFIX_GROUP_DEFAULTS\n if _PREFIX_GROUP_DEFAULTS is not None:\n return _PREFIX_GROUP_DEFAULTS\n\n out = {}\n for r in _load_mapping_df().itertuples(index=False, name=None):\n if not r or not r[0]:\n continue\n m = _PREFIX_GROUP_RE.match(str(r[0]).strip())\n if not m:\n continue\n vocab_id_s = str(r[5]).strip() if r[5] is not None else \"\"\n if not vocab_id_s.lstrip(\"-\").isdigit():\n continue\n person_role = (r[11] or \"\").strip()\n prefix = m.group(2).lower()\n out[(person_role, prefix)] = int(vocab_id_s)\n _PREFIX_GROUP_DEFAULTS = out\n return _PREFIX_GROUP_DEFAULTS\n\n\ndef _index_by_field() -> dict[tuple[str, str], list[MappingRow]]:\n global _FIELD_INDEX\n if _FIELD_INDEX is not None:\n return _FIELD_INDEX\n\n index: dict[tuple[str, str], list[MappingRow]] = {}\n for row in load_mapping_rows():\n index.setdefault((row.person_role, row.source_field), []).append(row)\n _FIELD_INDEX = index\n return _FIELD_INDEX\n\n\n# (source_field, field_name, concept_id) rows that should be treated as an \"Integer\n# example\" placeholder (substitute the real value in) even though their\n# Target_Concept_Name isn't literally \"Integer example\". apgar5's value_as_number row\n# has a real concept id (3004221, \"5 minute Apgar Score\") in that slot instead of the\n# placeholder pattern used elsewhere (oegest_comb/dbwt/combgest), so without this it\n# stores the literal concept id 3004221 instead of the actual apgar score. Flagged in\n# questions_for_spreadsheet_owner.md -- not a settled fact, best current guess.\n_INTEGER_PLACEHOLDER_OVERRIDE = {\n (\"apgar5\", \"value_as_number\", 3004221),\n (\"apgar10\", \"value_as_number\", 3016162),\n}\n\n# no_mmorb (\"no maternal morbidity reported\") answers the same underlying survey\n# question as the mm_* fields (its own answer concept is literally \"None of the\n# above (maternal morbidity)\"), and would pick up the \"Variables beginning with\n# mm_\" header row's observation_concept_id (43533805) via the normal prefix-group\n# match below -- except its own field name starts with \"no_\", not \"mm_\", so the\n# prefix never matches. Applied directly instead. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_OBSERVATION_CONCEPT_OVERRIDE = {\n (\"Mother\", \"no_mmorb\"): 43533805,\n # no_abnorm (\"no abnormal conditions of the newborn\") is the same underlying\n # survey question as the ab_* fields (its own answer concept is literally \"None\n # of the above\"), and would pick up the \"Variables beginning with ab_\" header\n # row's observation_concept_id (43533835) via the normal prefix-group match --\n # except its own field name starts with \"no_\", not \"ab_\", so the prefix never\n # matches. Applied directly instead, same shape as no_mmorb above. Not a settled\n # fact -- see questions_for_spreadsheet_owner.md.\n (\"Child\", \"no_abnorm\"): 43533835,\n # no_congen (\"no congenital anomalies of the newborn checked\") is the same\n # underlying survey question as the ca_* fields, and would pick up the\n # \"Variables beginning with ca_\" header row's observation_concept_id\n # (43533804) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ca_\". Same shape as no_mmorb/no_abnorm above.\n (\"Child\", \"no_congen\"): 43533804,\n # no_lbrdlv (\"no characteristics of labor and delivery reported\") is the\n # same underlying survey question as the ld_* fields, and would pick up the\n # \"Variables beginning with LD_\" header row's observation_concept_id\n # (43533836) via the normal prefix-group match -- except its own field name\n # starts with \"no_\", not \"ld_\". Same shape as no_mmorb/no_abnorm/no_congen\n # above.\n (\"Mother\", \"no_lbrdlv\"): 43533836,\n # wtgain's own \"99 = Unknown or not stated\" row supplies only a\n # qualifier_concept_id (4129922, \"unknown\"), unlike its real-value row\n # (00-97), which supplies observation_concept_id (40759199) directly.\n # Applied directly so the unknown case still identifies itself as a\n # pregnancy weight-gain observation, same concept as the real-value case.\n (\"Mother\", \"wtgain\"): 40759199,\n}\n\n# (person_role, source_field, field_name, concept_id from spreadsheet) ->\n# replacement concept_id. fagecomb's Child-role \"99 = Unknown or Not Stated\"\n# row supplies observation_concept_id=0 (\"No matching concept\") -- unlike\n# every other 0-row in the spreadsheet (bfacil/fhispx/mracehisp/dmar/attend\n# etc.), all of which land in an *answer* column (value_as_concept_id,\n# ethnicity_concept_id, provider_concept_id) where 0 legitimately means \"the\n# answer is Other/Unknown,\" this one lands in the *identifying*\n# observation_concept_id column, so it doesn't say what the observation is\n# about at all. Remapped to 4071357 (\"Paternal Age\"), the same concept\n# fagecomb's own Child-role real-value row already uses -- matching how\n# fagecomb's Father-role row already reuses one concept (4265453) for both\n# its real-value and its own 99-unknown case. Not a settled fact -- see\n# questions_for_spreadsheet_owner.md.\n_CONCEPT_ID_REMAP = {\n (\"Child\", \"fagecomb\", \"observation_concept_id\", 0): 4071357,\n # itran (\"Infant Transferred\") and ilive (\"Infant Living at Time of Report\") both\n # mapped to observation_concept_id 43533781 in the spreadsheet (questions_for_\n # spreadsheet_owner.md #7) -- confirmed by the spreadsheet owner as genuinely\n # different concepts (an ATHENA hierarchy display issue caused the mix-up).\n # itran's corrected code is 43533782 (\"Infant was transferred within 24 hr of\n # delivery\"); ilive's correct code is still pending confirmation, so ilive is\n # left unchanged at 43533781 for now.\n (\"Child\", \"itran\", \"observation_concept_id\", 43533781): 43533782,\n}\n\n# (source_field, sentinel value) pairs where the raw value is NCHS's own\n# \"unknown or not stated\" placeholder rather than a real measurement --\n# confirmed via each field's own Values=99 row (wtgain: \"Unknown or not\n# stated\"; fagecomb: \"Unknown or Not Stated\" / \"Age unknown\"; previs:\n# \"Unknown or Not Stated\"). Excluded from the value_as_number backfill below\n# so \"unknown\" doesn't get recorded as a literal, implausible fact (e.g.\n# wtgain=99 read as \"gained 99 lbs\", fagecomb=99 read as \"father is 99\").\n# previs=99 found via the same check while fixing wtgain -- not itself\n# reported, flagged here for visibility.\n_UNKNOWN_VALUE_SENTINEL = {\n (\"fagecomb\", 99),\n (\"wtgain\", 99),\n (\"previs\", 99),\n}\n\n\ndef get_records(person_role: str, source_field: str, value):\n \"\"\"\n Returns {target_table: {field_name: concept_id, ...}, ...} for every\n target table that has at least one matching mapping row for this\n (person_role, source_field, value). Rows with no value match (e.g. the\n field is blank/not reported) contribute nothing.\n \"\"\"\n source_field = source_field.lower()\n candidates = _index_by_field().get((person_role, source_field), [])\n out: dict[str, dict[str, int]] = {}\n for row in candidates:\n if row.matcher.matches(value):\n concept_id = row.concept_id\n is_placeholder = (\n row.concept_name.strip().lower() == \"integer example\"\n or (row.source_field, row.field_name, row.concept_id) in _INTEGER_PLACEHOLDER_OVERRIDE\n )\n if is_placeholder:\n try:\n concept_id = int(str(value).strip())\n except ValueError:\n continue\n concept_id = _CONCEPT_ID_REMAP.get(\n (person_role, source_field, row.field_name, concept_id), concept_id\n )\n out.setdefault(row.target_table, {})[row.field_name] = concept_id\n\n if \"OBSERVATION\" in out and \"observation_concept_id\" not in out[\"OBSERVATION\"]:\n override = _OBSERVATION_CONCEPT_OVERRIDE.get((person_role, source_field))\n if override is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = override\n else:\n prefix = source_field.split(\"_\")[0]\n default = _prefix_group_defaults().get((person_role, prefix))\n if default is not None:\n out[\"OBSERVATION\"][\"observation_concept_id\"] = default\n\n return out\n\n\n\n\nDEFAULT_TYPE_CONCEPT_ID = 0 # TODO: confirm correct *_type_concept_id with vocabulary before use\n\nWEEKDAY_CODE_TO_PY = {1: 6, 2: 0, 3: 1, 4: 2, 5: 3, 6: 4, 7: 5} # 1=Sunday..7=Saturday -> Python Monday=0..Sunday=6\n\n\ndef _int_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return int(s)\n except ValueError:\n return None\n\n\ndef _float_or_none(value):\n s = \"\" if value is None else str(value).strip()\n if s == \"\":\n return None\n try:\n return float(s)\n except ValueError:\n return None\n\n\ndef _impute_child_dob(dob_yy, dob_mm, dob_wk):\n year, month, wk = _int_or_none(dob_yy), _int_or_none(dob_mm), _int_or_none(dob_wk)\n if year is None or month is None:\n return None, None, None\n if wk is None or wk not in WEEKDAY_CODE_TO_PY:\n return year, month, None\n target_py_weekday = WEEKDAY_CODE_TO_PY[wk]\n first_of_month = dt.date(year, month, 1)\n offset = (target_py_weekday - first_of_month.weekday()) % 7\n return year, month, 1 + offset\n\n\ndef _parse_dob_time(dob_tt) -> tuple[int, int] | None:\n \"\"\"DOB_TT is a raw 4-digit HHMM code (e.g. '0830'); '9999' means not stated.\"\"\"\n s = \"\" if dob_tt is None else str(dob_tt).strip()\n if len(s) != 4 or not s.isdigit() or s == \"9999\":\n return None\n hour, minute = int(s[:2]), int(s[2:])\n if not (0 <= hour <= 23 and 0 <= minute <= 59):\n return None\n return hour, minute\n\n\ndef dob_date_for(rec):\n year, month, day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n if year and month and day:\n return dt.date(year, month, day)\n return None\n\n\ndef _subtract_months(base_date, months):\n total = base_date.year * 12 + (base_date.month - 1) - months\n year, month = divmod(total, 12)\n day = min(base_date.day, 28)\n return dt.date(year, month + 1, day)\n\n\nclass IdGenerator:\n # Dense, chunk-safe id allocation: exactly 3 roles per record, so\n # record_idx*3 + offset + 1 produces 1,2,3,4,5,6,... with no gaps,\n # while still being computable independently per record (no shared\n # counter needed across chunks/workers).\n PERSON_ROLE_OFFSET = {\"Child\": 0, \"Mother\": 1, \"Father\": 2}\n\n @staticmethod\n def person_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def visit_occurrence_id(record_idx, role):\n return record_idx * 3 + IdGenerator.PERSON_ROLE_OFFSET[role] + 1\n\n @staticmethod\n def fact_id(record_idx, local_counter):\n # 500-id block per record: the destination CDM's fact id columns\n # (observation_id etc.) are 32-bit INTEGER (max 2,147,483,647) and the\n # full Nat2022 file has 3,676,029 records, so the block size must keep\n # the largest id under 2^31 (500 tops out at ~1.84e9) while exceeding\n # the most facts one record produces in a table (~60 observations).\n return (record_idx + 1) * 500 + local_counter\n\n\nPROVIDER_DIMENSION = [\n {\"provider_concept_id\": 38004446, \"provider_source_value\": \"Physician\"},\n {\"provider_concept_id\": 38003822, \"provider_source_value\": \"Osteopathic Practitioner\"},\n {\"provider_concept_id\": 38004482, \"provider_source_value\": \"CNM/CM\"},\n {\"provider_concept_id\": 38003807, \"provider_source_value\": \"Other midwife\"},\n {\"provider_concept_id\": 0, \"provider_source_value\": \"Other/Unknown\"},\n]\nPROVIDER_ID_BY_CONCEPT = {row[\"provider_concept_id\"]: i + 1 for i, row in enumerate(PROVIDER_DIMENSION)}\n\n\n\n_CHILD_FIELDS = [\n # dmeth_rec deliberately excluded -- spreadsheet owner confirmed it's a collapsed\n # duplicate of rdmeth_rec (dmeth_rec=1 combines rdmeth_rec 1/2/5, dmeth_rec=2\n # combines rdmeth_rec 3/4/6) and to prefer rdmeth_rec, which has more granularity.\n # See questions_for_spreadsheet_owner.md #9.\n # mtran deliberately excluded -- spreadsheet owner confirmed mtran (\"Mother\n # Transferred\") belongs on the mother's own record only, not the child's; itran\n # (\"Infant Transferred\") is the child's own, separate fact. See new_findings.md.\n \"ab_anti\", \"ab_aven1\", \"ab_aven6\", \"ab_nicu\", \"ab_seiz\", \"ab_surf\",\n \"attend\", \"bfed\", \"ca_anen\", \"ca_cchd\", \"ca_cdh\", \"ca_cleft\", \"ca_clpal\",\n \"ca_disor\", \"ca_downs\", \"ca_gast\", \"ca_hypo\", \"ca_limb\", \"ca_mnsb\", \"ca_omph\",\n \"dmar\", \"dplural\", \"fagecomb\", \"ilive\", \"itran\", \"ld_indl\",\n \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\", \"no_abnorm\",\n \"no_congen\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\", \"pay\", \"precare\",\n \"previs\", \"rdmeth_rec\", \"setorder_r\", \"wic\", \"apgar5\", \"apgar10\",\n \"combgest\", \"dbwt\",\n]\n_MOTHER_FIELDS = [\n # dlmp_mm/dlmp_yy deliberately excluded -- combined into one hand-computed\n # OBSERVATION row below instead of two independent per-field rows.\n # dmeth_rec deliberately excluded -- see _CHILD_FIELDS comment above; same\n # rdmeth_rec-preferred resolution applies to Mother's own copy.\n \"attend\", \"cig0_r\", \"cig1_r\", \"cig2_r\", \"cig3_r\",\n \"dmar\", \"illb_r\", \"ilop_r\", \"ip_chlam\", \"ip_gon\", \"ip_hepatb\", \"ip_hepatc\",\n \"ip_syph\", \"ld_anes\", \"ld_antb\", \"ld_augm\", \"ld_chor\", \"ld_indl\", \"ld_ster\",\n \"m_ht_in\", \"mager\", \"mar_p\", \"me_pres\", \"me_rout\", \"me_trial\", \"meduc\",\n \"mm_aicu\", \"mm_mtr\", \"mm_plac\", \"mm_rupt\", \"mm_uhyst\", \"mrace15\", \"mtran\",\n \"no_lbrdlv\", \"no_mmorb\", \"no_risks\", \"ob_ecvf\", \"ob_ecvs\", \"oegest_comb\",\n \"pay\", \"precare\", \"previs\", \"rdmeth_rec\", \"rf_artec\", \"rf_cesar\",\n \"rf_ehype\", \"rf_fedrg\", \"rf_gdiab\", \"rf_ghype\", \"rf_inftr\", \"rf_pdiab\",\n \"rf_phype\", \"rf_ppterm\", \"sex\", \"setorder_r\", \"wic\", \"wtgain\", \"bmi\",\n \"combgest\", \"dbwt\", \"pwgt_r\", \"dplural\",\n]\n_FATHER_FIELDS = [\"fagecomb\", \"feduc\", \"frace15\"]\n\n\ndef _observation_date_for(field: str, value, dob_date: dt.date | None) -> dt.date | None:\n # illb_r/ilop_r: codes 0-3 = plural delivery (use child's dob); 4-300 = months\n # since last live birth/pregnancy outcome (subtract from child's dob). See\n # module docstring.\n if field in (\"illb_r\", \"ilop_r\") and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 4 <= v <= 300:\n return _subtract_months(dob_date, v)\n # precare (Mother): spreadsheet's own Notes on this row say \"Impute by Subtracting\n # value from birth month in months\" -- same subtract-months formula as illb_r/ilop_r,\n # applied to precare's real 1-10 range (0 = no prenatal care, handled by its own\n # separate row and never reaches here).\n if field == \"precare\" and dob_date is not None:\n v = _int_or_none(value)\n if v is not None and 1 <= v <= 10:\n return _subtract_months(dob_date, v)\n return dob_date\n\n\n# Spreadsheet NBER Field names that don't match the raw bronze column name for the\n# same field (source_field.upper() -> real raw column). Confirmed against both\n# node-load-bronze's FIELDS list and UserGuide2022.pdf: the real fields are\n# CA_DOWN/IP_HEPB/IP_HEPC, not CA_DOWNS/IP_HEPATB/IP_HEPATC. Without this, rec.get()\n# always misses and these 3 fields silently never produce any record, on any real\n# data -- verified via _iter_field_records() returning [] for all three. See\n# questions_for_spreadsheet_owner.md.\n_RAW_FIELD_ALIAS = {\n \"ca_downs\": \"CA_DOWN\",\n \"ip_hepatb\": \"IP_HEPB\",\n \"ip_hepatc\": \"IP_HEPC\",\n}\n\n\ndef _iter_field_records(rec: dict, dob_date: dt.date | None):\n \"\"\"\n Yields (target_table, role, fact_dict) for every field/value/role match\n found via the spreadsheet-driven mapping, across all of OBSERVATION,\n MEASUREMENT, CONDITION, PROCEDURE, PAYER_PLAN_PERIOD, and PROVIDER.\n\n Each of the 8 build_<table>() functions below filters this same stream\n down to its own table, rather than duplicating the field-matching logic\n once per table -- the per-table split is purely about which node emits\n which rows, not a second independent implementation of the mapping.\n \"\"\"\n for role, fields in ((\"Child\", _CHILD_FIELDS), (\"Mother\", _MOTHER_FIELDS), (\"Father\", _FATHER_FIELDS)):\n for field in fields:\n value = rec.get(_RAW_FIELD_ALIAS.get(field, field.upper()))\n if value is None or str(value).strip() == \"\":\n continue\n matches = get_records(role, field, value)\n for table, cols in matches.items():\n if table == \"OBSERVATION\":\n if field in (\"mager\", \"fagecomb\", \"previs\", \"precare\", \"wtgain\") and \"value_as_number\" not in cols:\n # spreadsheet gives observation_concept_id (4028487 \"Maternal age\"\n # for mager, 4071357 \"Paternal Age\" for fagecomb/Child, 4265453\n # \"Age\" for fagecomb/Father, 46270506/43533800 for previs, 40771565\n # for precare, 40759199 \"Pregnancy weight.gain.current\" for wtgain)\n # but no paired value row; the field's own numeric value is the\n # natural value_as_number -- except when that value is itself one\n # of NCHS's \"unknown or not stated\" sentinels (see\n # _UNKNOWN_VALUE_SENTINEL), where backfilling would record the\n # sentinel as if it were a real measurement.\n backfill_value = _int_or_none(value)\n if (field, backfill_value) not in _UNKNOWN_VALUE_SENTINEL:\n cols = {**cols, \"value_as_number\": backfill_value}\n if field == \"oegest_comb\" and \"unit_concept_id\" not in cols:\n # spreadsheet's unit row for oegest_comb has Values=\"wk\" (a literal\n # string), which only matches if the raw value were literally \"wk\" --\n # it never matches oegest_comb's real numeric week value, so the\n # unit was never actually being set. oegest_comb's unit is always\n # weeks (concept 8511), so applied unconditionally here as an\n # exception rather than via the normal value-matching path.\n cols = {**cols, \"unit_concept_id\": 8511}\n yield \"OBSERVATION\", role, {\n \"observation_date\": _observation_date_for(field, value, dob_date),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"MEASUREMENT\":\n if field in (\"combgest\", \"dbwt\") and \"unit_concept_id\" not in cols:\n # same issue as oegest_comb's unit (see OBSERVATION branch above):\n # the spreadsheet's unit row has a literal Values (\"wk\"/\"g\") that\n # never matches the field's real numeric value, so the unit was\n # never actually being set. combgest is always weeks (8511), dbwt\n # is always grams (8504) -- applied unconditionally as an exception.\n cols = {**cols, \"unit_concept_id\": 8511 if field == \"combgest\" else 8504}\n if field == \"pwgt_r\" and \"unit_concept_id\" not in cols:\n # pwgt_r has no unit row in the spreadsheet at all -- its own\n # NBER_Concept_Name (\"Pre-pregnancy Weight Recode Weight in pounds\")\n # and UserGuide2022.pdf (position 292-294) both confirm the unit is\n # always pounds, same as wtgain's own unit (8739 \"pounds (US)\").\n # Applied unconditionally as an exception, same treatment as\n # combgest/dbwt/oegest_comb above. Not a settled fact -- see\n # questions_for_spreadsheet_owner.md.\n cols = {**cols, \"unit_concept_id\": 8739}\n if field in (\"m_ht_in\", \"bmi\", \"pwgt_r\", \"wtgain\") and \"value_as_number\" not in cols:\n # same shape as the OBSERVATION-table numeric-value gap above (mager/\n # fagecomb/previs/precare/wtgain): the spreadsheet has a row supplying\n # measurement_concept_id for the field's real-value range, but no row\n # at all supplying value_as_number -- the actual measured number\n # (height in inches, BMI, pre-pregnancy weight in pounds) was never\n # being recorded. wtgain's own value_as_number is already recorded on\n # OBSERVATION (see above); this only covers its separate, still-\n # incomplete MEASUREMENT row (missing measurement_concept_id -- see\n # questions_for_spreadsheet_owner.md).\n cols = {**cols, \"value_as_number\": _float_or_none(value)}\n yield \"MEASUREMENT\", role, {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"CONDITION\":\n yield \"CONDITION\", role, {\n \"condition_start_date\": dob_date,\n \"condition_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PROCEDURE\":\n yield \"PROCEDURE\", role, {\n \"procedure_date\": dob_date,\n \"procedure_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n **cols,\n }\n elif table == \"PAYER_PLAN_PERIOD\":\n # end_date left null rather than also set to dob_date -- the source\n # data has no real coverage-period dates at all (only \"which payer\"),\n # so forcing both dates to the birth date would assert a false\n # one-day coverage period. start_date must stay non-null (OMOP\n # requires it) and dob_date is the only date this data has to offer;\n # end_date left open is more honest than a fabricated end. Flagged\n # for confirmation in questions_for_spreadsheet_owner.md -- not a\n # settled fact.\n yield \"PAYER_PLAN_PERIOD\", role, {\n \"payer_plan_period_start_date\": dob_date,\n \"payer_plan_period_end_date\": None,\n **cols,\n }\n elif table == \"PROVIDER\":\n provider_id = PROVIDER_ID_BY_CONCEPT.get(cols.get(\"provider_concept_id\"))\n yield \"PROVIDER_USAGE\", role, {\"provider_id\": provider_id}\n\n # hand-computed OBSERVATION/MEASUREMENT rows: these fields' value_as_number row has a\n # non-numeric \"Integer\" placeholder Target_Vocabulary_Id (not a real vocab_id), so\n # load_mapping_rows() can't load it as a normal MappingRow -- but each field also has its\n # own separate, valid, numeric concept_id row (e.g. priorlive -> 3018989) that\n # get_records() resolves correctly. None of these 4 fields are in _MOTHER_FIELDS, so that\n # concept_id was never being looked up at all, leaving these rows without the concept_id\n # OMOP requires. Merged into one row per field here. previs is NOT handled here -- it's in\n # _CHILD_FIELDS/_MOTHER_FIELDS, so it already gets its concept_id row via the field-family\n # loop above; its value_as_number backfill happens there instead (see the OBSERVATION\n # branch above) to avoid emitting a second, duplicate row for the same fact.\n for field in (\"priorlive\", \"priordead\", \"rf_cesarn\"):\n v = _int_or_none(rec.get(field.upper()))\n if v is not None:\n fact = {\n \"observation_date\": dob_date,\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": v,\n }\n fact.update(get_records(\"Mother\", field, v).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # dlmp_mm/dlmp_yy -> one combined OBSERVATION row (spreadsheet's own Notes on\n # dlmp_mm: \"Combine dlmp_yy and dlmp_mm and assume 1st day of month\"), instead of\n # two independent per-field rows sharing the same concept (3002314). dlmp_yy's own\n # Values (\"2020\") is a single literal example, not a real range, so the concept is\n # looked up via dlmp_mm's row (Values \"1 to 12\") instead. DLMP_MM=99/DLMP_YY=9999\n # are the official \"unknown or not stated\" sentinels -- no date computed for those.\n dlmp_month = _int_or_none(rec.get(\"DLMP_MM\"))\n dlmp_year = _int_or_none(rec.get(\"DLMP_YY\"))\n if dlmp_month is not None and 1 <= dlmp_month <= 12 and dlmp_year is not None and dlmp_year != 9999:\n fact = {\n \"observation_date\": dt.date(dlmp_year, dlmp_month, 1),\n \"observation_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n fact.update(get_records(\"Mother\", \"dlmp_mm\", dlmp_month).get(\"OBSERVATION\", {}))\n yield \"OBSERVATION\", \"Mother\", fact\n\n # priorterm -> MEASUREMENT (spreadsheet's own table choice, see module docstring)\n priorterm = _int_or_none(rec.get(\"PRIORTERM\"))\n if priorterm is not None:\n fact = {\n \"measurement_date\": dob_date,\n \"measurement_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n \"value_as_number\": priorterm,\n }\n fact.update(get_records(\"Mother\", \"priorterm\", priorterm).get(\"MEASUREMENT\", {}))\n yield \"MEASUREMENT\", \"Mother\", fact\n\n\ndef build_persons(rec: dict, record_idx: int) -> list[dict]:\n persons = []\n\n dob_year, dob_month, dob_day = _impute_child_dob(rec.get(\"DOB_YY\"), rec.get(\"DOB_MM\"), rec.get(\"DOB_WK\"))\n birth_datetime = None\n if dob_year is not None and dob_month is not None and dob_day is not None:\n # DOB_TT unknown/unparseable (incl. NCHS's \"9999\" sentinel) leaves\n # birth_datetime null rather than assuming midnight -- birth_datetime is\n # nullable in the destination schema, so no time is a truer statement than\n # an assumed one. See questions_for_spreadsheet_owner.md #15.\n dob_time = _parse_dob_time(rec.get(\"DOB_TT\"))\n if dob_time is not None:\n birth_datetime = dt.datetime(dob_year, dob_month, dob_day, dob_time[0], dob_time[1])\n child = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Child\"),\n \"year_of_birth\": dob_year,\n \"month_of_birth\": dob_month,\n \"day_of_birth\": dob_day,\n \"birth_datetime\": birth_datetime,\n }\n child.update(get_records(\"Child\", \"sex\", rec.get(\"SEX\")).get(\"PERSON\", {}))\n persons.append(child)\n\n mager = _int_or_none(rec.get(\"MAGER\"))\n mother = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Mother\"),\n \"year_of_birth\": (dob_year - mager) if (dob_year is not None and mager is not None) else None,\n }\n mother.update(get_records(\"Mother\", \"sex\", \"F\").get(\"PERSON\", {}))\n for field in (\"mrace6\", \"mhispx\", \"mracehisp\"):\n mother.update(get_records(\"Mother\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(mother)\n\n fagecomb = _int_or_none(rec.get(\"FAGECOMB\"))\n if fagecomb == 99:\n # NCHS's own \"unknown or not stated\" sentinel for FAGECOMB, not a real\n # age -- confirmed via the spreadsheet's own fagecomb=99 rows (\"Unknown\n # or Not Stated\" / \"Age unknown\"). Excluded here so the father's\n # year_of_birth isn't computed as if 99 were a real age.\n fagecomb = None\n father = {\n \"person_id\": IdGenerator.person_id(record_idx, \"Father\"),\n \"year_of_birth\": (dob_year - fagecomb) if (dob_year is not None and fagecomb is not None) else None,\n }\n father.update(get_records(\"Father\", \"sex\", \"M\").get(\"PERSON\", {}))\n for field in (\"frace6\", \"fhispx\", \"fracehisp\"):\n father.update(get_records(\"Father\", field, rec.get(field.upper())).get(\"PERSON\", {}))\n persons.append(father)\n\n return persons\n\n\ndef build_visits(rec: dict, record_idx: int, dob_date: dt.date | None) -> list[dict]:\n visits = []\n for role in (\"Child\", \"Mother\"):\n match = get_records(role, \"bfacil\", rec.get(\"BFACIL\"))\n if \"VISIT_OCCURRENCE\" not in match:\n continue\n visit = {\n \"visit_occurrence_id\": IdGenerator.visit_occurrence_id(record_idx, role),\n \"person_id\": IdGenerator.person_id(record_idx, role),\n \"visit_start_date\": dob_date,\n \"visit_end_date\": dob_date,\n \"visit_type_concept_id\": DEFAULT_TYPE_CONCEPT_ID,\n }\n # spreadsheet field name is \"Visit_Concept_id\" (case-as-published); normalize here\n for k, v in match[\"VISIT_OCCURRENCE\"].items():\n visit[k.lower()] = v\n visits.append(visit)\n return visits\n\n\n_ID_COLUMN = {\n \"OBSERVATION\": \"observation_id\",\n \"MEASUREMENT\": \"measurement_id\",\n \"CONDITION\": \"condition_occurrence_id\",\n \"PROCEDURE\": \"procedure_occurrence_id\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period_id\",\n}\n\nSQL_TABLE_NAME = {\n \"PERSON\": \"person\",\n \"VISIT_OCCURRENCE\": \"visit_occurrence\",\n \"OBSERVATION\": \"observation\",\n \"MEASUREMENT\": \"measurement\",\n \"CONDITION\": \"condition_occurrence\",\n \"PROCEDURE\": \"procedure_occurrence\",\n \"PAYER_PLAN_PERIOD\": \"payer_plan_period\",\n}\n\nTABLE_COLUMNS = {\n \"PERSON\": [\"person_id\", \"year_of_birth\", \"month_of_birth\", \"day_of_birth\", \"birth_datetime\",\n \"gender_concept_id\", \"race_concept_id\", \"ethnicity_concept_id\"],\n \"VISIT_OCCURRENCE\": [\"visit_occurrence_id\", \"person_id\", \"visit_start_date\", \"visit_end_date\",\n \"visit_type_concept_id\", \"visit_concept_id\"],\n \"OBSERVATION\": ['person_id', 'observation_id', 'observation_date', 'observation_type_concept_id',\n 'observation_concept_id', 'qualifier_concept_id', 'unit_concept_id',\n 'value_as_concept_id', 'value_as_number'],\n \"MEASUREMENT\": ['person_id', 'measurement_id', 'measurement_date', 'measurement_type_concept_id',\n 'measurement_concept_id', 'unit_concept_id', 'value_as_concept_id', 'value_as_number'],\n \"CONDITION\": ['person_id', 'condition_occurrence_id', 'condition_start_date', 'condition_type_concept_id',\n 'condition_concept_id', 'condition_status_concept_id'],\n \"PROCEDURE\": ['person_id', 'procedure_occurrence_id', 'procedure_date', 'procedure_type_concept_id',\n 'procedure_concept_id'],\n \"PAYER_PLAN_PERIOD\": ['person_id', 'payer_plan_period_id', 'payer_plan_period_start_date',\n 'payer_plan_period_end_date', 'payer_concept_id'],\n}\n\n\ndef build_facts(rec: dict, record_idx: int, dob_date: dt.date | None) -> dict[str, list[dict]]:\n \"\"\"\n One pass over _iter_field_records per record, bucketed by target table --\n _iter_field_records computes matches for every table in a single\n generator pass. Covers OBSERVATION/MEASUREMENT/CONDITION/\n PROCEDURE/PAYER_PLAN_PERIOD only -- PERSON and VISIT_OCCURRENCE are\n built separately (build_persons/build_visits) since they're driven by\n a handful of targeted get_records() calls, not this field-family scan.\n \"\"\"\n counters: dict[str, int] = {}\n out: dict[str, list[dict]] = {table: [] for table in _ID_COLUMN}\n for table, role, fact in _iter_field_records(rec, dob_date):\n if table not in _ID_COLUMN:\n continue # e.g. PROVIDER_USAGE -- not one of this node's tables\n counters[table] = counters.get(table, 0) + 1\n out[table].append({\n \"person_id\": IdGenerator.person_id(record_idx, role),\n _ID_COLUMN[table]: IdGenerator.fact_id(record_idx, counters[table]),\n **fact,\n })\n return out\n\n\ndef _sanitize_cache_id(raw_id: str) -> str:\n \"\"\"\n Mirrors the platform's own Dataset.cacheId derivation\n (portal/src/dataset/entity/dataset.entity.ts: sanitizeIdForCacheId) --\n hyphens aren't valid in a bare SQL/DuckDB identifier, so the Dataset's\n UUID gets converted the same way here before use as a trex cache_id.\n \"\"\"\n cleaned = raw_id.replace(\"-\", \"_\")\n return f\"_{cleaned}\" if cleaned[:1].isdigit() else cleaned\n\n\ndef exec(myinput):\n \"\"\"\n Gold transform: reads the staging CSV written by node-load-bronze in\n chunks, builds the OMOP rows in Python, writes each chunk's rows\n per-table to a CSV in the shared trex-volume staging dir, and loads each\n with one `INSERT INTO <table> (cols) SELECT * FROM read_csv('<trex\n path>')` -- trex's DuckDB reads the file straight from disk at native\n speed. Each load statement is atomic, and row counts are verified per\n table at the end; a mismatch raises.\n \"\"\"\n import csv\n import time\n import uuid\n\n logger = get_run_logger()\n\n global _MAPPING_DF_FROM_NODE\n mapping_input = myinput.get(MAPPING_CSV_NODE)\n if mapping_input is None or mapping_input.result is None:\n raise RuntimeError(\n f\"Expected mapping spreadsheet input from upstream csv node {MAPPING_CSV_NODE!r}. \"\n \"Connect that node to transform_facts (node-facts).\"\n )\n if not isinstance(mapping_input.result, pd.DataFrame):\n raise RuntimeError(\n f\"Upstream mapping input {MAPPING_CSV_NODE!r} must be a pandas DataFrame, got \"\n f\"{type(mapping_input.result).__name__}\"\n )\n _MAPPING_DF_FROM_NODE = mapping_input.result.copy()\n\n _validate_mapping_against_cdm(load_mapping_rows(), logger)\n cache_catalog = _sanitize_cache_id(dataset_id)\n dbdao = DBDao(database_code=destination_database_code, cache_id=cache_catalog, dialect=SupportedDatabaseDialects.TREX)\n dest_schema = f\"{cache_catalog}.{destination_schema_name}\"\n\n # Input: the staging CSV written by node-load-bronze. The .complete marker\n # holds the row count and only exists if bronze finished; csv-without-\n # marker means an interrupted bronze run and is refused.\n src_fname = f\"{cache_catalog}_nat2022_raw.csv\"\n src_csv_path = os.path.join(STAGING_DIR_FLOW, src_fname)\n src_complete_path = src_csv_path + \".complete\"\n if not (os.path.exists(src_csv_path) and os.path.exists(src_complete_path)):\n raise RuntimeError(\n f\"Staging CSV from node-load-bronze not found or incomplete \"\n f\"({src_csv_path}; marker present: {os.path.exists(src_complete_path)}). \"\n f\"Run node-load-bronze first -- a CSV without its .complete marker means \"\n f\"the bronze run was interrupted mid-write.\"\n )\n with open(src_complete_path) as f:\n total = int(f.read().strip())\n\n for sql_table in SQL_TABLE_NAME.values():\n # destination_schema_name.<table> is created by the OMOP CDM plugin, not here --\n # truncate rather than drop so we never touch its schema/table definition.\n dbdao.truncate_table(dest_schema, sql_table)\n\n size = int(chunk_size)\n\n # Self-healing sweep: staging files are deleted right after each bulk\n # load, but a hard-killed run orphans its in-flight file. Current-run\n # filenames get a fresh run_token below, so anything matching these\n # prefixes now is stale by construction; other datasets' files\n # (different prefix) are never touched.\n table_prefixes = tuple(f\"{cache_catalog}_{t}_\" for t in SQL_TABLE_NAME.values())\n removed = 0\n for f in os.listdir(STAGING_DIR_FLOW):\n if f.startswith(table_prefixes) and f.endswith(\".csv\"):\n try:\n os.remove(os.path.join(STAGING_DIR_FLOW, f))\n removed += 1\n except OSError:\n pass\n if removed:\n logger.info(f\"Removed {removed} stale staging file(s) left by a previous interrupted run\")\n\n run_token = uuid.uuid4().hex\n\n def _bulk_load(sql_table, columns, values, tag):\n \"\"\"One CSV in the shared staging dir + one read_csv INSERT for this table-chunk.\"\"\"\n fname = f\"{cache_catalog}_{sql_table}_{tag}_{run_token}.csv\"\n fpath = os.path.join(STAGING_DIR_FLOW, fname)\n with open(fpath, \"w\", newline=\"\") as f:\n w = csv.writer(f)\n w.writerow(columns)\n for row in values:\n w.writerow([\"\" if v is None else v for v in row])\n col_list = \", \".join(f'\"{c}\"' for c in columns)\n try:\n dbdao.execute_sql(\n f\"INSERT INTO {dest_schema}.{sql_table} ({col_list}) \"\n f\"SELECT * FROM read_csv('{STAGING_DIR_TREX}/{fname}', header=true, all_varchar=true)\"\n )\n finally:\n os.remove(fpath)\n\n expected_counts = {table: 0 for table in SQL_TABLE_NAME}\n processed = 0\n # Sequential single-pass read of the staging CSV -- flat cost per chunk,\n # order preserved (ROW_IDX rides along as a column and record_idx comes\n # from its stored value, not from position). dtype=str + keep_default_na\n # =False gives '' for missing values, which the transform treats the same\n # as NULL.\n for chunk_idx, src_chunk in enumerate(pd.read_csv(src_csv_path, dtype=str, keep_default_na=False, chunksize=size)):\n t_chunk = time.time()\n rows_by_table = {table: [] for table in SQL_TABLE_NAME}\n for rec in src_chunk.to_dict(\"records\"):\n record_idx = int(rec[\"ROW_IDX\"])\n dob_date = dob_date_for(rec)\n rows_by_table[\"PERSON\"].extend(build_persons(rec, record_idx))\n rows_by_table[\"VISIT_OCCURRENCE\"].extend(build_visits(rec, record_idx, dob_date))\n built = build_facts(rec, record_idx, dob_date)\n for table, rows in built.items():\n rows_by_table[table].extend(rows)\n t_transform = time.time() - t_chunk\n\n chunk_rows = 0\n for table, sql_table in SQL_TABLE_NAME.items():\n columns = TABLE_COLUMNS[table]\n values = [tuple(row.get(col) for col in columns) for row in rows_by_table[table]]\n if not values:\n continue\n expected_counts[table] += len(values)\n chunk_rows += len(values)\n t_tbl = time.time()\n _bulk_load(sql_table, columns, values, str(chunk_idx))\n logger.info(f\" {sql_table}: {len(values):,} rows bulk-loaded in {time.time() - t_tbl:.1f}s\")\n\n processed += len(src_chunk)\n done = processed\n logger.info(\n f\"{done:,}/{total:,} records -> {chunk_rows:,} OMOP rows \"\n f\"(transform {t_transform:.0f}s, chunk total {time.time() - t_chunk:.0f}s)\"\n )\n\n if processed != total:\n raise RuntimeError(f\"staging CSV row count mismatch: marker says {total:,}, read {processed:,}\")\n\n mismatches = []\n for table, sql_table in SQL_TABLE_NAME.items():\n actual = int(dbdao.execute_sql(f\"SELECT COUNT(*) FROM {dest_schema}.{sql_table}\", fetch=True)[0][0])\n if actual != expected_counts[table]:\n mismatches.append(f\"{sql_table}: expected {expected_counts[table]:,}, table has {actual:,}\")\n if mismatches:\n raise RuntimeError(\"row-count verification failed -- \" + \"; \".join(mismatches))\n logger.info(\"Row-count verification passed for all tables: \"\n + \", \".join(f\"{SQL_TABLE_NAME[t]}={expected_counts[t]:,}\" for t in SQL_TABLE_NAME))\n\n return (f\"Loaded {total} records into person, visit_occurrence, observation, measurement, \"\n f\"condition_occurrence, procedure_occurrence, payer_plan_period\")" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge Checklist
Please review this list and check any items that require additions or modifications beyond your core changes. Reviewers can also use it to help confirm that nothing was missed.
Test if the flow runs successfully as an imported json file
Test if the flow runs successfully as template
DATAFLOW_TEMPLATE_BRANCHenvironment variable in docker-compose.yaml to the branch being reviewed and successfully run the flowInspect the nodes in the flow (either as a template or imported json file):
[Currently not visible in UI]