Create lab_cleaning.R

[chadhunt2]

Enhance lab cleaning with sirfunctions-compatible output, EPID parsing, and future-date handling

Summary

This PR adds a standalone clean_lab_data() workflow to tidypolis that mirrors the output shape and key derived fields produced by sirfunctions::clean_lab_data(). It supports both WHO-style lab inputs with MasterKey and regional lab extracts, harmonizes them into a consistent output schema, computes lab turnaround metrics, fills EPID component fields, optionally backfills geography from AFP data, and removes future dates from lab date fields.

This PR also adds load_lab_data() as a helper for loading raw CSV or Excel lab files.

What clean_lab_data() does

clean_lab_data() is the top-level lab cleaning function. It accepts:

clean_lab_data(
  lab_data,
  start_date,
  end_date,
  afp_data = NULL,
  ctry_name = NULL,
  lab_locs_path = NULL,
  use_edav = TRUE,
  save_rda_path = NULL
)
1. Detects the input schema
The function checks whether the input has MasterKey.

If MasterKey is present, it treats the input as WHO-style lab data.

If MasterKey is absent, it treats the input as a regional lab extract.

It then routes the input through the appropriate internal cleaner.

2. Converts and cleans date fields
The cleaner converts date-like columns to date/date-time objects and removes impossible future dates by changing them to NA.

This is done before turnaround metrics are calculated, so future dates do not create invalid interval values.

3. Filters to the analysis date range
The function uses start_date and end_date to keep records within the requested analysis period.

4. Computes lab turnaround metrics
The cleaned output includes derived timing columns such as:

days.collect.lab
days.lab.culture
days.seq.ship
days.seq.rec.res
days.itd.arriveseq
days.itd.seqres
days.lab.seq
days.coll.sent.field
days.sent.field.rec.nat
days.rec.nat.sent.lab
days.sent.lab.rec.lab
days.rec.lab.culture

5. Adds rolling-year fields
The output includes rolling analysis-year fields:

date_interval
year_label
analysis_year_start
analysis_year_end
rolling_period
year_number

6. Harmonizes WHO and regional schemas
Regional lab extracts and WHO-style lab extracts can use different column names. This function adds aliases and normalizes names so downstream users get a consistent sirfunctions-compatible output.

Examples include:

DateOfOnset → ParalysisOnsetDate
DateStoolSentToLab → StoolDateSentToLab
DateFinalCellCultureResults → DateFinalCellCultureResult
DateSeqResult → DateofSequencing
whoregion → who.region

7. Populates EPID component fields
The function populates:

epid_ctry
epid_prov
epid_dist
epid_04
epid_05

These can come from:
delimited EPID strings, for example NGA-001-002-003-004
compact EPID strings, for example NGA001002003004
existing underscore columns, for example epid_ctry
existing dot-separated columns, for example epid.ctry
AFP-linked component fields, when available

8. Optionally backfills geography from AFP data
If afp_data is provided, the function uses matching EPIDs to backfill:

ctry
prov
dist
adm0guid
adm1guid
adm2guid

This matches the general behavior of sirfunctions::clean_lab_data() by using AFP data to improve geographic completeness.

9. Applies lab routing updates
The function applies sequencing and culture lab routing updates, including date-based sequencing-capacity changes and lab routing overrides.

10. Returns a stable sirfunctions-compatible output schema
The final output is selected into a fixed column order matching the sirfunctions-style lab output schema. Missing expected columns are added as NA, so downstream code can rely on the same column names being present.

11. Optionally saves the cleaned output
If save_rda_path is supplied, the cleaned object is saved as:

lab_data
inside the .rda file.

Example:

result <- clean_lab_data(
  lab_data = lab_data,
  start_date = "2024-01-01",
  end_date = "2024-12-31",
  afp_data = afp_data,
  save_rda_path = "lab_data_cleaned.rda"
)
Then later:

load("lab_data_cleaned.rda")
View(lab_data)
New load_lab_data() helper
This PR also adds:

load_lab_data(lab_data_path, sheet_name = NULL)
It supports:

.csv files via readr::read_csv()

.xlsx files via readxl::read_excel()

If readxl is not installed and an Excel file is requested, the function gives a clear error message.

How to test this PR
Run the new lab cleaning tests
Download this branch of tidypolis
pak::pak("CDCgov/tidypolis@311-lab-cleaning-functions")

From the repo root in R:

devtools::load_all()
testthat::test_file("tests/testthat/test-lab_cleaning.R")
Run all package tests
devtools::test()
Manually test WHO-style input
devtools::load_all()

lab_data <- data.frame(
  MasterKey = c("mk-1", "mk-1"),
  DateOfOnset = c("2024-01-01", "2024-01-01"),
  DateStoolCollected = c("2024-01-03", "2024-01-03"),
  DateStoolReceivedinLab = c("2024-01-08", "2024-01-08"),
  DateFinalCellCultureResults = c("2024-01-12", "2024-01-12"),
  DateIsolateRcvdForSeq = c("2024-01-15", "2024-01-15"),
  DateSeqResult = c("2024-01-20", "2099-01-20"),
  DateFinalrRTPCRResults = c("2024-01-13", "2024-01-13"),
  DateStoolSentfromField = c("2024-01-04", "2024-01-04"),
  DateStoolReceivedNatLevel = c("2024-01-05", "2024-01-05"),
  DateStoolSentToLab = c("2024-01-06", "2099-01-06"),
  CaseOrContact = c("1-Case", "2-Contact"),
  seq.lab = c("CDC-Atlanta", "CDC-Atlanta"),
  culture.itd.lab = c("CDC-Atlanta", "CDC-Atlanta"),
  country = c("NIGERIA", "NIGERIA")
)

result <- clean_lab_data(
  lab_data = lab_data,
  start_date = "2024-01-01",
  end_date = "2024-12-31"
)

names(result)
View(result)
Things to check:

result$DateofSequencing
result$StoolDateSentToLab
result$days.lab.seq
The future dates from 2099 should be changed to NA.

Manually test regional input
devtools::load_all()

lab_data <- data.frame(
  EPID = "NGA-001-002-003-004",
  SpecimenNumber = 1,
  CaseContactCode = "1-Case",
  Name = "NIGERIA",
  CaseDate = "2024-01-01",
  ParalysisOnsetDate = "2024-01-01",
  DateStoolCollected = "2024-01-03",
  StoolDateSentToLab = "2024-01-06",
  DateStoolReceivedinLab = "2024-01-08",
  FinalCellCultureResult = "Negative",
  DateFinalCellCultureResult = "2024-01-12",
  FinalITDResult = "Negative",
  DateFinalrRTPCRResults = "2024-01-13",
  DateIsolateSentforSequencing = "2024-01-14",
  ReportDateSequenceResultSent = "2024-01-21",
  DateIsolateRcvdForSeq = "2024-01-15",
  DateLArmIsolate = "2024-01-16",
  DateRArmIsolate = "2024-01-17",
  ResultNPENT = 0,
  DateofSequencing = "2024-01-20",
  DateNotificationtoHQ = "2024-01-22",
  WILD1 = NA,
  VDPV1 = NA,
  VDPV2 = NA,
  VDPV3 = NA
)

lab_locs_path <- tempfile(fileext = ".csv")
write.csv(
  data.frame(
    country = "NIGERIA",
    seq.capacity = "Sequencing capacity",
    seq.lab = "CDC-Atlanta",
    culture.itd.lab = "CDC-Atlanta"
  ),
  lab_locs_path,
  row.names = FALSE
)

result <- clean_lab_data(
  lab_data = lab_data,
  start_date = "2024-01-01",
  end_date = "2024-12-31",
  lab_locs_path = lab_locs_path,
  use_edav = FALSE
)

result[, c("epid_ctry", "epid_prov", "epid_dist", "epid_04", "epid_05")]
Expected EPID components:

epid_ctry = "NGA"
epid_prov = "001"
epid_dist = "002"
epid_04   = "003"
epid_05   = "004"
Test with actual AFP data
result <- clean_lab_data(
  lab_data = lab_data,
  start_date = "2024-01-01",
  end_date = "2024-12-31",
  afp_data = raw.data$afp
)

dplyr::glimpse(result)

result |>
  dplyr::summarise(
    missing_epid_ctry = sum(is.na(epid_ctry) | epid_ctry == ""),
    missing_epid_prov = sum(is.na(epid_prov) | epid_prov == ""),
    missing_epid_dist = sum(is.na(epid_dist) | epid_dist == ""),
    missing_ctry = sum(is.na(ctry) | ctry == ""),
    missing_prov = sum(is.na(prov) | prov == ""),
    missing_dist = sum(is.na(dist) | dist == "")
  )
Test coverage added
The new tests cover:

WHO-schema cleaning

output column names/order

preservation of case/contact rows

turnaround metric calculations

future-date trimming

.rda save behavior

EPID parsing from delimited EPID strings

EPID parsing from compact EPID strings

EPID component backfill from AFP data

dot-separated EPID component aliases

deriving EPID components before AFP matching

Notes for reviewers
This implementation intentionally does not call sirfunctions::clean_lab_data() directly. It mirrors the relevant output structure and field derivations inside tidypolis.

The final output is forced into a stable sirfunctions-compatible schema so downstream code can rely on the same columns being present for WHO and regional lab extracts.

Future dates are converted to NA before interval calculations and again after final alias creation, so derived alias columns are also protected.

If EDAV access is unavailable for regional data, users can provide a local lab locations CSV with lab_locs_path and set use_edav = FALSE.

Suggested validation before merge
devtools::load_all()
testthat::test_file("tests/testthat/test-lab_cleaning.R")
devtools::test()
If available, also compare against a sirfunctions run:

sirfunctions_result <- sirfunctions::clean_lab_data(
  lab_data = lab_data,
  start_date = start_date,
  end_date = end_date,
  afp_data = raw.data$afp
)

tidypolis_result <- clean_lab_data(
  lab_data = lab_data,
  start_date = start_date,
  end_date = end_date,
  afp_data = raw.data$afp
)

setdiff(names(sirfunctions_result), names(tidypolis_result))
setdiff(names(tidypolis_result), names(sirfunctions_result))

dplyr::bind_rows(
  sirfunctions = sirfunctions_result,
  tidypolis = tidypolis_result,
  .id = "source"
) |>
  dplyr::group_by(source) |>
  dplyr::summarise(
    rows = dplyr::n(),
    missing_epid_ctry = sum(is.na(epid_ctry) | epid_ctry == ""),
    missing_epid_prov = sum(is.na(epid_prov) | epid_prov == ""),
    missing_epid_dist = sum(is.na(epid_dist) | epid_dist == "")
  )

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bfd462cb14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Pass regional cleaner arguments in declared order

The clean_lab_data_regional() call swaps use_edav and lab_locs_path by position, so the default path (lab_locs_path = NULL, use_edav = TRUE) is passed as use_edav = NULL and lab_locs_path = TRUE. In the current implementation this drives get_lab_locs() into the CSV branch and attempts to read TRUE as a file path, which breaks regional cleaning before any data processing occurs. Use named arguments (or reorder the positional args) so use_edav and lab_locs_path are not inverted.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve existing country values when AFP data is absent

When afp_data is NULL, impute_missing_lab_geo() unconditionally sets ctry to NA and returns. Downstream logic filters by ctry when ctry_name is provided and also derives WHO region from this field, so this branch effectively disables country filtering and can force incorrect region output whenever callers rely on default afp_data = NULL. This should keep any pre-existing country signal (or fall back to country/Name) instead of blanking ctry for every row.

Useful? React with 👍 / 👎.