451 update lab functions to pull in cleaned lab data

[chadhunt2]

Pull Request: Load cleaned lab data from EDAV and refactor KPI lab timeliness helper
Summary
This PR updates the KPI and desk-review lab-data workflow so downstream code uses the canonical cleaned lab dataset from EDAV instead of re-running clean_lab_data() inside templates or KPI helper functions.

The main changes are:

get_constant("CLEANED_LAB_DATA") now points to Data/lab/lab_data.rda which is the cleaned lab data used in the kpi report and desk reviews.

KPI template generation now loads cleaned lab data from EDAV with:

edav_io("read", file_loc = get_constant("CLEANED_LAB_DATA"))
generate_kpi_lab_timeliness() now expects already-cleaned lab_data and no longer calls clean_lab_data() internally.

afp_data remains in generate_kpi_lab_timeliness() as afp_data = NULL for backwards compatibility.

The desk-review skeleton now loads cleaned lab data from EDAV and filters rows using either EPID or EpidNumber, depending on which column is present.

Documentation for generate_kpi_lab_timeliness() was updated to describe the cleaned-lab-data input contract.

Motivation
The previous KPI workflow could re-run lab cleaning in multiple places, which created risk of duplicated processing or failures when the supplied lab data was already cleaned.

This PR makes the workflow more explicit:

The cleaned lab dataset is loaded from EDAV.

KPI and desk-review functions operate on that cleaned dataset.

generate_kpi_lab_timeliness() only adds KPI-specific timeliness fields and no longer attempts to clean lab data.

Changes

1. Updated cleaned lab data constant
   get_constant("CLEANED_LAB_DATA") now resolves to:

"Data/lab/lab_data.rda"
This lets templates and workflows consistently refer to the canonical cleaned lab data file through the existing constant.

2. Updated KPI template lab-data loading
   The KPI template now loads cleaned lab data directly from EDAV:

lab_data <- edav_io("read", file_loc = get_constant("CLEANED_LAB_DATA"))
The previous template behavior of calling clean_lab_data() has been removed.

The template also builds lab range checks using:

lab_kpi_check <- sirfunctions:::generate_kpi_lab_timeliness(
lab_data,
start_date,
end_date
)
3. Refactored generate_kpi_lab_timeliness()
generate_kpi_lab_timeliness() now accepts already-cleaned lab data:

generate_kpi_lab_timeliness(lab_data, start_date, end_date, afp_data = NULL)
The internal call to clean_lab_data() was removed.

The function now only adds KPI-specific lab timeliness columns such as:

days.lab.culture

days.culture.itd

days.seq.ship

days.seq.rec.res

days.itd.res.seq.res

t1

t2

t3

t4

t5

afp_data is retained as an optional ignored argument for backwards compatibility with existing callers.

4. Updated desk-review skeleton
   The desk-review skeleton now loads cleaned lab data from EDAV:

lab_data <- sirfunctions::edav_io(
"read",
file_loc = get_constant("CLEANED_LAB_DATA")
)
It then detects whether the lab identifier column is named EPID or EpidNumber:

lab_epid_col <- intersect(c("EPID", "EpidNumber"), names(lab_data))[1]
If neither column exists, the template stops with a clear error:

if (is.na(lab_epid_col)) {
stop("Lab data must contain either EPID or EpidNumber.")
}
Then it filters out records missing the identifier:

lab_data <- lab_data |>
dplyr::filter(!is.na(.data[[lab_epid_col]]))
This avoids failures when cleaned lab data contains EpidNumber instead of EPID.

5. Updated documentation
   The generated documentation for generate_kpi_lab_timeliness() now states that:

lab_data should already be cleaned.

afp_data is deprecated/ignored and kept only for backwards compatibility.

The function returns cleaned lab data with KPI timeliness columns added.

How to test

1. Load the local package
   From the sirfunctions repo root:
   pak::pak("CDCgov/sirfunctions@451-update-lab-functions-to-pull-in-cleaned-lab-data")

devtools::load_all(".")
2. Confirm the EDAV lab-data path
get_constant("CLEANED_LAB_DATA")
Expected:

[1] "Data/lab/lab_data.rda"
3. Load lab data
Option A: Use a local test copy
loaded_objects <- load("C:/Users/id/Downloads/lab_data.rda")

if (!"lab_data" %in% loaded_objects) {
lab_data <- get(loaded_objects[1])
}

stopifnot(is.data.frame(lab_data))
Option B: Read from EDAV
lab_data <- edav_io("read", file_loc = get_constant("CLEANED_LAB_DATA"))

stopifnot(is.data.frame(lab_data))
4. Validate desk-review lab-data filtering
lab_epid_col <- intersect(c("EPID", "EpidNumber"), names(lab_data))[1]

stopifnot(!is.na(lab_epid_col))

lab_data_dr <- lab_data |>
dplyr::filter(!is.na(.data[[lab_epid_col]]))

stopifnot(is.data.frame(lab_data_dr))
stopifnot(!any(is.na(lab_data_dr[[lab_epid_col]])))
This confirms the desk-review skeleton logic works whether the identifier column is named EPID or EpidNumber.

5. Validate desk-review lab timeliness summaries
   start_date <- "2022-01-01"
   end_date <- "2026-06-23"

lab.timeliness.ctry <- generate_lab_timeliness(
lab_data,
"ctry",
start_date,
end_date
)

lab.timeliness.prov <- generate_lab_timeliness(
lab_data,
"prov",
start_date,
end_date
)

stopifnot(is.data.frame(lab.timeliness.ctry))
stopifnot(is.data.frame(lab.timeliness.prov))

nrow(lab.timeliness.ctry)
nrow(lab.timeliness.prov)
Example observed result:

nrow(lab.timeliness.ctry)
[1] 48

nrow(lab.timeliness.prov)
[1] 48
6. Validate KPI lab timeliness helper
lab_kpi_check <- sirfunctions:::generate_kpi_lab_timeliness(
lab_data,
start_date,
end_date
)

expected_cols <- c(
"days.lab.culture",
"days.culture.itd",
"days.seq.ship",
"days.seq.rec.res",
"days.itd.res.seq.res",
"t1",
"t2",
"t3",
"t4",
"t5"
)

stopifnot(length(setdiff(expected_cols, names(lab_kpi_check))) == 0)
Expected:

setdiff(expected_cols, names(lab_kpi_check))
character(0)
This confirms the helper adds all expected KPI lab interval columns without requiring clean_lab_data().

7. Validate KPI C4 table output
   raw.data <- get_all_polio_data()

c4 <- generate_c4_table(
lab_data = lab_data,
afp_data = raw.data$afp,
start_date = start_date,
end_date = end_date
)

stopifnot(is.list(c4))
stopifnot(all(c("itd_lab_summary", "seq_lab_summary") %in% names(c4)))
stopifnot(is.data.frame(c4$itd_lab_summary))
stopifnot(is.data.frame(c4$seq_lab_summary))

nrow(c4$itd_lab_summary)
nrow(c4$seq_lab_summary)
Example observed result:

nrow(c4$itd_lab_summary)
[1] 115

nrow(c4$seq_lab_summary)
[1] 48
Note: generate_c4_table() returns a list with two data frame components, not a single data frame.

8. Optional plot smoke test
   Install the optional plotting dependency if needed:

install.packages("ggh4x")
Then run:

p <- generate_lab_culture_violin(
lab_data = lab_data,
afp_data = raw.data$afp,
start_date = start_date,
end_date = end_date
)

stopifnot(inherits(p, "ggplot"))

print(p)
You may see warnings such as:

Groups with fewer than two datapoints have been dropped.
Computation failed in stat_ydensity().
These indicate sparse groups for violin density estimation and do not necessarily mean the refactor failed if the plot object is created.

Validation performed
The following targeted checks were run successfully with a local cleaned lab_data.rda test file:

generate_lab_timeliness() returned data frames for both country and province levels.

generate_kpi_lab_timeliness() returned all expected interval columns.

generate_c4_table() returned a list containing itd_lab_summary and seq_lab_summary.

generate_lab_culture_violin() returned a ggplot object after installing ggh4x.

Observed outputs included:

nrow(lab.timeliness.ctry)
[1] 48

nrow(lab.timeliness.prov)
[1] 48

setdiff(expected_cols, names(lab_kpi_check))
character(0)

nrow(c4$itd_lab_summary)
[1] 115

nrow(c4$seq_lab_summary)
[1] 48

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 2.00%. Comparing base (b0bca89) to head (6416eac).
⚠️ Report is 16 commits behind head on hotfix.

Files with missing lines	Patch %	Lines
R/dr.lab.functions.R	0.00%	3 Missing ⚠️
R/kpi.main.functions.R	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           hotfix    #460      +/-   ##
=========================================
- Coverage    2.00%   2.00%   -0.01%     
=========================================
  Files          38      38              
  Lines       15079   15080       +1     
=========================================
  Hits          302     302              
- Misses      14777   14778       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.