Make historical RDS data optional when running preprocessing - #294
Open
chadhunt2 wants to merge 7 commits into
Open
Make historical RDS data optional when running preprocessing #294chadhunt2 wants to merge 7 commits into
chadhunt2 wants to merge 7 commits into
Conversation
mcuadera
requested changes
Mar 2, 2026
mcuadera
left a comment
mcuadera
left a comment
Collaborator
There was a problem hiding this comment.
We're still requiring the use of files located in "core_files_to_combine." Please see my comments.
Comment on lines
1386
to
1392
| cli::cli_alert_warning(paste0( | ||
| "Please request the following file(s) from the SIR team", | ||
| "and move them into: ", core_files_folder_path | ||
| )) | ||
| for (file in missing_static_files) { | ||
| cli::cli_alert_info(paste0(file, "\n")) | ||
| } |
Collaborator
There was a problem hiding this comment.
This should be removed since we're making the historical datasets optional
Comment on lines
5403
to
5464
| cli::cli_process_start("Generating combined AFP dataset", | ||
| msg_done = "Generated combined AFP dataset" | ||
| ) | ||
|
|
||
| # Find all AFP files in Core_Ready_Files and core_files_to_combine | ||
| afp_files_main <- dplyr::tibble("name" = tidypolis_io( | ||
| io = "list", | ||
| file_path = paste0(polis_data_folder, "/", output_folder_name), | ||
| full_names = TRUE | ||
| )) |> | ||
| dplyr::filter(grepl(paste0("^.*(afp_linelist).*(\\", output_format, ")$"), name)) |> | ||
| dplyr::pull(name) | ||
|
|
||
| afp_files_combine <- dplyr::tibble("name" = tidypolis_io( | ||
| io = "list", | ||
| file_path = paste0(polis_data_folder, "/core_files_to_combine"), | ||
| full_names = TRUE | ||
| )) |> | ||
| dplyr::filter(grepl("^.*(afp_linelist).*(.rds)$", name)) |> | ||
| dplyr::pull(name) | ||
|
|
||
| # Combine AFP files | ||
| if (length(afp_files_combine) > 0) { | ||
| invisible(capture.output( | ||
| afp_to_combine <- purrr::map_df(afp_files_combine, ~ tidypolis_io( | ||
| io = "read", | ||
| file_path = .x | ||
| )) |> | ||
| dplyr::mutate( | ||
| stool1tostool2 = as.numeric(stool1tostool2), | ||
| datenotificationtohq = lubridate::parse_date_time( | ||
| datenotificationtohq, | ||
| c("dmY", "bY", "Ymd", "%Y-%m-%d %H:%M:%S") | ||
| ) | ||
| ) | ||
| )) | ||
|
|
||
| invisible(capture.output( | ||
| afp_new <- purrr::map_df( | ||
| afp_files_main, | ||
| ~ tidypolis_io(io = "read", file_path = .x) | ||
| ) | ||
| )) | ||
|
|
||
| afp_combined <- dplyr::bind_rows(afp_new, afp_to_combine) | ||
|
|
||
| # Export combined AFP dataset | ||
| invisible(capture.output( | ||
| tidypolis_io( | ||
| obj = afp_combined, | ||
| io = "write", | ||
| file_path = paste0( | ||
| polis_data_folder, | ||
| "/", output_folder_name, "/", | ||
| "afp_linelist_", | ||
| min(afp_combined$dateonset, na.rm = TRUE), | ||
| "_", | ||
| max(afp_combined$dateonset, na.rm = TRUE), | ||
| output_format | ||
| ) | ||
| ) | ||
| )) |
Collaborator
There was a problem hiding this comment.
We're still using the historical datasets to get an AFP linelist. We want to make this non-required
Comment on lines
5519
to
5585
| cli::cli_process_start("Generating combined Other Surveillance dataset", | ||
| msg_done = "Generated combined Other Surveillance dataset" | ||
| ) | ||
|
|
||
| # Find all other surveillance files in Core_Ready_Files and core_files_to_combine | ||
| other_files_main <- dplyr::tibble("name" = tidypolis_io( | ||
| io = "list", | ||
| file_path = paste0(polis_data_folder, "/", output_folder_name), | ||
| full_names = TRUE | ||
| )) |> | ||
| dplyr::filter(grepl(paste0("^.*(other_surveillance).*(\\", output_format, ")$"), name)) |> | ||
| dplyr::pull(name) | ||
|
|
||
| other_files_combine <- dplyr::tibble("name" = tidypolis_io( | ||
| io = "list", | ||
| file_path = paste0(polis_data_folder, "/core_files_to_combine"), | ||
| full_names = TRUE | ||
| )) |> | ||
| dplyr::filter(grepl("^.*(other_surveillance).*(.rds)$", name)) |> | ||
| dplyr::pull(name) | ||
|
|
||
| # Combine other surveillance files | ||
| if (length(other_files_combine) > 0) { | ||
| invisible(capture.output( | ||
| other_to_combine <- purrr::map_df(other_files_combine, ~ tidypolis_io( | ||
| io = "read", | ||
| file_path = .x | ||
| )) | ||
| )) | ||
|
|
||
| other_to_combine <- other_to_combine |> | ||
| dplyr::mutate( | ||
| stool1tostool2 = as.numeric(stool1tostool2), | ||
| datenotificationtohq = lubridate::parse_date_time( | ||
| datenotificationtohq, | ||
| c("dmY", "bY", "Ymd", "%Y-%m-%d %H:%M:%S") | ||
| ) | ||
| ) | ||
|
|
||
| invisible(capture.output( | ||
| other_new <- purrr::map_df( | ||
| other_files_main, | ||
| ~ tidypolis_io(io = "read", file_path = .x) | ||
| ) | ||
| )) | ||
|
|
||
| other_combined <- dplyr::bind_rows(other_new, other_to_combine) | ||
|
|
||
| # Export combined other surveillance dataset | ||
| invisible(capture.output( | ||
| tidypolis_io( | ||
| obj = other_combined, | ||
| io = "write", | ||
| file_path = paste0( | ||
| polis_data_folder, | ||
| "/", output_folder_name, | ||
| "/other_surveillance_type_linelist_", | ||
| min(other_combined$yronset, na.rm = TRUE), | ||
| "_", | ||
| max(other_combined$yronset, na.rm = TRUE), | ||
| output_format | ||
| ) | ||
| ) | ||
| )) | ||
|
|
||
| cli::cli_process_done() | ||
| } |
Collaborator
There was a problem hiding this comment.
we're still using the historical dataset to make the "other" linelist
| #' @returns `tibble` sia.clean.01 All historical SIA data | ||
| #' @keywords internal | ||
| #' | ||
| s3_sia_combine_historical_data <- function(sia.new, polis_data_folder) { |
Collaborator
There was a problem hiding this comment.
this whole function shouldn't be needed anymore, and wherever it's used. however, please double check that sia.06 contains sia data in 2000-2019. Our primarily goal is to see whether these "historical" datasets are something we need to run preprocessing properly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated init.R to give a warning if historical data files are not present instead of aborting.