Skip to content

❇️ Add ability to set throttle limits when performing parallel requests to the POLIS API - #309

Open
mcuadera wants to merge 29 commits into
devfrom
dev-call-url-using-curl
Open

❇️ Add ability to set throttle limits when performing parallel requests to the POLIS API#309
mcuadera wants to merge 29 commits into
devfrom
dev-call-url-using-curl

Conversation

@mcuadera

@mcuadera mcuadera commented Mar 27, 2026

Copy link
Copy Markdown
Collaborator

Closes #308

There are several changes this PR introduces, in anticipation of POLIS rate limiting requests in 2026. The following changes have been made in this new approach of pulling data from the API:

  1. Deprecate call_single_url() and call_urls() in favor of call_urls_in_parallel(). This decision was made because the old way we were pulling data did not respect rate limits. This can overload the POLIS API when pulling especially large files.
  2. Change functions that depended on either call_single_url() or call_urls(). The functions that had depended on them are: get_table_data(), get_table_ids(), run_single_table_diagnostic().
  3. Change function documentation that reference the two deprecated functions to use call_urls_in_parallel(). This includes create_table_urls().
  4. Pull call_urls() from utils.R into its own script, call_urls.R. This allows us to easily find where this function is at, as well as create paired unit tests.
  5. Update the unit tests for call_urls() and call_single_url(). There were changes in the behavior of call_single_url() that had impacted the unit tests of the function (and consequently call_urls(), since it depended on call_single_url()). These were updated to ensure unit tests now pass.
  6. Create unit tests for call_urls_in_parallel().
  7. Revise update_table_data() so that the number of IDs downloaded should equal the number of expected records for a table. Otherwise, we will keep redownloading the IDs of the table. This prevents our reference from erroneously labelling IDs as "deleted" if for some reason there was a failure in downloading that specific ID or IDs.
  8. Add lifecycle tags to the two deprecated functions so users are aware. For more details, please see this guide on how we communicate life cycle changes of our functions.

The function definition has a working example of how to run this function. Please see ?call_urls_in_parallel() so you can test it. In addition, I request the reviewers to perform the following tasks:

  1. Clone your own version of the POLIS folder in the sandbox in EDAV. Name this POLIS <your initials here>.
  2. Load the tidypolis R project and ensure you are in the correct branch dev-call-url-using-curl.
  3. Load via library(devtools) and run load_all() to source the functions.
  4. Run init_tidypolis() in your Sandbox POLIS folder. It's important that the runs are performed in independent folders.
  5. Run ?call_urls_in_parallel() to check the documentation and find the example.
  6. Run the example code.
  7. From step 4, you should've generated a variable called urls. Use this vector and pass it unto call_urls() and observe the behavior.
  8. Once tested, explore the unit test file associated with the new function. Run the unit test file by using this code: testthat::test_file("tests/testthat/test-call_urls_in_parallel.R"). I recommend using AI to explain what the tests are doing, but if anything remains unclear let me know.

Please focus your reviews on the .R files. Half of the changed files in this PR are just documentation changes that are read-only.

mcuadera added 29 commits March 25, 2026 11:33
allows us to call urls in parallel in posit workbench as it doesn't rely on the futures package and using background sessions nor cores for parallelization
only way to know we have a reliable record of reference
also silence deprecation warning when running testthat
and silence warnings during unit tests for call_urls
and generates a useful warning
and verify that the expected warning get emitted. please see _snaps to see the actual warning message emitted.
improves performance when there are many pages to bind, rather than continuously growing the the api_data tibble
@mcuadera
mcuadera requested a review from chadhunt2 March 27, 2026 16:24
@mcuadera mcuadera self-assigned this Mar 27, 2026
@mcuadera mcuadera added the enhancement New feature or request label Mar 27, 2026
@mcuadera

Copy link
Copy Markdown
Collaborator Author

Please see this document on how the approach differs, with included Mermaid sequence diagrams:

comparing new way to pull data from POLIS.pdf

The sequence diagrams were generated by copy and pasting the functions to the CDC EDAV Chatbot using the ChatGPT 5.2 model, with the prompt: "generate mermaid sequence diagram for my <call_urls()>/<call_urls_in_parallel()> function". This will give you code and you must make sure that you have Positron and the "Mermaid" extension installed so you can save this code as a .mmd file. Then, you can preview and download the mermaid diagram as a svg file. I've saved these files under our dev folder, in "Adhoc Analyses and Code/call_urls_mermaid_scripts".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant