❇️ Add ability to set throttle limits when performing parallel requests to the POLIS API - #309
Open
mcuadera wants to merge 29 commits into
Open
❇️ Add ability to set throttle limits when performing parallel requests to the POLIS API#309mcuadera wants to merge 29 commits into
mcuadera wants to merge 29 commits into
Conversation
allows us to call urls in parallel in posit workbench as it doesn't rely on the futures package and using background sessions nor cores for parallelization
only way to know we have a reliable record of reference
also silence deprecation warning when running testthat
and silence warnings during unit tests for call_urls
and generates a useful warning
and verify that the expected warning get emitted. please see _snaps to see the actual warning message emitted.
improves performance when there are many pages to bind, rather than continuously growing the the api_data tibble
Collaborator
Author
|
Please see this document on how the approach differs, with included Mermaid sequence diagrams: comparing new way to pull data from POLIS.pdf The sequence diagrams were generated by copy and pasting the functions to the CDC EDAV Chatbot using the ChatGPT 5.2 model, with the prompt: "generate mermaid sequence diagram for my <call_urls()>/<call_urls_in_parallel()> function". This will give you code and you must make sure that you have Positron and the "Mermaid" extension installed so you can save this code as a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #308
There are several changes this PR introduces, in anticipation of POLIS rate limiting requests in 2026. The following changes have been made in this new approach of pulling data from the API:
call_single_url()andcall_urls()in favor ofcall_urls_in_parallel(). This decision was made because the old way we were pulling data did not respect rate limits. This can overload the POLIS API when pulling especially large files.call_single_url()orcall_urls(). The functions that had depended on them are:get_table_data(),get_table_ids(),run_single_table_diagnostic().call_urls_in_parallel(). This includescreate_table_urls().call_urls()fromutils.Rinto its own script,call_urls.R. This allows us to easily find where this function is at, as well as create paired unit tests.call_urls()andcall_single_url(). There were changes in the behavior ofcall_single_url()that had impacted the unit tests of the function (and consequentlycall_urls(), since it depended oncall_single_url()). These were updated to ensure unit tests now pass.call_urls_in_parallel().update_table_data()so that the number of IDs downloaded should equal the number of expected records for a table. Otherwise, we will keep redownloading the IDs of the table. This prevents our reference from erroneously labelling IDs as "deleted" if for some reason there was a failure in downloading that specific ID or IDs.lifecycletags to the two deprecated functions so users are aware. For more details, please see this guide on how we communicate life cycle changes of our functions.The function definition has a working example of how to run this function. Please see
?call_urls_in_parallel()so you can test it. In addition, I request the reviewers to perform the following tasks:POLIS <your initials here>.tidypolisR project and ensure you are in the correct branchdev-call-url-using-curl.library(devtools)and runload_all()to source the functions.init_tidypolis()in your Sandbox POLIS folder. It's important that the runs are performed in independent folders.?call_urls_in_parallel()to check the documentation and find the example.urls. Use this vector and pass it untocall_urls()and observe the behavior.testthat::test_file("tests/testthat/test-call_urls_in_parallel.R"). I recommend using AI to explain what the tests are doing, but if anything remains unclear let me know.Please focus your reviews on the
.Rfiles. Half of the changed files in this PR are just documentation changes that are read-only.