Repository navigation
Conversation
… years
ERS renamed the CSV members of two machine-readable zips:
Vegetables_Pulses.csv -> vegetables-pulses-data.csv, and in the livestock
and meat zip MeatSD.csv -> Meat-Supply-Disappearance.csv and
MeatStats.csv -> Meat-Statistics.csv. Reading the old names raised KeyError
for every vegetables and pulses table and five livestock tables. The readers
now try the new name, then the old one, case-insensitively; the CSV columns
are unchanged.
The farm household income tables now publish preliminary years such as
2025P. parse_year read only the forecast suffix F, so int("2025P") failed;
it now strips P too and the sort key orders P like F, after the plain year.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
USDA ERS changed two machine-readable downloads, and three openbb-government-us readers broke:
Vegetables_Pulses.csvis nowvegetables-pulses-data.csv. Every table raisedKeyError: "There is no item named 'Vegetables_Pulses.csv' in the archive".MeatSD.csvis nowMeat-Supply-Disappearance.csv, andMeatStats.csvis nowMeat-Statistics.csv. The supply-and-disappearance table and the four MeatStats tables (slaughter, production, live and dressed weights, cold storage) raised the sameKeyError.2025P.parse_yearstripped only the forecast suffixF, soint("2025P")raisedValueError.Change
ers_vegetables_and_pulses.py:CSV_MEMBERS = ("vegetables-pulses-data.csv", "Vegetables_Pulses.csv"). The reader takes the first one in the archive, matching case-insensitively.ers_livestock_and_meat_domestic_data.py: a catalog entry'smembercan be a tuple of names, new name first. The reader matches them the same way. The other two members are unchanged.ers_farm_household_income_and_characteristics.py:parse_yearstripsForP, andyear_sort_keyputs aPyear after the plain year, as it does forF.The CSV columns are unchanged, so parsing is untouched.
Tests
test_usda_vegetables_and_pulses.py::test_afetch_tableandtest_usda_livestock_and_meat_domestic_data.py::test_afetch_tableare now parametrized over the new member name and the old one.test_usda_farm_household_income_and_characteristics.py::test_parse_year_preliminary_suffix(new).test_catalog_contentstests assert the new constants:CSV_MEMBERSand the MeatStats tuple. They still count 10 tables, 7 members and 4 MeatStats tables.Without the change, the vegetables test module fails at import (no
CSV_MEMBERS). The livestock new-name case, the livestock catalog test and the2025Ptest fail. With the change, all of them pass. The provider suite passes: 3,259 tests (Python 3.12).Results from a deployment
We run these as an overlay on the 2.0.0 wheels in OpenBQ's Docker image. Before the change, a sweep of every widget returned "missing archive member" errors for the vegetables and livestock widgets, and
'2025P'errors for farm household income. With it, they return:Not included
Only these three readers are covered.
🤖 Generated with Claude Code