Skip to content

fix(government_us): read USDA ERS renamed zip members and preliminary years - #7755

Open
artcashin wants to merge 2 commits into
openbq-org:developfrom
artcashin:fix/government-us-usda-renamed-files
Open

artcashin wants to merge 2 commits into
openbq-org:developfrom
artcashin:fix/government-us-usda-renamed-files

Conversation

@artcashin

@artcashin artcashin commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

USDA ERS changed two machine-readable downloads, and three openbb-government-us readers broke:

  • Vegetables and pulses. The zip member Vegetables_Pulses.csv is now vegetables-pulses-data.csv. Every table raised KeyError: "There is no item named 'Vegetables_Pulses.csv' in the archive".
  • Livestock and meat. MeatSD.csv is now Meat-Supply-Disappearance.csv, and MeatStats.csv is now Meat-Statistics.csv. The supply-and-disappearance table and the four MeatStats tables (slaughter, production, live and dressed weights, cold storage) raised the same KeyError.
  • Farm household income. The year columns now include a preliminary year, 2025P. parse_year stripped only the forecast suffix F, so int("2025P") raised ValueError.

Change

  • ers_vegetables_and_pulses.py: CSV_MEMBERS = ("vegetables-pulses-data.csv", "Vegetables_Pulses.csv"). The reader takes the first one in the archive, matching case-insensitively.
  • ers_livestock_and_meat_domestic_data.py: a catalog entry's member can be a tuple of names, new name first. The reader matches them the same way. The other two members are unchanged.
  • ers_farm_household_income_and_characteristics.py: parse_year strips F or P, and year_sort_key puts a P year after the plain year, as it does for F.

The CSV columns are unchanged, so parsing is untouched.

Tests

  • test_usda_vegetables_and_pulses.py::test_afetch_table and test_usda_livestock_and_meat_domestic_data.py::test_afetch_table are now parametrized over the new member name and the old one.
  • test_usda_farm_household_income_and_characteristics.py::test_parse_year_preliminary_suffix (new).
  • The two test_catalog_contents tests assert the new constants: CSV_MEMBERS and the MeatStats tuple. They still count 10 tables, 7 members and 4 MeatStats tables.

Without the change, the vegetables test module fails at import (no CSV_MEMBERS). The livestock new-name case, the livestock catalog test and the 2025P test fail. With the change, all of them pass. The provider suite passes: 3,259 tests (Python 3.12).

Results from a deployment

We run these as an overlay on the 2.0.0 wheels in OpenBQ's Docker image. Before the change, a sweep of every widget returned "missing archive member" errors for the vegetables and livestock widgets, and '2025P' errors for farm household income. With it, they return:

  • vegetables and pulses (default table): 56 rows;
  • livestock and meat: 204 rows;
  • farm household income: 27 rows.

Not included

Only these three readers are covered.

🤖 Generated with Claude Code

artcashin and others added 2 commits October 9, 2026 19:07
… years

ERS renamed the CSV members of two machine-readable zips:
Vegetables_Pulses.csv -> vegetables-pulses-data.csv, and in the livestock
and meat zip MeatSD.csv -> Meat-Supply-Disappearance.csv and
MeatStats.csv -> Meat-Statistics.csv. Reading the old names raised KeyError
for every vegetables and pulses table and five livestock tables. The readers
now try the new name, then the old one, case-insensitively; the CSV columns
are unchanged.

The farm household income tables now publish preliminary years such as
2025P. parse_year read only the forecast suffix F, so int("2025P") failed;
it now strips P too and the sort key orders P like F, after the plain year.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants