Tolerate OWID population column renames - #21
Merged
Merged
Conversation
OWID rewrites the population export in place and has renamed the value
column once already ("Population (historical estimates)" -> "Population"),
which broke both hardcoded pins at data-refresh -- the notebook path via
a silent rename no-op surfacing as a bare KeyError two lines later. The
column now resolves through owid_population_column() (newest spelling
first, diagnostic error on unknown), and notebook 103's ingestion filter
is an explicit ISO3-or-world-key allow-list that reports what it drops,
instead of relying on downstream source intersections to catch synthetic
aggregate codes. Regression tests cover both spellings and the failure
mode. The durable fix remains the pending Zenodo population vintage,
which converts this source from unversioned to a pinned sha256.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
renameno-op that surfaced as a bareKeyErrortwo lines later.owid_population_column()helper (newest spelling first, diagnosticDataProcessingErroron an unrecognised column) instead of two hardcoded string pins.Code.notna()check that would have silently admitted any futureOWID_*synthetic aggregate code into the intermediate artifact.fetch-datauser following the README hits this exact failure path, which is also the likely route a JOSS reviewer takes.natscenalready carries the consumer-side equivalent of this allow-list pattern; this brings the ingestion side in line.Lint & Test Results
ruff checkon the three changed files reports 2 pre-existing findings (UP038,isinstancetuple style) on lines untouched by this diff; no new findings introduced.Test Plan
pytest tests/unit/iamc_historical/test_owid_population_column.py -v-- 4/4 passed