Skip to content

chore(data): remove third-party raw data from git — fetch instead - #20

Merged
setupelz merged 2 commits into
mainfrom
chore/remove-raw-data
Aug 3, 2026
Merged

setupelz merged 2 commits into
mainfrom
chore/remove-raw-data

Conversation

@setupelz

@setupelz setupelz commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • Untrack 14 third-party raw data files (PRIMAP, CMIP7, WDI, UNU-WIDER Gini/WDI, UN/OWID population, LULUCF, GCB bunkers, AR6 Gidden scenarios); they stay on disk and are fetched via fair-shares fetch-data.
  • Add exact-path .gitignore entries (no directory wildcards) so CITATION.md files stay tracked in each directory.
  • data/regions/*.csv, data/rcbs/*, and data/scenarios/iamc_example/*.xlsx remain tracked (own compilations / cleared example).

Lint & Test Results

  • Lint: ruff check . — 116 pre-existing findings, unchanged from main (unrelated to this change, out of scope)
  • Tests: pytest tests/ -v — 742 passed, 11 deselected
  • pytest tests/unit/test_data_registry.py -q — 58 passed (committed-data-tree test class skips cleanly with data untracked)

Test Plan

  • git status shows 14 staged deletions + modified .gitignore, nothing else
  • Files verified still present on disk after git rm --cached
  • git ls-files data/ | grep -i citation — 10 CITATION.md files still tracked
  • git ls-files data/ — no remaining .nc/.xlsx/.csv/.feather/.zip except data/regions/*.csv and data/scenarios/iamc_example/*.xlsx
  • Full test suite green

- fair-shares fetch-data reconstitutes the tree from empty (verified
  today: 206 MB, 13 files, all pinned checksums match; GCB is the one
  licence-gated manual download)
- files stay on disk for local work; .gitignore keeps them untracked
- CITATION.md files, own compilations (rcbs, regions) and the cleared
  example scenario stay tracked
- restricted sources (WIID CC BY-NC-SA, the AR6-derived ensemble) are
  no longer redistributed by the repo; clones shrink by ~200 MB
@setupelz

setupelz commented Aug 3, 2026

Copy link
Copy Markdown
Owner Author

CI is red on all three Python versions (3.11/3.12/3.13) — 10 failures, not a flake:

  • 6 in tests/integration/test_python_api_reproduces_notebook.py (co2-ffi-EPC, co2-ffi-ECPC, all-ghg-EPC-free-rider, all-ghg-EPC-co2-debit, all-ghg-PCA-capability, all-ghg-ex-ECPC)
  • 4 in tests/unit/test_rcb_generator_config.py (test_rcb_pathways_default_generator, test_rcb_pathways_explicit_generator, test_non_rcb_pathways_without_generator, test_integration_config_and_source_id)

All fail identically: ConfigurationError: Emissions data file not found: data/emissions/primap-202503/Guetschow_et_al_2025-PRIMAP-hist_v2.6.1_final_no_rounding_13-Mar-2025.nc.

Root cause: .github/workflows/test.yml states its own now-invalidated assumption — "Auto-fetch is disabled in tests (tests/conftest.py), so CI never downloads data." That design assumed the raw data files were present in the checkout because they were git-tracked. This PR untracks them, so a fresh CI checkout has no data and these 10 tests (which exercise real emissions data, not just registry/config plumbing) fail at config-validation time.

tests/unit/test_data_registry.py (the class this operation was verified against) does skip cleanly — 58/58 pass. The 10 failures are outside that class and have no existing skip-on-missing-data marker (checked: only pyam-availability skips exist elsewhere in the suite).

Local pytest tests/ -q shows 742 passed because the files are still present on disk locally — that check can't catch this, since the change's entire effect is on a fresh checkout with no local data cache.

This needs a maintainer decision before merge, since either option is a design change, not a git operation:

  1. Add a fetch-data step to CI (can't fully replicate local — GCB is licence-gated/manual per this PR's own commit message, so CI would still need a strategy for that source or would need to skip GCB-dependent tests specifically).
  2. Mark these 10 tests as requiring real data and skip them in CI (drops notebook-reproduction integration coverage from CI, which may be load-bearing e.g. for the JOSS paper).

Not merging until this is resolved — main is currently green (last run: success) and this PR would turn it red.

@setupelz
setupelz merged commit 1d1450e into main Aug 3, 2026
3 checks passed
setupelz added a commit that referenced this pull request Aug 3, 2026
chore(data): remove third-party raw data from git — fetch instead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant