Summary
The provenance model landed in #595 records one CATALOG_ID per wave directory. That assumes a 1:1 mapping between WB catalog entries and our wave dirs. It is not 1:1 in either direction, and the failures are confident false claims, not unknowns — which makes them the dangerous kind.
Failure 1 — one directory, MANY catalog entries (live, causing false "missing")
Tanzania/2008-15/ is a multi-round folder (see .claude/skills/multi-round-waves.md): one directory holding four logical waves.
Country('Tanzania').waves
-> ['2008-09', '2010-11', '2012-13', '2014-15', '2019-20', '2020-21']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ all four live in 2008-15/
Tanzania/2008-15/Documentation/SOURCE.org
#+CATALOG_ID: 3814 <- ONE id
#+CATALOG_IDNO: TZA_2008-2014_NPS-UPD_v01_M (the uniform-panel re-release)
The four individual NPS catalog entries (76, 1050, 2252, 2862) are not recorded anywhere. So discover_waves(), matching on id, cannot see that we hold them and reports all four as missing with local_status='no' — a confident false claim, not an honest unknown.
This is the single largest source of false "missing wave" rows in the census (found while building #597 / PR #599).
Failure 2 — one study, MANY catalog entries (invisible to id-matching entirely)
The WB catalogues some studies twice, in two repositories, under two different ids, with nothing in the metadata linking them:
| held as |
lsms id |
duplicate entry |
same survey? |
Malawi/2010-11/ |
1003 |
central 3016 |
yes — IHS3 |
South Africa/1993/ |
297 (ZAF_1993_IHS) |
datafirst 902 (ZAF_1993_PSLSD) |
yes |
Id-matching cannot detect this: we record 1003, the catalog offers 3016, no field connects them. Naive cross-repository widening therefore calls both missing. Confirmed by test in PR #599.
Why this matters more than it looks
#595 fixed a real bug — discover_waves was matching on a reconstructed label, which was wrong ~35% of the time. Matching on catalog id is strictly better. But it inherited an unstated assumption, and the failure mode is the same one we have been chasing all along: not a gap, but a confident wrong answer.
Absent provenance is a gap. Wrong provenance is a lie, and it is the harder failure. (Cf. Nigeria/2018-19/SOURCE.org recording 3827 — the Living Standards Survey — while the directory holds GHS-Panel W4, 3557.)
Any "honest denominator" built on a 1:1 assumption is confidently wrong for exactly the countries with the most complex holdings.
Suggested fix
- Allow multiple ids per wave dir.
#+CATALOG_ID: becomes a list (or repeats). Tanzania/2008-15/ records 3814, 76, 1050, 2252, 2862 — the release plus the constituent rounds. provenance.py already parses org keyword lines; this is mostly a list-ification.
- Record catalog-entry aliases. A small git-tracked equivalence map (
1003 ≡ 3016, 297 ≡ 902) so a duplicate re-release does not read as a gap. Keep it config and evidenced — each alias should cite why the two entries are the same survey.
- Distinguish "not held" from "cannot tell". Where a directory's provenance is structurally incomplete (multi-round, aliased), discovery must report
unknown, never a confident no. Silence must not masquerade as knowledge — and neither must a partial record.
Provenance
Found by the #597 discovery-widening agent (PR #599), which explicitly declined to fix it as out of scope and flagged it instead. Correct call — it is a distinct defect in #595, not a discovery bug.
Summary
The provenance model landed in #595 records one
CATALOG_IDper wave directory. That assumes a 1:1 mapping between WB catalog entries and our wave dirs. It is not 1:1 in either direction, and the failures are confident false claims, notunknowns — which makes them the dangerous kind.Failure 1 — one directory, MANY catalog entries (live, causing false "missing")
Tanzania/2008-15/is a multi-round folder (see.claude/skills/multi-round-waves.md): one directory holding four logical waves.The four individual NPS catalog entries (76, 1050, 2252, 2862) are not recorded anywhere. So
discover_waves(), matching on id, cannot see that we hold them and reports all four as missing withlocal_status='no'— a confident false claim, not an honestunknown.This is the single largest source of false "missing wave" rows in the census (found while building #597 / PR #599).
Failure 2 — one study, MANY catalog entries (invisible to id-matching entirely)
The WB catalogues some studies twice, in two repositories, under two different ids, with nothing in the metadata linking them:
lsmsidMalawi/2010-11/central3016South Africa/1993/ZAF_1993_IHS)datafirst902 (ZAF_1993_PSLSD)Id-matching cannot detect this: we record 1003, the catalog offers 3016, no field connects them. Naive cross-repository widening therefore calls both missing. Confirmed by test in PR #599.
Why this matters more than it looks
#595 fixed a real bug —
discover_waveswas matching on a reconstructed label, which was wrong ~35% of the time. Matching on catalog id is strictly better. But it inherited an unstated assumption, and the failure mode is the same one we have been chasing all along: not a gap, but a confident wrong answer.Any "honest denominator" built on a 1:1 assumption is confidently wrong for exactly the countries with the most complex holdings.
Suggested fix
#+CATALOG_ID:becomes a list (or repeats).Tanzania/2008-15/records3814, 76, 1050, 2252, 2862— the release plus the constituent rounds.provenance.pyalready parses org keyword lines; this is mostly a list-ification.1003 ≡ 3016,297 ≡ 902) so a duplicate re-release does not read as a gap. Keep it config and evidenced — each alias should cite why the two entries are the same survey.unknown, never a confidentno. Silence must not masquerade as knowledge — and neither must a partial record.Provenance
Found by the #597 discovery-widening agent (PR #599), which explicitly declined to fix it as out of scope and flagged it instead. Correct call — it is a distinct defect in #595, not a discovery bug.