Release v0.9.0: development → master#663
Merged
Merged
Conversation
Phase 3 of GH #218. Rewrite Panama food_acquired to the canonical long form (index (t,i,j,u,s); columns [Quantity, Expenditure, Price]; materialize: make) and register it, add a minimal sample table so v joins at API time, and drop the dead derived script. Derived food_{expenditures,prices,quantities} now auto-surface via the framework's _FOOD_DERIVED. i/j swap fixed: legacy mapped form/hogar(household)->j and item->i; the new wave scripts map household->i, item->j. Country('Panama').food_acquired() now gives n_i=8634 households >> n_j=81 items. s-axis driven by the SOURCE-FLAG columns, not the obtained-quantity bucket (fixes the round-1 refutation). Per wave the non-purchased 15-day quantity (ga110a / gai10a / s11a10a) is split by the survey's yes/no source flags: - own production (ga111a / gai111 / s11a11a) -> s='produced' - gift / donation (ga111b / gai112 / s11a11b) -> s='inkind' - part-of-payment + business (ga111c/d, gai113/114, s11a11c/d) -> s='other' The MONTHLY recall bucket (ga109a / gai09a / s11a9a) is deliberately NOT used: it overlaps ~100% with the 15-day bucket and summing both double-counts. Purchased Expenditure reconciles to raw per wave (sentinel-masked): - 1997 356079.56 == raw ga106c<9999 356079.56 (exact) - 2003 396676.87 == raw gai06c minus the open-ended "otro" catch-all slots (which cannot map to a named item) 396676.88 (exact on the harmonizable universe; full raw 397466.44) - 2008 547159.13 == raw s11a6c<99999 547159.13 (exact) sample table (index (i,t), v=upm formatted via panama.py's v()): 1997 from INFORM.DTA (v only -- no weight var), 2003 from E03BASE.DTA (the food file lacks upm) with weight=factor, 2008 from 02hogar.dta with weight=factor. food_items.org: add the 2003 spelling for Condimentos (was '---', a gap; the named item now harmonizes, closing the 2003 reconciliation). Verified (LSMS_COUNTRIES_ROOT=worktree, LSMS_NO_CACHE=1): food_acquired index (t,v,i,j,u,s) with v joined; is_this_feature_sane().ok True (only the framework-joined-v warn); derived tables non-empty; Panama present (modal, not excluded) in the full Feature('food_acquired')() (18 countries) and in Feature('food_acquired')(['Panama','Uganda']). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #581 added the Panama config + registration but omitted Panama from the Currency: section, so test_currency.py::test_every_country_has_currency[Panama] failed (Panama missing from Currency: section). Panama's ENV survey is denominated in Balboas (PAB), pegged 1:1 to USD; USD also circulates (dual currency), mirroring the Liberia entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nical feat(#218): canonical Panama food_acquired + register (actual 15-day recall, sample from upm)
The `blessed` tier was already fully implemented -- loader, tier
promotion, colour, the `★` glyph, docs -- but its file was a 21-byte
header, so the top of the ladder had never once been used.
`sane` and `blessed` answer different questions, and the gap between them
is where quiet errors live:
sane the automated checks passed. No human has necessarily looked
at a single number.
blessed a human read the actual numbers for that cell and believes
them.
A feature can build cleanly, pass every sanity check, and still be wired
to the wrong source column or carry an unverified unit conversion. For
anything feeding published analysis, `sane` is not enough.
Adopting the tier cost zero library code:
- blessed.csv schema extended to
`country,feature,wave,blessed_by,date,note`. `load_blessed()` reads
via `r.get(...)`, so the provenance columns are free -- and they are
what make a blessing auditable rather than a bare assertion.
- The rule, documented in CLAUDE.md and docs/guide/coverage.md: if you
used a cell in real analysis and looked at its numbers, bless it in the
same PR.
Seeded empty, deliberately. Blessings accrete; they are never
bulk-seeded. An empty blessing file is honest -- bulk-blessing the 1208
`sane` cells would make `blessed` a synonym for `sane`, destroy the only
distinction the tier draws, and assert a human review that never
happened.
Two things found while wiring it:
- CLAUDE.md never mentioned the coverage matrix at all (it landed in
June and the guide never learned about it). Added a section.
- `load_blessed()` calls `pd.read_csv(..., keep_default_na=False)` with
no `comment=` argument, so a `#` comment line in blessed.csv parses as
a *row* -- a phantom blessed cell. Documented; file kept header-only
rather than touching the loader.
Also records the corollary that bit two live issues: a `sane` cell is not
proof an issue is fixed. The matrix is a cold build, so it structurally
cannot see #588's warm-cache-only divergence, and it does not check
currency labels (#589). Both look green and are genuinely open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…teristics is the only site; BF-2014 French labels are a config fix (reuse the EHCVM mapping: idiom), the per-t all-NaN guard is the one 'new' piece Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
… grader) The coverage matrix measures, but it does not measure right. A census (2026-07-11) found that 595 of its 1849 cells -- 32% -- carry a tier that is wrong or unactionable, and that its errors bias in BOTH directions: it cries wolf about what we have, and stays silent about what we lack. A workplan built on those numbers is built on sand. This is the first half of fixing the instrument. Two defects, and the sequencing between them matters. ## 1. The wave-slice grader applied a COUNTRY-level check (128 cells) `coverage_matrix.grade_feature` grades a wave by slicing the country frame on `t` and calling `is_this_feature_sane` on the slice -- but `_check_no_all_null_columns` is a *country-level* check. A question simply not fielded in a given wave is legitimately all-null within that wave's slice, and was graded `builds` (a defect). It is a grading bug, not a data bug: of the 124 columns it flagged, ZERO are all-null country-wide. Every one is populated in some other wave. Fix: `is_this_feature_sane` gains an `extra_optional=` kwarg (default None -> historical behaviour byte-for-byte; it has many callers). The slice grader passes the set of columns populated somewhere in the parent frame. A column that is all-null across the WHOLE country still (correctly) fails -- we relaxed the slice, not the country. Verified on Malawi/Uganda/Niger: 434 cells, 37 changed, all `builds`-> `sane`, none unexplained by `no_all_null_columns`, ZERO regressions, and the 5 genuinely-broken cells (`no_null_index_levels` -- the `v`-join failure) correctly stayed `builds`. The fix is surgical. ### The sequencing rule this taught us A regrade is NOT a fix. Re-tiering 128 cells to `sane` would have SILENTLY BURIED two genuine gaps that the buggy grading was catching only by accident. Both were inventoried as issues BEFORE this landed: GH #591 Nigeria `food_prices()` silently drops ~99% of rows in 6 of 8 waves. A `Quantity == 0` sentinel (never decoded to NA) makes `Price = Expenditure/0 = inf`, which transformations.py:933 deletes. 675-1106 rows survive out of 120k-160k. All 6 cells were graded `sane`. GH #592 18 waves across 6 countries have ZERO GPS in cluster_features. Whenever a fix changes how cells are GRADED rather than what they CONTAIN, first inventory what the old grading was catching by accident. ## 2. `absent` was a black hole (456 cells) `absent` said only "not declared for this wave", conflating states that could not be more different -- so the number could never reach zero, and every probe's findings evaporated into the prober's head. Verdicts now live in `.coder/coverage/absent_verdicts.csv`: todo data IS there; config missing. Stays `absent` (still work) but is now sized and SOURCED. asked-not-distributed the instrument asked; the shipped extract lacks the variables. An ACQUISITION problem -- a different queue. CLOSES the cell. not-asked genuinely never asked. CLOSES the cell. unsure a required check could not run. Stays in the queue and RECORDS WHY. Silence is never evidence. `asked-not-distributed` is not a hypothetical: Albania 2004's questionnaire lists MODULE 2 DURABLE GOODS, and the variables are simply not in the shipped .dta. Data-side checks cannot see that, which is why C4 (the questionnaire) is mandatory before any permanent close. ### Evidence is load-bearing, and this already went wrong `load_verdicts()` REFUSES a closing verdict with an empty `evidence` field. A closing verdict is a permanent, unsupervised write; an unevidenced negative is unfalsifiable and therefore permanent whether or not it is true. `Albania/_/data_scheme.yml` asserted "earlier waves have no shocks module". It is FALSE. Verified against source: Albania 2005's `migrationE_cl.dta` carries `m6e_q00 = 'Type of Shock Code'` with ten shock types; 2008 is identical; 2003 and 2004 carry "faced any shocks in last 12 months". The claim was wrong, nobody could catch it (nothing recorded HOW it was reached), and it silently suppressed ~5 cells. Comment corrected; the four waves are seeded as `todo` with their file+variable citations, and 2002 as an honest `unsure` with its blocker named. A 38-cell pilot returned 25 todo / 13 unsure / **0 not-asked** -- so `absent` is not hiding "the survey never asked", it is hiding a backlog. This mechanism does not shrink the work. It makes the work knowable. Tests: 12 new (grader exemption, country-wide-null still fails, back-compat default, all four verdicts, and the evidence refusal). Docs in CLAUDE.md + docs/guide/coverage.md. Design: slurm_logs/DESIGN_coverage_discipline_2026-07-11.org Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ole waves
`roster_to_characteristics()` is handed the *country-level* concat of every
wave's `household_roster`, so a residence-duration column contributed by one
wave is unioned (all-NaN) onto every other wave. Three defects followed, each
silently returning NOTHING for a wave whose roster is fully populated:
D1 The resolution was an if/elif chain over the columns *present*, not
*populated*. Ethiopia carries both MonthsAway (W1-W3) and WeeksAway
(W4-W5, when the questionnaire switched to weeks -- exactly as CLAUDE.md
documents). MonthsAway won the elif, resolved to all-NaN on W4-W5, and
both waves vanished. -> per-ROW coalesce across the three sources,
priority MonthsSpent > MonthsAway > WeeksAway.
D2 No all-NaN guard. CotedIvoire 1985-89 and Mali 2021-22 never asked a
residence question; their MonthsSpent column exists only because a later
wave supplies it. The keep-mask was all-False -> 5 waves vanished.
-> per-`t` fallback to the documented "count everyone" behaviour ("countries
without any residence column are unaffected", CLAUDE.md), applied at wave
granularity, and it now warns rather than deleting silently.
D3 Burkina Faso 2014's MonthsSpent (EMC B3A) is the raw French 6-month binary
'6 mois ou plus' / 'moins de 6 mois'; to_numeric(errors='coerce') NaN'd the
wave away. -> fixed in CONFIG (the wave's data_info.yml `mapping:`),
following the EHCVM Oui/Non -> 12/0 pattern. No French in library code.
The filter itself is untouched wherever it has data: the fallback fires only
for a wave with no usable residence datum at all. Census over all 34 countries
that expose household_characteristics: the 8 dead wave-cells come back (+34,974
households) and all 88 other waves are byte-identical, Uganda included -- so the
1315-HH Uganda drift the filter was introduced to fix stays fixed.
Ledger: .coder/ledger/roster-characteristics-wave-deletion.md
Tests: tests/test_roster_residence_filter.py (6 of them fail pre-fix)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…the matrix The six defects in the coverage instrument, the four-way adjudication of `absent`, the NOT-ASKED evidence standard (and the pilot that refuted my original premise: 25 todo / 13 unsure / 0 not-asked), and a workplan ordered by leverage. Cited by PR #593. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tier, and stop a scoped run from destroying the snapshot Three more defects in the coverage instrument, plus one found while fixing them. ## Defect 4 — eleven countries were INVISIBLE `catalog.countries()` admits a directory only if it carries `_/data_scheme.yml`, so country dirs holding downloaded microdata and no config never appeared in the matrix at all. That hid **11 countries and 35 waves of already-downloaded data** (Afghanistan, Bosnia-Herzegovina, Brazil, Bulgaria, KenyaLPS, Kyrgyz Republic, Nicaragua, Panama, Peru, Rwanda, Tanzania_Kegera) -- ten of which Ethan has confirmed are in remit. New `unconfigured` tier + `unconfigured_countries()`. They are now visibly red rather than absent from the report. A denominator that omits the work you have not started is not a denominator. ## Defect 6 — `broken` was mis-tiered for the known-no-microdata countries All 8 `broken` cells in the cube were Armenia (2) and Nepal (6) -- both already documented in CLAUDE.md as having no source data in the repo. They cannot build; that is a data-availability fact, not a defect. Declared in the canonical `data_info.yml` (config, not code -- GH #436) and graded `n/a`. This matters beyond tidiness: the most alarming tier in the ladder contained *nothing actionable*, so a genuine build failure would have been lost among them. ## Found while fixing the above: a scoped run DESTROYED the snapshot `save_snapshot()` wrote wholesale, so `make matrix C="Uganda"` -- which grades only Uganda -- silently erased every cell it did not grade. `docs/guide/coverage.md` documents exactly that as the "spot refresh" procedure, immediately followed by `git add .coder/coverage/latest.csv && git commit`. Following the guide verbatim therefore committed a 67-cell snapshot over the authoritative 1849-cell one. I did it to myself while testing the renderer. `save_snapshot(..., merge=True)` (now the default) upserts on `(country, feature, wave)` and leaves un-graded cells untouched. A full sweep still rewrites everything it grades, so the authoritative run is unaffected -- it simply no longer depends on *not* being scoped. **A partial measurement must never be able to erase a complete one.** ## Notes - Two full-suite failures investigated, neither caused by this work: `test_feature_ghana_per_wave` is pre-existing and already tracked as **GH #589** (its title is literally that test name); the Albania `sample` failure was a parquet-cache race from running three heavy builds concurrently right after a `data_scheme.yml` edit invalidated Albania's content hash. Passes in isolation. - 11 new tests (34 in test_coverage_matrix.py total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s wrong Addendum #3 claimed "`PYTHONPATH` alone does NOT redirect imports of `lsms_library` to a worktree". That is FALSE -- PYTHONPATH does beat the `.pth`. The wrong claim sent agents to a needless mitigation (build a fresh venv inside the worktree, or verify by static diff only). The ACTUAL trap is `python <script>.py`. Python sets `sys.path[0]` to the SCRIPT's own directory, not cwd -- so `cd`-ing into the worktree does not protect a script run: cwd never enters `sys.path` at all, the `.pth`'s main-repo path wins, and you import the MAIN checkout while believing you are testing the worktree. Verified (cwd = worktree throughout): python -c "import lsms_library" PYTHONPATH unset -> worktree (cwd is sys.path[0]) python bench/scan.py PYTHONPATH unset -> MAIN CHECKOUT <-- the trap python bench/scan.py PYTHONPATH=<wt> -> worktree This is nastier than the old text described, because the agent has done everything that LOOKS right (cd'd into the worktree, run the worktree's own script) and still tested the wrong code -- silently. It has already cost a worker agent a full wasted audit run (found while reviewing PR #594). Since the failure is silent and looks like success, the guidance now says to ASSERT it rather than assume it: import lsms_library assert 'worktrees' in lsms_library.__file__, lsms_library.__file__ Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ethan's call: keep 12/0. The reasoning is not obvious and someone will
otherwise "fix" it later, so it now lives above the mapping itself and in the
ledger (§6, open question -> resolution):
- B3A's variable label is a DURATION question ("durant combien de mois [NOM] a
vecu dans le menage"), but the released variable is collapsed to a 6-month
binary: {1: '6 mois ou plus', 2: 'moins de 6 mois', 9: 'Valeur manquante'},
distributed 77,268 / 943 / (332 NaN).
- So 'moins de 6 mois' does NOT mean "lived zero months" -- those 943 people
(1.2%) DID live in the household, for 1-5 months. Read literally, mapping
them to 0 is false.
- That is where B3A differs from the EHCVM precedent it borrows: s01q12
("lived continuously 6+ months? Oui/Non") is a MEMBERSHIP test, where Non->0
is faithful. B3A is a binned duration.
- We keep 12/0 anyway because the 0 encodes "not a de-facto household member"
(the standard LSMS 6-month rule), not a claim about months lived -- and
consistency with the EHCVM family wins.
- 12/6 was rejected partly because 6 is the one value the respondent explicitly
ruled out. If we ever keep these members, the honest fill is 3 (the 1-5
month interval midpoint), not 6.
Comment-only + ledger; no behaviour change (tests/test_roster_residence_filter.py
still 13/13, including the mapping assertion it reads back out of the YAML).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…ves on it
discover_waves() decided whether we already held a WB catalog entry by
rebuilding a wave *label* from the entry's year range and string-matching it
against directory names. Labels collide, so this was wrong in both
directions:
* False positive -- Nigeria/2018-19/ holds GHS-Panel Wave 4 (id 3557). The
Living Standards Survey (id 3827) also spans 2018-2019, rendered to the
same "2018-19" label, and was therefore reported as already held. It is
not. Ethiopia 2671 (Land & Soil Experimental Research) was likewise
masked by ESS W2 (2247).
* False negative -- WB id 1001 (UGA_2005-2009_UNPS) renders as "2005-10",
matching no directory, so it looked missing even though we hold it, split
across Uganda/2005-06/ and Uganda/2009-10/.
Wave provenance is now explicit and machine-readable. lsms_library/
provenance.py extends the SOURCE.org that add_wave() already writes -- rather
than introducing a parallel mechanism -- with org keyword lines (#+CATALOG_ID,
#+CATALOG_IDNO, #+PROVENANCE_SOURCE in {worldbank, external, unknown}, ...).
The original URL stays the file's first URL, so the legacy _read_source_url()
reader is unaffected, and human-written prose is preserved verbatim under a
"* Notes (preserved ...)" heading. The writer is idempotent.
discover_waves() now matches on the recorded catalog id. Where no id is
recorded it falls back to the old label heuristic but reports
local_status='unknown' with local=False -- silence must not masquerade as
knowledge, and an unverified claim must not read as "held". `local` remains a
bool, so existing callers are unaffected.
SOURCE.org turns out to be load-bearing: Country.waves (country.py) falls back
to scanning for subdirectories containing Documentation/SOURCE.org when a
country declares no explicit wave list, so *creating* one silently promotes a
directory into a wave. The backfill therefore only ever UPDATES an existing
SOURCE.org, or CREATES one in a directory the library already declares a wave;
13 directories that are not declared waves (a stray Benin/2018-2019/ duplicate,
Bulgaria's five, Afghanistan's two, Brazil's, four KenyaLPS) are reported and
left alone. Country.waves is verified byte-identical to pristine for all 47
countries.
Also fixed:
* Serbia and Montenegro -> SCG (ids 80, 81 match our 2002/ and 2003/); it
previously had no ISO code and returned [] with an error.
* GhanaLSS/GhanaSPS (both GHA) and Tanzania/Tanzania_Kegera (both TZA) are
disambiguated by an idno_pattern in the new _COUNTRY_CATALOG registry;
each previously saw the other's waves as its own missing waves.
* EthiopiaRHS (IFPRI/Harvard Dataverse) and KenyaLPS are marked
discoverable=False with a reason, rather than merely left out.
Backfilled via scripts/backfill_wave_provenance.py: of 136 wave dirs, 123 are
declared waves and were stamped -- 118 resolved (52 from a recorded
/catalog/{id} URL, 48 by resolving a WB Microdata DOI against the catalog's
doi field, 17 external, 1 manual override) and 5 recorded as unknown. The
unknowns are declared waves with no SOURCE.org; none is guessed from its
directory name -- that heuristic is the bug being removed.
The single manual override is Nigeria/2018-19, whose SOURCE.org recorded 3827
(the LSS) while the directory holds sect*_plantingw4 / sect*_harvestw4 /
nga_*_y4 -- i.e. GHS-Panel W4 (3557), consistent with every sibling Nigeria
wave. The superseded URL is retained in the record.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…" claim was FALSE I wrote, in the design doc, in commit 8dcd2bc's message, and in PR #593: "none of the 136 wave directories records the WB catalog id it came from ... not one pre-existing wave has one (find ... -name SOURCE.org -> 0)" That is false. **117 of the 136 waves had a SOURCE.org all along.** My `find` used `-maxdepth 3`; the file lives at `{Country}/{wave}/Documentation/SOURCE.org` -- depth **4**. The command returned 0 because it could not see the files, not because they were absent. I ran one command, it returned zero, and I called it proof. That is exactly the *unevidenced negative* this very design doc spends its evidence-standard section warning against -- the same failure as Albania's "earlier waves have no shocks module". The rule applies to me too: silence is not evidence, and a null result from an instrument you have not validated is a null result *about the instrument*. The real defect is WORSE than the one I described. The provenance is not merely missing -- in places it is WRONG: Nigeria/2018-19/Documentation/SOURCE.org -> catalog 3827 (Nigeria LSS) Nigeria/2018-19/Data/ -> aux_plantingw4, aux_harvestw4 = GHS-Panel W4 (3557) A backfill that TRUSTED SOURCE.org would have cemented that error rather than fixed it. Absent provenance is a gap; wrong provenance is a lie, and it is the harder failure. PR #595 therefore validates rather than trusts, and keeps `none` (definitively not a WB dataset) distinct from `unknown` (we don't know) -- collapsing those would recreate the `absent`-black-hole mistake of Defect 1 one level down. Also records a trap found while implementing it: **SOURCE.org is load-bearing.** `Country.waves` (country.py:1452) falls back to scanning for dirs containing `Documentation/SOURCE.org`, so *creating* one PROMOTES a directory into a wave. Stamping all 136 broke `test_sample.py::test_covers_all_waves[Benin]`, because `Benin/2018-2019/` is a stray duplicate of `Benin/2018-19/` and stamping it promoted the duplicate into a wave `sample()` does not cover. Metadata that silently changes the shape of the data model is not metadata. Credit: caught by the wave-provenance agent (PR #595), not by me. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`Benin/2018-2019/` was a `Data/`-only staging duplicate of the configured
`Benin/2018-19/` wave. Removed at Ethan's direction (2026-07-12).
Verified safe before deleting:
- all **52** DVC sidecars are **md5-identical** to `2018-19/Data/`'s;
- **zero** files unique to either directory;
- no `_/` config, no `Documentation/`, not in `Country('Benin').waves`;
- referenced by nothing but the `Benin/_/CONTENTS.org` note describing it.
The DVC blobs are content-addressed and shared, so removing the duplicate
sidecars orphans no data.
## It was not merely clutter — it was a trapdoor
`Country.waves` (`country.py:1452`) falls back to scanning for directories
containing `Documentation/SOURCE.org`, so **writing a SOURCE.org promotes a
directory into a wave**. A provenance-backfill pass that stamped every
directory therefore turned this duplicate into a wave that `sample()` does not
cover, breaking `test_sample.py::test_covers_all_waves[Benin]` (found in
PR #595). Removing the directory closes that trapdoor rather than working
around it.
Post-delete: `Country('Benin').waves == ['2018-19']`, `sample()` (8012, 5),
`pytest tests/test_sample.py -k Benin` 9 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 0: fix the coverage ruler (blessed tier + absent verdicts + wave-slice grader)
…letion fix(household_characteristics): residence filter was silently deleting 8 entire wave-cells
feat(provenance): record each wave's WB catalog id; match discover_waves on it
Every wave dir that ships a SOURCE.org now records a terminal answer: 123
stamped, 0 unknown. Each of the five resolutions is validated against the
directory's actual contents and the live WB catalog -- a year match alone
proves nothing (that is what put the wrong id in Nigeria/2018-19).
Resolved to a catalog id:
* Niger/2011-12 -> 2050. Data/ contains a directory literally named
NER_2011_ECVMA_v01_M_Stata8 -- the WB idno itself.
* Senegal/2021-22 -> 6278. 58 ehcvm_*_sen2021 files; the only SEN EHCVM-2
entry (4297 is the 2018-19 wave).
* Nepal/2003-04 -> 74. Matched on idno + year range only: this wave's
Data/ is EMPTY (Nepal's microdata is NSO-hosted, not WB). So provenance
is KNOWN while the data is ABSENT -- two different facts, and this work
exists to stop conflating them.
Reclassified unknown -> none (a terminal state, not a gap):
* GhanaSPS/2013-14, GhanaSPS/2017-18. EGC-ISSER (Yale/ISSER) panel waves 2
and 3, not WB datasets. A GHA query across the ENTIRE WB catalog (all
collections, 152 entries) returns exactly one GSPS study -- 2534, the
2009-2010 wave -- and nothing later; the file naming (01a_consent.dta,
00_comm_info.dta) is the EGC-ISSER instrument. Leaving these 'unknown'
would mean someone re-searches them forever: the same absent-vs-unknown
black hole this work closes, one level down.
New #+PROVENANCE_VALIDATION: field records how strongly an id is corroborated
-- content-validated (the wave's own data files confirm it) vs catalog-only
(catalog metadata alone; no local data to check, as for Nepal). "Right for
the right reason" and "right by coincidence" must not look identical.
_declared_waves() now folds wave_folder_map values in: Country.waves is not
always a list of directory names (Nigeria's are rounds -- 2010Q3, 2011Q1;
Tanzania's are logical waves), and provenance is recorded per directory.
Caught by the new zero-unknown test.
Rebased onto development, which deleted the stray Benin/2018-2019/ duplicate
this branch surfaced. Country.waves verified identical to pristine for all 47
countries. Full suite: 3638 passed, 0 failed (test_currency.py::
test_feature_ghana_per_wave deselected -- it fails identically on development).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
The state between "obtainable" and "does not exist": the data EXISTS, we KNOW
where it is, and we CANNOT get it because the acquisition channel is broken.
Nepal (3 waves) NSO Nepal hosts the microdata, not the WB. Their site is
unreachable and no human contact has been reached.
Albania 1996 directory + WB catalog entry (2311, ALB_1996_EWS) both exist;
zero data files.
## I had this wrong, and the mistake was instructive
Two commits ago I graded these `n/a` under "Countries Without Microdata"
(Defect 6). But `n/a` means "no readiness applies" -- it reads as *nothing to
see here*, and it BURIED a gap we actively want. That is the exact lumping
error this module exists to prevent, committed by its author, in the fix for a
different lumping error.
`blocked` is *visible* (it counts against us) but *not actionable by us* --
nobody closes it by trying harder, and an RA who picks it up will waste a day
rediscovering that the NSO site is down. It sorts below the real defects and
above the settled tiers.
## The load-bearing part is the probe, not the label
A blocker is a CONDITION THAT CHANGES. A broken website heals; a dead contact
gets replaced. **"We hope we'll find it someday" is not a plan; a scheduled
recheck is.** So `load_blocked()` REFUSES a row with no `probe` URL (it could
never be rechecked, and would silently become the permanent close it explicitly
is not) or no `blocker` (an unexplained block is indistinguishable from
neglect). `bin/coverage_refresh.sbatch` -- which already self-resubmits, is
connectivity-gated, and already opens GitHub issues -- re-probes them.
## Two further bugs this flushed out
**Blockedness is a property of the SOURCE, not of a build.** I first wired it
only into the build-failure branch, so `make matrix-coverage` reported
`blocked=0` while four sources were plainly blocked. We do not need to try and
fail to build Nepal to know NSO's site is down. Hoisted; it now applies to the
cheap coverage-only layer.
**A blocked source we do not MODEL as a wave still has to appear.**
`Albania/1996` has a directory, Documentation, and a catalog id -- but no `_/`
and no `Data/`, so `Country.waves` never enumerates it and every per-wave loop
skipped it in silence. The blocked list is AUTHORITATIVE about sources we want;
it does not defer to whether the model happens to know about them. Otherwise
the denominator quietly omits exactly the waves we most want and cannot get.
## "Countries Without Microdata" is now EMPTY, and that is the point
Both of its entries were misclassified live gaps, each hidden behind a label
that reads "nothing to see here":
Nepal -> `blocked` (above).
Armenia -> NOT MISSING AT ALL. `discover_waves()` defaults to
collection='lsms', but Armenia's **18 waves of Integrated Living
Conditions Survey (2001-2018)** sit under repositoryid='central' --
invisible to our discovery, not absent from the world. Plausibly
the largest actionable acquisition gap in the repo. Filed as
**GH #597**, which also covers South Africa's General Household
Survey 2015-2025 (repositoryid='datafirst', 10 waves).
#597 matters beyond those two countries: we are teaching the matrix to report an
honest denominator of waves the WB publishes that we lack -- and that fix is
INCOMPLETE while `discover_waves` itself can only see one collection. We would
have built a denominator that is confidently wrong.
The ruler was miscalibrated. So was the ruler's own instrument.
Tests: 10 new (blocked beats n/a; probe/blocker refusal; un-modelled wave still
appears; a genuine build failure is still `broken`). 49 passing in
test_coverage_matrix.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 0 was meant to make the denominator honest. It did -- and the honest denominator is far larger than the matrix ever suggested, because the instrument could not see most of it. Armenia (18 ILCS waves, repo=central) and South Africa (10 GHS waves, repo=datafirst) are both IN REMIT (Ethan, 2026-07-12) and both structurally invisible to discover_waves(collection='lsms'). The queue as now known: ~43 waves to ACQUIRE, ~35 already-downloaded waves to CONFIGURE, on top of the 456 absent cells. None of this is new work appearing. It is work that was always there and could not be counted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ources Backlog reconciliation, and a prior-art miss I should own: **GH #552 (June 2026) already audited WB coverage gaps "tiered by obtainability"** — Tier 1 (WB-direct) / Tier 2 (NSO-redirect) / Tier 3 (licensed, fee-gated) — and even cites the Nepal->NSO precedent. I built the `blocked` tier without reading it. CLAUDE.md tells me to run the prior-art-ledger skill at the start of a non-trivial task; I skipped it. The reconciliation was productive in BOTH directions. ## What #552 gives us: `blocker_kind` #552's tiering is the right axis, and it belongs in the machinery as STRUCTURE rather than prose, because the kinds route to completely different people: nso-redirect the WB catalogues it but redirects to a national statistics office whose channel is broken. -> chase a human. fee-gated obtainable, for money and permission (Vietnam GSO: USD 200-2000 + consent). -> a BUDGET decision, not an engineering one. NEVER queue this for an RA. unconfirmed we believe it is blocked but have not proven it; it may be a plain obtainable TODO (Albania 1996). The honest state when you have not actually checked. "The site is down" and "it costs $2,000" are not the same blocker. Flattening them is how a money question silently becomes an engineering backlog item that nobody can ever close. blocked_sources.csv grows from 4 rows to 8, absorbing #552's Tier 2/3: Jamaica (JSLC, STATIN redirect), Ecuador (ECV 1995/1998, INEC redirect), and Vietnam (VLSS/VHLSS x5, GSO, fee-gated). They are now git-tracked, probed, and re-checked — not a snapshot in an issue that ages quietly. ## What we give #552: two of its claims are stale, one is wrong in fact - **"Kyrgyz Republic — wholly-missing"**: NOT missing. 4 waves, 146 DVC sidecars (~378 MB) are already downloaded; only the CONFIG is absent. It is one of the 10 `unconfigured` countries (#593). An add-feature job, not an add-wave one — a materially cheaper class of work than #552 assumes. - **"South Africa: no later WB-LSMS waves"**: true within `collection='lsms'`, FALSE in fact. The General Household Survey 2015-2025 sits under `repositoryid='datafirst'`. Same blind spot hides Armenia's 18 ILCS waves under `repositoryid='central'`. Both now in remit; see GH #597. #552's conclusions are correct GIVEN ITS INSTRUMENT, and its instrument could only see one collection. *A negative result from an uncalibrated instrument is a result about the instrument.* That is the third time that exact lesson has landed in this series -- twice on me. Also closed as verified-fixed during the sweep: #571 (Guatemala), #572 (Serbia), #573 (Panama — its data_scheme.yml landed today in a0ba3e1), #574 (GhanaSPS food_acquired; its sibling #579 stays open — GhanaSPS.sample() still raises). And corrected #587, whose stated hypothesis for its largest bloc (Nigeria's 22 `dropped` cells as "false positives") is refuted: the built `t` values are valid wave labels from the OPPOSITE PP/PH round. They are real half-missing features, not a reporting artefact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g the bookkeeping
Organizing principle: **the GitHub backlog should hold exactly what the coverage
matrix cannot.** Any issue that IS a cell is duplicate bookkeeping -- the
Phase-1 sweep re-derives it, sized and sourced, whether or not an issue exists.
Maintaining both is how a backlog becomes a museum.
the matrix owns WHICH cells are missing / broken / blocked (the WHAT)
the backlog owns WHY a cell is hard, cross-cutting bugs, and everything the
matrix CANNOT SEE
That last clause is not a throwaway. The grader has known blind spots, and they
are exactly where issues earn their keep: #588 (Uganda earnings is sane COLD and
broken WARM -- the matrix is a cold build and structurally cannot see it) and
#589 (currency labels -- it does not check them).
Track what the instrument cannot see. Let the instrument track the rest.
## Severity rule: order by HARM, not effort or age
1 silently wrong the user gets a wrong number and does not know. The only
UNRECOVERABLE class -- it propagates into published work.
2 silently missing the user gets less data than they think.
3 loudly broken it raises. Annoying, harmless. A crash is a gift.
4 untidy noise, dead code, ergonomics.
## Top of the board: #503 (Guyana), and it is not close
`i = [ED, HH]` CONFLATES DISTINCT HOUSEHOLDS. The source is at (ED, HH, SN)
granularity: COVERN.dta has 1502 unique (ED,HH) vs 1807 unique (ED,HH,SN). So
roughly 300 real households are silently merged into others.
That is not missing data. It is WRONG DATA THAT LOOKS FINE -- it builds clean
and passes every sanity check. Every Guyana number we have produced is suspect,
and nothing in the matrix flags it. The purest instance of the session's theme:
`sane` is not `blessed`.
## And #537 is that theme INVERTED
`validate_acquisition_source` has ZERO call sites, while its docstring asserts it
is wired into `_finalize_result`. A validation we believe is running, and is
not: an unevidenced POSITIVE. Same disease, opposite sign. A guarantee that
does not execute is worse than no guarantee, because it is trusted.
## Also
- Wave 4 is DECISIONS, not tickets (#575 purchased-value-only; #226; #279;
#438/#439; Vietnam's USD 200-2000 fee gate). An RA cannot close these by
working harder, and leaving a judgement in an engineering queue is how it
silently becomes a default.
- A large fraction of the ~29 open issues are not defects at all -- they are
cells, decisions, or acquisitions filed as issues because there was nowhere
else to put them. Now there is.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(provenance): resolve the last 5 unknown waves from evidence (0 unknown)
feat(coverage): the `blocked` tier — wanted, known, unobtainable (+ empty out "Countries Without Microdata")
…ually lives
`discover_waves()` searched a single hard-coded collection (`lsms`). Whole
household-survey series that the World Bank publishes under OTHER repositories
were therefore not "not yet fetched" but structurally unfindable:
* Armenia's Integrated Living Conditions Survey -- 18 annual waves,
2001-2018 -- sits in `central`. `lsms` carries only the 1996 Household
Budget Survey. Armenia was recorded as a country "without microdata".
* South Africa's General Household Survey -- 21 waves, 2002-2025 -- sits in
`datafirst`. `lsms` carries only the 1993 Integrated Household Survey.
That is 39 waves of in-remit data invisible behind a default argument, and it
made the coverage matrix's "honest denominator" (#593, #595) confidently wrong:
it reported no missing waves for Armenia while 18 sat in the catalog.
The fix is a per-country `repositories` list on CountryCatalog, defaulting to
("lsms",), unioned and deduplicated on the catalog id. 34 of 36 countries are
unaffected.
Deliberately NOT a removal of the collection filter. Searching everything
inflates results 30-400x (Findex, DHS, Afrobarometer, enterprise surveys;
`datafirst` alone returns 320 South African rows -- censuses, election studies,
school registers, media surveys). That trades a false-negative problem for a
false-positive one, and the second is worse: it buries the real gaps in noise.
So widening is paired with an `idno_pattern` that pins the survey series --
the lever #595 already built. The two compose: `repositories` says where to
look, `idno_pattern` says what counts. Both are config, not heuristics.
The pin also defends against a trap that id-matching alone cannot see: the WB
catalogues some studies TWICE, in two repositories, under two different ids.
`central` 3016 (MWI_2010_IHS-III_v01_M_v01_A_ML) is the same Malawi IHS3 as
`lsms` 1003, which we hold as Malawi/2010-11/; `datafirst` 902
(ZAF_1993_PSLSD) is the same 1993 survey as `lsms` 297 (ZAF_1993_IHS), which we
hold as South Africa/1993/. Nothing in the catalog metadata links those pairs.
Naive widening reports both as missing waves. They are not.
Armenia stays listed under "Countries Without Microdata" -- we still hold zero
data files, and delisting it would flip its coverage cells from `n/a` to
`broken`. What changes is the reason: an acquisition backlog, not an absence.
No data is acquired here; that is a separate `add-wave` job. `Country.waves` is
byte-identical to pristine for all 36 countries / 108 wave dirs.
Refs #597.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
Ethan's rulings on #597: * South Africa -- configure the GHS *series* (21 waves, 2002-2025), not the 2015-2025 range. Already what this branch did; no change. * Liberia HIES (central 2563, 2986) -- IN REMIT. Configured here. Liberia now searches ("lsms", "central") pinned by r"_(HIES|NHFS)_", which yields exactly three rows: the two HIES waves (missing) and the National Household Forest Survey (id 3787, which we hold as 2018-19/). The `central` noise -- energy-access framework, survey of public servants, Afrobarometer, and the global WLD_* rows tagged to every country -- is excluded. The NHFS is in the pattern for IDENTITY, not remit: it is the catalog entry backing a wave dir we hold, so discovery must be able to report it as held. By the remit predicate proposed in the PR, a forest-resources survey is NOT in remit -- Liberia is held-but-out-of-remit, the same class as KenyaLPS. This is where the two axes visibly diverge, and the divergence is deliberate: `idno_pattern` answers "is this catalog row this country's?", not "is this survey in remit?". Do not conflate them. The "widened => must pin a series" invariant is now asserted over EVERY country in _COUNTRY_CATALOG, not just the two named ones, so a future widening cannot silently skip the pin. Census: 75 rows; 47 missing in-remit waves (Armenia 18, South Africa 21, Liberia 2, plus 6 already visible in `lsms`). Still no data acquired. Refs #597. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…time Ethan's rulings on #597: South Africa takes the GHS **and** the IES + LCS (28 waves). Each series is now acquired for a STATED reason: GHS (21) roster, housing, education, employment. NO consumption module -- it cannot populate `food_acquired`. IES (5) income + expenditure + acquisition diary. LCS (2) living conditions + expenditure diary. IES and LCS are what feed the demand-estimation path. So the ZAF pin becomes r"_(IHS|GHS|IES|LCS)_" -- 29 rows, zero over-match, against the 320 ZAF rows `datafirst` actually returns. CAPABILITY PRE-POPULATES `not_asked` ------------------------------------ When SA-GHS lands, `South Africa/food_acquired/2015` grades `absent`, someone files it as a gap, and a probe (label sweep + sibling differential + questionnaire) eventually rediscovers a fact the WB catalog told us the day we acquired it. That is archaeology to recover knowledge we already had. `lsms_library/capability.py` records what a *series* measures when it is acquired. A `SeriesCapability` names the features it `provides` and `lacks`; `proposed_absent_verdicts()` turns each `lacks` entry into rows for `.coder/coverage/absent_verdicts.csv` in the schema `load_verdicts()` already reads. The probe sweep then only adjudicates cells inherited WITHOUT a capability record -- a shrinking set, not a permanent tax. AND IT MAY NOT CLOSE A CELL ON A CATALOG BLURB ---------------------------------------------- This is the whole discipline. A capability asserted from a cataloguer's topic list is NOT C4. Closing a cell on it would be the Albania mistake with better paperwork -- an unevidenced negative is unfalsifiable, and therefore permanent whether or not it is true. So each record carries a `validation` level, and the verdict is DERIVED from it rather than chosen by the caller: catalog-only WB topics / abstract -> `unsure` (no close) data-validated C1 label sweep negative -> `unsure` (no close) questionnaire-validated C4: the questionnaire read -> `not-asked` (closes) `data-validated` deliberately does not close either: a negative label sweep is exactly as consistent with `asked-not-distributed` as with `not-asked`. That is precisely why C4 is mandatory. Every record shipped here is `catalog-only`, so every one proposes `unsure`. The 21 GHS `food_acquired` rows it would emit are all `unsure`, `checks_run` empty, evidence prefixed `[catalog-only; C4 NOT run]`. Upgrading a series to `questionnaire-validated` -- one RA, one PDF -- is what converts them into a permanent `not-asked`: a bounded human step per SERIES, not per cell. `capability.audit()` enforces the invariant; `tests/test_capability.py` proves a catalog-only record cannot close a cell *through the real `load_verdicts()`*, and that the questionnaire upgrade does. No verdict rows are written -- SA is not acquired. The mechanism is proven, the shape is up for review. Rebased onto development: #596 resolved the Nepal/Niger/Senegal/Nigeria provenance gaps I had reported, and #598's `blocked` tier supersedes my data_info.yml edit (its emptied "Countries Without Microdata" map is taken wholesale). Census: 79 rows, 54 missing in-remit waves. Refs #597. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtARb9se2un1DLFJrrpjHe
…-canonical values (#602) `data_info.yml` declares accepted values via `spellings` blocks, but nothing enforced them. `_enforce_canonical_spellings()` maps known variants and never rejects unknown ones, and no sanity check read the declarations at all. Uganda's `sample.Rural` was the literal string '0' for 2,263 households (72% of the 2005-06 wave), so `df[df.Rural == 'Rural']` silently returned ZERO rows -- and `is_this_feature_sane(...).ok` returned True. The sweep is the deliverable; the check is how you do it. Sweeping all 34 countries x every spellings-constrained (table, column) through the live API found 10 violations, of which Uganda was the LEAST severe. The check --------- * `diagnostics._check_declared_spellings` -- status `fail`, naming offending values + row counts. Wired into both `is_this_feature_sane` and `validate_feature`. * Keyed on **(table, column)**, never column name: `housing.Tenure` is a legitimately different vocabulary (dwelling tenure) in ~12 countries and `housing` declares none; a name-keyed check emits ~12 false failures. * Membership is tested AFTER applying the variant map, so the check is correct on both pre-`_finalize_result` cached parquets (which legitimately hold known variants such as Malawi's `rural`/`RURAL`) and post-finalize API frames. * Reads `Columns` itself rather than reusing `country._load_canonical_spellings()`, whose `if variant_map:` guard DROPS columns whose variant lists are all empty -- `Affinity`, `Tenure`, `TenureSystem` would otherwise go unchecked. Schema ------ * Declare `sample:` under `Columns:`. It was absent entirely, which made `_enforce_canonical_spellings` a TOTAL no-op on `sample()` -- not merely failing to reject unknowns, but never mapping the known variants. That is the Tajikistan bug: 9,020 rows of lowercase 'rural'/'urban' next to canonical values, so `== 'Rural'` returned 2,419 of 8,209 true rural households. `Rural` is declared WITHOUT `required: true` (many samples have no such column). * Extend the `Rural` vocabulary with the French (EHCVM) / Portuguese labels ('Urbain', 'Urbano', ...), and with `Informal` -- South Africa 1993's third settlement stratum, kept VISIBLE rather than guessed into Urban (the source's own `metro` variable groups it with the non-metro areas). Data fixes (each a distinct root cause) --------------------------------------- * Uganda 2005-06: YAML key TYPE mismatch. Raw `urban` is object-dtype but MIXED -- python int 0 (2263) + str 'urban' (860). The declared key was the STRING '0', which never matches. Commit 9959b9f ("close GH #163 item 3") introduced exactly that string key and sat silently dead for three months. Fix: `0`. * Malawi cluster_features: the `mapping:` stanzas converted good labels into 0/1 and got the DIRECTION wrong -- the Cross_Sectional and Panel files disagree on case, so 2016-17/2019-20 mapped rural households onto BOTH codes, SIGN-FLIPPING the indicator between waves. Every raw value is already canonical or a declared variant, so the mappings are deleted and the label passed through. Agreement with the (correct) sample.Rural goes 0.0% -> 87-100% in every wave. * India 1997-98: `Rural` was derived from `stratum` via a float-keyed map against a string column, so it never fired and the raw stratum label ('UP-other', ' B-qual', ...) reached the user as `Rural` for 100% of households. `stratum` is a state x phase stratum (collinear with `state`) carrying NO urban/rural information, so `Rural` is DELETED, not invented. * Mali 2017-18 / GhanaLSS 2016-17: French strata labels, and a TRAILING SPACE ('Urban ') that looks right in a value_counts but compares false. * GhanaLSS 1991-92: the `rural` code->label table claimed 1 = 'Semi-urban'. loc2=1 nests exactly onto loc3 in {1,2} (Accra 459 + Other Urban 1119 = 1578), so code 1 is the union of the two URBAN strata. Latent (not currently reaching sample() in a clean build) but provably wrong; fixed in the org + CONTENTS. Remaining, deliberately not force-fitted ---------------------------------------- Five plot_features Tenure/TenureSystem pairs (Albania, Cambodia, Kosovo, Timor-Leste, Tajikistan) ship raw unharmonized survey labels. The check correctly FAILS them; they are enumerated in `tests.test_declared_spellings.KNOWN_UNHARMONIZED` and xfailed rather than downgraded to `warn`. Mapping them needs the questionnaire codebooks, and guessing would replace a visibly wrong value with an invisibly wrong one -- the exact class-1 failure this issue is about. BEFORE -> AFTER (isolated LSMS_DATA_DIR, race-free): violations 10 -> 5 (all 5 enumerated + xfailed) Uganda (Rural=='Rural') 0 -> 2263 of 3123 Malawi cf vs sample agreement 0.0% -> 87.2/100/90/97.5/96.1% by wave Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_ids.py `GhanaLSS/_/panel_ids.py` built its rename map with a bare `dict(zip(cur_i, prev_i))`. `PANELC.DAT` is PERSON-level and records two GLSS1 households that SPLIT into two GLSS2 households each, so that dict is silently MANY-TO-ONE: both split-offs are told to rename onto the same 1987-88 id. `id_walk` renames both, and `_normalize_dataframe_index`'s `groupby().first()` / additive `sum()` collapses them -- inside `load_from_waves`, i.e. BEFORE the L2-country parquet is written, so the loss is baked into the cache and every warm read is silent. Not a "rename-onto-occupied" collision as #548 states: the targets (101332, 114008) are 1987-88 ids and are absent from the live 1988-89 id set, so #536's `cur_set` guard is a no-op here. The guard that applies is the one this script bypassed: `local_tools.update_id`'s `_N` split suffix (the Malawi IHPS convention). Route the linkage through `local_tools.panel_ids()` (the Burkina pattern) instead of hand-rolling the maps. Damage (cold rebuild, before -> after): food_acquired (i=101332, t=1988-89) Expenditure 15,850 -> 5,570 15,850 was two different households' food summed into one (5,570 for HH 204932 + 10,280 for HH 204922); 101332_1 now carries 10,280. household_roster: 101332 was reported as a 6-person household whose members were entirely HH 204922's -- HH 204932 (an 83-year-old head + 3 others) was ERASED. Now 101332 = 4 persons, 101332_1 = 6. rows: sample 56,328->56,330 (+2), interview_date +2, household_roster +10, individual_education +3, food_acquired +35 un-summed tuples (-> food_expenditures +29, food_prices/quantities +35). GH#323 cold-build collapse warnings: 2/2/10/8 -> 0/0/0/0. Every row outside the two split lineages is byte-identical, including all of 1987-88. housing / plot_features / cluster_features / food_security: unchanged. `panel_ids.json` is byte-identical (the chain stores pre-suffix prev-ids, per `local_tools.panel_ids`); `updated_ids.json` changes exactly 2 of 714 entries. Also: - GhanaLSS carries #536's full-cover `cur_set` guard too (0 hits today, from Y00A.DAT), plus post-conditions that raise if the map is ever non-injective or collides with a live id -- turning a future class-1 silent defect into a crash. - Burkina_Faso `cur_set` built from the full 2021-22 cover rather than panel-only candidates (the unfixed #536 pattern; latent because its cover is 100% panel today). Both JSONs byte-identical, 95 skipped chains unchanged. - `diagnostics._check_panel_ids_targets_exist` now accepts `base_N` split ids as consistent -- that IS the library's convention. Malawi flips fail -> pass (1410/5686 = 25% false "inconsistent" entries removed). `local_tools` (id_walk / update_id / _close_id_map) is deliberately UNTOUCHED: the framework is correct; one bespoke script refused to use it. Verification: new tests in tests/test_panel_id_collisions.py fail on pre-fix code (3 failed) and pass after; the cross-country injectivity invariant flags only GhanaLSS -- the other 8 shipped updated_ids.json pass, so the negative results are about the data, not the instrument. Full pytest on an isolated data root: 3229 passed, 3 failed -- the same 3 that fail on pristine development (test_feature_ghana_per_wave / GH #589, 2x CotedIvoire/cluster_features). bench/feature_audit/scan.py over GhanaLSS: identical before and after (56 records, same 4 pre-existing `assembly_nonempty` fails). NOT fixed here (needs its own issue): Niger's 59 duplicate (i,t) tuples in 2014-15 are NOT this bug -- they are pre-walk, from `EXTENSION` in {0,1,2} in ECVMA2_MS00P1.dta while the wave's household id omits EXTENSION. pre-walk dups == post-walk dups == 59; `updated_ids['2014-15']` has 0 many-to-one targets and 0 rename-onto-occupied entries. Panel_ids.py cannot fix it. Prior-art ledger: .coder/ledger/548-panelids.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rebase Uganda sample baseline Follow-ups from the full-suite run, each a consequence of the #602 fixes: * India `_/data_scheme.yml` still declared `Rural: str` under `sample` after the column was removed from the wave YAML, so `_check_declared_columns` (and hence `is_this_feature_sane`) failed India/sample on a column that intentionally no longer exists. Removed the declaration; documented in India/_/CONTENTS.org. * `test_india_sample_has_no_fabricated_rural` asserted the wrong thing: 'UP-other' is a LEGITIMATE stratum label (only the Bihar pair carried a leading space). It now asserts Rural is absent and `strata` == the four true labels, whitespace normalised. * `tests/fixtures/uganda_baseline.json`: `var/sample.parquet` content_hash rebased. Uganda 2005-06's 2,263 '0' values are now 'Rural', so the parquet's contents legitimately change. Shape [24362, 6], columns, dtypes and index_names are all UNCHANGED -- the diff is exactly one line (the hash). Full suite, isolated LSMS_DATA_DIR: 3 failed / 3239 passed / 9 xfailed. Pristine development, same harness: 3 failed / 3224 passed / 4 xfailed. The 3 failures are IDENTICAL and pre-existing (test_feature_ghana_per_wave = GH #589; CotedIvoire/cluster_features x2). Zero new failures. The +5 xfails are the enumerated KNOWN_UNHARMONIZED plot_features pairs. Regression net (123 country/table pairs, live API, both branches): 112 BYTE-IDENTICAL; 7 changed, every one an intended fix; row counts unchanged everywhere (only India loses the fabricated Rural column). Feature('cluster_features'): 41376 rows / 32 countries / 0 dup-warnings on BOTH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(Tanzania): describe a cluster from inside it; re-key 2020-21 (GH #323, Site 2)
… claim Adversarial review found that this PR's four tests all pass on plain `development` WITHOUT the config change, so they pinned an invariant rather than testing the fix. Cause: commit `c8c25f68` (GH #602/#605) had already added `Urbano, urbano, URBANO` to `Columns.cluster_features.Rural.spellings`, and it is an ancestor of this branch. `_enforce_canonical_spellings` therefore repairs the value at API time, so `Country(...).cluster_features()` is canonical with or without this diff. Every API-level assertion is blind to the thing being fixed. The config change is still worth keeping: it fixes the EXTRACTION, so the stored L2-wave parquet is canonical rather than relying on a read-time net. That matters because the parquet is a consumer-visible artifact (`docs/guide/ caching.md`), and CLAUDE.md notes cached parquets hold PRE-transformation data -- so anything the net repairs is invisible in storage. Adds `test_wave_parquet_stores_canonical_rural`, which reads the wave parquet directly and so bypasses the net. Measured cold in an isolated `LSMS_DATA_DIR`: with the fix stored Rural = {'Rural': 3396, 'Urban': 2014} without the fix stored Rural = {'Rural': 3396, 'Urbano': 2014} Note on method: my first negative control was INVALID -- I used `git stash`, which found nothing because the config change is committed on this branch, so I rebuilt with the fix still in place and got a false "no difference". Reverting the file to `origin/development` is what actually removes it. Recording this because a negative control that silently tests nothing is the exact failure this commit exists to fix. The module docstring now states plainly that the API-level assertions do NOT discriminate, and which single test does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Assembles the per-country documentation sweep into one reviewable evidence
document at slurm_logs/POPULATION_STATEMENTS_2026-07-21.org, recording what
each wave's own documentation says its sample represents.
Scope: 111 waves across 38 country directories. 104 have a documented
statement of some kind; 7 have none. On the stricter reading -- "does any
document NAME the population this sample represents?" -- the answer is no for
10 waves, because 3 more record NOT FOUND for a universe while offering only a
fallback coverage claim.
Standing rule the sweep obeys: quotes are verbatim (sic spellings, source
typography and internal contradictions preserved; every PDF-extraction repair
declared), gaps are recorded as gaps with the full search record so the claim
is falsifiable, and nothing is inferred from the data -- no .dta file was
opened. Where inheriting a sibling wave's universe would have been defensible
(Mali / Niger / Senegal 2021-22 EHCVM-2 panels) it was declined and the entry
says so.
Structure:
1. what this is, how it was gathered, and the rule it obeys
2. summary table (country, wave, source_type, confidence, universe tag) --
the universe tag is marked throughout as an EDITORIAL reading, not a quote
3. the verbatim record per wave: population statement, translation where
present, exclusions called out separately, source + locator, and the
extractor's evidence notes
4. the gaps, unsoftened, with what was searched for each
5. what the evidence suggests, with disagreements flagged and NOT resolved
Findings worth the maintainer's attention: 41 of 111 waves' population record
exists only in WB catalog metadata, not in this repo, and three of those have
a <universe> element that is present and empty -- so a record built by linking
would track a moving target. A field literally labelled "Universe" exists for
only 47 of 111 waves. The single most repeated sentence in the corpus is WB
NADA boilerplate appearing in 15 waves (including Ethiopia and Nigeria, not
just the WAEMU countries), so its presence attests a metadata template rather
than a statistical office's claim. 19 waves subtract a named region by design;
exactly one declares an urban exclusion (Ethiopia ESS1). Specialization mostly
appears INSIDE waves, not as whole waves. Section 5.7 lists 17 unresolved
disagreements, including one where the editorial tag contradicts the wave's own
UNIVERSE field (Mali 2018-19), flagged rather than adjudicated.
No proposal is made. Per #601's own note, a proposal encoding a judgement
about what the library will pool on a user's behalf is a direction decision for
Ethan.
Refs #603, #601.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
`to_parquet` serialized object columns as
`astype(str).astype('string[pyarrow]').replace({'nan': None, 'None': None,
'<NA>': None})` -- it stringified FIRST and then tried to recover the nulls by
matching the resulting characters. After `astype(str)` a genuine missing and a
legitimate value spelling 'None' are the same characters, so the recovery is
wrong in principle. `None` was the canonical library label for "no education",
so every never-schooled person was nulled IN THE ACT OF WRITING the cache, and
`_finalize_result`'s `dropna(how='all')` then deleted the row -- the table has
exactly one column, so a nulled value is a deleted person.
Measured cold, isolated LSMS_DATA_DIR, both directions:
Guatemala individual_education 20,678 / 20,678 -> 29,527 (+8,849, 30%)
Tajikistan individual_education 59,297 / 54,462 -> 59,790 (+493 / +5,328)
Ethiopia individual_education 63,139 / 62,939 -> 63,181 (+42 / +242)
Guatemala was cold == warm == wrong, so no A/B comparison could see it and the
coverage matrix graded it `sane`. All three are now cold == warm, pinned as a
test.
Changes:
* local_tools.to_parquet -- capture the null mask BEFORE the cast, restore it
after. The pyarrow guard itself is KEPT: a genuinely mixed object column
still raises ArrowTypeError under pandas 3.0.2 / pyarrow 23, and narrowing
the block would change the stored dtype of homogeneous non-str object columns
(silently changing returned data corpus-wide -- the very failure class this
issue is about).
* LSMS_CACHE_SCHEMA 4 -> 5. The loss happened in the act of writing, so every
existing L2-wave and L2-country parquet already has the nulls baked in, and
the per-table input hashes are unchanged. Without the bump nothing rebuilds
and the fix is invisible on exactly the machines where the corruption is
present. Forces one full rebuild; this changes returned data.
* Canonical label `None` -> `No education` (38 rows across 20
harmonize_education tables, canonical_education_labels.org, 3 wave
data_info.yml inline mappings, China/_/china.py,
Uganda/_/_education_helpers.py). A canonical value colliding with the null
literal under str()/YAML/CSV/parquet keeps re-arming the trap. `None` /
`NONE` / `none` stay as accepted VARIANTS in a new data_info.yml `spellings`
block, so historical caches and un-updated countries resolve at API time.
* Ethiopia 2013-14 `individual_education` df_edit hook: one row of 12,583
carries a bare unlabelled float (76.0) that cannot be mapped in the org table
(mixed-text Original Label => string keys can never match a float source).
Folded onto `Unknown`, the vocabulary's own "unmappable" sentinel. Zero
effect on the row count; on the current base (post-#644) it is what keeps the
API frame inside its own declared vocabulary. Surfaced by
declaring the vocabulary.
* country.py: `dropna(how='all')` now logs how many rows it removed. This is
where a value corruption becomes a silent deletion, and 30% of a table
vanished here without a word.
Blast radius swept: 28 data-bearing countries cold (all cold == warm, all
in-vocabulary) + 107 freshly-built parquets audited against the declared
vocabulary (0 failures). Nigeria/food_acquired is the one other behaviour
change: its deliberate `u = fillna('None')` sentinel now survives as the
literal instead of becoming a NaN index key -- same row count (1,005,656) and
identical Quantity_kg. Guyana / South Africa `housing.Toilet == 'None'`
measured identical before and after (those columns arrive as `str`, so the
block never fired).
Tests: tests/test_gh645_to_parquet_null_coercion.py (17). Negative control
against unmodified origin/development: 14 of 17 fail. GH #323 guards re-run
green (no `aggregation:` key, no reducer).
Ledger: .coder/ledger/645-to-parquet-null-coercion.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
`pytest.raises(Exception)` would have passed on any error at all, including one caused by the test's own frame construction -- which is exactly the trap this file exists to guard against (an earlier draft of this very test built a mis-indexed frame and silently asserted nothing). Name the two errors pyarrow actually raises for a mixed object column, and write into a TemporaryDirectory instead of `tempfile.mktemp`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
First CI run on this PR failed at collection with BOTH import spellings:
from tests.conftest import requires_s3 -> ModuleNotFoundError: 'tests.tests'
from conftest import requires_s3 -> ImportError: cannot import name
'requires_s3' from 'conftest'
(<repo root>/conftest.py)
The repo has conftest.py at BOTH the root and in tests/, and under the CI
rootdir the root one wins the bare name while 'tests' resolves relative to
tests/ itself. This is a live trap for #648's helper: tests/conftest.py's own
docstring recommends 'from conftest import aws_creds_available', which no
module had yet exercised and which does not work in CI.
Load it by file path instead — import-mode agnostic, and still #648's single
source of truth rather than a fourth private copy of the creds check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
…inert The note's diagnosis is right and is kept verbatim: wave 2's asset module is a PER-UNIT roster, `item_seq` is exactly the level the canonical (t,i,j) grain drops, and `first()` destroys N147,297,485 (25.6%) in each quarter. Its PRESCRIPTION was wrong, and is now a tracked file telling the next reader to do something that cannot work. Edit 1 -- `index_info assets -> (t,i,j,item_seq)` -- is INERT FOR NIGERIA. Verified by experiment (PR #629, 2026-07-22): `item_seq` declared in 2012-13's `idxvars` AND `final_index`, wave rebuilt cold, and the frame still returns index ['i','t','j'] with identical rows and Value. The `dfs:` merge in `Wave.grab_data` drops the level BEFORE `final_index` applies -- that is GH #323 SITE 4 (PR #627), not Site 1. Nigeria cannot emit `item_seq` at all today. Worse than inert: making it canonical would leave edit 2 as dead code AND silently disable the GH #498 guard for all 25 assets countries, none of which carry the level either. Edit 2 -- `_ADDITIVE_MEASURE_COLUMNS['assets'] = ('Value',)` -- is correct and is implemented in PR #629. The note now also records the trap found while doing it, because it generalises to any future registration: adding a table to that dict ALSO SILENCES ITS GRAIN AUDIT. Harmless while `food_acquired` was the sole entry (its collapse reconciles every column it carries); `assets` carries `Age`, which no reducer reconciles at (t,i,j), so the naive registration converted a loud 13,447-destroyed-rows-per-round report into silence -- #323 reintroduced by its own fix. "Age wants mean" is SUPERSEDED and marked so: the grain contract's P2 requires every cell to hold an OBSERVED value or NA (a mean is neither), and core's last aggregation was retired at Site 2. Both are 2026-07-13 decisions that postdate the note. Comment-only; `Country('Nigeria').data_scheme` parses unchanged (24 tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-cartesian # Conflicts: # lsms_library/countries/Malawi/2010-11/_/data_info.yml # lsms_library/countries/Malawi/2019-20/_/data_info.yml # lsms_library/countries/Malawi/_/CONTENTS.org # tests/test_gh323_malawi_cluster_key.py
…r instead PR #648 added `tests/conftest.py` and told test modules to do: from conftest import aws_creds_available That does not work, and I shipped it merged. There is a tracked **repo-root** `conftest.py`, so the bare name resolves to THAT file, which has no such function. The other obvious spelling, `from tests.conftest import ...`, fails as `tests.tests` under pytest's import mode here. Both were tried on CI by the #645 work and both failed — the exact class of breakage #648 exists to prevent, in #648 itself. Fix: make the guard reachable WITHOUT an import. * register a `requires_s3` marker in `pytest_configure`; * add `pytest_collection_modifyitems` to skip marked items when credentials are absent. A test module now writes: pytestmark = pytest.mark.requires_s3 and imports nothing. The docstring records both broken spellings so the next person does not re-derive them. Verified both directions: creds present : 1 passed (marker does not over-skip) creds absent : 1 skipped (CI's shape, reason reported) The automatic `pytest_runtest_makereport` net from #648 is unchanged and still catches modules that forget the marker entirely; this adds the explicit, declarative path that was advertised but unusable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…olumn present" after the transforms Two review findings on the site-4 guard. The code was judged correct; these are the two places where it disagreed with machinery that had since merged. 1. `_merge_subframes` read `LSMS_GRAIN_STRICT` with a bare `os.environ.get` truthiness test. PR #614 merged `_grain_strict()` into this same module (`.lower() in {"1","true","yes"}`), and the private copy was NOT bit-identical: measured, `LSMS_GRAIN_STRICT=0` made site 4 RAISE while sites 1-2 stayed in warn mode. One lever, one reader. 2. The dropped-sub-df required-column check judged presence between `set_index` and `apply_derived` -- before `derived:`, `drop:` and the per-request `df_edit` hook -- while its country-level twin `_assert_built_required_columns` deliberately judges post-finalize. A required column supplied by a transformer or a wave hook would have been reported ENTIRELY ABSENT. Latent today; latent is a property of today's configs, not of the code. Moved below the hook. Also: the error message recommended `optional: true` "if they are genuinely unavailable for this wave", which misrepresents a COUNTRY-grain escape hatch as per-wave. It now says so, and points at the column name as the real fix -- in all ten cells this guard caught, the data existed (casing, wrong key, or a sibling file). Tests: +5, two of them proved discriminating by reverting the change. - test_grain_strict_is_read_through_the_one_shared_predicate: 4 failures with the old env read (0/false/no/off). - test_required_column_supplied_by_the_wave_hook_is_not_reported_absent: RuntimeError with the check in its old position. - test_hard_error_says_optional_true_is_country_wide pins the message. 28 pass; no skip, no xfail, no microdata -- the module builds a synthetic country in tmp_path, so none of it can go vacuous in a data-free CI job. Refs #323, #515, #614. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
…measure the census A false claim in a tracked file outlives the PR thread. Each correction below was re-derived here, not copied from the review. 1. `merge_how: left` prose (CLAUDE.md, the `sample` skill, the source comment) said the `outer` orphans "collapse together into one phantom null-keyed row". They do not. Ethiopia 2013-14, two isolated cold processes differing only in that key: `outer` 5,287 wave rows / 25 null-`v`; `left` 5,262 / 0; the delivered table is 433 clusters with sum(Latitude) 4070.3702 EITHER WAY, because the cluster-grain collapse DELETES null keys. `merge_how: left` is not a data fix -- it stops manufacturing rows for site 1 to delete, and stops the merge widening `District` from int8 to float64. Its cost is the lost signal `outer` gave you. (The "phantom 434th EA" story is not false, it is about the PRE-#628 config where `df_geo` was keyed on `v`. Ethiopia/_/CONTENTS.org and the 2013-14 `merge_how` comment still tell that story beside the new config; flagged for #628's owner, not edited here.) 2. Ledger §6 opened "67 `dfs:` merges exercised across 19 countries (the other 21 declare none)". The declaration count is statically checkable and is **76 blocks / 20 countries** at the branch base (80 / 20 after merging development). 67 was the number EXECUTED. Two quantities reported as one. 3. Ledger §6's Niger row said the geo file "has no lat/lon column at all" and prescribed `optional: true`. Half true and wholly the wrong conclusion: `NER_HouseholdGeovars_Y1.dta` has none, but the sibling `NER_EA_Offsets.dta` in the same DVC-tracked directory is 271 x [grappe, LAT_DD_MOD, LON_DD_MOD], 270 unique grappes, lat 11.876-18.747 N -- re-verified from source here. This is exactly the unevidenced "no module here" claim CLAUDE.md warns against. 4. Ledger §7's "0 of the 67 sub-frames is drawn from a `_pp_`/`_ph_` round file" overstated a FILENAME test. Nigeria's `df_main` is `Post Planting Wave 1/.../secta_plantingw1.dta`. The substantive conclusion (no cross-round cartesian at this site) rests on `t` being constant within every `dfs:` merge, and stands. Also corrected: §2/§5 (LSMS_GRAIN_STRICT IS on development now -- reuse `_grain_strict()`), §4 (the required-column check's blast radius was three countries and ten cells, not one), §9 (the three raises are fixed; the reason the suite was blind is `test_table_structure`'s read-only contract, NOT the v0.8.0 hash missing a core change -- see §11.5), §10 (sequencing discharged: #625, #628, #644 all merged; all three countries re-verified cold). New §11 answers each review finding, including one DISPUTED with measurement (§11.5: `country.py` build-path code IS in the v0.8.0 hash -- editing only `_merge_subframes`'s body moves `build_transforms_fingerprint`, and Nigeria's warm parquet grades `stale`). New §12 re-runs the whole census on the merged tree: 80 blocks / 90 merges exercised / **3 cartesian cells left**, whose phantom counts (183,812 + 171,230 + 59) reproduce §6 to the row. Refs #323, #515. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
fix(Nigeria): correct the assets KNOWN OPEN DEFECT note — its edit 1 is inert
docs(Tanzania): back-casting is a country fact, not a food_acquired quirk (+ dispatcher rule)
docs(#603): population-statement survey — all 111 waves, verbatim
fix(#323): Guinea-Bissau is Lusophone — French Urbain keys matched nothing, 169 clusters off-schema
fix(#323): add a `condition` index level to Uganda crop_production
fix(#323): assets Value is additive — recovers ₦294,594,970 (25.6% of Nigeria W2, × 2 rounds)
fix(#645): to_parquet stops deleting real data via string-matched nulls
fix(#648): the documented conftest import does not work — use a marker instead
…xisting on development `pytest tests/` on the merged branch: 2 failed, 3935 passed, 142 skipped. Both failures are the CotedIvoire/cluster_features pair, and both reproduce on a clean origin/development worktree with none of this PR's code — verified by running exactly those two node ids there, not by assertion. Zero new failures. Recorded with the §9 caveat attached: a green suite is necessary and not sufficient here, because `test_table_structure` reads cached parquets and never asks for a build. The evidence for this PR is the cold 90-merge census (§12). Refs #323. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
…in 38 country CONTENTS.org For each country, a new "Sampling universe, exclusions and sub-samples" section records *what the survey documentation claims the sample represents* -- verbatim, with a file/page or catalog-field citation for every quote, and with gaps recorded as gaps (naming what was searched) rather than filled by inference. This is EVIDENCE ONLY. No schema, YAML key, or code change is proposed here: GH #603 owns the representation decision. Nothing in these sections is inferred from the .dta files -- what a survey claims and what its microdata contain are kept strictly apart. Placement follows each file's existing structure: sections sit beside the country's existing "Sampling Design" / "Survey Program" material where that exists, and above the issue list otherwise. Malawi and Uganda are filed one level down as the last subsection of their existing "Sampling Design" section. Four countries had no _/CONTENTS.org and get one (Afghanistan, Armenia, Peru, Serbia and Montenegro). Where the new evidence contradicts text already in a file, both sides are quoted and the disagreement is flagged in place -- nothing existing was edited or deleted (Azerbaijan weights, CotedIvoire CILSS panel shape, Ethiopia ESS1 coverage, GhanaLSS GLSS1/2 rotating panel, Guinea-Bissau vague/grappe, Mali EHCVM-vs-EAC-I, Nepal panel, Nigeria wt_wave5, Uganda refresh cadence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
fix(#323) Site 4: the dfs: merge MANUFACTURES the duplicates — 4.9M phantom rows (UNBLOCKED: Ethiopia/Niger/Nigeria configs merged)
docs(#601): record the sampling universe, exclusions and sub-samples in 38 country CONTENTS.org
Lands on development so the development -> master release merge (#663) carries it, matching the docs/releases/v0.8.0.md precedent. Leads with the cache rebuild, because both invalidating changes ship in this release deliberately together (LSMS_CACHE_SCHEMA 4->5 and grab_data's build_transform fingerprint) so users rebuild once rather than twice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(release): v0.9.0 notes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release merge for v0.9.0.
developmentatfb749f03, CI green, 135 commits / 45 merges ahead ofmaster.Closes #548, #597, #602, #645
Headline
to_parquetwas nulling the literal strings'None'/'nan'/'<NA>', and'None'is the canonical "no education" label —dropna(how='all')then deleted the row. Guatemala recovers 8,849 rows (30% ofindividual_education), Tajikistan 5,328, Ethiopia 242. Closes #645.dfs:outer merge that manufactures the duplicate rows every other site then collapses.assetsValueis additive — ₦294,594,970 recovered for Nigeria (25.6% of W2 asset value), with the grain audit kept loud rather than silenced.spellingsare now enforced; 6 countries were shipping non-canonical values. Closes #602.discover_waves()searched onlycollection='lsms'. Closes #597.Plus GH #323 country fixes across Uganda, Tanzania, Malawi, Benin, Togo, Ethiopia, Niger, Nigeria, Albania, Pakistan, India and Guinea-Bissau, and the #637 key-soundness reviews.
Two changes each invalidate every cached parquet, and they are in this release deliberately together so the rebuild happens once rather than twice:
LSMS_CACHE_SCHEMA4 → 5grab_data's@build_transformfingerprintBugs fixed here that are live on
mastertodaysample().Ruralreturned the literal'0'for 2,263 rural households, sodf[df.Rural == 'Rural']returned zero rows.Urbain/Urbanospellings were absent, leaving unharmonised French incluster_features.CLAUDE.md's.pth/PYTHONPATHguidance was inverted, sending contributors to build a needless fresh venv.Documentation
CONTENTS.orgfiles.food_acquiredquirk.Deliberately NOT closed
#323, #601 and #637 remain open — each has real work outstanding (Nigeria's cartesian cells, the population-record design, the remaining
.first()inventory). Listing them here would auto-close umbrellas that are not resolved.