fix(Malawi): 2004-05's cluster key was an intra-TA sequence number (GH #323, config-only)

[ligon]

Malawi's half of GH #323 Site 2, config-only. Refs #323.

No file under lsms_library/*.py is touched, no aggregation: key is added (dead config, forbidden by D1), no reducer is declared and nothing is blanked to <NA>. Every number below is a cold build against an isolated LSMS_DATA_DIR (fresh tree, dvc-cache symlinked to the shared L1) with LSMS_COUNTRIES_ROOT pointed at the worktree and asserted before any run — LSMS_NO_CACHE=1 alone was not trusted, because the collapse is baked into the cache it poisoned.

---

The finding: two different diseases, and only one is a broken key

cluster_features is declared (t, v) but all five waves extract it from the household cover page, so the frame arrives at household grain and _collapse_to_cluster_grain reduces it with .first(). Measured per column, per cluster, on the frame captured inside that function before it runs:

wave	contested cells before	after	clusters	verdict
2004-05	164 / 330 (49.7%)	0 / 1,692	110 → 564	broken key
2010-11	0 / 3,840	0 / 3,840	768	already correct
2013-14	765 / 1,020 (75.0%)	765 / 1,020	204	tracked panel dispersal
2016-17	78 / 4,405	89 / 4,400	880	Region 1→0; GPS 0→779 clusters
2019-20	250 / 4,095 (6.1%)	250 / 4,095	819	tracked panel dispersal
total	1,257 / 13,690	1,104 / 15,047	2,782 → 3,235

Two of the five waves now emit no GrainCollapseWarning at all; before, four did.

(The brief's baseline was 1,255 — the 2-cell difference is in 2004-05 and bears on nothing.)

1. 2004-05 — ea is not a cluster id, it is a sequence number

ea is the EA's position within its Traditional Authority: 110 distinct values for IHS2's 564 enumeration areas. Keyed on it, roughly five unrelated EAs — different TAs, different districts, different regions — merge into one v. Measured at source:

key	groups	region	district	reside
ea	110	66	79	19
(dist, ea)	447	0	0	8
(dist, ta, ea)	564	0	0	0
psu	564	0	0	0

The (dist, ea) row is the instructive one: it is the "make v a district composite" reflex — Uganda's fix — and here it is wrong, because it still merges EAs that share a district but sit in different TAs. The right key needed no composite at all: psu is already a column in sec_a.dta, the fully qualified 8-digit code region(1)+district(2)+TA(2)+EA(3), 564 values with exactly 20 households each (the IHS2 design). It is also the same 8-digit keyspace the IHS3/IHS4/IHS5 waves use for ea_id, so v now has one format across the whole country.

What the broken key was actually doing, at the API

	v: ea	v: psu
clusters	110	564
districts in cluster_features	20	26
settlement strata in cluster_features	1 — all Rural	2

.first() reached a rural household in every one of the 110 merged groups, so IHS2's entire urban stratum was absent from cluster_features, and 6 of Malawi's 26 districts had vanished outright. Neither was visible at the API: the table looked complete.

Declared identically in sample and cluster_features — they cannot be allowed to drift, or _join_v_from_sample matches nothing.

Frame vintage, stated so nobody reads too much into it: 2004-05 shares only 3 codes with 2010-11 (1998 vs 2008 census frame). The format is common to all five waves; the code sets are not. A cross-wave join on v is meaningful from 2010-11 on and not before, and this change neither creates nor claims a panel link.

2. 2013-14 and 2019-20 — the residual is dispersal, and it is not re-keyed

ea_id in the IHPS waves is the IHS3 baseline EA (case_id is literally ea_id + household number), the panel EA sets nest cleanly — 2019-20-PN = 2016-17-PN = 102 ⊂ 2013-14 = 204 ⊂ 2010-11 = 768 — and the survey tracks movers and split-offs wherever they go.

The GPS settles it. 188 of 204 EAs in 2013-14 have households at different coordinates, which is exactly the broken-key signature. It is not one:

Restrict to households that did not move (dist_to_IHS3location ≤ 1 km) and every one of the 204 EAs collapses to exactly one coordinate.

LAT_DD_MOD is the EA-level displaced fix of the household's current location — not per-household jitter — so all of the within-EA spread (median 145 km, max 775 km) is dispersal. Corroborating: 588 of 4,000 households report a district different from the one encoded in their own ea_id, and they sit a median 68 km from their IHS3 location against 0.03 km for the 3,412 that match.

Why the 75% headline is not what it looks like. With ~20 households per EA and 13% of them district-movers, 1 − 0.87²⁰ ≈ 94% of EAs are expected to contain at least one mover. A high cell rate is a property of the metric applied to a tracked panel, not evidence of a merged key.

Why not re-key anyway. Splitting 2013-14 on (district, ea_id) gives 626 groups of median one household — 422 singleton pseudo-clusters — desynchronises v from sample and from the IHS3 baseline, and destroys the panel's link to its own sampling design. Same precedent as Uganda's comm (PR #634 §6.5). So these residuals are reported, not forced to zero.

2019-20's 250 cells are all in the panel half, on the same mechanism (747 of 3,178 households off their EA's modal district, 652 of them >1 km from their IHS3 location). Its 102 panel EAs share no ea_id with the 717 cross-sectional ones — no cross-half collision.

3. 2016-17 had GPS for 0 of its 880 clusters, and the cause was a merge key

df_main builds Cross_Sectional i as cs_i(case_id) ('cs-17-' + format_id) to keep it apart from the Panel's y3_hhid. df_geo declared the raw case_id, so its merge key matched nothing: the how='outer' join produced 12,447 orphan geo rows next to 14,955 coordinate-less main rows. The two files are exactly 1:1 on case_id (12,447 = 12,447, none on either side alone), so mirroring the transform makes the join exact — 779 of 880 clusters now carry coordinates (the other 101 are IHPS panel EAs, for which IHS4 publishes no geovariables).

Honest cost: 7 of those 779 EAs have households whose IHS4 lat_modified differs within the EA — IHS4's displacement is nearly but not exactly EA-constant — so Latitude/Longitude show 7 contested cells where before there were 0, because before there was no coordinate at all. That is the whole of the 78 → 89 movement in the table above.

This is not a cardinality fix, and 2016-17 is not on PR #627's cartesian list: a non-matching key produces orphans, not a product.

Separately, 2016-17's single Region conflict was a spelling: EA 30305580 is sampled in both halves (a genuine shared EA, same district), and the CS file writes Southern where the Panel writes South. The existing region table — which already collapses them at API time, so this is output-neutral — is now applied at extraction, so the collapse stops reading a spelling difference as a disagreement about the cluster. 1 → 0.

4. Site 4: measured, and deliberately left alone — for #627

2010-11 and 2019-20 key a household-grain geovariables file on ea_id and merge it on the cluster key v, so pd.merge pairs every household with every geo row sharing that v:

wave	households	geo rows	merged	phantom
2010-11	12,271	12,271	196,083	183,812
2019-20	14,612	11,434	185,842	171,230

The load-bearing claim, measured rather than argued: a cartesian repeats values, it does not invent them, so it cannot change a contested-cell count. The one-line config fix (idxvars: {i: case_id} + merge_on: [i]; both file pairs are 1:1 on case_id) was implemented and measured — contested cells 0 and 250, identical either way — and then reverted, so PR #627's cardinality guard keeps these two cells as evidence. #627 owns the decision. The fix is recorded in each wave's data_info.yml next to the declaration that causes it, and a test fails if either wave silently starts merging on i without telling #627.

5. Rural needed nothing

The pre-#602 inline mapping: blocks were both live and inverted — rural: 0 in one file and RURAL: 1 in another, in the same wave, so the two halves of 2016-17 and 2019-20 coded rural households with opposite values. GH #602 already removed all five on development. Verified against the pre-collapse frames: Rural arrives as the raw label and _enforce_canonical_spellings normalises it. Nothing added here; noted only so the next reader does not re-derive it.

6. The invariant

sample.v ≡ cluster_features.v in every wave after the change: 0 cluster_features (t, v) pairs unknown to sample, 0 orphaned sample clusters.

wave	sample v	cluster_features v	unknown to sample	orphaned
2004-05	564	564	0	0
2010-11	768	768	0	0
2013-14	204	204	0	0
2016-17	880	880	0	0
2019-20	819	819	0	0

7. Tests

tests/test_gh323_malawi_cluster_key.py — 14 tests, all config-level; exercises no core aggregation. They pin the fix and the diagnosis: test_2013_14_within_cluster_spread_is_tracked_movers_not_a_broken_key fails if the evidence for leaving the IHPS waves un-re-keyed ever stops holding.

• On this branch: 14 passed.
• Negative control: against a pristine origin/development config tree (exported with git archive, cold data dir), 7 of the 14 fail — including every load-bearing one.
• Malawi-relevant slice (-k "malawi or Malawi or 323 or schema or sample or cluster"): 680 passed, 58 skipped, 4 xfailed, 1 failed.

Pre-existing failure, not caused or fixed here (core is out of scope): tests/test_gh323_explicit_reducers.py::test_core_does_not_dispatch_the_reducers greps lsms_library/country.py for the substring collapse_to_cluster_grain and finds PR #617's private _collapse_to_cluster_grain. This branch is config-only and does not touch country.py. Same failure is reported on PR #634.

8. Deferred

• Site 4 / PR fix(#323) Site 4: the dfs: merge MANUFACTURES the duplicates — 4.9M phantom rows (UNBLOCKED: Ethiopia/Niger/Nigeria configs merged) #627 — §4 above.
• Extraction-time region/district harmonisation is applied only where it removed a measured conflict (2016-17 Region). Doing it everywhere is output-neutral and tidier, but it is a diff across five waves for zero measured gain.
• 2016-17's 101 and 2019-20's 102 panel clusters have no coordinates: IHS4/IHS5 publish geovariables for the cross-sectional half only. Pre-existing, unrelated to bug: framework silently drops rows on duplicate canonical index (groupby().first()) #323.
• IHPS-2016 mixes baseline and current geography in one file — district is baseline (0 of 2,508 households outside their EA's modal district) while reside is current (277 differ, 246 of them flagged mover). That is the source's inconsistency; it accounts for all 75 of 2016-17's remaining contested cells, and it is documented rather than papered over.

Diagnosis preserved in Malawi/_/CONTENTS.org and .coder/ledger/323-malawi-key.md.

🤖 Generated with Claude Code

[ligon]

Adversarial review — APPROVE_WITH_NOTES

I tried to break this rather than confirm it. Every number below is my own cold measurement: two git worktrees (origin/fix/323-malawi-config and origin/development), each with its own isolated LSMS_DATA_DIR (fresh tree, dvc-cache symlinked to the shared L1), LSMS_COUNTRIES_ROOT + PYTHONPATH pointed at the worktree and asserted in-process. No dvc CLI was invoked.

The headline numbers reproduce

Contested cells measured by monkeypatching _collapse_to_cluster_grain read-only to capture the pre-collapse frame, nunique(dropna=True) > 1 per (cluster, column):

wave	dev (mine)	PR branch (mine)	PR claims
2004-05	164/330 {Region 66, District 79, Rural 19}	0/1,692	164 → 0 ✓
2010-11	0/3,840	0/3,840	✓
2013-14	765/1,020	765/1,020	✓
2016-17	76/4,400 {Region 1, Rural 75}	89/4,400 {Region 0, Rural 75, Lat 7, Lon 7}	78/4,405 → 89/4,400 (see note 1)
2019-20	250/4,095	250/4,095	✓
total	1,255	1,104	✓

At the API, cold: 2004-05 goes 110 → 564 clusters, 20 → 26 districts, {Rural} → {Rural, Urban}; 2016-17 Latitude non-null goes 0 → 779 clusters; sample.v ≡ cluster_features.v with 0 unknown / 0 orphaned in all five waves; GrainCollapseWarning count 4 → 3.

What I attacked, and what held

1. "Did the new key score zero by splitting real entities?" No. At source, psu ≡ (dist, ta, ea) is an exact bijection — 564 ↔ 564, max(psu per (dist,ta,ea)) == 1 and max((dist,ta,ea) per psu) == 1 — with exactly 20 households in every one of the 564 groups. psu[-3:] == ea for all 564. So it is neither a split nor a coarsening of the real EA; it is the same partition, spelled with the qualified code. (dist, ea) — the "district composite" reflex — really does give 447 groups and 8 still-contested reside cells, so the PR's rejection of it is right.

2. "Invariance by missingness?" No. psu / ea / region / dist / reside / ta have zero NaN across all 11,280 rows. And the urban stratum genuinely exists: 1,440 urban households in 72 distinct PSUs, of which groupby('ea').first() surfaced 0. The claim that the old key erased the urban stratum is literally true.

3. "Did the fix move mass, not just rows?" It did not move mass at all. sample and household_characteristics are identical between the two trees for every wave once v is excluded — same row counts (11,280 / 11,276 for 2004-05), same i sets, DataFrame.equals True on every non-v column. Only the identifier changed.

4. "Are the 2016-17 coordinates attached to the right EAs?" Independent check, not self-consistency: each 2016-17 cluster's new coordinate vs the median coordinate of its own district in 2010-11 (a wave whose geo was already correct) → median 17.0 km, p90 41.7 km. Control with district labels shuffled → median 240.4 km. The join is right. Files confirmed 1:1: 12,447 = 12,447 case_id, 0 on either side alone. 7 of 779 EAs really do carry >1 lat_modified, as disclosed.

5. "Does the extraction-time Region mapping silently NaN anything?" No. The region table covers all five labels the two files use (North, Central, Centre, Southern, South), and cluster-level Region counts are identical between trees: {Southern 407, Central 306, North 167} = 880, no NaN. Output-neutral, as claimed. Corroborated at source: exactly one EA (30305580) is in both halves, CS spells Southern, Panel spells South.

6. "Is the 'leave 2013-14/2019-20 alone' story real?" The 2016-17 half of it checks out at source: 75 of 102 panel EAs (0 of 779 CS EAs) have >1 reside; panel households are 0/2,508 off their EA's modal district but 277 off its modal reside. The 2013-14 GPS argument is pinned by a test that I ran and that passes.

7. "Does the new test discriminate?" Yes — verified, not taken on trust. Running the PR's test file against a pristine origin/development config tree (same core, cold data dir): 7 failed, 7 passed, exactly the 7 load-bearing ones the PR names. On the branch: 14 passed.

8. "Does anything else break?" 8 relevant test modules (test_gh323_malawi_cluster_key, test_gh323_grain_contract, test_grain_collapse{,_cluster_features}, test_sample, test_schema_consistency, test_table_structure, test_gh323_explicit_reducers) on the branch, cold: 886 passed, 54 skipped, 4 xfailed, 0 failed.

9. Scope / D1. Diff is 4 wave data_info.yml + CONTENTS.org + ledger + one new test. No lsms_library/*.py. No aggregation: key, no reducer, no .first(skipna=False), nothing blanked to <NA>. The only two functional edits are identifiers. git log origin/development -- countries/Malawi shows no prior cluster-key fix — not duplicate work. Only cluster_features and community_prices carry v in Malawi's scheme, and community_prices excludes 2004-05, so nothing else needed re-declaring.

Notes (none blocking)

1. The 2-cell baseline is misattributed. The PR and ledger say "the brief's baseline was 1,255 — the 2-cell difference is in 2004-05". It is not: I reproduce 2004-05 at exactly 164. The difference is in 2016-17, where your 78/4,405 counts the NaN-v orphan-geo bucket the broken merge created (881 groups × 5 cols = 4,405; Lat/Lon contested inside the orphan group = +2), while my 76/4,400 drops it. Your number is arguably the more complete one — only the sentence attributing it to 2004-05 is wrong.
2. §7's "pre-existing failure" is stale. test_gh323_explicit_reducers.py::test_core_does_not_dispatch_the_reducers no longer exists on this branch's merge base (the module was rewritten by test(#323): state the grain contract as behaviour, not as names #636/fix(#323): the D1 guard had its polarity inverted — guard choosing, not naming #640); the whole file passes here. Worth deleting the paragraph so a future reader doesn't hunt for a failure that isn't there.
3. test_the_known_cartesian_geo_merges_are_documented_where_they_live is a tripwire that fails when the bug is fixed (it asserts merge_on == ['v']). That's deliberate and the docstring says so, but it means PR fix(#323) Site 4: the dfs: merge MANUFACTURES the duplicates — 4.9M phantom rows (UNBLOCKED: Ethiopia/Niger/Nigeria configs merged) #627 must edit this test as part of its fix — flagging so it isn't later read as a regression.
4. Cross-wave v collisions go 0 → 3. Under v: ea the 2004-05 keyspace was format-disjoint from IHS3+; now 3 codes coincide with 2010-11 across different census frames. Disclosed twice in the PR/CONTENTS, and inert in practice — _add_market_index keys on (i,t) / (t,v) and nothing joins v across waves — so informational only.

Verdict: APPROVE_WITH_NOTES. The diagnosis is correct, the identifier is the right one, the payload is conserved exactly, the tests discriminate, and the two documentation inaccuracies above are the only things I could find in a deliberate attempt to break it.