verify(#323): Ethiopia is GREEN under #627 (already fixed by #628) — unblocks #627; + one real individual_education index bug

[ligon]

Ethiopia is GREEN under #627. Verified, not assumed — and this PR deliberately carries no cluster_features config change, because the fix is already on development.

#627's blocker table is stale

#627's body lists Ethiopia — 3 cells — "config fix written but not yet filed". That body was written 2026-07-13. Commit 3488b791 (PR #628) landed the fix and is an ancestor of origin/development. Nobody had confirmed it against #627's actual core — the sibling agent who cleared Niger flagged exactly that. This PR is that confirmation.

Method: development merged with fix/323-site4-dfs-merge (git merge-tree → 59b34cc2) on PYTHONPATH, asserted at runtime via lsms_library.__file__; config read via LSMS_COUNTRIES_ROOT; isolated LSMS_DATA_DIR whose only warm tier is a symlinked dvc-cache.

Negative control — pre-3488b791 config, same core

Reproduces #627's census exactly:

wave	result	phantom rows
2011-12	RuntimeError — df_geo lacks lat_dd_mod (file has LAT_DD_MOD)	—
2013-14	65,508 rows, CARTESIAN	60,221
2015-16	57,786 rows, CARTESIAN	52,832
2018-19	RuntimeError — file has lat_mod, not lat_dd_mod	—
2021-22	RuntimeError — geo file has no ea_id at all	—

Current development config, same core

wave	2011-12	2013-14	2015-16	2018-19	2021-22
rows	3,969	5,262	4,954	6,770	4,959
cartesian cells	0	0	0	0	0

Country('Ethiopia').cluster_features() → 2,168 rows on a unique (t, v), 2,146 with Latitude (333 / 433 / 432 / 535 / 435 clusters). All 25 declared Ethiopia tables build: 0 raised, 0 cartesian, 0 dropped sub-dfs. #627's own 16 tests pass against this config tree.

Answering #627's open question directly: yes, 3488b791 cleared the cartesian cells too. The same re-key fixes the raise and the cartesian — which is why #627's census, taken before #628 landed, could not see it coming. 113,053 of #627's 4,907,774 phantom rows are already gone.

The one real gap found: individual_education (2013-14, 2015-16)

Not a #627 blocker. A #323 site-1 true positive, unfixed until now, found while verifying the above.

Both waves indexed education on the W1-baseline household_id / individual_id, while household_roster and shocks for the same waves correctly used the wave-native household_id2 / individual_id2.

• W2 — the baseline pair is blank for the urban refreshment sample (no W1 antecedent): 5,248 of 23,785 rows, all landing on the single tuple ('2013-14', '', ''). The framework's own audit said so:

  DESTROYED 5,247 of 23,785 rows whose values DISAGREE (1 conflicting index tuples) ... Additionally 5,248 row(s) carry NaN in a declared index level and are DELETED OUTRIGHT

• W3 — split-off households inherit their parent's baseline id (010501020100105031 and 010501088800105031, the 8888 marking the split-off, both carry baseline 01050100105031), so 16 rows in 8 pairs collided and first() merged two real, distinct people: one recorded 10th Grade Complete, the other 9th Grade Complete.

It also made education unjoinable to the roster: 0 of the built (i, pid) pairs matched a household_roster row in W2/W3, against 100% in W1/W4/W5.

Fixed by correcting the identifier, per the standing rule — never by declaring a reducer.

	before	after
country rows (identical conditions)	59,092	63,181 (+4,089)
2013-14 wave frame unique on (t, i, pid)	No	Yes (23,785/23,785)
2015-16 wave frame unique on (t, i, pid)	No	Yes (23,393/23,393)
GrainCollapseWarnings	2	0
W2 (i, pid) joinable to roster	0 / 8,505	12,583 / 12,583
W3 (i, pid) joinable to roster	0 / 12,597	12,599 / 12,599

id_walk + the existing panel_ids chain still map household_id2 back to the W1 baseline, so panel linkage is preserved — this is exactly what household_roster already relies on.

Deliberately NOT salvaged from the abandoned fix/323-ethiopia

• its aggregation: keys — dead config, contradicts D1. A test now pins that Ethiopia declares none, so a future salvage cannot revive them.
• its interview_date mapping.py hooks — measured redundant: _enforce_canonical_dtypes already honours the declared Int_t: datetime, and the table returns datetime64[us], 8,236 rows, 0 nulls on development. Adding them would be inert code that reads as a fix.
• its core patch — Design A, rejected under D1.
• its final_index: [t,v] + drop: [i] shape — wrong: Wave.cluster_features' GH Uganda cluster_features: wrong grain, stringified-float District, missing GPS wiring #161 collapse only fires when i is an index level. 3488b791 gets this right.

A CONTENTS.org claim that had gone false

#628 documented the collapse as taking mean() for Latitude/Longitude ("the cluster centroid") and first() for the rest. The GPS .mean() has since been retired — core aggregates nothing anywhere. Corrected, and do not assume the flip is a no-op here. Measured:

wave	EAs	EAs with >1 distinct household coordinate
2011-12	333	0
2013-14	433	38
2015-16	432	72
2018-19	516	0
2021-22	432	72

W5 EA 020103010100203 spans 1.79° of longitude (~200 km). Points that far apart are not one enumeration area — this is a cluster key that is not unique nationally (the Malawi 2013-14 pattern), not GPS jitter. Recorded, unfixed, GH #323 site 2.

Tests

21 new, asserted at the wave level — post-collapse index uniqueness holds by construction and passes with the bug fully present (the instrument note inherited from the Benin/Togo tests).

Negative control run: 21/21 pass against the current config; 17/21 FAIL against the pre-3488b791 tree on the same core. Of the 4 that still passed, 3 were data-backed cluster_features cases whose raise was shadowed by a stale L2-wave parquet — #627's "the green is a lie" warning, live. The config-level assertions failed 5/5 in every wave and cannot be shadowed.

Recorded, not fixed

• individual_education's country-level row count is order-dependent. Country('Ethiopia').individual_education() → 63,139; the same call after touching c.panel_ids → 62,939. The delta lands on waves this PR does not touch and reproduces in the pre-fix config. Cause: the lazy self.updated_ids probe in _aggregate_wave_data, whose except (...): pass records a failed first attempt as "attempted" and then skips id_walk for the rest of the process. The same call returns two different row counts depending on what the process touched first — a correctness problem. Worth its own issue.
• The 2013-14 wave-level panel_ids: block still points at sect_cover_ls_w2.dta — the livestock cover, holder-grain (verified: 3,812 rows / 3,670 households, vs the household cover's 5,262 unique). data_scheme.yml declares panel_ids as materialize: make so the country script wins, but "appears unused" is not evidence and deleting config on that basis is the move the repo forbids.

Ledger: .coder/ledger/323-ethiopia-config.md. Refs #323, #515. Unblocks #627.

🤖 Generated with Claude Code

[ligon]

Adversarial review — verdict: APPROVE_WITH_NOTES

I tried to break this and could not break the fix. Everything load-bearing reproduced
independently and cold. Two documentation numbers do not reproduce; details at the end.

Method. Two git worktrees (origin/development, this branch's head), core pinned via
PYTHONPATH and asserted at runtime on lsms_library.__file__, config via
LSMS_COUNTRIES_ROOT, and six separate isolated LSMS_DATA_DIRs whose only warm tier is
a symlinked dvc-cache. I also reconstructed #627's core myself (`git merge origin/development

• origin/fix/323-site4-dfs-merge) rather than take the PR's word for the "GREEN under #627" table. No dvc` CLI was invoked anywhere.

Re-derived, cold, and confirmed

Raw identifiers (read straight from the .dta, no library path):

file	rows	(household_id, individual_id) unique	blanks	(household_id2, individual_id2) unique
sect2_hh_w2.dta	23,785	18,538	5,248	23,785
sect2_hh_w3.dta	23,393	23,385 (16 rows / 8 pairs)	0	23,393

The W3 "two real, distinct people" claim is the one I most wanted to falsify — the house
rule is that a composite is only wrong when the rows describe different real entities, and
"same person double-punched into parent + split-off" would have made this a first() that was
merely completing. It isn't. Cross-checking the 8 colliding baseline person-ids against
sect1_hh_w3.dta:

baseline individual 0105010010503101 -> hh ...0201...  Head,           Male,   age 38
                                     -> hh ...8888...  Son/Daughter,   Male,   age 29
baseline individual 0105010010503105 -> hh ...0201...  Sister/Brother, Male,   age 13
                                     -> hh ...8888...  Grandchild,     Female, age 15

Different age, different relationship, in one case different sex. The split-off's roster
inherited the parent's baseline person-ids positionally. These are different people, the
collapse was destroying one, and the fix is the identifier — not a reducer. D1 respected.

Conservation. Post-fix the country-level row count per wave equals exactly the raw count
of non-null mapped attainment — no mass invented, no mass lost:

wave	raw non-null (post-mapping)	country rows before	after
2011-12	8,008	8,008	8,008
2013-14	12,583	8,505	12,583
2015-16	12,635	12,633	12,635
2018-19	16,027	16,027	16,027
2021-22	13,886	13,886	13,886

W3 gains exactly 2 rows — precisely the two collisions where both rows were non-null and
disagreed. The other 6 collisions were NaN-completions and are correctly left as completions,
not resurrected. The three untouched waves are unchanged to the row. That is the right shape:
the fix moves nothing it shouldn't.

Roster join — 0 / 8,505 and 0 / 12,633 before, 12,583 / 12,583 and 12,635 / 12,635
after, 100% in all five waves. Two GrainCollapseWarnings before, zero after. Confirmed that
no other table in either wave still uses a baseline id, so the wave is now internally
consistent (household_roster, shocks, individual_education all on _id2).

Negative control reproduces exactly. Reverting only the five Ethiopia data_info.yml to
3488b791^ on a cold data root: 17 failed, 4 passed — the PR's claim, and the same 4
(three cluster_features data-backed cells whose failure is shadowed — here by #515's
swallowed KeyError on the development core rather than by a stale parquet — plus the
aggregation: guard). The 5 config-level assertions failed 5/5 and cannot be shadowed.

Mutation test of the D1 guard. Injected aggregation: first under
individual_education in Ethiopia/_/data_scheme.yml →
test_no_dead_aggregation_key_in_ethiopia_config fails with
assert not ['individual_education']. It discriminates. Restored.

"GREEN under #627" independently confirmed. On my own merge of development +
fix/323-site4-dfs-merge, cold: 3,969 / 5,262 / 4,954 / 6,770 / 4,959 rows, 0 cartesian
warnings, 0 raises, cluster_features() → 2,168 rows unique on (t, v), 2,146 with
Latitude. Matches the PR's table digit for digit. (On plain development those wave counts
are 3,969 / 5,287 / 4,954 / 6,894 / 4,994 — the union, because merge_how: left is only
honoured under #627. Worth knowing when reading the table, not a defect in this PR.)

Also verified: GPS disagreement table (333/0, 433/38, 432/72, 516/0, 432/72 — exact); EA
010103010100106 = 3 distinct fixes, 0.1921° lat / 0.596° lon; EA 020103010100203 =
1.7923° lon; the GPS .mean() really is retired in _collapse_to_cluster_grain;
sect_cover_ls_w2.dta = 3,812 rows / 3,670 households vs the household cover's 5,262 unique.
Scope is clean — config + tests + docs only, no lsms_library/*.py, no aggregation: key, no
reducer. No duplicate of already-merged work (#320 added the table with the baseline ids;
nothing since has re-keyed it).

Findings (all LOW — documentation only, none blocking)

L1 — the order-dependence caveat does not reproduce. The body, CONTENTS.org and the
ledger all state as measured fact that Country('Ethiopia').individual_education() returns
63,139 alone but 62,939 after touching c.panel_ids, and call it "a correctness problem …
worth its own issue". I ran that exact sequence four ways — development core and #627 core ×
warm and cold, each on its own fresh isolated data root — and got 63,139 every time, in both
orders, with identical per-wave splits. Whatever produced 62,939 was a property of one
session's cache state, not of either core. As written, CONTENTS.org will send the next
person hunting a ghost. Suggest qualifying it ("observed once under cache state X; not
reproducible on a cold root") or dropping it. The advice it justifies — assert at the wave
level — is right regardless, and the tests do.

L2 — the headline row totals don't reproduce. "59,092 → 63,181 (+4,089) under identical
conditions" measures, for me, 59,059 → 63,139 (+4,080) — again on both cores, cold and
warm. The body's own per-wave numbers are also internally inconsistent with it (12,583 − 8,505
= +4,078, plus W3's +2, = +4,080, not +4,089; and my W3 pair is 12,633 → 12,635, not
12,597 → 12,599). Direction, sign and mechanism are all correct and nothing in the test suite
depends on these digits — the tests assert at the wave level, where I reproduced them exactly.
But per the PR's own standard, worth correcting or marking as environment-dependent.

L3 — informational, pre-existing, not caused by this PR. The 42-row gap between my
numbers and the PR's is a real thing worth someone's attention: in W2, 42 rows whose raw
hh_s2q05 is "Not educated" map to Preferred Label None in harmonize_education and
therefore leave the built table as missing. "Never attended" is a value, not an absence, and
it is silently indistinguishable from "not asked". Identical in both configs, so out of scope
here — but it deserves its own issue.

None of L1–L3 touches the correctness of the four changed YAML lines, which are right.