fix(GhanaLSS): GH #323 — make the silent index collapse declared-and-CHECKED (one instance; class stays open)

[ligon]

GH #323 — GhanaLSS instance (ONE instance; the class stays open)

_normalize_dataframe_index silently collapses a non-unique declared index with
groupby().first(). This PR fixes the GhanaLSS instance of that bug — 7 cells —
and adds the first actual enforcement mechanism for the aggregation: policy.

It does NOT close #323. #323 is a class (~14 countries / ~44 cells / ~150k rows;
Mali alone loses 32,026 of 37,175 roster rows). Closing the class on an instance fix is
precisely how this bug survived its first round. #323 stays open until the framework fix
lands.

---

What was silently dropped, and how many rows

Measured at the L2-wave parquet (the truth layer). The L2-country parquet
(var/) is written POST-collapse, so scanning it returns a false zero — the instrument
was validated on the known positives first (Mali/2014-15 household_roster dup = 32,026
exactly; Guyana/1992 housing dup = 311 exactly) before any GhanaLSS number was trusted.

table	wave	dup rows BEFORE	AFTER	root cause
cluster_features	1988-89	14,754	—	EXTRACTION_BUG → table removed
cluster_features	1987-88	never built	—	EXTRACTION_BUG → table removed
cluster_features	1991-92	4,158	4,158	INTENDED_AGGREGATION → declared + checked
cluster_features	1998-99	5,698	5,698	INTENDED_AGGREGATION → declared + checked
cluster_features	2005-06	8,107	8,107	INTENDED_AGGREGATION → declared + checked
cluster_features	2012-13	15,572	15,572	INTENDED_AGGREGATION → declared + checked
cluster_features	2016-17	933,584	933,584	INTENDED_AGGREGATION → declared + checked
food_security	2016-17	109 (+110 NaN-keyed)	0	PHANTOM_NAN_ROWS → fixed

Silent #323 warnings on a cold GhanaLSS build: 8 → 1 (the remaining one is sample,
pre-existing, a different root cause).

Rows recovered at the API = 0, honestly. One row per cluster is the correct grain for
(A)/(B); the 110 recovered food_security households are all-NA in every declared column
and are dropped by _finalize_result's dropna(how='all') (country.py:2217) — exactly as
it already drops 7 all-NA households that did have clust/nh populated. Claiming a
110-row recovery would be a vanity metric. The value here is that the collapse is no longer
silent and the fabrication path is disarmed.

---

Three root causes, fixed three different ways

One uniform fix would have baked in a wrong value.

(A) cluster_features, 5 waves — INTENDED_AGGREGATION, now DECLARED and ENFORCED.
Each wave points file: at a HOUSEHOLD-grain source and projects cluster-invariant columns.
Verified invariant in all five at the source (max distinct value per cluster = 1; zero
violations).

Key finding: aggregation: was already declared by 9 countries and read by ZERO lines of
code — grep for its consumption returns nothing; SkunkWorks/grain_aggregation_policy.org
item #4 is literally "not yet built". Declaring it would have been pure prose, and prose is
not enforcement. So it is now implemented: a new invariant reducer in
_normalize_dataframe_index (+ _assert_invariant_within_index) that verifies constancy
within the index group and RAISES otherwise. Proven fatal end-to-end through Country()
(projected household-varying nh as a declared-invariant column → ValueError,
"non-constant in 1000 of 1000 groups") — i.e. it is not swallowed by the wave loop's
except (KeyError, AttributeError). Guard cost: 0.30 s on the 934,584-row worst case.

(B) cluster_features, 1987-88 + 1988-89 — EXTRACTION_BUG (silently WRONG).
Region was wired to Y01A.DAT REGION = the person's REGION OF BIRTH. Confirmed 5 ways:
mapping.py's Region() and Birthplace() are byte-identical bodies over the same
region_dict; the waves' own code list runs to 11=Nigeria / 12=Ivory Coast / 13=Togo /
14=Burkina Faso (a Ghanaian EA cannot be located in Nigeria); REGION varies within one
household in 1,019/3,192 and 1,033/3,136 HH (up to 6 distinct values in ONE household);
within-cluster in 167/170 and 174/176; Y01A.DCT's variable order is the classic
birthplace→nationality pair. Collapsed, .first() labelled 9/170 and 7/176 clusters
"Nigeria".

No cluster-invariant region source exists in these waves → DROPPED rather than guessed
(class-2 silently-MISSING > class-1 silently-WRONG). Also dropped 1987-88's Age: AGEY (a
person's age, made into "the cluster's age"). 1987-88 was not in the reported cell list —
its parquet had never been built — so fixing only 1988-89 would have closed the instance and
left the class.

Correction to the original diagnosis (it mattered): the (B) fabrication is currently
MASKED — Region is 100% NaN (0/14,924) because get_categorical_mapping returns {}
for these waves (a separate bug). The first guard test was therefore vacuous (all-NA →
nunique=0 → passes trivially). Re-tested against the real birth-region values (the world
where region_dict is repaired — plausible, since that bug also breaks Birthplace) it
RAISES: non-constant in 174/176 and 167/170. A latent class-1 bug is armed behind a
class-2 bug; dropping Region costs zero rows today and defuses it.

(C) food_security 2016-17 — PHANTOM_NAN_ROWS.
110 tail rows have clust=NaN and nh=NaN (all 8 FIES items NaN) but a valid hid;
groupby's dropna dropped all 110 outright. Re-keyed on i: hid, which reconstructs
{clust}/{nh:02d} with 100.00% fidelity on all 13,899 good rows (byte-identical ids).

• BEFORE: 14,009 rows / 110 NaN i / 13,899 unique i
• AFTER: 14,009 rows / 0 NaN i / 14,009 unique i
• ids lost = 0; ids recovered = 110; payload of the 13,899 pre-existing households
  byte-identical (.equals() == True); all 110 recovered hids present in sample().
• FIES_score() already returns pd.NA when all 8 items are missing → no fabricated
  food-secure 0.

---

Proof nothing else moved

1. Dual-implementation byte-identity. Loaded BASE and FIXED _normalize_dataframe_index
   into one process and pushed every cached L2-wave parquet across all countries through
   both, with each country's real schema entry — 757 (country × wave × table) cells: 0
   differ, 0 non-GhanaLSS deltas. The framework change is data-inert everywhere else; it
   alters only warnings. (An independent red-team re-ran this over 729 cells / 32 countries:
   0 data differences, 0 raise/no-raise changes, 0 warnings gained; non-GhanaLSS bug: framework silently drops rows on duplicate canonical index (groupby().first()) #323 warning
   count 102 base / 102 fixed.)
2. The 9 countries that already declare aggregation: (Benin, CotedIvoire, Burkina_Faso,
   Senegal, Albania, Togo, Malawi, Guinea-Bissau, Niger) all declare {visit: first}.
   visit is an index level, never a column, and first ≠ invariant — doubly unaffected.
   Confirmed empirically in the sweep, not by grep.
3. Tests fail pre-fix (verified by stashing lsms_library/ in the worktree): 6 of 7 FAIL
   on pristine base, 7/7 pass post-fix. The 1 that passes both ways
   (test_undeclared_collapse_still_warns) is a regression guard, not a fix-detector.
4. Blast-radius suite: test_normalize_index_j_preserved.py, test_schema_consistency.py,
   test_table_structure.py, test_id_walk.py, test_add_market_index_dedup.py + the new
   file = 2,198 passed, 2 skipped, 0 failed (2,200 of the 3,427-test suite).
5. GH test_feature_ghana_per_wave failing — GhanaLSS food_expenditures per-wave currency labels #589: tests/test_currency.py::test_feature_ghana_per_wave FAILS IDENTICALLY on
   pristine base and with this branch — same assertion, AssertionError: assert ['GHC'] == {'GHC'} at test_currency.py:222 (both diffed). Not touched, not "fixed".
6. Shared parquet cache not poisoned: every worktree-config build ran under an isolated
   LSMS_DATA_DIR. Verified the shared cache still holds the base-config GhanaLSS parquets.
7. .pth trap defeated and asserted: every verification script does
   sys.path.insert(0, WT) + asserts 'worktrees' in lsms_library.__file__ and in
   countries_root().

API level (unchanged, as predicted): cluster_features 3,145 rows / 0 duplicate (t,v);
food_security 13,892 rows / 0 duplicates / 0 NaN i. Feature('cluster_features')(['GhanaLSS'])
assembles, and the base No rule to make target .../1987-88/_/cluster_features.parquet error is gone.
Base-drift is nil (git diff d572d8a9..development on country.py is empty).

---

Red-team verdicts — all three lenses, including the concerns that did NOT block

[correctness] passes = true, severity = concern.
Every load-bearing claim reproduces exactly under an independently-validated instrument.
Duplicate counts go to zero or to a declared, checked, non-vacuously verified aggregation.
The invariant guard is genuinely fatal through Country(). ONE UNDISCLOSED HOLE (concern,
not blocker): the new warning suppression at country.py:4225 keys only on undeclared
being empty — but groupby(level=...).first() also silently drops NaN-KEYED rows
(dropna=True), and _assert_invariant_within_index is itself a groupby, so it cannot see
them either. A table that declares all its columns invariant therefore loses NaN-keyed rows
with ZERO warning — which is exactly the phantom-NaN mechanism this same PR fixes in
food_security 2016-17. Not live for GhanaLSS (rows_with_NaN_key = 0 on all five wired
waves; invariant is declared nowhere else), so this branch loses no data — but it is a
regression in the #323 detector itself, inside the generic framework mechanism other fix-agents
are being pointed at. Flagged loudly here rather than laundered into silence.

[regression] passes = true, severity = nit.
Nothing else moves (see the 729/757-cell differential above). The scanner was rebuilt after
catching its own instrument bug — yaml.safe_load silently skipped the 14 countries whose
data_scheme.yml uses !make tags, including Mali and GhanaLSS themselves, yielding a
clean-looking 333-cell sweep with zero GhanaLSS rows — then re-validated on both required
positives. The newly-declared Ecological_zone: str is a genuine no-op (base var/ already
stored it as StringDtype). Nit, not a blocker: the warning gate changed from
if n_dropped: to if n_dropped and undeclared:, so a frame reaching the collapse with zero
non-index columns would suppress the #323 warning for every country regardless of
declaration — unreachable in all 729 cells today, but a new silent-drop path inside a #323 fix.

[soundness] passes = true, severity = concern.
The retained/re-keyed data is CORRECT, not merely present, and the fix guesses nowhere.
(C) hid is the framework's own composite id (format_id(hid) == f"{clust}/{nh:02d}" on
13,899/13,899, 0 mismatches; hid unique 14,009/14,009; the 110 are disjoint, all in sample(),
all 8 FIES items NaN in all 110). (A) the invariant declaration is true at the source.
(B) the Region drop is well-founded and refuses to guess. CONCERN (one-line hardening):
_assert_invariant_within_index uses nunique(dropna=True), so a declared-invariant column
that is 100% NA at check time yields nunique == 0 and passes vacuously — and because
undeclared is then empty, the fix also suppresses the pre-existing #323 warning.
GhanaLSS 1998-99 is a live instance: its cluster_features L2-wave parquet is 5,998 rows
with Region 0/5,998 and Rural 0/5,998 non-null (the same get_categorical_mapping → {}
defect found in 1988-89); base warned "had 5698 duplicate tuple(s)", the fix checks nothing and
says nothing. So the exact configuration the PR exists to catch — a person-level column wired as
a cluster attribute in a wave whose categorical mapping is broken — is the one the guard cannot
see, because the broken mapping NA-s out the evidence. No wrong data today (the source is
genuinely invariant) and it is fail-safe if the mapping is repaired, but the data_scheme.yml
comment's "Verified cluster-invariant in all five wired waves (max distinct value per cluster =
1)" is 0, not 1, at the layer the guard actually inspects.

Both concerns are recorded here deliberately rather than fixed in this PR: hardening the
detector (NaN-keyed rows; vacuous all-NA nunique) belongs with the framework fix for the
class, where it can be validated against all ~44 cells rather than GhanaLSS's 7.

---

Refs GH #323 (do not close it). Do not merge without human review.

🤖 Generated with Claude Code

[ligon]

Superseded by #622, which carries the same two GhanaLSS fixes with the rejected +84-line country.py patch stripped.

Per the #323 consolidation (slurm_logs/DESIGN_grain_collapse_sites_2026-07-13.org), the core fix is owned centrally by #614 (Site 1) and #617 (Site 2) — this branch's core patch was one of 17 mutually distinct patches to country.py the sweep produced, and Design A ("declare a reducer") was rejected.

The country-level diagnosis here was right and is preserved intact in #622 — including the finding that Region: REGION was the person's region of birth (the wave's code list includes Nigeria, Ivory Coast, Togo).

Closing in favour of #622.