bug: framework silently drops rows on duplicate canonical index (groupby().first())

[ligon]

_finalize_result collapses a non-unique canonical index via groupby().first(), silently dropping rows. Found in assets (PR #319): Iraq 2012 (durable_serial -> 25,248 rows) and Albania 2002 (479) silently lost household holdings while still passing is_this_feature_sane. Fixed per-country by summing within (t,i,j), but ANY feature whose source has multiple rows per canonical-index tuple is at risk of silent data loss. Suggest: warn/error on silent dedup, and/or a documented per-feature duplicate-aggregation policy (e.g. assets: Quantity=sum, Value=sum, Age=min).

[ligon]

Fixed by the above PR (merged to development): silent dup-index drops now emit a RuntimeWarning (verified on a synthetic dup). Note: a full per-feature duplicate-aggregation policy is still the long-term fix; this surfaces the loss.

[ligon]

Reopening. This class is alive in 14 countries and it was one merge away from having zero trackers.

An adversarial red-team of the Guyana fix (#503) went looking for the class rather than the instance, and found it. I have independently verified the headline numbers from source.

Country('Mali').household_roster() silently drops 86% of the 2014-15 wave

Mali 2014-15 household_roster
  wave parquet (the truth)     :  37,175 rows
  what the API actually returns:   5,149 rows
  SILENTLY DROPPED             :  32,026 rows   (86.1%)

32,026 people vanish from Mali's roster with no warning. Mali is an EHCVM country in active use.

44 cells / 14 countries / ~150k rows

Verified sample (duplicate-index rows in the L2-wave parquet, i.e. before the collapse):

country	wave	table	rows	dup index	%
Mali	2014-15	household_roster	37,175	32,026	86.1%
Kazakhstan	1996	sample	7,224	5,228	72.4%
Kosovo	2000	housing	7,047	4,182	59.3%
Nigeria	2012Q3 + 2013Q1	assets	—	14,493 ×2
Tanzania	2008-15	interview_date	—	12,710
Malawi	2004-05	shocks	203,044	11,077	5.5%
Ethiopia	2013-14	individual_education	—	5,247
Niger	2014-15	sample/housing/food_security + roster/education	—	59 / 210
Guyana	1992	housing	1,819	311	17.1%

Mechanism: _normalize_dataframe_index's silent groupby().first() on a non-unique declared index.

Why this stayed invisible — and it is worth understanding

The L2-country parquet is written POST-collapse; the L2-wave parquet holds the truth. So the #323 warning fires only on a cold build. In normal warm operation the corruption is already baked into the cache and the warning never appears again.

The bug hides behind the cache that the bug poisoned.

The governance failure, which is the reason I am reopening rather than filing new

• bug: framework silently drops rows on duplicate canonical index (groupby().first()) #323 (this issue) — CLOSED.
• [feature-audit] food_acquired / all: Canonical index over ['t', 'v', 'i', 'j', 'u', 's'] had 34591 duplicate tuple(s); collapsed via groupby().first(), dropping those rows (p... #514 — CLOSED.
• Guyana sample/roster conflate distinct households: i=[ED,HH] needs SN (household identity) #503 (Guyana) — the ONLY open issue in the entire 33-issue backlog matching this class.

Had we merged the Guyana fix and closed #503 — which is exactly what was about to happen — this class would have had no tracker at all. A 14-country, ~150k-row silent-data-loss bug would have become invisible, with every one of its issues closed and a green tick against it.

That is not a hypothetical. The PR was written, tested, red-teamed on correctness and regression, and both of those lenses passed. Only the completeness lens — "does this fix the class or just the instance?" — caught it.

A fix that closes one instance of a systemic bug and leaves the rest is worse than no fix, because the issue is now closed and the bug is harder to find.

Actions

• Guyana sample/roster conflate distinct households: i=[ED,HH] needs SN (household identity) #503 must not close until this is tracked. Commented there.
• The Guyana identity fix itself is correct and should land — on its own terms it is good work. It is being reworked (its assets crosswalk ships 312 households on a positional guess whose guard is provably vacuous — see Guyana sample/roster conflate distinct households: i=[ED,HH] needs SN (household identity) #503).
• A separate defect surfaced alongside: phantom NaN-i rows from a how='outer' merge (country.py:1032) — Ethiopia 2013-14 individual_education 5,248 rows; Guyana 1992 sample 488. Filing separately.

Credit

Found by the completeness red-team lens, which also disclosed its own instrument errors rather than burying them: its first scan ran against the var/ (post-collapse) tier and returned zero hits — including for Guyana, the known positive. It recognised that as a false negative from an unvalidated instrument, rebuilt against the wave tier, and re-validated on the known positive before trusting a single number.

A negative result from an instrument you have not validated is a result about the instrument.