Summary
A how='outer' merge at lsms_library/country.py:1032 manufactures rows with a NaN household id (i) — households that do not exist.
Confirmed instances:
- Ethiopia 2013-14
individual_education — 5,248 phantom rows
- Guyana 1992
sample — 488 phantom rows
Why it matters
A NaN-i row is not a household. It pollutes counts, joins, and any groupby on i. And it interacts badly with the groupby().first() collapse in _normalize_dataframe_index (see #323), because a block of NaN-i rows collapses to a single phantom household.
Provenance
Surfaced by the adversarial red-team of the Guyana fix (#503), which worked around it Guyana-locally and noted in its own PR that it "deserves its own issue" — and then no such issue was filed. Filing it, because a defect named in a PR body and nowhere else is a defect that will be lost.
Related: #323 (silent groupby().first() collapse, reopened — alive in 14 countries), #503 (Guyana household identity).
Summary
A
how='outer'merge atlsms_library/country.py:1032manufactures rows with a NaN household id (i) — households that do not exist.Confirmed instances:
individual_education— 5,248 phantom rowssample— 488 phantom rowsWhy it matters
A NaN-
irow is not a household. It pollutes counts, joins, and any groupby oni. And it interacts badly with thegroupby().first()collapse in_normalize_dataframe_index(see #323), because a block of NaN-irows collapses to a single phantom household.Provenance
Surfaced by the adversarial red-team of the Guyana fix (#503), which worked around it Guyana-locally and noted in its own PR that it "deserves its own issue" — and then no such issue was filed. Filing it, because a defect named in a PR body and nowhere else is a defect that will be lost.
Related: #323 (silent
groupby().first()collapse, reopened — alive in 14 countries), #503 (Guyana household identity).