Nigeria food_prices() silently drops ~99% of rows in 6 of 8 waves (Quantity==0 sentinel → inf → dropped)

[ligon]

Summary

Country('Nigeria').food_prices() (the default units='kgvalue') returns 675–1,106 rows for waves whose food_acquired holds 120k–160k rows. Roughly 99% of Nigeria's price data silently disappears for 6 of its 8 waves — with no warning, no NaN, and no error.

The coverage matrix grades all of these cells sane.

Evidence

food_prices() rows vs. food_acquired rows, per wave:

wave	food_acquired rows	food_prices rows	survived
2010Q3	73,237	61,967	85%
2011Q1	71,246	59,619	84%
2012Q3	121,750	675	0.6%
2013Q1	134,097	161	0.1%
2015Q3	147,516	509	0.3%
2016Q1	137,856	426	0.3%
2018Q3	158,470	1,106	0.7%
2019Q1	161,484	1,044	0.6%

Root cause

From 2012Q3 onward, ~40% of food_acquired rows carry Quantity == 0 together with Expenditure > 0 — i.e. "spent money, acquired zero". This is a 0-as-missing sentinel that was never decoded to pd.NA.

wave	rows	Quantity == 0	%
2010Q3	73,237	132	0.2%
2011Q1	71,246	154	0.2%
2012Q3	121,750	52,081	42.8%
2013Q1	134,097	56,772	42.3%
2015Q3	147,516	61,967	42.0%
2016Q1	137,856	56,706	41.1%
2018Q3	158,470	63,270	39.9%
2019Q1	161,484	65,688	40.7%

Note the sentinel is absent in 2010Q3/2011Q1 (0.2%) and appears abruptly at 2012Q3 — a wave-level encoding change that was never handled.

The chain:

1. Quantity == 0 ⇒ _apply_kg_conversion (transformations.py:609) yields Quantity_kg == 0.
2. food_prices_from_acquired computes Price = Expenditure / Quantity_kg = inf (transformations.py:915).
3. transformations.py:933 — v[['Price']].replace([0, np.inf, -np.inf], np.nan).dropna() — silently drops every one of them.

Step 3 is the design choice that turns a data-encoding bug into invisible data loss: the rows do not come back as NaN (which a user would notice), they simply cease to exist.

Why this was invisible

• food_acquired.Quantity_kg is all-null in the cached table for these waves, but that is a red herring — _apply_kg_conversion fills it from the unit→factor map at derivation time.
• The sanity checks pass, so all 6 cells are graded sane in the coverage matrix. sane means "the automated checks passed", not "a human read the numbers". This issue is the canonical argument for the blessed tier.

Suggested fix

1. Decode the sentinel at source — Quantity == 0 with Expenditure > 0 should be pd.NA, in the Nigeria wave config/scripts. That is the actual defect.
2. Make the silent drop loud — transformations.py:933 should not discard inf without trace. At minimum warn with a count; consider returning NaN rather than dropping, so the row survives and the gap is visible.
3. Add a regression test asserting Nigeria food_prices() retains an order-of-magnitude-plausible row count per wave.

Also affects

Likely food_quantities() (summing 0s understates totals). food_expenditures() is unaffected (Expenditure is intact).

---

Found while auditing the coverage matrix (see slurm_logs/DESIGN_coverage_discipline_2026-07-11.org). Filed BEFORE the wave-slice grader fix, which would otherwise re-tier these cells and bury the evidence.