From 3488b791475b6e09b03da5c90df674ffbe2db1cc Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Mon, 13 Jul 2026 11:52:26 -0700 Subject: [PATCH] =?UTF-8?q?fix(#323):=20geo=20config=20for=20Ethiopia=20+?= =?UTF-8?q?=20Niger=20=E2=80=94=20unblock=20the=20site-4=20required-column?= =?UTF-8?q?=20raise=20(#627)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #627 makes the GH #515 optional-sub-df fallback RAISE when a dropped sub-dataframe costs a REQUIRED declared column. That is correct, and it means `Country('Ethiopia').cluster_features()` and `Country('Niger').cluster_features()` raise on that branch until their configs are fixed. This fixes them. Config only -- no `lsms_library/*.py`. All ten cells are TRUE POSITIVES. Every one was diagnosed by opening the source file and listing its real columns; none was guessed. Nine of ten are case (b) -- the column EXISTS and the YAML named it wrong -- so they are WIRED UP, not deleted. ETHIOPIA -- the household geovariables file is renamed in almost every wave: 2011-12 Pub_ETH_HouseholdGeovariables_Y1 LAT_DD_MOD / LON_DD_MOD (UPPERCASE) 2013-14 Pub_ETH_HouseholdGeovars_Y2 lat_dd_mod / lon_dd_mod 2015-16 ETH_HouseholdGeovars_y3 lat_dd_mod / lon_dd_mod 2018-19 ETH_HouseholdGeovariables_Y4 lat_mod / lon_mod (no `_dd_`) 2021-22 eth_householdgeovariables_y5 lat_dd_mod / lon_dd_mod, and the file has NO ea_id AT ALL W1 (casing), W4 (dropped `_dd_` infix) and W5 (missing `ea_id` KEY -- the coordinates themselves were always there, 4,847/4,890 non-null) are the three raising cells. All three are case (b). W2 and W3 resolved their columns but merged two HOUSEHOLD-grain frames on the CLUSTER key `v` -- a cartesian product (2013-14: 65,508 rows, 60,221 phantom; 2015-16: 57,786 rows, 52,832 phantom). Same edit fixes both. All five waves now merge on the HOUSEHOLD id (1:1 -- measured unique and blank-free in BOTH frames in every wave) and carry `i` in `final_index`, so `Wave.cluster_features`'s existing GH #161 collapse reduces to the (t, v) EA grain with mean() for Latitude/Longitude (the cluster centroid) and first() for Region/District/Rural. This is the Malawi 2013-14 pattern. W2/W3 use `household_id2`, NOT `household_id`: the latter is the W1 baseline id, blank for households new to the panel, so re-keying to it would trade an EA cartesian for a NULL-KEY cartesian. `merge_how: left` in all five -- the geo files carry households the cover page does not (W2 25, W4 124, W5 35), and under `outer` those orphans arrive with a null `v` and collapse into one phantom cluster. NOTE: this deliberately does NOT reuse `fix/323-ethiopia`'s config, which used `final_index: [t, v]` + `drop: [i]` and leaned on a declared `aggregation:` key in data_scheme.yml. Per D1 of DESIGN_grain_collapse_sites_2026-07-13.org that key is DEAD CONFIG and is deliberately not honoured by the core collapse. Under that config the frame would stay household-grain on a non-unique (t, v) index and Site 1 would collapse it with first() -- giving an arbitrary household's latitude instead of the cluster centroid. NIGER 2011-12 -- the brief said "the geo file has no lat/lon at all", and the honest fix would then be `optional: true`. VERIFIED, AND THAT IS A TRAP. `NER_HouseholdGeovars_Y1.dta` indeed has no coordinates: its 43 columns are derived raster covariates only (dist_road, af_bio_*, srtm, twi, sq1..sq7, EVI/NDVI). But the coordinates ship in a SIBLING FILE in the same directory -- `NER_EA_Offsets.dta`, the WB's standard offset-EA-centroid release: 271 rows x (grappe, LAT_DD_MOD, LON_DD_MOD), 270 non-null grappes, coordinates inside Niger's bounding box, ALREADY DVC-TRACKED. Every grappe in the household file is in it and vice versa (0 orphans both ways). So Niger is case (b), not case (a): the data is real, available, and merely mis-addressed. Marking it `optional:` would have DELETED A REAL COLUMN and called it a cleanup. Wired up instead -- all 270 clusters now carry coordinates where they previously carried none. The offsets file is already EA-grain, so the merge on `v` is many-to-one, not a cartesian. `merge_how: left` drops the file's one trailing null-grappe row, which `outer` would turn into a phantom cluster. Niger 2014-15 ships no geovariables/offsets file of any kind, so its Latitude/Longitude are honestly absent rather than mis-addressed. Not wired, and that is correct. VERIFIED (site-4 code via PYTHONPATH, this config via LSMS_COUNTRIES_ROOT, LSMS_NO_CACHE=1 -- cold, no cached parquet, import asserted): development (today) this branch Ethiopia 2,169 rows / 866 lat 2,168 rows / 2,146 lat Niger 1,599 rows / 1,048 lat 1,599 rows / 1,318 lat Ethiopia by wave, clusters carrying Latitude: 2011-12 0 -> 333 2013-14 434 -> 433 2015-16 432 -> 432 2018-19 0 -> 516 2021-22 0 -> 432 Niger 2011-12: 0 -> 270 Three of Ethiopia's five waves had ZERO coordinates and now have them; Niger 2011-12 likewise. +1,280 Ethiopian and +270 Nigerien clusters gain real GPS. NO DATA LOST. The only row that disappears is Ethiopia 2013-14 434 -> 433 clusters, and that row is EA 030412088800105 -- present in the geo file, absent from the cover page, its single household absent too. It was a PHANTOM cluster manufactured by the outer merge on `v`. Dropping it is the correction, not a loss. Every other per-wave count is identical. Both countries build with 0 cartesian warnings, 0 dropped sub-dfs, 0 grain-collapse warnings, and a unique (t, v) index. Every YAML-path table in both countries still builds under LSMS_GRAIN_STRICT=1. All 16 tests in #627's tests/test_gh323_site4_dfs_merge.py pass against this config. Coordinates sanity-checked against each country's real bounding box: Ethiopia lat 3.46-14.48 N / lon 33.43-47.31 E; Niger lat 11.80-19.07 N / lon 0.28-13.78 E. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TqDLDHYG32Z57ipikJMywZ --- .../Ethiopia/2011-12/_/data_info.yml | 32 +++++++++++-- .../Ethiopia/2013-14/_/data_info.yml | 30 +++++++++++- .../Ethiopia/2015-16/_/data_info.yml | 27 ++++++++++- .../Ethiopia/2018-19/_/data_info.yml | 31 ++++++++++-- .../Ethiopia/2021-22/_/data_info.yml | 28 ++++++++++- .../countries/Ethiopia/_/CONTENTS.org | 48 +++++++++++++++++++ .../countries/Niger/2011-12/_/data_info.yml | 38 +++++++++++++-- lsms_library/countries/Niger/_/CONTENTS.org | 43 +++++++++++++++++ 8 files changed, 260 insertions(+), 17 deletions(-) diff --git a/lsms_library/countries/Ethiopia/2011-12/_/data_info.yml b/lsms_library/countries/Ethiopia/2011-12/_/data_info.yml index cd565d605..7c375cf6c 100644 --- a/lsms_library/countries/Ethiopia/2011-12/_/data_info.yml +++ b/lsms_library/countries/Ethiopia/2011-12/_/data_info.yml @@ -1,6 +1,21 @@ Country: Ethiopia Wave: 2011-12 +# cluster_features (GH #323 site 4). Both sources are HOUSEHOLD-grain, so +# they are merged on the household id (1:1 -- household_id is unique and blank- +# free in BOTH frames: 3,969 cover rows / 3,917 geo rows, 0 keys duplicated in +# both) and reduced to the (t, v) EA grain by Wave.cluster_features' existing +# GH #161 collapse -- mean() for Latitude/Longitude (the cluster centroid), +# first() for Region/District/Rural. That is why `i` is in `final_index`: +# the collapse only fires when `i` is an index level. Merging on the CLUSTER +# key v instead made the join many-to-many (333 EAs shared by ~3.9k households +# on each side) -- a cartesian product. +# +# The geo file spells its coordinates LAT_DD_MOD / LON_DD_MOD (UPPERCASE) in +# THIS wave only; asking for the lowercase names raised a KeyError that the +# GH #515 optional-sub-df fallback swallowed, silently serving cluster_features +# with Latitude/Longitude 100% absent. Verified against the source: the file +# carries LAT_DD_MOD/LON_DD_MOD non-null for all 3,917 rows. cluster_features: dfs: - df_main @@ -9,6 +24,7 @@ cluster_features: file: sect_cover_hh_w1.dta idxvars: v: ea_id + i: household_id myvars: Region: saq01 District: saq02 @@ -30,15 +46,23 @@ cluster_features: df_geo: file: Pub_ETH_HouseholdGeovariables_Y1.dta idxvars: - v: ea_id + i: household_id myvars: - Latitude: lat_dd_mod - Longitude: lon_dd_mod + # UPPERCASE in W1 -- see the header comment. + Latitude: LAT_DD_MOD + Longitude: LON_DD_MOD merge_on: - - v + - i + # The cover page is authoritative for which households exist; the geo file + # is a strict enrichment. `left` keeps orphan geo households (households + # the cover does not carry) from arriving with a null v and collapsing + # into one phantom cluster. W1 has 0 such orphans, but the declaration is + # uniform across the five waves, which do (W2 25, W4 124, W5 35). + merge_how: left final_index: - t - v + - i household_roster: file: sect1_hh_w1.dta diff --git a/lsms_library/countries/Ethiopia/2013-14/_/data_info.yml b/lsms_library/countries/Ethiopia/2013-14/_/data_info.yml index ae5422ddd..df3f80228 100644 --- a/lsms_library/countries/Ethiopia/2013-14/_/data_info.yml +++ b/lsms_library/countries/Ethiopia/2013-14/_/data_info.yml @@ -1,6 +1,25 @@ Country: Ethiopia Wave: 2013-14 +# cluster_features (GH #323 site 4). Both sources are HOUSEHOLD-grain, so +# they are merged on the household id (1:1) and reduced to the (t, v) EA grain +# by Wave.cluster_features' existing GH #161 collapse -- mean() for +# Latitude/Longitude (the cluster centroid), first() for Region/District/Rural. +# That is why `i` is in `final_index`: the collapse only fires when `i` is an +# index level. +# +# Merging on the CLUSTER key v instead made the join many-to-many and produced +# a CARTESIAN PRODUCT: 5,262 cover households x 5,287 geo households over 433 +# EAs -> 65,508 rows, of which 60,221 were phantom. The downstream +# groupby().first() then tidied the evidence away, so the table reported clean. +# +# The key MUST be household_id2 (the W2 re-identified id), not household_id: +# household_id is the W1 baseline id and is BLANK for households new to the +# panel, so it is non-unique on the empty value -- re-keying to it would trade +# an EA cartesian for a NULL-KEY cartesian (pd.merge matches nulls). Verified: +# household_id2 is unique and blank-free in BOTH frames (5,262 / 5,287 rows, +# 0 keys duplicated in both). This matches the wave's v (ea_id2) and the +# i that sample/household_roster emit -- see _/CONTENTS.org. cluster_features: dfs: - df_main @@ -9,6 +28,7 @@ cluster_features: file: sect_cover_hh_w2.dta idxvars: v: ea_id2 + i: household_id2 myvars: Region: saq01 District: saq02 @@ -31,15 +51,21 @@ cluster_features: df_geo: file: Pub_ETH_HouseholdGeovars_Y2.dta idxvars: - v: ea_id2 + i: household_id2 myvars: Latitude: lat_dd_mod Longitude: lon_dd_mod merge_on: - - v + - i + # The cover page is authoritative for which households exist; the geo file + # is a strict enrichment and carries 25 households the cover does not (and + # a 434th EA the cover has never heard of). `left` keeps those orphans + # from arriving with a null v and collapsing into one phantom cluster. + merge_how: left final_index: - t - v + - i panel_ids: file: diff --git a/lsms_library/countries/Ethiopia/2015-16/_/data_info.yml b/lsms_library/countries/Ethiopia/2015-16/_/data_info.yml index 17f70ee64..6dc44849e 100644 --- a/lsms_library/countries/Ethiopia/2015-16/_/data_info.yml +++ b/lsms_library/countries/Ethiopia/2015-16/_/data_info.yml @@ -1,6 +1,22 @@ Country: Ethiopia Wave: 2015-16 +# cluster_features (GH #323 site 4). Both sources are HOUSEHOLD-grain, so +# they are merged on the household id (1:1) and reduced to the (t, v) EA grain +# by Wave.cluster_features' existing GH #161 collapse -- mean() for +# Latitude/Longitude (the cluster centroid), first() for Region/District/Rural. +# That is why `i` is in `final_index`: the collapse only fires when `i` is an +# index level. +# +# Merging on the CLUSTER key v instead made the join many-to-many and produced +# a CARTESIAN PRODUCT: 4,954 x 4,954 households over 432 EAs -> 57,786 rows, of +# which 52,832 were phantom. The downstream groupby().first() then tidied the +# evidence away, so the table reported clean. +# +# The key is household_id2 (the W2 re-identified id, which W3 inherits) -- the +# only id this geo file carries, and the i that sample/household_roster emit +# for this wave (see _/CONTENTS.org). Verified unique and blank-free in BOTH +# frames (4,954 / 4,954 rows, 0 keys duplicated in both). cluster_features: dfs: - df_main @@ -9,6 +25,7 @@ cluster_features: file: sect_cover_hh_w3.dta idxvars: v: ea_id2 + i: household_id2 myvars: Region: saq01 District: saq02 @@ -31,15 +48,21 @@ cluster_features: df_geo: file: ETH_HouseholdGeovars_y3.dta idxvars: - v: ea_id2 + i: household_id2 myvars: Latitude: lat_dd_mod Longitude: lon_dd_mod merge_on: - - v + - i + # The cover page is authoritative for which households exist; the geo file + # is a strict enrichment. `left` keeps orphan geo households from arriving + # with a null v and collapsing into one phantom cluster. W3 has 0 such + # orphans, but the declaration is uniform across the five waves, which do. + merge_how: left final_index: - t - v + - i panel_ids: # Built by bespoke script (materialize: make in data_scheme.yml). diff --git a/lsms_library/countries/Ethiopia/2018-19/_/data_info.yml b/lsms_library/countries/Ethiopia/2018-19/_/data_info.yml index d011c9079..4a28f6b72 100644 --- a/lsms_library/countries/Ethiopia/2018-19/_/data_info.yml +++ b/lsms_library/countries/Ethiopia/2018-19/_/data_info.yml @@ -1,6 +1,21 @@ Country: Ethiopia Wave: 2018-19 +# cluster_features (GH #323 site 4). Both sources are HOUSEHOLD-grain, so +# they are merged on household_id (1:1 -- unique and blank-free in BOTH frames: +# 6,770 cover rows / 6,804 geo rows, 0 keys duplicated in both) and reduced to +# the (t, v) EA grain by Wave.cluster_features' existing GH #161 collapse -- +# mean() for Latitude/Longitude (the cluster centroid), first() for +# Region/District/Rural. That is why `i` is in `final_index`: the collapse +# only fires when `i` is an index level. Merging on the CLUSTER key v instead +# was a latent cartesian (535 EAs shared by ~6.8k households on each side); it +# never fired only because the geo sub-df never loaded at all -- see below. +# +# W4 spells its coordinates lat_mod / lon_mod -- NO '_dd_' infix, unlike every +# other ESS wave. Asking for lat_dd_mod raised a KeyError that the GH #515 +# optional-sub-df fallback swallowed, silently serving cluster_features with +# Latitude/Longitude 100% absent. Verified against the source: the file +# carries lat_mod/lon_mod non-null for 6,637 of its 6,804 rows. cluster_features: dfs: - df_main @@ -9,6 +24,7 @@ cluster_features: file: sect_cover_hh_w4.dta idxvars: v: ea_id + i: household_id myvars: Region: saq01 District: saq02 @@ -24,15 +40,22 @@ cluster_features: df_geo: file: ETH_HouseholdGeovariables_Y4.dta idxvars: - v: ea_id + i: household_id myvars: - Latitude: lat_dd_mod - Longitude: lon_dd_mod + # W4 drops the '_dd_' infix -- see the header comment. + Latitude: lat_mod + Longitude: lon_mod merge_on: - - v + - i + # The cover page is authoritative for which households exist; the geo file + # is a strict enrichment and carries 124 households the cover does not. + # `left` keeps those orphans from arriving with a null v and collapsing + # into one phantom cluster. + merge_how: left final_index: - t - v + - i household_roster: file: sect1_hh_w4.dta diff --git a/lsms_library/countries/Ethiopia/2021-22/_/data_info.yml b/lsms_library/countries/Ethiopia/2021-22/_/data_info.yml index c83007d29..42d58a836 100644 --- a/lsms_library/countries/Ethiopia/2021-22/_/data_info.yml +++ b/lsms_library/countries/Ethiopia/2021-22/_/data_info.yml @@ -1,6 +1,22 @@ Country: Ethiopia Wave: 2021-22 +# cluster_features (GH #323 site 4). Both sources are HOUSEHOLD-grain, so +# they are merged on household_id (1:1 -- unique and blank-free in BOTH frames: +# 4,959 cover rows / 4,890 geo rows, 0 keys duplicated in both) and reduced to +# the (t, v) EA grain by Wave.cluster_features' existing GH #161 collapse -- +# mean() for Latitude/Longitude (the cluster centroid), first() for +# Region/District/Rural. That is why `i` is in `final_index`: the collapse +# only fires when `i` is an index level. +# +# The W5 geo file has NO ea_id column AT ALL -- it is keyed on household_id +# only (52 columns, checked). So the old `v: ea_id` idxvar on df_geo asked it +# for a column it does not have; the KeyError was swallowed by the GH #515 +# optional-sub-df fallback and cluster_features was served with +# Latitude/Longitude 100% absent. The COORDINATES themselves were never +# missing: lat_dd_mod / lon_dd_mod are present and non-null for 4,847 of the +# 4,890 rows. It was the KEY that was wrong, not the data -- so this is wired +# up, NOT marked optional. Joining on household_id lets the cover supply v. cluster_features: dfs: - df_main @@ -9,6 +25,7 @@ cluster_features: file: sect_cover_hh_w5.dta idxvars: v: ea_id + i: household_id myvars: Region: saq01 District: saq02 @@ -24,17 +41,24 @@ cluster_features: urban: Urban 2. URBAN: Urban df_geo: + # Keyed on household_id ONLY -- this file carries no ea_id. file: eth_householdgeovariables_y5.dta idxvars: - v: ea_id + i: household_id myvars: Latitude: lat_dd_mod Longitude: lon_dd_mod merge_on: - - v + - i + # The cover page is authoritative for which households exist; the geo file + # is a strict enrichment and carries 35 households the cover does not. + # `left` keeps those orphans from arriving with a null v and collapsing + # into one phantom cluster. + merge_how: left final_index: - t - v + - i panel_ids: file: diff --git a/lsms_library/countries/Ethiopia/_/CONTENTS.org b/lsms_library/countries/Ethiopia/_/CONTENTS.org index 76815e757..49459af13 100644 --- a/lsms_library/countries/Ethiopia/_/CONTENTS.org +++ b/lsms_library/countries/Ethiopia/_/CONTENTS.org @@ -454,6 +454,54 @@ We therefore DEFER Latitude/Longitude -- they are NOT emitted. Note the GPS-*measured field area* (square metres; =pp_s3q05_a= W1-W3, =s3q08= W4-W5) is NOT redacted and is the primary Area source. +This applies to =plot_features= ONLY. The HOUSEHOLD geovariables +files are a different release and DO carry usable (offset) coordinates +-- see the next section. Do not conflate the two. + +** Household geovariables: four spellings across five waves (GH #323) + +=cluster_features= gets its Latitude/Longitude from the per-wave +household geovariables file. The coordinates are the WB's standard +*modified* (randomly offset) decimal degrees -- real and usable at the +EA/cluster grain, and NOT redacted like the plot GPS above. + +The catch: the WB renamed the coordinate columns in almost every wave, +and W5 dropped the EA key entirely. All four variants are live: + +| Wave | Geovariables file | Lat / Lon columns | Keys present | +|---------+---------------------------------------+----------------------------+-------------------------| +| 2011-12 | =Pub_ETH_HouseholdGeovariables_Y1= | =LAT_DD_MOD= / =LON_DD_MOD= (UPPERCASE!) | household_id, ea_id | +| 2013-14 | =Pub_ETH_HouseholdGeovars_Y2= | =lat_dd_mod= / =lon_dd_mod= | household_id(2), ea_id(2) | +| 2015-16 | =ETH_HouseholdGeovars_y3= | =lat_dd_mod= / =lon_dd_mod= | household_id2, ea_id2 | +| 2018-19 | =ETH_HouseholdGeovariables_Y4= | =lat_mod= / =lon_mod= (no =_dd_=!) | household_id, ea_id | +| 2021-22 | =eth_householdgeovariables_y5= | =lat_dd_mod= / =lon_dd_mod= | household_id ONLY -- *no ea_id* | + +Every one of these files is HOUSEHOLD-grain, not EA-grain. So +=cluster_features= merges the cover page to the geo file on the +HOUSEHOLD id (1:1; unique and blank-free in both frames in all five +waves) and lets =Wave.cluster_features='s GH #161 collapse reduce to +the =(t, v)= EA grain -- =mean()= for Latitude/Longitude (the cluster +centroid), =first()= for Region/District/Rural. This is why =i= is in +each wave's =final_index=: that collapse only fires when =i= is an +index level. + +Merging on the CLUSTER key =v= instead -- which is what the config used +to do -- is a many-to-many join and produces a CARTESIAN PRODUCT +(2013-14: 5,262 x 5,287 households over 433 EAs -> 65,508 rows, 60,221 +of them phantom; 2015-16: 57,786 rows, 52,832 phantom). The downstream +=groupby().first()= then tidied the evidence away, so the table +reported clean. Use =household_id2= for W2/W3, NOT =household_id=: +the latter is the W1 baseline id and is BLANK for households new to the +panel, so re-keying to it would trade an EA cartesian for a NULL-KEY +cartesian. + +=merge_how: left= is declared in all five waves: the cover page is +authoritative for which households exist, and the geo file carries +households the cover does not (W2 25, W4 124, W5 35). Under the +default =outer= those orphans arrive with a null =v= and collapse into +one phantom cluster -- W2 gained a spurious 434th EA +(=030412088800105=) exactly this way. + ** Area and the local-unit gap Area is GPS square-metres / 10000 -> hectares where present diff --git a/lsms_library/countries/Niger/2011-12/_/data_info.yml b/lsms_library/countries/Niger/2011-12/_/data_info.yml index 0402ca371..126c35339 100644 --- a/lsms_library/countries/Niger/2011-12/_/data_info.yml +++ b/lsms_library/countries/Niger/2011-12/_/data_info.yml @@ -66,14 +66,46 @@ cluster_features: urbain: Urban URBAIN: Urban df_geo: - file: NER_2011_ECVMA_v01_M_Stata8/NER_HouseholdGeovars_Y1.dta + # GH #323 site 4 -- THE COORDINATES WERE IN THE WRONG FILE. + # + # This used to point at NER_HouseholdGeovars_Y1.dta and ask it for + # lat_dd_mod / lon_dd_mod. That file has 43 columns and NOT ONE of + # them is a coordinate: it ships only DERIVED raster covariates + # (dist_road, dist_market, af_bio_*, srtm, twi, sq1..sq7, EVI/NDVI + # series). The KeyError was swallowed by the GH #515 optional-sub-df + # fallback, and cluster_features has been served with + # Latitude/Longitude 100% ABSENT ever since. + # + # The coordinates ship in a SIBLING file, NER_EA_Offsets.dta -- the + # WB's standard offset-EA-centroid release: 271 rows x (grappe, + # LAT_DD_MOD, LON_DD_MOD), 270 non-null grappes, coordinates inside + # Niger's bounding box (lat 11.88-18.75 N, lon 0.40-13.70 E). It is + # already DVC-tracked alongside the rest of the wave. + # + # So this is NOT a "the survey never measured GPS" case and the + # columns are NOT marked `optional:` -- the data is real, available, + # and merely mis-addressed. Wired up. + # + # Grain: EA_Offsets is one row per grappe (270 unique / 270), while + # df_main (ecvmamen_p1) is household-grain (3,968 rows over the same + # 270 grappes). So the merge on the cluster key v is many-to-ONE -- + # NOT a cartesian (0 keys duplicated in BOTH frames). Every grappe in + # df_main is present in the offsets file and vice versa (0 orphans + # either way). The coordinate is an EA-level constant, so the + # downstream collapse to the (t, v) cluster grain is exactly lossless. + file: NER_2011_ECVMA_v01_M_Stata8/NER_EA_Offsets.dta idxvars: v: grappe myvars: - Latitude: lat_dd_mod - Longitude: lon_dd_mod + Latitude: LAT_DD_MOD + Longitude: LON_DD_MOD merge_on: - v + # `left` is load-bearing: the offsets file has ONE trailing row with a + # null grappe (and null coordinates). Under the default `outer` that row + # arrives as a phantom cluster with a null v; `left` drops it, since + # df_main is authoritative for which clusters exist. + merge_how: left final_index: - t - v diff --git a/lsms_library/countries/Niger/_/CONTENTS.org b/lsms_library/countries/Niger/_/CONTENTS.org index 23390f7fb..b7ea4ddd8 100644 --- a/lsms_library/countries/Niger/_/CONTENTS.org +++ b/lsms_library/countries/Niger/_/CONTENTS.org @@ -15,6 +15,49 @@ See CLAUDE.md "EHCVM countries" for the cross-country pattern. Pre-EHCVM waves (2011-12, 2014-15) follow their own conventions; see the wave-level =data_info.yml= files for index definitions. +* Cluster coordinates: the 2011-12 GPS is in =NER_EA_Offsets.dta= (GH #323) + +=cluster_features= Latitude/Longitude by wave: + +| Wave | Source file | Lat / Lon columns | Grain | +|---------+------------------------------------------+----------------------------------------------------+----------------| +| 2011-12 | =NER_EA_Offsets.dta= | =LAT_DD_MOD= / =LON_DD_MOD= | grappe (EA) | +| 2014-15 | -- none shipped -- | -- | -- | +| 2018-19 | =grappe_gps_ner2018.dta= | =coordonnes_gps__Latitude= / =__Longitude= | grappe (EA) | +| 2021-22 | (see wave =data_info.yml=) | | grappe (EA) | + +THE TRAP, recorded because it cost real time and nearly cost a real +column: the 2011-12 config used to point =df_geo= at +=NER_HouseholdGeovars_Y1.dta= and ask it for =lat_dd_mod= / +=lon_dd_mod=. *That file has no coordinates at all.* Its 43 columns +are DERIVED raster covariates only -- =dist_road=, =dist_market=, +=af_bio_*=, =srtm=, =twi=, =sq1..sq7=, the EVI/NDVI series. There is +no latitude and no longitude anywhere in it. The resulting =KeyError= +was swallowed by the GH #515 optional-sub-df fallback, so +=cluster_features= was served with Latitude/Longitude 100% ABSENT for +this wave, behind a warning nobody read. + +It is tempting to conclude "the survey has no GPS -- mark it +=optional:=". *That conclusion is wrong.* The coordinates ship in a +SIBLING file in the very same directory: =NER_EA_Offsets.dta=, the WB's +standard offset-EA-centroid release -- 271 rows x (=grappe=, +=LAT_DD_MOD=, =LON_DD_MOD=), 270 non-null grappes, coordinates inside +Niger's bounding box (lat 11.88-18.75 N, lon 0.40-13.70 E), already +DVC-tracked. Every grappe in the household file is present in it and +vice versa (0 orphans either way). The file was DVC-tracked and +available the whole time; only the config was pointing at the wrong +one. All 270 clusters now carry real coordinates. + +Because the offsets file is already at the EA grain, the merge on =v= +is many-to-ONE (not a cartesian) and the collapse to =(t, v)= is +exactly lossless -- the coordinate is an EA-level constant. +=merge_how: left= is load-bearing: the offsets file has one trailing +null-grappe row which =outer= would turn into a phantom cluster. + +2014-15 genuinely ships no geovariables/offsets file of any kind (the +wave directory contains none), so its Latitude/Longitude are honestly +absent -- not mis-addressed. It is NOT wired, and that is correct. + * Household Presence / MonthsSpent The EHCVM questionnaire (used in the 2018-19 and 2021-22 waves) does