Surfaced while testing the LSMS-ISA urban-exclusion hypothesis for GH #603, then re-verified cold (isolated LSMS_DATA_DIR, dvc-cache symlinked, LSMS_COUNTRIES_ROOT pinned and asserted) because the first measurement was taken on a warm cache.
Measured
GhanaLSS.sample().Rural (COLD, origin/development)
{nan: 21026, "Rural": 20263, "Urban": 15041}
nulls: 21026 of 56330 (37%)
Four of seven waves carry no Rural value at all. The 1991-92 wave additionally uses a non-canonical Semi-urban, which is not in the declared domain for this column.
Why it matters
Rural is the axis that separates a national household survey from a rural-only one — it is the single most load-bearing column for the population-record work in #603/#654, and the one that cleanly distinguishes LSMS-ISA (12.7–55% urban) from a genuinely rural survey (EthiopiaRHS, 0.0% urban in all 8 waves).
With 37% of rows null, any urban/rural analysis of GhanaLSS silently drops more than a third of the sample — and the drop is not random, it is four whole waves.
Semi-urban is a second, smaller problem: a third category that is neither Rural nor Urban. Whether it should map to one of them, or whether the canonical domain should admit it, is a judgement — but it should not sit outside the declared vocabulary unrecorded. Note lsms_library/data_info.yml declares the domain via Columns.*.Rural.spellings, and _enforce_canonical_spellings applies it at API time.
Not investigated
Which four waves, and whether the source .dta carries the information under a different column name (the common cause elsewhere in this library — cf. Nigers LAT_DD_MOD casing, Malawi 2016-17s case_id/cs_i mismatch). Both are cheap to check and would decide between "not collected" and "not wired".
Method note worth heeding
Of four Rural-related defects reported in the same sweep, two were warm-cache artifacts that do not reproduce cold (Uganda 2005-06s literal "0", fixed by GH #602; Malis unharmonised French, canonical cold). Verify against a fresh data root before acting on any of them. LSMS_NO_CACHE=1 alone is not sufficient — it is soft for script-path L2-wave parquets.
Refs #603, #654, #602.
Surfaced while testing the LSMS-ISA urban-exclusion hypothesis for GH #603, then re-verified cold (isolated
LSMS_DATA_DIR, dvc-cache symlinked,LSMS_COUNTRIES_ROOTpinned and asserted) because the first measurement was taken on a warm cache.Measured
Four of seven waves carry no
Ruralvalue at all. The 1991-92 wave additionally uses a non-canonicalSemi-urban, which is not in the declared domain for this column.Why it matters
Ruralis the axis that separates a national household survey from a rural-only one — it is the single most load-bearing column for the population-record work in #603/#654, and the one that cleanly distinguishes LSMS-ISA (12.7–55% urban) from a genuinely rural survey (EthiopiaRHS, 0.0% urban in all 8 waves).With 37% of rows null, any urban/rural analysis of GhanaLSS silently drops more than a third of the sample — and the drop is not random, it is four whole waves.
Semi-urbanis a second, smaller problem: a third category that is neitherRuralnorUrban. Whether it should map to one of them, or whether the canonical domain should admit it, is a judgement — but it should not sit outside the declared vocabulary unrecorded. Notelsms_library/data_info.ymldeclares the domain viaColumns.*.Rural.spellings, and_enforce_canonical_spellingsapplies it at API time.Not investigated
Which four waves, and whether the source
.dtacarries the information under a different column name (the common cause elsewhere in this library — cf. NigersLAT_DD_MODcasing, Malawi 2016-17scase_id/cs_imismatch). Both are cheap to check and would decide between "not collected" and "not wired".Method note worth heeding
Of four
Rural-related defects reported in the same sweep, two were warm-cache artifacts that do not reproduce cold (Uganda 2005-06s literal"0", fixed by GH #602; Malis unharmonised French, canonical cold). Verify against a fresh data root before acting on any of them.LSMS_NO_CACHE=1alone is not sufficient — it is soft for script-path L2-wave parquets.Refs #603, #654, #602.