Skip to content

fix(#323): GhanaLSS cluster Region was the person's region of BIRTH (code list includes Nigeria)#622

Open
ligon wants to merge 2 commits into
developmentfrom
fix/323-ghanalss-config
Open

fix(#323): GhanaLSS cluster Region was the person's region of BIRTH (code list includes Nigeria)#622
ligon wants to merge 2 commits into
developmentfrom
fix/323-ghanalss-config

Conversation

@ligon

@ligon ligon commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Config-only. Zero changes to the access path. Supersedes PR #609 (same fix, minus its rejected +84-line country.py patch), per slurm_logs/DESIGN_grain_collapse_sites_2026-07-13.org.

Updated 2026-07-21 after adversarial review (FIX_FIRST). Every data number below reproduced exactly, cold, in two isolated LSMS_DATA_DIRs; the code did not move. Three claims used to justify the fix were wrong and are corrected — see "Corrections after review" at the bottom. All corrections are comment-only: parsed YAML is unchanged (verified by loading before/after and comparing objects), and the tests still discriminate.

Fix 1 — a Ghanaian EA cannot be in Nigeria

The 1987-88 and 1988-89 cluster_features blocks are deleted. They were wired to the wrong source column:

  • Region: REGION is the person's REGION OF BIRTH. The wave's own code list includes 11 = Nigeria, 12 = Ivory Coast, 13 = Togo. mapping.py's Region() and Birthplace() are the same function body over the same region_dict, and the same column is also mapped as the roster's Birthplace.

Verified: development builds 14,924 rows carrying a Region column for 1988-89; this branch builds 0. (Those 14,924 values are all-NA today, because get_categorical_mapping returns {} for the wave — the fabrication is armed, not firing. Repairing region_dict, needed anyway for Birthplace, arms it. Deleting the block costs nothing now and defuses it.)

On "but an EA could be in Nigeria if we're following movers" (@ligon, above) — answered by the data, not by assertion. Code 11 is populated: 236 persons in 1988-89. But they sit in 87 of the 170 clusters, and no cluster is more than 17.5% foreign-coded; zero clusters are 100%. If REGION were the EA's location, an EA coded 11 would have all its residents coded 11. GLSS also never followed movers — each round is an independent nationally representative cross-section of EAs drawn from the Ghanaian census frame (GhanaLSS/_/CONTENTS.org).

Fix 2 — 110 households collapsed onto one phantom tuple

2016-17 food_security rekeyed i: [clust, nh]i: hid. The old key produced NaN for 110 households whose clust/nh were missing, and all 110 collapsed onto a single phantom (t, NaN) index tuple.

Verified at wave level: 13,899 distinct i + 110 NaN → 14,009 distinct, 0 NaN. Re-derived from source: hid == f"{clust}/{nh:02d}" on 13,899 of 13,899 well-keyed rows, the 110 recovered ids are disjoint from those 13,899, and all 110 rows are NA on all eight FIES items.

Reviewer note — this moves no country-level API numbers

Worth knowing before you look for a delta: the 110 recovered households are all-NA, so _finalize_result's dropna(how='all') drops them at the API boundary either way. The fix removes the phantom at the source. It was verified at the wave level precisely because the country level cannot see it.

Structural check

GhanaLSS spans two source-format eras (fixed-width .DAT/.DCT for 1987–89 vs .dta later). The highest risk was the hid rekey silently changing the household keyspace — it does not: ids are byte-identical (70001/01), with 0 orphans against sample and household_roster.

(Note: GhanaLSS is not the PP/PH GhanaSPS — a near-miss on name alone.)

Also surfaced, pre-existing and not from this branch: 1998-99 cluster_features fails to build on development too.


Corrections after review (2026-07-21)

HIGH — "no cluster-invariant region source exists" was FALSE

1988-89/_/data_info.yml asserted, as permanent justification for the drop, that "a sweep of all 93 .DAT files in this wave's Data/ found no cluster-invariant region-like column". That is the Albania-precedent anti-pattern CLAUDE.md bans by name, and it is not merely unevidenced — it is contradicted. I re-ran the sweep: REGION appears in three of the 93 files.

measurement value
1988-89 files with a REGION column Y01A.DAT (roster), HEALTH.DAT (231×404), DRUG.DAT (169×191)
offset resolving CL1..CL5CLUST +2000 — 166/168 (HEALTH), 167/169 (DRUG) land in the roster's 170-cluster set; 0 and +1000 land 0
clusters covered by HEALTH ∪ DRUG 168 of 170
clusters with cross-facility disagreement 14
HEALTH vs DRUG agreement, shared clusters 154 of 165
REGION range HEALTH/DRUG 0..9 vs roster 1..11 → 0-indexed against the wave's 1-indexed region table
REGION + 1 == cluster's modal birth-region 160 of 166
mean / median share of a cluster's residents BORN in its HEALTH region 0.793 / 0.855

That last row is the cleanest confirmation of the PR's own thesis — the ~20% gap between "born here" and "lives here" is migration, which is exactly why birthplace cannot stand in for location. But the categorical negative had to go. The comments now say a candidate exists, unvalidated, not wired — a todo, not a closure.

Not wiring it is still right: 14 ambiguous clusters, 2 uncovered, and CL + 2000 is inferred from value ranges rather than documented — the .DCT files carry positions and names only, no variable labels. Confirming it needs the instrument, and per GhanaLSS/_/CONTENTS.org the GLSS1/GLSS2 questionnaires are scanned-image PDFs with no text layer, so check C4 is unrunnable for these two waves. That is precisely why a closing verdict here would have been unfalsifiable.

Partly disputed for 1987-88. Directly, the charge does not stick: a sweep of all 77 .DAT files in that wave (not 93 — that is 1988-89's count) finds REGION in exactly one file, the roster; COMM.DAT has no region column. Indirectly it does: 1988-89/Data/CLYR1YR2.DAT bridges the two waves — 85 linked (CLYR1, CLYR2) pairs, all 85 CLYR1 in 1987-88's 176 clusters and all 85 CLYR2 in 1988-89's 170; 84 of 85 reach a HEALTH/DRUG region, matching the 1987-88 cluster's modal birth-region in 77 of 84. Partial (48% of clusters) and inheriting every ambiguity above, so also a todo. Both readings are now recorded.

MEDIUM — the Age: AGEY characterization was FALSE

The old comment said AGEY "made a cluster's 'Age' the age of its first-listed person (range 17-80)". That never happened. 1987-88/_/mapping.py defines a table-level cluster_features(df) hook, dispatched by country.py as the table's df_edit; AGEY is its filter input (df.query("Age<12")) and the returned frame has columns ['Region', 'Rural'] — no Age, ever. Verified cold on the pre-fix config tree: Country('GhanaLSS')['1987-88'].grab_data('cluster_features')(0, 2), ['Region','Rural'], zero rows (both waves' region_dict is {}, so Region is all-NA and the hook's groupby(['t','v','Region']) drops every row).

So the real 1987-88 defect is modal birthplace of a cluster's under-12s — the very guess the 1988-89 comment says it refuses to make. It is now named, and a landmine note sits on the surviving dead hook, since re-adding the block re-arms it.

LOW ×2 — wrong wave's numbers

  • "development builds 14,924 rows … 1987-88 likewise" (ledger) — no. Development already builds 0 rows for 1987-88. Only 1988-89 has the delta. Corrected.
  • "Y00A.DAT's LOC … varies within cluster in 3 of 170 clusters" in the 1987-88 comment — 3/170 is 1988-89's figure; 1987-88 is 5 of 176. Also stronger and true: LOC is 1 for 3,142 of 3,147 households, i.e. near-constant, and its .DCT gives no label — so it is not a "plausible Rural indicator" at all. Still not wired.

Tests

Unchanged in substance; docstrings corrected (they carried the same false "no region variable exists in either wave's 93 source files" claim). They discriminate: against the origin/development config tree via LSMS_COUNTRIES_ROOT all 3 fail with their intended messages; against this branch all 3 pass. That is a real negative control — a separate origin/development checkout, not a git stash.

Follow-up, deliberately not done here

Wiring / adjudicating HEALTH.DAT + DRUG.DAT REGION as 1988-89 cluster_features.Region (bridged to 1987-88 via CLYR1YR2.DAT) needs the four absent-cell checks, and C4 is blocked on the image-only GLSS1/2 questionnaires. Noted in GhanaLSS/_/CONTENTS.org and both waves' data_info.yml; not guessed.

Refs #323. Supersedes #609.

🤖 Generated with Claude Code

@ligon

ligon commented Jul 20, 2026

Copy link
Copy Markdown
Owner Author

Claim is that "A Ghanaian enumeration area cannot be located in Nigeria." Need to confirm this--an enumeration area could be in Nigeria if we're following movers.

…d_security on hid (#323)

Config/script only: no change to lsms_library/country.py, feature.py or
local_tools.py, and no data_scheme `aggregation:` key.  Per Ethan's decisions
in slurm_logs/DESIGN_grain_collapse_sites_2026-07-13.org (D1: core does not
aggregate), the core patch on fix/323-ghanalss is dropped -- Site 1 is owned by
PR #614 and Site 2 by its own PR.  The original branch is left intact as the
audit trail.

Refs #323

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ligon
ligon force-pushed the fix/323-ghanalss-config branch from 9431506 to feb53dd Compare July 22, 2026 02:56
@ligon

ligon commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

Adversarial review — FIX_FIRST

Everything numeric in this PR reproduced exactly, cold, in two isolated LSMS_DATA_DIRs (dvc-cache symlinked, LSMS_NO_CACHE=1), development vs. this branch. The direction of both fixes is right and the change is scope-clean. Two claims used to justify the fix are false, and one of them is the Albania-precedent anti-pattern this repo has an explicit rule against. Both are comment-only edits — no data changes.


What I reproduced (all exact)

claim measured
1988-89 Y01A.DAT REGION varies within HH in 1,019/3,192, within cluster 167/170 1,019/3,192, 167/170 ✓ (14,924 rows / 3,192 HH / 170 clusters)
1987-88: 1,033/3,136, 174/176 1,033/3,136, 174/176 ✓ (15,492 / 3,136 / 176)
region code list includes 11=Nigeria, 12=Ivory Coast, 13=Togo ✓ (and code 11 is populated: 236 persons in 1988-89)
Region()Birthplace() over the same region_dict; region_dict == {} ✓ both waves
dev builds 14,924 Region rows for 1988-89; branch 0 ✓ dev (14924, 1), Region.notna() == 0; branch: not wired
2016-17 food_security wave: 13,899 distinct i + 110 NaN → 14,009 distinct, 0 NaN ✓ exactly
hid == f"{clust}/{nh:02d}" 100% fidelity 13,899/13,899, 0 mismatches
110 all-NA on all 8 FIES items, 110 distinct, disjoint, all in sample() ✓ all four; sample('2016-17') has exactly 14,009 ids → 0 orphans
5-wave cluster-invariance table in data_scheme.yml ✓ all row/cluster counts exact (4,523/365 … 934,584/1,000), max distinct per cluster = 1, n>1 = 0 — and not invariance-by-missingness: none of region/loc2/ez is all-NA in source

Conservation. Cold country-level frames, dev vs. branch, .equals():

cluster_features   (3145,3) vs (3145,3)  index equal ✓  frame equal: Ecological_zone only
food_security    (13892,9) vs (13892,9)  frame equal ✓
sample           (56330,5) vs (56330,5)  frame equal ✓
interview_date   (56340,1) vs (56340,1)  frame equal ✓
housing          (49994,8) vs (49994,8)  frame equal ✓

The lone Ecological_zone delta is the declaration taking effect: object/nanstring/<NA>, values identical where non-null (1,200 both sides). That's an improvement.

Tests discriminate. 3 pass on the branch; run against the development config tree (LSMS_COUNTRIES_ROOT=<dev-worktree>) all 3 fail with the intended messages. Real negative control, verified. test_gh323_grain_contract + test_gh323_explicit_reducers + test_schema_consistency + test_table_structure + this file: 350 passed cold.

House rules. D1 clean — diff is countries/ config + one test + the ledger; no lsms_library/*.py, no aggregation: key, no reducer. Not a duplicate: git log origin/development -- countries/GhanaLSS/ has no prior touch of cluster_features or food_security (#421 was 1991-92 Birthplace).


HIGH — the "no cluster-invariant region source exists" claim is false

1988-89/_/data_info.yml states, as permanent justification for the drop:

"a sweep of all 93 .DAT files in this wave's Data/ found no cluster-invariant region-like column, and CLUST does not encode region"

Two of those 93 files carry exactly that. HEALTH.DAT and DRUG.DAT (community health-facility / drug-availability modules) both have REGION plus CL1..CL5 — the clusters each facility serves. CL + 2000 is the wave's CLUST (166 of 168 codes land in the roster's cluster set), and REGION is 0-indexed against the wave's own 1-indexed region table.

HEALTH+DRUG cover 168 of 170 clusters   (14 with a cross-facility disagreement)
HEALTH vs DRUG agree on 156 of 167 shared clusters
REGION+1 == cluster's modal birth-region in 159 of 166
mean share of a cluster's residents BORN in its HEALTH region: 0.798 (median 0.855)

That 0.80 is itself the cleanest confirmation of your main thesis — ~20% migrants is exactly the gap between "born here" and "lives here". But it means the categorical negative is wrong, and 1987-88's "No cluster-invariant region source exists in this wave" is weaker than stated too: 1988-89/Data/CLYR1YR2.DAT maps year-1↔year-2 clusters (176/176 and 170/170 both resolve), bridging 84 of 85 linked 1987-88 clusters to a region.

This is the shape CLAUDE.md bans by name — "Never write an unevidenced 'no module here' claim, in a verdict file or in a YAML comment" — with the added problem that here it isn't merely unevidenced, it's contradicted. Merged, it closes two cluster_features cells permanently on a false premise. I'm not asking you to wire HEALTH.REGION (it needs the four checks; the 14 ambiguous clusters are real). I'm asking the comment to say what is true: a candidate exists in HEALTH/DRUG, unvalidated, not wired — i.e. a todo, not a closure.

MEDIUM — the Age: AGEY characterization is false

"Age: AGEY … Collapsed to one row per cluster it made a cluster's 'Age' the age of its first-listed person (range 17-80)"

That never happened. 1987-88/_/mapping.py defines a table-level cluster_features(df) hook, dispatched as the df_edit (country.py:802:983). It does:

youngsters = df.query("Age<12")
foo = youngsters.reset_index().groupby(['t','v','Region']).count()
... drop_duplicates(subset=['t','v'], keep='first') ...
return foo[['Region','Rural']]

AGEY is a filter input, and the returned frame has no Age column. Cold on development, Wave('1987-88').grab_data('cluster_features') is (0, 2) with columns ['Region','Rural'] — zero rows, no Age, ever. Two consequences:

  1. The real 1987-88 defect is modal birthplace of under-12s — precisely the guess the PR says it refuses to make ("Taking the cluster's MODAL birthplace would be a guess…"). It deserves naming; it's a better argument for the drop than the one given.
  2. That hook survives this PR as dead code with its docstring intact ("infers the region for each cluster via where most young kids have their birthplace"). Re-adding the block re-arms it. Worth a one-line note next to it, in the same spirit as the region_dict landmine you did document.

LOW — "1987-88 likewise" overstates

PR body: "development builds 14,924 birthplace-Region rows for 1988-89; this branch builds 0 … 1987-88 likewise." Development already builds 0 rows for 1987-88 (the df_edit above). Only 1988-89 has the 14,924. Fine as a deletion; not a 14,924→0 delta.

LOW — wrong wave's numbers in the 1987-88 comment

"Y00A.DAT's LOC … varies within cluster in 3 of 170 clusters"

3/170 is 1988-89's figure. 1987-88 is 5 of 176. (Measured both; LOC ∈ {1,2} in both waves.)


What I tried and could not break

  • Splitting real entities (hid rekey): the 110 recovered ids are disjoint from the 13,899, all 110 are in sample(), and the 13,899 reconstruct byte-identically — 14,009 rows → 14,009 groups, zero collisions. It is a strictly finer and correct key, not a key that scored zero by shattering entities.
  • Invariance by missingness: checked all five wired waves' region/loc2/ez for all-NA in the raw source. None is. The max distinct = 1 result is real, not vacuous.
  • Conservation: five country tables compared cell-by-cell; only the intended dtype change moved.
  • A test that passes both ways: it doesn't — 3/3 fail against the pre-fix config tree.
  • Silent loss from the deletion: country cluster_features is identical (3,145 rows) — those waves contributed 0 API rows on development too, as §7 of the ledger honestly states.
  • Ecological_zone fallout: not in canonical data_info.yml Columns, no other country declares it, and the 1,945 waves lacking ez build fine — no required-column assertion fires.
  • 1998-99's cluster_features failing to build is indeed pre-existing (absent from the country t-index on development too).

Verdict: FIX_FIRST — the code is right; correct the 1988-89 "no such column exists" claim (and the 1987-88 one), the AGEY characterization, and the 3/170 slip, then this is a clean merge.

ligon added a commit that referenced this pull request Jul 22, 2026
Comment-only.  Parsed YAML is byte-identical (verified: data_info /
data_scheme load to equal objects before and after), the tests still
discriminate, and no data moves.  Responds to the adversarial review on
PR #622, which reproduced every data number exactly and attacked the
claims used to justify the fix.

HIGH -- "no cluster-invariant region source exists" was FALSE, and it is
the Albania-precedent anti-pattern CLAUDE.md bans by name.  Re-ran the
sweep: REGION appears in THREE of 1988-89's 93 .DAT files, not one.
HEALTH.DAT and DRUG.DAT carry a facility-level REGION plus CL1..CL5 (the
clusters served); CL + 2000 is this wave's CLUST (166/168 and 167/169
land in the roster's cluster set; offsets 0 and +1000 land zero), the two
together cover 168 of 170 clusters, and REGION+1 equals the cluster's
modal birth-region in 160 of 166.  Mean share of a cluster's residents
BORN in its HEALTH region: 0.793 (median 0.855) -- the ~20% gap is the
migration that makes birthplace unusable as location, so this confirms
the PR's thesis while refuting its categorical negative.  1987-88 reaches
the same candidate for 84 of 85 clusters bridged by CLYR1YR2.DAT.  The
comments now record a `todo` (unvalidated: 14 ambiguous clusters, 2
uncovered, CL+2000 inferred not documented, and C4 is unrunnable because
the GLSS1/2 questionnaires are image-only PDFs) instead of a closure.

MEDIUM -- the `Age: AGEY` characterization was FALSE.  It never produced
a cluster "Age" column: it is the filter input to mapping.py's
table-level cluster_features(df) df_edit hook, which returns the modal
BIRTHPLACE of a cluster's under-12s.  Verified cold on the pre-fix tree:
grab_data('cluster_features') is (0, 2) ['Region','Rural'].  The real
1987-88 defect is that modal-birthplace guess -- a better argument for
the drop than the one given.  A landmine note now sits on the surviving
dead hook.

LOW -- development already builds 0 rows for 1987-88, so "1987-88
likewise" overstated a 14,924->0 delta that only 1988-89 has; and the
"LOC varies within cluster in 3 of 170" in the 1987-88 comment was
1988-89's figure (1987-88 is 5 of 176, and LOC is 1 for 3,142 of 3,147
households, so it is near-constant rather than a "plausible Rural
indicator").

Also recorded, answering the PR thread's movers objection: code 11
(Nigeria) is spread over 87 of 170 clusters with no cluster more than
17.5% foreign-coded and zero at 100% -- an EA coded 11 would have all its
residents coded 11.

Refs #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
@ligon
ligon force-pushed the fix/323-ghanalss-config branch from feb53dd to 1161bb9 Compare July 22, 2026 04:22
Comment-only.  Parsed YAML is byte-identical (verified: data_info /
data_scheme load to equal objects before and after), the tests still
discriminate, and no data moves.  Responds to the adversarial review on
PR #622, which reproduced every data number exactly and attacked the
claims used to justify the fix.

HIGH -- "no cluster-invariant region source exists" was FALSE, and it is
the Albania-precedent anti-pattern CLAUDE.md bans by name.  Re-ran the
sweep: REGION appears in THREE of 1988-89's 93 .DAT files, not one.
HEALTH.DAT and DRUG.DAT carry a facility-level REGION plus CL1..CL5 (the
clusters served); CL + 2000 is this wave's CLUST (166/168 and 167/169
land in the roster's cluster set; offsets 0 and +1000 land zero), the two
together cover 168 of 170 clusters, and REGION+1 equals the cluster's
modal birth-region in 160 of 166.  Mean share of a cluster's residents
BORN in its HEALTH region: 0.793 (median 0.855) -- the ~20% gap is the
migration that makes birthplace unusable as location, so this confirms
the PR's thesis while refuting its categorical negative.  1987-88 reaches
the same candidate for 84 of 85 clusters bridged by CLYR1YR2.DAT.  The
comments now record a `todo` (unvalidated: 14 ambiguous clusters, 2
uncovered, CL+2000 inferred not documented, and C4 is unrunnable because
the GLSS1/2 questionnaires are image-only PDFs) instead of a closure.

MEDIUM -- the `Age: AGEY` characterization was FALSE.  It never produced
a cluster "Age" column: it is the filter input to mapping.py's
table-level cluster_features(df) df_edit hook, which returns the modal
BIRTHPLACE of a cluster's under-12s.  Verified cold on the pre-fix tree:
grab_data('cluster_features') is (0, 2) ['Region','Rural'].  The real
1987-88 defect is that modal-birthplace guess -- a better argument for
the drop than the one given.  A landmine note now sits on the surviving
dead hook.

LOW -- development already builds 0 rows for 1987-88, so "1987-88
likewise" overstated a 14,924->0 delta that only 1988-89 has; and the
"LOC varies within cluster in 3 of 170" in the 1987-88 comment was
1988-89's figure (1987-88 is 5 of 176, and LOC is 1 for 3,142 of 3,147
households, so it is near-constant rather than a "plausible Rural
indicator").

Also recorded, answering the PR thread's movers objection: code 11
(Nigeria) is spread over 87 of 170 clusters with no cluster more than
17.5% foreign-coded and zero at 100% -- an EA coded 11 would have all its
residents coded 11.

Refs #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
@ligon
ligon force-pushed the fix/323-ghanalss-config branch from 1161bb9 to cfb5f2e Compare July 22, 2026 04:23
@ligon

ligon commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

FIX_FIRST applied — cfb5f2e4, comment-only

All three findings are correct and are fixed. I re-derived every number I wrote down rather than copying yours; where we differ I say so. Nothing about the fix changed: the parsed YAML is provably unchanged (loaded data_info / data_scheme before and after and compared objects — SAME for all four files), and the tests still discriminate.

(Housekeeping: my first push briefly force-updated the branch onto an older base — the branch had been rebased onto 45aee170 and I had built on the pre-rebase commit. Repaired immediately by resetting to feb53dd7 and cherry-picking. feb53dd7 is intact as the parent of cfb5f2e4; nothing was lost.)


HIGH — accepted. The categorical negative was false, and it is gone.

I re-ran the sweep myself rather than trusting yours. REGION appears in three of 1988-89's 93 .DAT files, not one:

measurement mine yours
files with a REGION column Y01A.DAT, HEALTH.DAT (231×404), DRUG.DAT (169×191) same
offset resolving CL1..CL5CLUST +2000; 166/168 HEALTH, 167/169 DRUG land in the roster's cluster set (0 and +1000 land 0) 166/168
HEALTH ∪ DRUG cluster coverage 168 of 170, 14 with cross-facility disagreement same, same
HEALTH vs DRUG agreement 154 of 165 shared 156 of 167
REGION+1 == modal birth-region 160 of 166 159 of 166
mean / median share born in the HEALTH region 0.793 / 0.855 0.798 / 0.855

The three small gaps are mode tie-breaking and whether the shared-cluster set is restricted to roster clusters; the substance is identical. And you're right that the 0.79 is the best confirmation of the PR's own thesis — the ~20% gap between "born here" and "lives here" is the migration that makes birthplace unusable as location.

The comments now say what is true: a candidate exists, unvalidated, not wired — a todo, not a closure. They also record why validation is blocked, which is the part that makes a closing verdict unfalsifiable here: the .DCT files carry positions and names only (verified — no variable labels), so CL + 2000 is inferred from value ranges rather than documented, and per GhanaLSS/_/CONTENTS.org the GLSS1/GLSS2 questionnaires are scanned-image PDFs with no text layer — check C4 is unrunnable for these two waves.

Partly disputed, for 1987-88. Directly, the charge does not stick: I swept all 77 .DAT files in that wave — not 93; 93 is 1988-89's count — and REGION appears in exactly one, the roster. COMM.DAT (137 rows) has no region column. Indirectly your point stands, and I reproduced it: CLYR1YR2.DAT gives 85 linked (CLYR1, CLYR2) pairs, all 85 CLYR1 in 1987-88's 176 clusters and all 85 CLYR2 in 1988-89's 170, with 84 of 85 reaching a HEALTH/DRUG region (matching that cluster's modal birth-region in 77 of 84). One correction to your framing: the values are already prefixed (1001 / 2001), so there is no offset to apply — and "176/176 and 170/170 resolve" holds for the wave cluster sets, but the bridge reaches only 85 of 176 1987-88 clusters (48%), so it is a partial candidate. Both readings are now in the file.

MEDIUM — accepted, and it is worse than you said

Reproduced exactly. 1987-88/_/mapping.py defines a table-level cluster_features(df) hook dispatched as the table's df_edit; AGEY is its filter input and the returned frame is ['Region','Rural'] — no Age, ever. Cold on the pre-fix tree, grab_data('cluster_features') is (0, 2).

The extra bit: the reason it is (0, 2) and not (176, 2) is that get_categorical_mapping returns {} for 1987-88 as well as 1988-89 (I measured both; the ledger previously named only 1988-89). Region is all-NA, so the hook's groupby(['t','v','Region']) drops every row. Same landmine, both waves: repair region_dict and 1988-89 starts fabricating via .first() while 1987-88 starts fabricating via modal-birthplace-of-under-12s. The comment now names that modal-birthplace guess as the real 1987-88 defect — you're right that it is the better argument for the drop — and a landmine note sits on the surviving dead hook.

LOW ×2 — both accepted, both re-measured

  • "1987-88 likewise" — no. Development already builds 0 rows for 1987-88; only 1988-89 has the 14,924→0 delta. Fixed in the ledger and the PR body. I also tightened the 14,924 claim itself: those Region values are all NA (notna() == 0), so the fabrication is armed, not firing.
  • "LOC … 3 of 170" in the 1987-88 comment — 3/170 is 1988-89's; 1987-88 is 5 of 176. Confirmed. Stronger and also true: LOC is 1 for 3,142 of 3,147 households in 1987-88 (3,191 of 3,194 in 1988-89) and its .DCT gives no label, so calling it "a plausible Rural indicator" was itself an overclaim. Corrected to "near-constant, meaning unverified, not wired".

@ligon's question — "an EA could be in Nigeria if we're following movers"

Fair challenge, and "it can't" is not an adequate answer, so here is the data. Code 11 is populated — 236 persons in 1988-89 — but they sit in 87 of the 170 clusters, and no cluster is more than 17.5% foreign-coded (>= 11); zero clusters are 100%. If REGION were the EA's location, an EA coded 11 would have all of its residents coded 11. Separately, GLSS never followed movers: every round is an independent nationally representative cross-section of EAs drawn from the Ghanaian census frame (GhanaLSS/_/CONTENTS.org, "Survey Program"). This is now strand 3 in 1988-89/_/data_info.yml.

Tests

Substance unchanged — they already discriminated and I re-verified it (3/3 fail against the origin/development config tree via LSMS_COUNTRIES_ROOT, 3/3 pass here; a separate checkout, not a git stash). Their docstrings carried the same false "no region variable exists in either wave's 93 source files" claim, so those are corrected, and a note now states explicitly that the suite discriminates and how that was checked.

Sweep on this branch: test_ghanalss_cluster_features_gh323 + test_gh323_grain_contract + test_gh323_explicit_reducers + test_grain_collapse_cluster_features + test_schema_consistency + test_table_structure600 passed, 2 failed. Both failures are pre-existing and untouched by this branch: GhanaLSS/sample and India/sample fail the new declared_spellings check on Rural ('Semi-urban' ×1,578 for GhanaLSS, from GLSS3's loc2; 'UP-other' / ' B-other' / … for India). The diff touches nothing named sample — the single occurrence of the word in the whole diff is inside a comment. Noting, not fixing; out of scope here.

Follow-up, deliberately not done

Wiring/adjudicating HEALTH.DAT + DRUG.DAT REGION as 1988-89 cluster_features.Region, bridged to 1987-88 via CLYR1YR2.DAT, needs the four absent-cell checks and C4 is blocked on the image-only questionnaires. Recorded as a TODO heading in GhanaLSS/_/CONTENTS.org, in both waves' data_info.yml, and in a new "Corrections after adversarial review" section of .coder/ledger/323-ghanalss.md. Not guessed.

@ligon

ligon commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

Independent verification of the FIX_FIRST remediation (cfb5f2e4) — READY

I did not take the remediation report on trust. Every number below I re-derived from source in a fresh pair of worktrees (cfb5f2e4 and origin/development), reading the .DAT/.DCT blobs directly through _ensure_dvc_pulled / _dvc_cache_path — no dvc CLI. Cold builds used an isolated LSMS_DATA_DIR with dvc-cache symlinked, plus LSMS_NO_CACHE=1.

Negative control — re-run by me, not inherited

A separate git worktree at origin/development, LSMS_COUNTRIES_ROOT pointed at its config tree (not a git stash, which would do nothing to a committed change):

branch config tree       : 3 passed
origin/development tree  : 3 failed
  test_early_waves_do_not_map_birthplace_as_cluster_region[1987-88]  -> "wired to Y01A.DAT REGION, the person-level region of BIRTH"
  test_early_waves_do_not_map_birthplace_as_cluster_region[1988-89]  -> same
  test_food_security_keys_on_complete_hid                            -> "must key on the complete `hid` column"

The tests discriminate.

Finding-by-finding

HIGH — the categorical negative — FIXED, and the correction is itself correct.

I re-ran the sweep independently (parsing all 93 .DCT sidecars, then reading the .DATs, which are comma-delimited with a header row, not fixed-width):

measurement mine in the file
1988-89 files whose dictionary declares REGION Y01A, HEALTH, DRUG (3 of 93) same
HEALTH / DRUG shape 231x404 / 169x191 same
CL{1..5} offset landing in the roster's cluster set +2000: 166/168, 167/169; +0 and +1000: 0/168, 0/169 same
HEALTH u DRUG cluster coverage 168 of 170 same
clusters with cross-facility disagreement 14 same
REGION range vs roster 0..9 vs 1..11 same
roster grain 14,924 persons / 3,192 HH / 170 clusters same
HEALTH.DCT contents positions + names only, no variable labels same

So the deleted claim really was contradicted, and the replacement (todo, not closure, with the C4 blocker named) is accurate. I also confirmed the C4 blocker is pre-existing, not invented for this PR: "GLSS1/2 questionnaires are scanned-image PDFs with no text layer" is already in GhanaLSS/_/CONTENTS.org on development (line 361), recorded for individual_education.

The 1987-88 partial dispute holds. 1987-88 has 77 .DAT files, not 93, and REGION appears in exactly oneY01A, the roster. COMM.DAT is 137x311 with no region-like column. The CLYR1YR2.DAT bridge reproduces exactly: 85 linked pairs, values already prefixed (1001/2001, so no offset to apply — the correction to my framing is right), all 85 CLYR1 in 1987-88's 176 clusters and all 85 CLYR2 in 1988-89's 170 -> at most 48% coverage. "Partial candidate", as the file now says.

MEDIUM — the AGEY characterization — FIXED. Cold wave-level build against the pre-fix config tree:

1987-88  grab_data('cluster_features') -> (0, 2)      ['Region','Rural']   Region.notna()=0
1988-89  grab_data('cluster_features') -> (14924, 1)  ['Region']           Region.notna()=0

No Age column, ever. And I confirm the extra bit the remediation added: get_categorical_mapping(..., 'region') returns {} for both waves (relationship too) — so the ledger's S8 correction is right and the landmine note on the surviving mapping.py hook is warranted.

LOW x2 — FIXED. development already builds 0 rows for 1987-88 (so "likewise" was wrong; only 1988-89 has the 14,924->0 delta). And Y00A.DAT's LOC:

1987-88   LOC=1 for 3,142 / 3,147   varies within cluster in  5 of 176
1988-89   LOC=1 for 3,191 / 3,194   varies within cluster in  3 of 170

3/170 was indeed 1988-89's figure; the file now carries 5/176 for 1987-88.

Comment-only, and scope

Parsed-YAML equivalence feb53dd7 -> cfb5f2e4, loaded with the same loader the library uses: SAME for all four (1987-88, 1988-89, 2016-17 data_info.yml, _/data_scheme.yml). The corrections provably cannot have moved data.

Scope clean: outside countries/ the diff touches only .coder/ledger/323-ghanalss.md and tests/test_ghanalss_cluster_features_gh323.py. No lsms_library/*.py. No aggregation: key anywhere in GhanaLSS YAML (grep '^\s*aggregation:' -> none; the ledger mentions are prose recording that it was stripped). No reducer. D1 clean.

No false claim survives in a tracked file: the only remaining occurrences of the old wording are inside explicit "That is FALSE and is corrected here" quotations in 1988-89/_/data_info.yml and the ledger.

Two small things, neither blocking

  1. 1,019 vs 1,018 within-household REGION variation (1988-89, strand 4). The one household of difference is CLUST=2493, NH=12: one member coded 6, three coded .. Counting it as "varies" treats missingness as a distinct region, which cuts against this repo's own rule that NaN is absence, not contradiction; on the non-missing reading it is 1,018. 1987-88's 1,033 is convention-insensitive, so the two waves' figures are computed under different conventions. The argument is untouched either way (max 6 distinct in one household is real regardless).
  2. The HEALTH-vs-DRUG agreement statistic is genuinely method-sensitive: I get 151/165, 155/165 or 157/167 depending on tie-breaking and whether the shared set is restricted to roster clusters (the file says 154/165, I originally said 156/167). Already disclosed as tie-breaking; it is a reason not to wire the candidate, so nothing rests on the exact value.
  3. Correction to the sweep note on the thread: the pre-existing declared_spellings failure I reproduce is GhanaLSS/cluster_features (Rural: 'Urban ' x439, 'Semi-urban' x100), not GhanaLSS/sample; India passes in my -k "GhanaLSS or India" run. It fails identically on origin/development (1 failed, 111 passed both trees), so the "pre-existing and out of scope" call is right — only the node label was off.

Verdict: READY. All three findings are genuinely addressed, the 1987-88 partial dispute is evidenced and correct, the negative control is real, and the corrections are provably comment-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant