design(#601): the population record — declaring what a sample represents [DECISION NEEDED]#603
design(#601): the population record — declaring what a sample represents [DECISION NEEDED]#603ligon wants to merge 1 commit into
Conversation
…epresents)
A proposal, not an implementation. It changes the semantics of a PUBLIC API
(`Feature()`) and encodes a judgement about what the library is willing to pool
on a user's behalf. That is a direction decision, and it is Ethan's.
I also nearly shipped the WRONG one, which is the best argument for asking.
## What I got wrong
I proposed `frame: national | subnational | specialized`, with `Feature()`
defaulting to `national`.
Ethan: "the lsms-isa countries focus on agriculture, and so often exclude urban
by design."
That kills it. A `national`-only default would have SILENTLY EXCLUDED THE
LSMS-ISA COUNTRIES -- the core of the library. I would have replaced a
silent-inclusion bug with a silent-exclusion bug: same disease, sign flipped.
## The distinction I had missed
Two different things a sample can be, and they must not share a label:
documented frame restriction Ethiopia ESS W1 (rural + small towns); the ISA
ag-focused designs. STILL a general-purpose
welfare survey of the country. -> INCLUDE,
record the restriction, warn on heterogeneity.
different target population Liberia NHFS (forest-adjacent communities).
NOT a general-purpose survey of Liberia.
-> FENCE: exclude from Feature() by default.
"% rural" is not the discriminator and never was. The discriminator is WHAT
POPULATION THE INSTRUMENT IS TRYING TO MEASURE.
## Why it is load-bearing, not an oddity
Ethiopia's universe changes ACROSS WAVES: urban households go 503 (2011-12) to
3,655 (2018-19). ESS W1 covered rural + small towns; later waves added urban
centres. So anyone running a panel on Ethiopia W1->W5 is pooling different
populations -- household fixed effects across a universe that shifted underneath
them. Every one of those waves is graded `sane`.
Liberia made the gap visible. Ethiopia makes it load-bearing: a core ISA
country, one of the most-used panels in the library.
It also settles the shape -- the universe is a property of the (country, wave)
cell, not the country.
## The connection
This is the same move as the rest of the week, pushed one step upstream. The
`not-asked` mechanism ADJUDICATES absence after the fact. The capability record
(PR #599) and the population record DECLARE it at acquisition time. Same
discipline -- stop lumping, demand evidence for terminal states -- applied when
the knowledge is cheapest instead of rediscovered by a probe months later.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
67f2a99 to
637e94b
Compare
Population-statement survey: all 111 waves, assembledThe per-country documentation sweep is now one reviewable document — It records, for every wave the library ships, what that survey's own documentation says its sample represents: verbatim quotes, exclusions called out in their own block for every wave, and the gaps left as gaps with the full search record. Headline counts
Both gap counts are correct; they answer different questions. 7 is the The rule the sweep obeyed
Universe-tag distributionThe tag is an editorial reading, not a quote — no document uses these words. It exists so 111 rows can be scanned.
Five things the evidence shows
Also worth noting: "declared universe" and "what the weights represent" come apart in real documents (Ethiopia ESPS-5 weights represent the population excluding Tigray; Nigeria 2018-19 is "representative of areas of Nigeria that were accessible during 2018/19"; Tanzania 2019-20 declares a national universe over an Extended-Panel cohort). And three surveys' documentation denies a well-defined universe outright — China 1995-97 ("not a rigorous random sample drawn from a well-defined population"), EthiopiaRHS ("these data are not nationally representative"), Albania 1996 (purposive over-sampling of poor areas). The library has no way to express that today, and those are the cells most dangerous to pool. No proposal is made. §5.7 of the document lists 17 unresolved disagreements, including one where the editorial tag contradicts the wave's own Full summary table (111 waves)
Expand — country, wave, source_type, confidence, universe tag
|
Is there a de facto DEFAULT urban exclusion in the LSMS-ISA countries?No. Not declared, and not realised. Exactly one wave in the entire 111-wave library declares an urban restriction (Ethiopia ESS1 2011-12), and it was lifted at the very next wave. Every other LSMS-ISA wave declares a national, explicitly urban-inclusive universe, and the realised samples are 20–55% urban. But the intuition is not wrong — it is aimed at the wrong object. What is agriculture-shaped in these surveys is module administration, not sample coverage. In its most extreme form, measured here for the first time: Ethiopia ESS4 (2018-19) and ESPS-5 (2021-22) administer the agriculture questionnaire to zero of their 3,655 / 2,674 urban households, while urban households are the majority of the sample. That is the precise statement, and it is very different from "excludes urban". Of the three claims in play, the evidence supports claim 3 (module applicability) and refutes claims 1 and 2 as characterisations of ISA design. 0. First: a correction to the evidence base (#654 / the sweep document)The maintainer's pushback on §5.2 is correct and I want to record why, because #603 should not build on the tag ladder as drawn. §5.2 says the 28 All eight Uganda waves were demoted on that basis. Their actual text (§3, verbatim):
And Uganda 2005-06 goes further than any tag suggests:
with strata of representativeness recorded across every UNPS wave as "Kampala City, Other Urban Areas, Central Rural, Eastern Rural, Western Rural, Northern Rural". Urban is a design stratum, not an omission. Uganda declares a national household universe. A second, independent problem with the ladder, found while checking the first. One identical sentence —
— is the population statement for 14 waves across 10 countries (Benin ×2, Burkina Faso, Côte d'Ivoire, Ethiopia ×2, Guinea-Bissau, Mali, Niger, Nigeria ×3, Senegal, Togo), and appears 30 times in the document overall. It is World Bank catalog boilerplate. Yet it earns the top tier ( (Where the wiggle room actually lives — see §5 below — is the household definition, not the scope.) 1. The LSMS-ISA country list, confirmed from the repoEthiopia, Malawi, Mali, Niger, Nigeria, Tanzania, Uganda, Burkina Faso — your list is right.
2. Per-country-per-wave: declared universe / realised urban share / ag-module coverage
Not one wave has an urban share near zero. The range is 12.7% (Ethiopia ESS1 — the one declared exclusion) to 55%. Median ≈ 33%. 3. The headline finding: Ethiopia ESS4 / ESPS-5 administer agriculture to rural EAs onlyThe The library config ( This is corroborated by the declared design, which the sweep records but does not connect: ESS rural EAs are "a subsample of the 2018 AgSS EA sample; 10 agricultural + 2 non-agricultural households per rural EA, 15 households per urban EA". The agricultural frame linkage exists only on the rural side. And it is a change. ESS1–ESS3 did put urban (small-town) households through the ag modules — 174 / 414 / 411 of them. ESS4 stopped. So within one country the library now spans two different module regimes under one feature name, with no record of the difference. This is exactly the kind of fact #603 exists to represent, and it is invisible in every wave-level universe statement. 4. Did the design change over time? Yes — twice, in opposite directions
Elsewhere the trend is uniformly toward more urban: Tanzania added a 545-household urban booster in 2020-21 (
5. Where the ambiguity actually is: the household definitionScope is well declared. The unit is where ISA waves genuinely differ:
Three materially different rules are in play (de jure vs. residence-based vs. age-gated eligibility), and Tanzania's "any member aged 15+, excluding live-in servants" is a genuine restriction on which households qualify — not just which people get interviewed. Uganda's silence is the real gap, and it is a gap about the unit, which is exactly the maintainer's "wiggle room". 6. Does the answer differ between ISA and the rest of the library? Yes — sharply
The contrast that matters for #603: the library does contain genuinely rural-only surveys, and they do not look like this. EthiopiaRHS is 0.0% urban across every wave and every household — a flat, unambiguous signal. China 1995-97 and India 1997-98 are the same shape (§5.3 records both as rural in fact while declining to write it as a universe). The ISA countries look nothing like them. Whatever representation #603 chooses needs to separate those two populations cleanly, and the realised urban share does so without ambiguity. Also worth noting: Nigeria 2012-13's 7. Data-quality issues surfaced in passing (not the assignment; reporting rather than dropping)
Happy to file these separately if useful; none of them changes any conclusion above. 8. Method / caveats
|
docs(#603): population-statement survey — all 111 waves, verbatim
…in 38 country CONTENTS.org For each country, a new "Sampling universe, exclusions and sub-samples" section records *what the survey documentation claims the sample represents* -- verbatim, with a file/page or catalog-field citation for every quote, and with gaps recorded as gaps (naming what was searched) rather than filled by inference. This is EVIDENCE ONLY. No schema, YAML key, or code change is proposed here: GH #603 owns the representation decision. Nothing in these sections is inferred from the .dta files -- what a survey claims and what its microdata contain are kept strictly apart. Placement follows each file's existing structure: sections sit beside the country's existing "Sampling Design" / "Survey Program" material where that exists, and above the issue list otherwise. Malawi and Uganda are filed one level down as the last subsection of their existing "Sampling Design" section. Four countries had no _/CONTENTS.org and get one (Afghanistan, Armenia, Peru, Serbia and Montenegro). Where the new evidence contradicts text already in a file, both sides are quoted and the disagreement is flagged in place -- nothing existing was edited or deleted (Azerbaijan weights, CotedIvoire CILSS panel shape, Ethiopia ESS1 coverage, GhanaLSS GLSS1/2 rotating panel, Guinea-Bissau vague/grappe, Mali EHCVM-vs-EAC-I, Nepal panel, Nigeria wt_wave5, Uganda refresh cadence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr
1 participant
A proposal, not an implementation. It changes the semantics of a public API (
Feature()) and encodes a judgement about what the library is willing to pool on a user's behalf. That is a direction decision. @ligon — this one is yours.I also nearly shipped the wrong one an hour ago, which is the best argument for asking.
What I got wrong
I proposed
frame: national | subnational | specialized, withFeature()defaulting tonational.That kills it. A
national-only default would have silently excluded the LSMS-ISA countries — the core of the library. I'd have replaced a silent-inclusion bug with a silent-exclusion bug. Same disease, sign flipped.The distinction I'd missed
Two different things a sample can be, and they must not share a label:
Feature()should do"% rural" is not the discriminator, and never was. The discriminator is what population the instrument is trying to measure.
This maps onto the SCOPE predicate the discovery agent derived independently in #599: Liberia NHFS fails "general-purpose welfare instrument"; Ethiopia W1 passes it with a documented restriction.
Why it's load-bearing, not an oddity
Ethiopia's universe changes across waves:
Urban goes 503 → 3,655. ESS W1 covered rural + small towns; later waves added urban centres. Anyone running a panel on Ethiopia W1→W5 is pooling different populations — household fixed effects across a universe that shifted underneath them. Every one of those waves is graded
sane.Liberia made the gap visible. Ethiopia makes it load-bearing — a core ISA country, one of the most-used panels in the library.
It also settles the shape: the universe is a property of the
(country, wave)cell, not the country.The judgement call
Record the universe per
(country, wave)— target population, deliberate exclusions, what the weights inflate to, and the source. Config, not prose. (Liberia/_/CONTENTS.orgalready describes the problem correctly, in plain English, and it changed nothing. Prose is not enforcement.)Then
Feature():df.attrs) — the universe travels with the data.specializedframes by default, opt-in to include. ← This is the contestable piece.For: Liberia is silently corrupting cross-country analysis today, and the people most likely to be harmed are the least likely to read a
CONTENTS.org.Against: it's paternalistic, and a default that drops data is exactly the silent behaviour this whole exercise has been fighting.
I lean toward fencing, because the failure is asymmetric — a spurious warning costs a user thirty seconds; a silently pooled forest survey costs them a published result. But it's a call about what the library is for.
The connection worth noticing
This is the same move as the rest of the week, one step upstream.
not-askedadjudicates absence after the fact. The capability record (#599) and the population record declare it at acquisition time. Same discipline — stop lumping, demand evidence for terminal states — applied when the knowledge is cheapest, instead of rediscovered by a probe months later.Full design:
slurm_logs/DESIGN_population_record_2026-07-12.org🤖 Generated with Claude Code