diff --git a/CLAUDE.md b/CLAUDE.md index c526fc721..e0fc993cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,6 +51,13 @@ The returned DataFrame prepends a `country` index level. For **config**-only edits (`countries/{C}/_/...` — YAML/scripts/`.org`, the common case), `LSMS_COUNTRIES_ROOT=/lsms_library/countries` is the cleaner lever: the live package reads the worktree's config tree (GH #436) and library-code identity stops mattering. A fresh in-worktree venv is **not** needed for either case. See the `.pth`-pinned package imports pitfall in the scrum-master-hpc skill. 4. *Savio compute nodes only*: `.venv` is typically a symlink to `/local/jobNNN/venv` (node-local SSD) and goes stale whenever you land on a different node. Recovery recipe lives in `.venv.lustre/README_WHY_THIS_EXISTS.md` at the repo root. **Do not just grab `.venv.lustre/bin/python`** — every import will round-trip through Lustre. Follow the README's **adopt-or-build recipe** (adopt any importable `/local/job*/venv` already on the node → tar-pipe build only if none → reclaim the stale ones), which gives cross-job persistence without root. Note: a stable `/local/$USER/` path is NOT creatable by us (`/local` is `root:root`); a genuinely persistent path needs an HPC-support ticket (README "Admin endgame"). **Opt-in fast path**: `bin/savio_venv.sh` packages the venv as a single squashfs image (`.venv.sqfs`, ~255 MB / one Lustre inode vs ~33k files) and mounts it per job via apptainer's `squashfuse_ll` (`mount`/`umount`/`update` subcommands) — kills the Lustre-MDS load and sidesteps the per-job reaper; see `docs/savio_venv.md` for the model + how to rebuild the image when deps change. This guidance is Savio-specific; other environments (login nodes, laptops, non-HPC clusters) use a normal in-tree `.venv/` and this paragraph doesn't apply. + 5. **Any agent scoped to a country MUST be told to read `countries/{C}/_/CONTENTS.org` first — put it in the prompt.** That file is where this repo records per-country *idiosyncrasies*: identifier conventions, survey-design oddities, known defects, and decisions already taken with their reasons. The `add-feature` skill already says "Read existing documentation FIRST … **Start here**", but a dispatched agent only reads what its brief tells it to, so this is the **dispatcher's** responsibility. + - Read the `LOGBOOK`/`WAITING`/`TODO` entries too, not just the prose: **a CLOSED GitHub issue can still have a live caveat parked there** (Tanzania `#114` is closed; its `CONTENTS.org` entry is still `WAITING`). + - Instruct the agent that if its finding **contradicts** `CONTENTS.org`, that is a *result to report and quote*, not a licence to assume the file is stale. + - Instruct it to **add** any genuine idiosyncrasy it discovers that is not recorded. That is what the file is for. + - **Why a *dispatcher* rule, when two rules already said this?** The `add-feature` skill says "Read existing documentation FIRST … **Start here**", and §"A script is a complication" says "BEFORE editing any script-path table, read the country's `_/CONTENTS.org` … Then say what you found." Both were already binding on the 2026-07-21 Uganda `crop_production` task (a script-path table) and **neither bit**, because they live in files an agent skims rather than in the brief it is handed. A rule an agent must have internalised is not enforcement; a checklist item in the prompt is. Put it in the prompt, and require the agent to **name the idiosyncrasy it found** — "I checked" is not an answer. + - Cost of skipping this, 2026-07-21: an agent concluded Tanzania's `shocks` key was broken because its `i` values shared none with `household_roster`; the dispatcher endorsed it, then flagged it as a regression, then withdrew both — because `Tanzania/_/CONTENTS.org` already documented the NPS back-casting design in full. Two rounds of confident, wrong framing from skipping one file. + ## Repository Layout - Country root-level symlinks (e.g. `Uganda -> lsms_library/countries/Uganda`) are convenience only; actual config lives under `lsms_library/countries/`. - **DVC repository is rooted at `lsms_library/countries/`, NOT the top-level repo.** `.dvc/`, remotes, and credentials all live there. Run `dvc` CLI commands from that directory or they fail with missing-remote errors. diff --git a/lsms_library/countries/Tanzania/_/CONTENTS.org b/lsms_library/countries/Tanzania/_/CONTENTS.org index cdbdd9b05..f1661fed0 100644 --- a/lsms_library/countries/Tanzania/_/CONTENTS.org +++ b/lsms_library/countries/Tanzania/_/CONTENTS.org @@ -1,5 +1,87 @@ #+title: Contents +* Survey design --- read before touching ANY 2008-15 table +:PROPERTIES: +:CUSTOM_ID: uphi-backcasting +:END: + +Tanzania's NPS handles split-off households differently from most --- +possibly all --- other surveys in this library, and the difference is +*not* confined to any one table. It was first written up under +[[https://github.com/ligon/LSMS_Library/issues/114][#114]] in the +context of =food_acquired= (see [[#issue-114][that entry]]), which has +made it easy to mistake for a food-specific quirk. It is not. + +** Split-offs are back-cast into EARLIER waves + +In most panels a household that forms mid-panel simply appears from +that wave onward. In the NPS's UPD4 construction, when a child leaves +and founds a new household, *new observations are created in the +earlier waves too*: the ancestor household's record is carried back +under the descendant's own panel identifier. + +Two identifiers coexist in the 2008-15 multi-round files, and their +Stata labels say exactly what they are: + +| variable | label | +|-----------+------------------------------------------------| +| =UPHI= | Uniform Panel Household Identifier at NPSY4 | +| =r_hhid= | NPS HHID: y1_hhid, y2_hhid, y3_hhid, y4_hhid | + +So =UPHI= identifies the panel household *as constituted at Round 4*, +propagated backwards; =r_hhid= is the round-specific household. The +relationship is many-to-many, and it is *deliberate*. + +Measured on =upd4_hh_r.dta= (2026-07-21): + +| round | rows | distinct =UPHI= | distinct =r_hhid= | +|-------+---------+-----------------+-------------------| +| 1 | 20,039 | 5,594 | 2,956 | +| 2 | 155,097 | 8,163 | 3,924 | +| 3 | 189,962 | 9,998 | 5,010 | +| 4 | 84,337 | 4,961 | 4,341 | + +In round 1, *1,526 =r_hhid= values carry more than one =UPHI=, up to +11*. One 2008-09 household appears once per descendant household it +eventually produces. + +** Why this matters for every table, not just food + +Any 2008-15 table inherits a choice, whether or not its author made it +consciously: + +- key on =UPHI= :: one row per (descendant household, round). A + household-level fact is repeated once per descendant --- the row + count reflects the Round-4 panel, not the interviewed households. +- key on =r_hhid= :: one row per (interviewed household, round). The + descendant linkage is discarded. + +Neither is "the" right answer, and *neither produces a signal that the +usual duplicate-index instruments can see*: + +- keyed on =UPHI=, the index is unique, so no collapse warning fires + --- absence of duplicates is *not* evidence the key is sound here; +- keyed on =r_hhid=, the back-cast copies collapse, and because their + payloads are identical the collapse is value-lossless and therefore + silent too. + +What the tables do today (2026-07-21) is *not* consistent: + +| table | key | note | +|------------------+----------------+--------------------------------------------------| +| =food_acquired= | =r_hhid= | de-duplicated; row counts 408,707 $\to$ 228,281 | +| =shocks= (2008-15) | =UPHI= | one row per descendant-household-round | + +*This inconsistency is unresolved.* It is a grain decision about +published tables and belongs to the maintainer; see the discussion on +[[https://github.com/ligon/LSMS_Library/pull/651][PR #651]]. Until it +is settled, do not "fix" one table's key to match the other as a side +effect of unrelated work --- and do not conclude from a zero duplicate +count that a 2008-15 key is sound. + +Related: the Round-4 sub-panel split is a separate matter, documented +at [[#panel-design][Panel design note]]. + * Issues ** WAITING Missing food quantities in 2019-20 and 2020-21 (#112) :LOGBOOK: @@ -463,6 +545,18 @@ the community↔cluster link. ** WAITING Duplicate rows in food_acquired from r_hhid/UPHI matching in 2008-15 (#114) +:PROPERTIES: +:CUSTOM_ID: issue-114 +:END: + +#+begin_quote +*Scope note (2026-07-21).* The =r_hhid= / =UPHI= many-to-many analysed +below is a *country-level* property of the NPS panel construction, not a +=food_acquired= quirk. It affects every 2008-15 table. See +[[#uphi-backcasting][Survey design]] at the top of this file; this entry +records only how it surfaced in =food_acquired=. +#+end_quote + :LOGBOOK: - Note taken on [2026-04-01 Tue] \\ Documented as GitHub issue #114. Root cause analysis added below.