ligon · ligon · Jul 22, 2026 · Jul 22, 2026 · Jul 22, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -51,6 +51,13 @@ The returned DataFrame prepends a `country` index level.
      For **config**-only edits (`countries/{C}/_/...` — YAML/scripts/`.org`, the common case), `LSMS_COUNTRIES_ROOT=<worktree>/lsms_library/countries` is the cleaner lever: the live package reads the worktree's config tree (GH #436) and library-code identity stops mattering.  A fresh in-worktree venv is **not** needed for either case.  See the `.pth`-pinned package imports pitfall in the scrum-master-hpc skill.
   4. *Savio compute nodes only*: `.venv` is typically a symlink to `/local/jobNNN/venv` (node-local SSD) and goes stale whenever you land on a different node.  Recovery recipe lives in `.venv.lustre/README_WHY_THIS_EXISTS.md` at the repo root.  **Do not just grab `.venv.lustre/bin/python`** — every import will round-trip through Lustre.  Follow the README's **adopt-or-build recipe** (adopt any importable `/local/job*/venv` already on the node → tar-pipe build only if none → reclaim the stale ones), which gives cross-job persistence without root.  Note: a stable `/local/$USER/<repo>` path is NOT creatable by us (`/local` is `root:root`); a genuinely persistent path needs an HPC-support ticket (README "Admin endgame").  **Opt-in fast path**: `bin/savio_venv.sh` packages the venv as a single squashfs image (`.venv.sqfs`, ~255 MB / one Lustre inode vs ~33k files) and mounts it per job via apptainer's `squashfuse_ll` (`mount`/`umount`/`update` subcommands) — kills the Lustre-MDS load and sidesteps the per-job reaper; see `docs/savio_venv.md` for the model + how to rebuild the image when deps change.  This guidance is Savio-specific; other environments (login nodes, laptops, non-HPC clusters) use a normal in-tree `.venv/` and this paragraph doesn't apply.
 
+  5. **Any agent scoped to a country MUST be told to read `countries/{C}/_/CONTENTS.org` first — put it in the prompt.**  That file is where this repo records per-country *idiosyncrasies*: identifier conventions, survey-design oddities, known defects, and decisions already taken with their reasons.  The `add-feature` skill already says "Read existing documentation FIRST … **Start here**", but a dispatched agent only reads what its brief tells it to, so this is the **dispatcher's** responsibility.
+     - Read the `LOGBOOK`/`WAITING`/`TODO` entries too, not just the prose: **a CLOSED GitHub issue can still have a live caveat parked there** (Tanzania `#114` is closed; its `CONTENTS.org` entry is still `WAITING`).
+     - Instruct the agent that if its finding **contradicts** `CONTENTS.org`, that is a *result to report and quote*, not a licence to assume the file is stale.
+     - Instruct it to **add** any genuine idiosyncrasy it discovers that is not recorded.  That is what the file is for.
+     - **Why a *dispatcher* rule, when two rules already said this?**  The `add-feature` skill says "Read existing documentation FIRST … **Start here**", and §"A script is a complication" says "BEFORE editing any script-path table, read the country's `_/CONTENTS.org` … Then say what you found."  Both were already binding on the 2026-07-21 Uganda `crop_production` task (a script-path table) and **neither bit**, because they live in files an agent skims rather than in the brief it is handed.  A rule an agent must have internalised is not enforcement; a checklist item in the prompt is.  Put it in the prompt, and require the agent to **name the idiosyncrasy it found** — "I checked" is not an answer.
+     - Cost of skipping this, 2026-07-21: an agent concluded Tanzania's `shocks` key was broken because its `i` values shared none with `household_roster`; the dispatcher endorsed it, then flagged it as a regression, then withdrew both — because `Tanzania/_/CONTENTS.org` already documented the NPS back-casting design in full.  Two rounds of confident, wrong framing from skipping one file.
+
 ## Repository Layout
 - Country root-level symlinks (e.g. `Uganda -> lsms_library/countries/Uganda`) are convenience only; actual config lives under `lsms_library/countries/`.
 - **DVC repository is rooted at `lsms_library/countries/`, NOT the top-level repo.** `.dvc/`, remotes, and credentials all live there. Run `dvc` CLI commands from that directory or they fail with missing-remote errors.

diff --git a/lsms_library/countries/Tanzania/_/CONTENTS.org b/lsms_library/countries/Tanzania/_/CONTENTS.org
@@ -1,5 +1,87 @@
 #+title: Contents
 
+* Survey design --- read before touching ANY 2008-15 table
+:PROPERTIES:
+:CUSTOM_ID: uphi-backcasting
+:END:
+
+Tanzania's NPS handles split-off households differently from most ---
+possibly all --- other surveys in this library, and the difference is
+*not* confined to any one table.  It was first written up under
+[[https://github.com/ligon/LSMS_Library/issues/114][#114]] in the
+context of =food_acquired= (see [[#issue-114][that entry]]), which has
+made it easy to mistake for a food-specific quirk.  It is not.
+
+** Split-offs are back-cast into EARLIER waves
+
+In most panels a household that forms mid-panel simply appears from
+that wave onward.  In the NPS's UPD4 construction, when a child leaves
+and founds a new household, *new observations are created in the
+earlier waves too*: the ancestor household's record is carried back
+under the descendant's own panel identifier.
+
+Two identifiers coexist in the 2008-15 multi-round files, and their
+Stata labels say exactly what they are:
+
+| variable  | label                                          |
+|-----------+------------------------------------------------|
+| =UPHI=    | Uniform Panel Household Identifier at NPSY4    |
+| =r_hhid=  | NPS HHID: y1_hhid, y2_hhid, y3_hhid, y4_hhid   |
+
+So =UPHI= identifies the panel household *as constituted at Round 4*,
+propagated backwards; =r_hhid= is the round-specific household.  The
+relationship is many-to-many, and it is *deliberate*.
+
+Measured on =upd4_hh_r.dta= (2026-07-21):
+
+| round | rows    | distinct =UPHI= | distinct =r_hhid= |
+|-------+---------+-----------------+-------------------|
+|     1 |  20,039 |           5,594 |             2,956 |
+|     2 | 155,097 |           8,163 |             3,924 |
+|     3 | 189,962 |           9,998 |             5,010 |
+|     4 |  84,337 |           4,961 |             4,341 |
+
+In round 1, *1,526 =r_hhid= values carry more than one =UPHI=, up to
+11*.  One 2008-09 household appears once per descendant household it
+eventually produces.
+
+** Why this matters for every table, not just food
+
+Any 2008-15 table inherits a choice, whether or not its author made it
+consciously:
+
+- key on =UPHI= :: one row per (descendant household, round).  A
+  household-level fact is repeated once per descendant --- the row
+  count reflects the Round-4 panel, not the interviewed households.
+- key on =r_hhid= :: one row per (interviewed household, round).  The
+  descendant linkage is discarded.
+
+Neither is "the" right answer, and *neither produces a signal that the
+usual duplicate-index instruments can see*:
+
+- keyed on =UPHI=, the index is unique, so no collapse warning fires
+  --- absence of duplicates is *not* evidence the key is sound here;
+- keyed on =r_hhid=, the back-cast copies collapse, and because their
+  payloads are identical the collapse is value-lossless and therefore
+  silent too.
+
+What the tables do today (2026-07-21) is *not* consistent:
+
+| table            | key            | note                                             |
+|------------------+----------------+--------------------------------------------------|
+| =food_acquired=  | =r_hhid=       | de-duplicated; row counts 408,707 $\to$ 228,281  |
+| =shocks= (2008-15) | =UPHI=       | one row per descendant-household-round           |
+
+*This inconsistency is unresolved.*  It is a grain decision about
+published tables and belongs to the maintainer; see the discussion on
+[[https://github.com/ligon/LSMS_Library/pull/651][PR #651]].  Until it
+is settled, do not "fix" one table's key to match the other as a side
+effect of unrelated work --- and do not conclude from a zero duplicate
+count that a 2008-15 key is sound.
+
+Related: the Round-4 sub-panel split is a separate matter, documented
+at [[#panel-design][Panel design note]].
+
 * Issues
 ** WAITING Missing food quantities in 2019-20 and 2020-21 (#112)
 :LOGBOOK:
@@ -463,6 +545,18 @@ the community↔cluster link.
 
 
 ** WAITING Duplicate rows in food_acquired from r_hhid/UPHI matching in 2008-15 (#114)
+:PROPERTIES:
+:CUSTOM_ID: issue-114
+:END:
+
+#+begin_quote
+*Scope note (2026-07-21).*  The =r_hhid= / =UPHI= many-to-many analysed
+below is a *country-level* property of the NPS panel construction, not a
+=food_acquired= quirk.  It affects every 2008-15 table.  See
+[[#uphi-backcasting][Survey design]] at the top of this file; this entry
+records only how it surfaced in =food_acquired=.
+#+end_quote
+
 :LOGBOOK:
 - Note taken on [2026-04-01 Tue] \\
   Documented as GitHub issue #114.  Root cause analysis added below.