Release v0.9.0: development → master

.claude/skills/add-feature/SKILL.md

@@ -423,8 +423,8 @@ The corresponding `data_scheme.yml` entry:
 - **Subdirectory structure:** Some waves use `Data/Cross_Sectional/` and `Data/Panel/` subdirectories; others have files directly in `Data/`.
 - **Multi-round files:** A single `.dta` containing multiple survey rounds (e.g., Tanzania 2008-15) cannot be handled by `data_info.yml` alone — use a `.py` script.
 - **Missing columns across waves:** Earlier survey instruments may not include all variables. Columns absent from a wave's `data_info.yml` entry will be NaN in the output — this is expected.
-- **DVC-tracked files:** Pull data with `dvc pull {path}.dvc` before building. Run from the DVC root (`lsms_library/countries/`).
-- **DVC lock contention:** If `dvc pull` hangs with `Unable to acquire lock`, clear stale locks: `rm -f lsms_library/countries/.dvc/tmp/*.lock lsms_library/countries/.dvc/tmp/rwlock`, then retry.  For parallel work across agents, use git worktrees so each has its own DVC lock.
+- **DVC-tracked files:** You do **not** pre-pull anything. `get_dataframe('../Data/file.dta')` materializes the blob on first read, lock-free and directly from S3. For a non-tabular file (questionnaire PDF, Excel codebook) use `data_access.get_data_file('{Country}/{wave}/Documentation/foo.pdf')`, which returns a local `Path`. **Never invoke the `dvc` CLI** — see `CLAUDE.md` §"Data Access".
+- **DVC lock contention:** Should not arise, because the read path never takes `.dvc/tmp/lock`. If you are seeing it, something is shelling out to `dvc` — find and remove that call rather than working around it. **Do not `rm` the lock file**: it may belong to a live sibling process, and deleting it corrupts that writer. Worktrees do *not* give an agent its own DVC lock — the DVC repo is shared, which is exactly why the lock-free read path exists.
 - **Pre-ISA vs ISA waves:** Earlier waves (e.g., Malawi 2004-05 "IHS2") predate the LSMS-ISA standardization and often use completely different module letters and variable naming conventions. Module L might be "non-food expenditures" in 2004-05 but "durable goods" in 2010+. Always verify via the World Bank data dictionary — never assume module letters are stable across survey instruments.
 - **`convert_categoricals` — value-label decode (bit several features in 2026-06):** `get_dataframe()` (and the YAML path) decode Stata/SPSS value labels by **default** (`convert_categoricals=True`). This cuts both ways:
   - In a **script** (`materialize: make`) that assumes *numeric codes* — e.g. `df['itemcode'].astype('Int64').map(CODE_MAP)` or `pd.to_numeric(df['have_flag'])` — the default makes the column come back as the **label string** (`'Watches'`, `'Yes'`), so `int(...)`/`to_numeric` fails or NaNs out (→ empty melt, `No objects to concatenate`). Pass `get_dataframe(path, convert_categoricals=False)` to keep numeric codes.

.claude/skills/add-feature/food-acquired/units/SKILL.md

@@ -71,8 +71,10 @@ from the first source that has it:
    `slurm_logs/build_ehcvm_unit_codebook.py` if a new sibling is added.
 
 3. **The questionnaire / codebook.** Survey documentation (Excel `Unites`
-   sheets, IHPS "CODES FOR UNIT" PDF pages) lists the code→label map. Pull it
-   from `{Country}/{wave}/Documentation/` via DVC (`.venv/bin/dvc pull …`) and
+   sheets, IHPS "CODES FOR UNIT" PDF pages) lists the code→label map. Fetch it
+   with `data_access.get_data_file('{Country}/{wave}/Documentation/foo.pdf')`,
+   which returns a local `Path` (lock-free; **never** shell out to `dvc` — see
+   `CLAUDE.md` §"Data Access"), then
    read with `pdftotext` or `pd.read_excel`. **Watch the scheme:** the
    questionnaire's generic list may use *different numbering* than the data
    (EHCVM's `Unites` sheet is 1–57; the data codes are 100–700 — they do not

.claude/skills/add-feature/sample/SKILL.md

@@ -159,6 +159,89 @@ sample:
         - t
 ```
 
+Note `merge_on: i` — the **household** key, unique in both sub-frames. That is
+what makes this merge a join.
+
+### The `dfs:` merge contract (GH #323 site 4)
+
+**`merge_on` must be unique in at least one sub-frame.** If both sub-frames are
+finer-grained than the merge key, the merge is many-to-many and pandas emits a
+*cartesian product within each key group* — it MANUFACTURES rows that exist in
+no survey. The classic error is joining two **household**-grain frames on the
+**cluster** key `v`: Ethiopia's `cluster_features` did exactly this and produced
+65,508 rows in 2013-14 and 57,786 in 2015-16 from tables that have 433 and 432
+clusters. The downstream `groupby().first()` collapse then tidied it away, so
+the table looked perfectly clean.
+
+`Wave._merge_subframes` now checks this exactly (a key value duplicated in
+*both* sub-frames is the definition of the cartesian) and warns with the phantom
+row count; `LSMS_GRAIN_STRICT` makes it fatal. That variable is read through the
+same `_grain_strict()` predicate as sites 1–2, so all three sites agree on what
+counts as "strict" (`1`/`true`/`yes`, case-insensitive) — and turning it on in
+CI is gated on the whole of #323, not just this site. **Fix it by making the
+merge correct — re-key to `i`, or reduce a sub-frame to the merge-key grain in a
+wave script.** Never by aggregating the explosion away afterwards: core does not
+aggregate (`SkunkWorks/grain_aggregation_policy.org`), and a reducer applied to
+a cartesian only puts a signature on the corpse.
+
+#### `merge_how:` (optional, default `outer`)
+
+```yaml
+    merge_on:
+        - i
+    merge_how: left      # default is `outer`
+```
+
+Declare `merge_how: left` when the **primary** sub-df (`dfs[0]`) is authoritative
+for which rows exist and the others are strict enrichments. A geovariable file
+that carries households the cover page does not is the usual case: under `outer`
+those orphans arrive with a null value in every index level the cover owned.
+
+**Be accurate about what this buys you.** Measured on Ethiopia 2013-14 (geo file
+carries 25 households the cover page does not):
+
+| | wave rows | null-`v` rows | delivered clusters | ΣLatitude | `District` dtype |
+|---|---:|---:|---:|---:|---|
+| `outer` | 5,287 | 25 | 433 | 4070.3702 | float64 |
+| `left`  | 5,262 | 0  | 433 | 4070.3702 | int8 |
+
+The orphans do **not** "collapse into one phantom cluster" — the downstream
+cluster-grain collapse *deletes* them, because `groupby` drops null keys. The
+delivered values are identical. So `merge_how: left` is not a data fix; it is
+worth declaring because it stops manufacturing null-keyed rows for site 1 to
+delete, and because it stops the merge widening an integer column. It has a
+cost: under `outer` the site-1 grain report *told* you 25 geo households had no
+cover page. `left` drops them silently. If that signal matters for a wave, keep
+`outer` and let site 1 report it.
+
+#### A dropped sub-df that owned a required column is now a hard error
+
+The GH #515 fallback drops a secondary sub-df that fails to load and proceeds
+with a warning. That is right for a file that is *unavailable* (nothing you can
+edit fixes it). It is **wrong** for a file that loaded fine but does not carry
+the column your YAML names — a typo, a casing mismatch (`lat_dd_mod` vs
+`LAT_DD_MOD`), a renamed variable. Ethiopia lost `Latitude`/`Longitude` from
+three of five waves exactly that way, behind a warning nobody read. If such a
+drop leaves a column declared **required** in `data_scheme.yml` entirely absent,
+`grab_data` now raises. Presence is judged after `derived:`, `drop:` and the
+wave's `df_edit` hook, so a column the wave module supplies is not reported
+absent.
+
+**Fix the column name.** In all ten cells this guard caught, the data existed:
+Ethiopia and Nigeria were casing mismatches or a wrong key column, and Niger
+2011-12's coordinates were in a *sibling* file in the same directory
+(`NER_EA_Offsets.dta`, `LAT_DD_MOD`/`LON_DD_MOD`) while the file the YAML named
+had none.
+
+**Know what this guard is and is not.** It is *"a mis-named column in a `dfs:`
+sub-df is fatal"* — not *"a required column is never absent"*. It fires only
+when a sub-df was dropped, so a wave with no `dfs:` block can serve a declared
+column 100% absent and never trip it (Niger **2014-15** does: `Latitude` is
+declared `float`, 0 of 270 rows populated, no raise). And `optional: true` is a
+blunt escape hatch — `data_scheme.yml` is **country**-grain, so it disarms the
+column for *every* wave and every script-path build of that country while the
+check is per-wave. Use it only where the country genuinely never has the column.
+
 ### Multi-round files (Tanzania 2008-15 pattern)
 
 When a single `.dta` file contains multiple survey rounds with a `round` column, the YAML path cannot handle it --- use a Python script. The script reads the file, maps round numbers to wave labels, and splits panel vs refresh households for `panel_weight`:

.claude/skills/add-wave/SKILL.md

@@ -29,7 +29,81 @@ from lsms_library.data_access import discover_waves
 discover_waves("Ethiopia")
 ```
 
-Returns a list of dicts annotated with `"local": True/False`. Waves marked `local=False` are on the WB but not yet in the repo.
+Returns a list of dicts annotated with:
+
+| key            | meaning                                                                 |
+|----------------|-------------------------------------------------------------------------|
+| `local`        | `bool` — `True` only when a wave dir *records* this catalog id.          |
+| `local_status` | `"yes"` / `"no"` / `"unknown"` — the honest tri-state (see below).       |
+| `local_waves`  | the wave directories backing this entry.                                 |
+
+Waves with `local_status == "no"` are on the WB but not yet in the repo.
+
+**Matching is on the WB catalog id, not on the wave label.** Each wave dir
+records the catalog entry it came from in `Documentation/SOURCE.org`
+(`#+CATALOG_ID:` — see `lsms_library/provenance.py`). Label matching used to
+be the mechanism and was wrong in both directions: two different surveys can
+share a year range (Nigeria's GHS-Panel W4 **3557** and Living Standards
+Survey **3827** both span 2018–2019), and one catalog entry can span two of
+our wave dirs (Uganda **1001** covers both `2005-06/` and `2009-10/`).
+
+`local_status == "unknown"` means a directory whose label matches exists but
+has **no recorded WB catalog id** — so we cannot say whether it holds this
+study or a different one with the same year range. Such rows carry
+`local=False`: an unverified claim is treated as not-held, because wrongly
+believing we hold a survey is the failure mode that hides missing data.
+
+To (re)stamp provenance across the tree:
+
+```sh
+python scripts/backfill_wave_provenance.py --dry-run   # report only
+python scripts/backfill_wave_provenance.py             # write SOURCE.org
+```
+
+Countries that are not WB datasets (`EthiopiaRHS`, `KenyaLPS`) are marked
+`discoverable=False` in `_COUNTRY_CATALOG` and return `[]` with an explanatory
+log — deliberately, rather than being silently absent.
+
+#### Which repositories get searched (GH #597)
+
+`discover_waves()` searches the collections listed in the country's
+`repositories` field, **default `("lsms",)`**. The World Bank publishes whole
+series outside `lsms` — Armenia's Integrated Living Conditions Survey (18
+waves) is in `central`, South Africa's General Household Survey (21 waves) is
+in `datafirst` — and a lsms-only search cannot see them *at all*.
+
+**Adding a new country requires a one-time broad sweep** to learn which
+repositories hold its series (`_wb_catalog_search(code, collection=None)`
+searches everything). Curate the answer into config; do not infer it at
+runtime.
+
+> **Widening a country to a second repository REQUIRES an `idno_pattern` that
+> pins the survey series.** Dropping the collection filter inflates results
+> 30–400× (Findex, DHS, Afrobarometer, enterprise surveys; `datafirst` alone
+> returns 320 South African rows — censuses, election studies, school
+> registers). Worse, it resurfaces studies we *already hold* under a different
+> catalog id in another repository: `central` 3016 (`MWI_2010_IHS-III_..._A_ML`)
+> is the same Malawi IHS3 as `lsms` 1003, and `datafirst` 902 (`ZAF_1993_PSLSD`)
+> is the same 1993 survey as `lsms` 297. Nothing in the catalog metadata links
+> those pairs, so id-matching cannot catch them — only the series pin can.
+> A missing-wave list nobody trusts is worse than no list.
+
+```python
+"Armenia":      CountryCatalog("ARM", idno_pattern=r"_(HBS|ILCS)_",
+                               repositories=("lsms", "central")),
+"South Africa": CountryCatalog("ZAF", idno_pattern=r"_(IHS|GHS)_",
+                               repositories=("lsms", "datafirst")),
+"Liberia":      CountryCatalog("LBR", idno_pattern=r"_(HIES|NHFS)_",
+                               repositories=("lsms", "central")),
+```
+
+`idno_pattern` answers **"is this catalog row this country's?"** (identity), NOT
+**"is this survey in remit?"**.  The two diverge: Liberia's NHFS is in the
+pattern because it is the catalog entry backing a wave dir we *hold*, but a
+forest-resources survey is not in remit.  Do not conflate them.
+
+Note `GET /api/collections` returns HTTP 400 — collection ids cannot be
+enumerated via the API. Read them off the `repositoryid` field of search rows.
 
 ### Step 2: Add the wave
 
@@ -125,14 +199,9 @@ DVC layers config files: `config.local` overrides `config`. This keeps the reade
 
 ## Batched DVC Operations
 
-Always use batched `dvc add` + `dvc push` when adding multiple files. The `populate_and_push()` and `push_to_cache_batch()` functions do this automatically. On a cluster scratch filesystem, batched operations process 68 files in ~2.5 minutes vs ~90 minutes sequentially.
+Publish blobs with `push_to_cache_batch()` (or `populate_and_push()` / `add_wave()`, which wrap it). These batch the underlying `dvc add` + `dvc push` for you: on a cluster scratch filesystem, batched operations process 68 files in ~2.5 minutes vs ~90 minutes sequentially. They also route through `_run_dvc_with_lock_retry()`, so concurrent writers queue on the global lock with backoff + jitter instead of failing.
 
-If running DVC commands manually, follow CONTRIBUTING.org steps 8-9:
-```bash
-cd lsms_library/countries
-dvc add Ethiopia/2021-22/Data/*.dta
-dvc push
-```
+**Do not run `dvc add` / `dvc push` by hand**, and never `rm` a DVC lock file — see `CLAUDE.md` §"Data Access". Hand-rolled invocations take the global `.dvc/tmp/lock` with no retry, which is how a parallel sweep gets a wave half-pushed.
 
 ## Data Loading in Scripts
 

.coder/coverage/blessed.csv

@@ -1 +1 @@
-country,feature,wave
+country,feature,wave,blessed_by,date,note