Documentation/ should not be optional: 3 waves carry only a provenance stub, and add_wave() never fetches the real thing

[ligon]

Surfaced while gathering verbatim population/universe statements for GH #603 / #601.

The gap

Measured against origin/development, over the 109 declared waves:

	waves
substantive documentation present	104
only a provenance stub (SOURCE.org / LICENSE)	3

The three:

• Ethiopia/2021-22 — [LICENSE, SOURCE, SOURCE.org]
• Niger/2021-22 — [SOURCE.org]
• Senegal/2021-22 — [SOURCE.org]

All three are 2021-22 — i.e. the most recently added waves. That is the tell: this is not three unrelated omissions, it is a systematic gap in how waves get added.

For contrast, the sibling Senegal/2018-19/Documentation/ carries what a wave should have: ddi-documentation-english_microdata-4297.pdf, LICENSE.org, and the community questionnaires.

Root cause

data_access.add_wave() docstring:

Creates the directory structure (Data/, Documentation/, _/), writes SOURCE.org, downloads the Stata zip, extracts …

It creates Documentation/ and stamps provenance, but the only thing it downloads is the data. The DDI / BID / questionnaire files published alongside the survey in the WB catalog are never fetched. So every wave added through the supported path starts life with an empty-but-for-provenance Documentation/, and stays that way unless someone backfills by hand.

Why this matters more than tidiness

Documentation is not decoration in this repo — it is the only source for facts the data cannot tell you:

• The target population / universe. GH design(#601): the population record — declaring what a sample represents [DECISION NEEDED] #603 turns on what each sample purports to represent, and exclusions ("excludes urban", "excludes region X for security reasons", "excludes nomadic households") exist only in the documentation. You cannot recover an exclusion from a row count — a 98.8%-rural frequency is consistent with both a rural-by-design sample and an unlucky national one.
• Sampling weights. GH [umbrella] Rebuild GhanaSPS: registration, data refresh, feature gap #140 asks whether GhanaSPS 2013-14/2017-18 weights are in the BID PDFs or were never published; that question is unanswerable without the BIDs.
• Unit and condition code schemes. .claude/skills/add-feature/food-acquired/units/SKILL.md names the questionnaire as a primary decode source when .dta value labels are absent.
• absent verdict adjudication. docs/guide/coverage.md makes check C4 (the questionnaire) mandatory before closing a cell, because absence in the shipped .dta is not absence in the instrument. Without documentation, C4 cannot be run at all, so no absent cell in these waves can ever be honestly closed.

The immediate cost is live: a workflow gathering verbatim universe statements for #603 will return not-found for these three waves from local sources and must fall back to the WB catalog — which is slower, may not carry the same text, and is not archived with the wave.

Ask

1. Define a minimum Documentation/ standard — at least: provenance (SOURCE.org), licence, and the survey documentation proper (DDI/BID and questionnaire where published). Write it down where contributors will see it (CONTRIBUTING.org and the add-wave skill).
2. Make add_wave() fetch it, so the gap cannot recur through the supported path. The catalog id is already recorded in SOURCE.org, so the lookup is available.
3. Backfill the three waves above.
4. Consider a check — the coverage machinery already grades cells; a wave whose Documentation/ holds only a provenance stub could be surfaced the same way rather than discovered by accident, as this was.

Note on two directories that are NOT waves

While measuring I also hit two directories that look like waves and are not. Neither needs documentation; recording them so the next person does not re-derive them:

• Benin/2018-2019 — the duplicate-wave directory. Already removed on development (0 files tracked) but still present on master (52 files); it will clear at the next release merge.
• EthiopiaRHS/_dataverse_archive — 251 files, untracked, a local archive rather than a wave.

Refs #603, #601, #140.