From dee1d41d1e559cbce4dac8d234165547350f8328 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 14:20:15 +0000 Subject: [PATCH] Fix size-less RNG draws that made 15 columns constant; add tests and an answer key Twenty-two call sites across ten generators called numpy's RNG without a size argument, so the draw returned a scalar that broadcast across the whole column: owns_radio = rng.binomial(1, 0.55) # one draw, every row months_displaced = np.where(displaced, rng.exponential(14), 0) # one draw, every row Nothing raised, the row count was right and the file wrote. Fifteen columns were constants, among them four asset variables in `targeting` (a proxy-means-test dataset whose asset predictors did not vary) and the three dropout barriers in `girls_education`, all identically zero. The neighbouring `barrier_cost` was always fine because its probability is an array, which is what confirmed the diagnosis. None of it was visible because CI ran `python generate.py --list`, which imports no generator. All 36 modules were unexecuted by any automated check. Generators - Add the missing size argument at all 22 sites. Scalar draws that are genuinely parameters (per-arm compliance rate, per-country intercept, anything inside a per-row loop) are left alone. - rct_experiment: `spillover_risk` was 1 for every control and 0 for every treated unit, an exact alias of the treatment dummy, because randomisation was stratified within district so every district always held treated units. Add a village level, randomise 70% of villages into treatment and 30% into pure control, and give exposed controls a real +3% spillover. The flag now varies, and using contaminated controls biases the ITT toward zero by ~0.011 log points, which is the lesson the design should teach. - public_health: rebuild PHQ-9 as a sum of nine 0-3 items. The previous `9 * logistic(...) * 3` was bounded with no item-level variance, so the floor sat at 2 and `depression_severe` was identically zero in every draw. Now spans 0-27 with 14.6% at or above the moderate threshold and 1.45% at severe. - girls_education: schooling status is three-state. `dropped_out` was defined as the complement of `enrolled`, which both collapsed never-enrolled girls into dropouts and made the two columns perfect aliases. Also vectorise the distance loop and remove a dead `2 if True else 5` conditional. Tests and documentation - tests/: 233 tests. Every generator runs, is reproducible under a seed, and responds to a seed change; the registry matches the modules on disk; no column is constant or a perfect alias of another, with the legitimate exceptions (IRT item parameters, the poverty line) listed with reasons. - TRUTH.md: the answer key. Records which parameters are in the data and which estimand recovers each. The ITT is not a stable target because take-up is drawn U(0.65, 0.85) per run and moved between +0.089 and +0.203 across five seeds; the Wald LATE recovers theory to within 0.002. - CLAUDE.md: new, so the next session does not re-derive this. - CI: run the suite on 3.11 and 3.12, plus an end-to-end job writing every dataset to CSV and Parquet, since serialisation fails differently. - Pin requirements exactly. `pandas>=2.0` would pick up pandas 3, where text columns carry a `str` dtype and a check keyed on `dtype == object` silently skips every text column. - README: correct the output directory (`output/`, not `data/`), the Python floor (3.11), a dependency list naming `faker`, which nothing imports, and claims that the data mirrors the distributions of real surveys. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MrvR2NXsJFVRCeJZFCPuNL --- .github/workflows/ci.yml | 55 ++++++++++++-- .gitignore | 1 + CLAUDE.md | 105 +++++++++++++++++++++++++++ README.md | 73 ++++++++++++++++--- TRUTH.md | 85 ++++++++++++++++++++++ generators/advocacy_rights.py | 12 ++-- generators/field_survey_quality.py | 2 +- generators/gender_programme.py | 2 +- generators/girls_education.py | 48 +++++++++---- generators/governance.py | 2 +- generators/health_nutrition.py | 2 +- generators/humanitarian.py | 6 +- generators/media_development.py | 6 +- generators/public_health.py | 29 ++++++-- generators/rct_experiment.py | 75 +++++++++++++------ generators/social_protection.py | 2 +- generators/targeting.py | 10 +-- requirements-dev.txt | 2 + requirements.txt | 17 +++-- tests/conftest.py | 40 +++++++++++ tests/test_calibration.py | 71 ++++++++++++++++++ tests/test_generators_run.py | 43 +++++++++++ tests/test_no_degenerate_columns.py | 108 ++++++++++++++++++++++++++++ tests/test_rct_truth.py | 89 +++++++++++++++++++++++ 24 files changed, 801 insertions(+), 84 deletions(-) create mode 100644 CLAUDE.md create mode 100644 TRUTH.md create mode 100644 requirements-dev.txt create mode 100644 tests/conftest.py create mode 100644 tests/test_calibration.py create mode 100644 tests/test_generators_run.py create mode 100644 tests/test_no_degenerate_columns.py create mode 100644 tests/test_rct_truth.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b545537..e18108b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,20 +7,63 @@ on: branches: [main] jobs: - validate: + test: runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.11", "3.12"] steps: - uses: actions/checkout@v6 - - name: Set up Python + - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: - python-version: "3.11" + python-version: ${{ matrix.python-version }} + cache: pip - name: Install dependencies - run: | - pip install -r requirements.txt + run: pip install -r requirements-dev.txt - - name: List available generators + # This used to be the entire job. `--list` imports no generator, so all 36 + # modules were unexecuted by CI and a broken one would have gone unnoticed. + - name: Registry lists cleanly run: python generate.py --list + - name: Test suite + run: python -m pytest tests/ -q + + end_to_end: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-python@v6 + with: + python-version: "3.11" + cache: pip + - run: pip install -r requirements.txt + + # Writing the files is a separate failure surface from generating the + # frames: a column dtype that a frame holds happily can still fail to + # serialise, and parquet is stricter than csv about mixed types. + - name: Generate every dataset to CSV + run: python generate.py --rows 500 --output /tmp/out-csv + + - name: Generate every dataset to Parquet + run: python generate.py --rows 500 --format parquet --output /tmp/out-parquet + + - name: Every registered dataset produced a file + run: | + python - <<'PY' + import pathlib, sys + from generate import GENERATORS + missing = [] + for fmt, d in (("csv", "/tmp/out-csv"), ("parquet", "/tmp/out-parquet")): + for name in GENERATORS: + p = pathlib.Path(d) / f"{name}.{fmt}" + if not p.exists() or p.stat().st_size == 0: + missing.append(str(p)) + if missing: + sys.exit("missing or empty outputs:\n " + "\n ".join(missing)) + print(f"PASS - {len(GENERATORS)} datasets written in both formats") + PY diff --git a/.gitignore b/.gitignore index 34565b4..22cbabb 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ output/ *.parquet __pycache__/ *.pyc +.pytest_cache/ .venv/ .env *.egg-info/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..4bf7d58 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,105 @@ +# devdata-practice + +Synthetic datasets for practising development economics analysis. Thirty-six +generators, pure Python, no service dependencies, nothing fetched at runtime. + +## Commands + +```bash +pip install -r requirements-dev.txt +python generate.py --list # registry +python generate.py rct_experiment # one dataset to ./output +python generate.py --rows 500 # all of them, small +python -m pytest tests/ -q # ~2 min, mostly the degeneracy guard +``` + +## Layout + +`generate.py` holds `GENERATORS`, the registry mapping a dataset name to its +module, its size keyword and a description. `generators/` holds one module per +dataset, each exporting `generate(**kwargs) -> DataFrame`. `generators/utils.py` +holds the shared geography, missingness and id helpers. `TRUTH.md` is the answer +key. `tests/` is the guard. + +A module on disk that is not in `GENERATORS` is invisible to every user; one in +`GENERATORS` that is not on disk crashes. `tests/test_generators_run.py` compares +the two, so neither can happen quietly. + +## The bug this repository is shaped around + +`rng.binomial(1, 0.55)` **without a size argument returns a scalar**, which numpy +broadcasts across the whole column. Nothing raises, the row count is right, the +file writes, and the variable is a constant. It was present at nineteen call +sites across nine generators and produced fifteen constant columns, including +four asset variables in `targeting`, a proxy-means-test dataset whose asset +predictors did not vary. + +It appears in two shapes: + +```python +owns_radio = rng.binomial(1, 0.55) # every row: the same draw +months_displaced = np.where(displaced, rng.exponential(14), 0) # every displaced row: the same +``` + +The second shape is the dangerous one, because the column then holds *two* +values rather than one and so survives any "is this column constant" check. It +is why `tests/test_no_degenerate_columns.py` tests for perfect binary aliasing +as well as for constants. + +The discriminator when reading code: **is the probability a scalar or an array?** +`barrier_cost` in `girls_education` was always fine because its probability is +`0.35 - 0.10 * receives_scholarship`, an array, so numpy returned a vector. Its +neighbours on the surrounding lines took scalar probabilities and were broken. + +A scalar draw is correct when it is a *parameter* rather than a column: the +per-arm compliance rate in `rct_experiment`, the per-country intercept in +`panel_data`, anything inside a per-row loop. Do not add `size=n` to those. The +test to apply is whether the result becomes a column. + +## Watch out for + +- **Do not loosen a tolerance in `tests/` to make a failure go away.** The + numbers in `TRUTH.md` are what a learner checks their answer against. If a test + fails, either the generator changed or `TRUTH.md` is now lying. +- **The ITT in `rct_experiment` is not a fixed number.** Take-up is drawn + `U(0.65, 0.85)` per arm per run, so the ITT moved between +0.089 and +0.203 + across five seeds. Only the complier effect (Wald: ITT ÷ take-up) is a stable + target, and only against pure-control villages. +- **`spillover_risk` has a village level under it.** Villages are randomised + into treatment (70%) or pure control (30%); the flag marks control units inside + a treatment village. Before that existed, randomisation was stratified within + district, every district therefore contained treated units, and the flag was an + exact alias of the control dummy. If you flatten the design back to individual + randomisation, the flag becomes useless again. +- **PHQ-9 is a sum of nine 0–3 items, not a transformed latent.** The previous + `9 * logistic(...) * 3` construction was bounded and had no item-level variance: + the floor sat at 2 and `depression_severe` was identically zero in every draw. + If you touch the item severities, re-check the bands in `TRUTH.md`. +- **Requirements are pinned on purpose.** `pandas>=2.0` would pick up pandas 3, + where text columns carry a `str` dtype rather than `object`, so any check keyed + on `dtype == object` skips every text column without saying so. Raise the pins + deliberately and together, then re-run the suite. +- **Nothing here is real data.** Country names and GDP per capita in + `generators/utils.py` are plausible orders of magnitude, not current figures. + Do not cite them, and do not describe any generator as reproducing a survey. + +## Adding a generator + +1. Write `generators/.py` exporting `generate(n_x=..., seed=...) -> DataFrame`. +2. Register it in `GENERATORS` in `generate.py` with its size keyword and a + one-line description. +3. Run `python -m pytest tests/ -q`. The degeneracy guard runs against every + registered generator automatically, so you do not add a test for it. +4. If the generator encodes a parameter someone is meant to recover, put it in + `TRUTH.md` and assert it in `tests/`. + +## Testing + +`.github/workflows/ci.yml` runs the suite on 3.11 and 3.12, plus an end-to-end +job that writes every dataset to CSV and Parquet and checks the files are +non-empty. Serialisation is a separate failure surface from generation: parquet +is stricter than csv about mixed types, so a frame that builds happily can still +fail to write. + +CI used to run `python generate.py --list` and nothing else. That imports no +generator, so all thirty-six modules were unexecuted by any automated check. diff --git a/README.md b/README.md index 90a0bd6..9264172 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,15 @@ [![Website](https://img.shields.io/badge/Docs-varnasr.github.io%2Fdevdata--practice-blue)](https://varnasr.github.io/devdata-practice/) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) -[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue)](https://www.python.org/) +[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-blue)](https://www.python.org/) [![GitHub Last Commit](https://img.shields.io/github/last-commit/Varnasr/devdata-practice)](https://github.com/Varnasr/devdata-practice/commits/main) [![Part of ImpactMojo](https://img.shields.io/badge/Part%20of-ImpactMojo-orange)](https://www.impactmojo.in) **Realistic, large-scale practice datasets for development economics — 36 generators, 840,000+ rows.** -Built for researchers, students, and practitioners who need real-feeling data modelled on DHS, NFHS, ASER, and other major development survey frameworks. +Built for researchers, students and practitioners who need data shaped like a +real survey: the same variables, the same awkward missingness, the same +structure, without waiting on a data request. **Full documentation:** [varnasr.github.io/devdata-practice](https://varnasr.github.io/devdata-practice/) @@ -16,7 +18,17 @@ Built for researchers, students, and practitioners who need real-feeling data mo ## About -DevData Practice generates synthetic datasets that closely mirror the structure, variable distributions, and statistical properties of real development sector surveys. The data is designed for: +DevData Practice generates synthetic datasets built to the *shape* of development +sector surveys: the variables a DHS or LSMS instrument collects, laid out the way +it lays them out, with realistic missingness, partial compliance and attrition. + +It does not reproduce any survey's distributions, and the figures it produces are +not estimates of anything. A stunting rate here is a number the code was told to +produce, not a measurement. If you need to check a pipeline against published +figures, use a real recode and a published table. `TRUTH.md` says exactly which +parameters are in the data and which estimand recovers each one. + +The data is designed for: - **Learning** — practice data analysis, MEL, and econometrics without needing access to restricted datasets - **Teaching** — ready-made datasets for classroom exercises, workshops, and tutorials @@ -47,7 +59,8 @@ python generate.py --list python generate.py rct_experiment labor_market household_survey ``` -Generated files are saved to the `data/` directory as CSV files. +Generated files are saved to `./output` as CSV files. Use `--output` to change +the directory and `--format parquet` to change the format. --- @@ -107,13 +120,55 @@ devdata-practice/ ## Requirements ``` -pandas>=1.5.0 -numpy>=1.23.0 -scipy>=1.9.0 -faker>=15.0.0 +numpy==2.4.6 +pandas==3.0.6 +scipy==1.17.1 +pyarrow==25.0.1 ``` -Python 3.9 or higher. +Python 3.11 or higher. Versions are pinned exactly, not floored: see `CLAUDE.md` +for why. `requirements-dev.txt` adds pytest. + +An earlier version of this section listed `faker>=15.0.0`. Nothing in the +repository imports it. + +--- + +## Checking your answer + +Every generator encodes parameters on purpose. `TRUTH.md` records what they are +and which estimand recovers each one, so an exercise can be marked rather than +guessed at. + +The one worth reading before you use `rct_experiment`: its intention-to-treat +effect is **not** a fixed number. Take-up is drawn `U(0.65, 0.85)` for each arm on +each run, so the ITT moved between +0.089 and +0.203 log points across five seeds. +Only the complier effect is stable, and only when computed against the +pure-control villages (`spillover_risk == 0`). Controls inside a treatment village +receive a +3% spillover; using them as the comparison biases every ITT toward +zero by about 0.011 log points, which is the lesson the design exists to teach. + +## Testing + +```bash +pip install -r requirements-dev.txt +python -m pytest tests/ -q # 233 tests, about two minutes +``` + +CI runs the suite on Python 3.11 and 3.12, then writes every dataset to CSV and +Parquet and checks the files are non-empty. + +The largest part of the suite is a guard against degenerate columns, and it exists +because of a specific defect. `rng.binomial(1, 0.55)` without a size argument +returns a *scalar*, which numpy broadcasts across the whole column: nothing +raises, the row count is right, the file writes, and the variable is a constant. +Nineteen call sites across nine generators were affected. Among the fifteen +constant columns were four asset variables in `targeting`, a proxy-means-test +dataset whose asset predictors did not vary, and the three dropout barriers in +`girls_education`, all identically zero. + +It was invisible because CI ran `python generate.py --list`, which imports no +generator at all. Thirty-six modules were unexecuted by any automated check. --- diff --git a/TRUTH.md b/TRUTH.md new file mode 100644 index 0000000..0afbc66 --- /dev/null +++ b/TRUTH.md @@ -0,0 +1,85 @@ +# Ground truth + +A practice dataset you cannot check your answer against is a worked example with +the answer torn off. This file records the parameters the generators actually +use, which estimand recovers each one, and how close you should expect to get. +Everything here is asserted by `tests/`, so it cannot quietly drift away from the +code. + +Two warnings before the numbers. + +**Not every true parameter is a stable target.** In `rct_experiment` the take-up +rate is drawn `U(0.65, 0.85)` afresh for each arm on each run. The intention-to-treat +effect is therefore a different number for every seed. It ranged from +0.089 to ++0.203 log points across five seeds at n = 60,000. Only the complier effect is +stable. If you are checking your work against a fixed number, check the LATE. + +**Recovering a parameter is not the same as the parameter being there.** Several +of these are recoverable only under the identifying assumption the exercise is +meant to teach. The spillover figure below is recoverable because the design has +pure-control villages; drop them and it is not identified at all. + +## rct_experiment + +Outcome is `log(endline_consumption_usd) - log(baseline_consumption_usd)`. + +| Quantity | True value | How to recover it | +|---|---|---| +| Complier effect, cash_transfer | `log(1.15) + h` = **+0.1893** | Wald: ITT ÷ take-up, against pure controls | +| Complier effect, cash_plus_training | `log(1.22) + h` = **+0.2484** | same | +| Complier effect, training_only | `log(1.08) + h` = **+0.1265** | same | +| Spillover on untreated neighbours | `log(1.03)` = **+0.0296** | exposed controls minus pure controls | +| Common time trend | `log(1.03)` = +0.0296 | absorbed by any control group; not separately identified | + +`h = log(1 + 0.04 × P(female) + 0.06 × P(below-median baseline)) = +0.0496` is the +average heterogeneity loading among compliers. It is part of the complier effect +because the heterogeneity terms multiply only for units with `actually_treated == 1`. +If you compare against `log(1.15) = +0.1398` alone you will appear to over-recover +by about five log points, and the gap is this term, not an error. + +Observed across seeds 1, 42, 123, 777 and 2026 at n = 60,000: Wald means of ++0.1873, +0.2480 and +0.1248, all within 0.002 of theory. `tests/test_rct_truth.py` +asserts a tolerance of 0.01. + +**The control group you choose changes the answer.** Villages are randomised into +treatment (70%) or pure control (30%), then individuals within treatment villages +are randomised across arms. Control units inside a treatment village receive the ++3% spillover. Using all controls as the comparison therefore biases every ITT +toward zero by roughly 0.011 log points. Using `spillover_risk == 0` controls only +is the clean contrast. + +**`spillover_risk` used to be unusable.** Before this was fixed it equalled 1 for +every control and 0 for every treated unit, an exact alias of the treatment dummy, +because randomisation was stratified within district so every district always +contained treated units. Any regression including both dropped a collinear term. +`tests/test_no_degenerate_columns.py` now fails if any column collapses that way. + +## Calibration targets + +These are design targets for a synthetic teaching dataset, not estimates from a +named survey. They are asserted in `tests/test_calibration.py` so a future change +to a generator cannot silently move them. + +| Generator | Quantity | Target | +|---|---|---| +| `public_health` | PHQ-9 ≥ 10 (moderate or worse) | 12–18% | +| `public_health` | PHQ-9 ≥ 20 (severe) | 0.8–2.5% | +| `public_health` | PHQ-9 in 0–4 (minimal) | 45–65% | +| `public_health` | PHQ-9 support | reaches both 0 and the upper 20s | +| `girls_education` | mean distance to school, rural | 3.5–4.5 km | +| `girls_education` | mean distance to school, urban | 1.2–1.8 km | +| `rct_experiment` | take-up in each treatment arm | 0.60–0.90 | +| `targeting` | asset ownership (radio, mobile, bicycle) | matches its stated rate ± 0.05 | + +PHQ-9 is built the way the instrument is: nine items each scored 0–3 and summed. +The previous version mapped a single latent normal through `9 × logistic(...) × 3`, +which is bounded and has no item-level variance, so the floor sat at 2 and +`depression_severe` was identically zero in every draw. + +## What is deliberately *not* true here + +- No generator reproduces a real survey. Country names and GDP per capita in + `generators/utils.py` are plausible orders of magnitude, not current figures, + and should not be cited. +- Nothing here is a benchmark. If you need to check a pipeline against published + numbers, use a real recode with a published table. diff --git a/generators/advocacy_rights.py b/generators/advocacy_rights.py index e7ebcfc..f16840e 100644 --- a/generators/advocacy_rights.py +++ b/generators/advocacy_rights.py @@ -42,7 +42,7 @@ def generate(n_individuals: int = 15000, seed: int = 708) -> pd.DataFrame: # Land tenure owns_land = rng.binomial(1, _logistic(0.35, wealth - 0.1 * female, 0.3)) has_land_title = np.where(owns_land, rng.binomial(1, _logistic(0.25, wealth + 0.1 * urban.astype(float), 0.4)), 0) - land_dispute_experienced = np.where(owns_land, rng.binomial(1, 0.15), 0) + land_dispute_experienced = np.where(owns_land, rng.binomial(1, 0.15, n), 0) # Programme participation received_legal_aid = rng.binomial(1, 0.18, n) @@ -77,17 +77,17 @@ def generate(n_individuals: int = 15000, seed: int = 708) -> pd.DataFrame: dispute_resolved = np.where(sought_resolution.astype(bool), rng.binomial(1, 0.55 + 0.10 * received_legal_aid), 0) satisfied_with_outcome = np.where(dispute_resolved.astype(bool), - rng.binomial(1, 0.60), 0) + rng.binomial(1, 0.60, n), 0) # Access to justice barriers barrier_cost = np.where(experienced_dispute & ~sought_resolution.astype(bool), - rng.binomial(1, 0.40), 0) + rng.binomial(1, 0.40, n), 0) barrier_distance = np.where(experienced_dispute & ~sought_resolution.astype(bool), - rng.binomial(1, 0.25), 0) + rng.binomial(1, 0.25, n), 0) barrier_fear = np.where(experienced_dispute & ~sought_resolution.astype(bool), - rng.binomial(1, 0.30), 0) + rng.binomial(1, 0.30, n), 0) barrier_distrust = np.where(experienced_dispute & ~sought_resolution.astype(bool), - rng.binomial(1, 0.20), 0) + rng.binomial(1, 0.20, n), 0) # Civic participation voted_last_election = rng.binomial(1, _logistic(0.55, 0.05 * (age > 18).astype(float) + 0.1 * educ_years / 10, 0.3)) diff --git a/generators/field_survey_quality.py b/generators/field_survey_quality.py index 273a436..4eddc4e 100644 --- a/generators/field_survey_quality.py +++ b/generators/field_survey_quality.py @@ -178,7 +178,7 @@ def generate(n_surveys: int = 20000, seed: int = 814) -> pd.DataFrame: # --- Supervisor checks --- field_spot_checked = rng.binomial(1, 0.10, n) audio_recorded = rng.binomial(1, 0.25, n) - audio_reviewed = np.where(audio_recorded, rng.binomial(1, 0.40), 0) + audio_reviewed = np.where(audio_recorded, rng.binomial(1, 0.40, n), 0) # --- Data quality metrics --- missing_rate = np.where( diff --git a/generators/gender_programme.py b/generators/gender_programme.py index afa8a75..1198186 100644 --- a/generators/gender_programme.py +++ b/generators/gender_programme.py @@ -99,7 +99,7 @@ def generate(n_individuals: int = 25000, seed: int = 701) -> pd.DataFrame: ) unmet_need_fp = np.where( married & (age >= 15) & (age <= 49) & ~using_contraception.astype(bool), - rng.binomial(1, 0.22), 0 + rng.binomial(1, 0.22, n), 0 ) # --- Empowerment composite (WEAI-like, 0-1) --- diff --git a/generators/girls_education.py b/generators/girls_education.py index 8f8c818..e815150 100644 --- a/generators/girls_education.py +++ b/generators/girls_education.py @@ -49,13 +49,14 @@ def generate(n_girls: int = 20000, seed: int = 702) -> pd.DataFrame: receives_school_meals = rng.binomial(1, 0.35, n) in_safe_spaces_programme = rng.binomial(1, 0.20, n) - # Distance to school - distance_km = np.clip(rng.exponential(2 if True else 5, n) + (0 if True else 3), 0.1, 15) - for i in range(n): - if urban[i]: - distance_km[i] = np.clip(rng.exponential(1.5), 0.1, 8) - else: - distance_km[i] = np.clip(rng.exponential(4), 0.3, 15) + # Distance to school. Rural girls travel further. + # Previously this drew a full vector through a dead `2 if True else 5` + # conditional, discarded it, then overwrote every element in a Python loop. + distance_km = np.where( + urban.astype(bool), + np.clip(rng.exponential(1.5, n), 0.1, 8), + np.clip(rng.exponential(4.0, n), 0.3, 15), + ) # Safety feels_safe_route = rng.binomial(1, _logistic(0.60, ses + 0.3 * urban.astype(float) - 0.02 * distance_km, 0.3)) @@ -70,17 +71,32 @@ def generate(n_girls: int = 20000, seed: int = 702) -> pd.DataFrame: rng.binomial(1, _logistic(0.40, ses + 0.15 * receives_scholarship, 0.4)), 0) missed_school_menstruation = np.where( has_menstruated & ~has_sanitary_products.astype(bool), - rng.binomial(1, 0.55), 0 + rng.binomial(1, 0.55, n), 0 ) # Enrollment & attendance - enrolled = rng.binomial(1, _logistic( - 0.85, + # + # Schooling status is three-state, not two. A girl who never started school is + # not a dropout, and the policy response to the two is different: the first is + # an access problem, the second a retention problem. Modelling `dropped_out` as + # the complement of `enrolled` collapses them, and also made the two columns + # perfect aliases, so any model including both silently dropped a term. + ever_enrolled = rng.binomial(1, _logistic( + 0.94, + ses + 0.20 * parent_values_girls_edu - 0.04 * distance_km + + 0.10 * urban.astype(float), + 0.5 + )) + # Retention among girls who did start. + stays_enrolled = rng.binomial(1, _logistic( + 0.88, ses + 0.15 * receives_scholarship + 0.10 * parent_values_girls_edu - 0.03 * distance_km - 0.15 * (age >= 14).astype(float) + 0.08 * urban.astype(float), 0.5 )) + enrolled = (ever_enrolled.astype(bool) & stays_enrolled.astype(bool)).astype(int) + never_enrolled = (1 - ever_enrolled).astype(int) attendance_rate = np.where(enrolled, np.clip( rng.beta(6, 1.5, n) @@ -103,13 +119,14 @@ def generate(n_girls: int = 20000, seed: int = 702) -> pd.DataFrame: + 4 * in_safe_spaces_programme + rng.normal(0, 10, n), 0, 100 ).astype(int), np.nan) - # Dropout & barriers - dropped_out = (~enrolled.astype(bool)).astype(int) - barrier_marriage = np.where(dropped_out & (age >= 13), rng.binomial(1, 0.25), 0) - barrier_pregnancy = np.where(dropped_out & (age >= 14), rng.binomial(1, 0.12), 0) + # Dropout & barriers. A dropout started school and left; a never-enrolled girl + # is neither enrolled nor a dropout, so the two columns are no longer aliases. + dropped_out = (ever_enrolled.astype(bool) & ~enrolled.astype(bool)).astype(int) + barrier_marriage = np.where(dropped_out & (age >= 13), rng.binomial(1, 0.25, n), 0) + barrier_pregnancy = np.where(dropped_out & (age >= 14), rng.binomial(1, 0.12, n), 0) barrier_cost = np.where(dropped_out, rng.binomial(1, 0.35 - 0.10 * receives_scholarship), 0) barrier_distance = np.where(dropped_out, rng.binomial(1, np.clip(0.05 * distance_km, 0, 0.5)), 0) - barrier_household_chores = np.where(dropped_out, rng.binomial(1, 0.20), 0) + barrier_household_chores = np.where(dropped_out, rng.binomial(1, 0.20, n), 0) # Transition (primary grade 6 → secondary grade 7) at_transition = (grade == 6).astype(int) @@ -141,6 +158,7 @@ def generate(n_girls: int = 20000, seed: int = 702) -> pd.DataFrame: "has_sanitary_products": has_sanitary_products, "missed_school_menstruation": missed_school_menstruation, "enrolled": enrolled, + "never_enrolled": never_enrolled, "attendance_rate": np.round(attendance_rate, 3), "math_score": math_score, "literacy_score": literacy_score, diff --git a/generators/governance.py b/generators/governance.py index 4d6bcd7..cc45801 100644 --- a/generators/governance.py +++ b/generators/governance.py @@ -89,7 +89,7 @@ def generate(n_citizens: int = 15000, seed: int = 711) -> pd.DataFrame: # --- Information access --- knows_rti = rng.binomial(1, _logistic(0.15, 0.15 * educ_years / 10 + 0.1 * urban.astype(float), 0.4)) - used_rti = np.where(knows_rti, rng.binomial(1, 0.12), 0) + used_rti = np.where(knows_rti, rng.binomial(1, 0.12, n), 0) gets_info_radio = rng.binomial(1, 0.55, n) gets_info_social_media = rng.binomial(1, _logistic(0.30, 0.2 * urban.astype(float) + 0.1 * (age < 35).astype(float), 0.4)) gets_info_community_meeting = rng.binomial(1, 0.30 - 0.10 * urban.astype(float)) diff --git a/generators/health_nutrition.py b/generators/health_nutrition.py index e7bb6b5..68f4327 100644 --- a/generators/health_nutrition.py +++ b/generators/health_nutrition.py @@ -57,7 +57,7 @@ def generate(n_children: int = 35000, seed: int = 789) -> pd.DataFrame: rng.normal(3.1 + 0.1 * (wealth_quintile - 1), 0.45, n), 1.0, 5.5 ) low_birth_weight = (birth_weight_kg < 2.5).astype(int) - birth_order = np.clip(rng.poisson(2), 1, 10) + birth_order = np.clip(rng.poisson(2, n), 1, 10) # --- Anthropometrics (WHO z-scores) --- # Height-for-age (stunting if < -2) diff --git a/generators/humanitarian.py b/generators/humanitarian.py index b74f99a..f533498 100644 --- a/generators/humanitarian.py +++ b/generators/humanitarian.py @@ -88,12 +88,12 @@ def generate(n_individuals: int = 18000, seed: int = 712) -> pd.DataFrame: months_displaced = np.where( displaced, - np.clip(rng.exponential(14), 0.5, 72).round(0).astype(int), + np.clip(rng.exponential(14, n), 0.5, 72).round(0).astype(int), 0, ) times_displaced = np.where( displaced, - np.clip(rng.poisson(1.5), 1, 6), + np.clip(rng.poisson(1.5, n), 1, 6), 0, ) @@ -363,7 +363,7 @@ def generate(n_individuals: int = 18000, seed: int = 712) -> pd.DataFrame: ) complaint_resolution_days = np.where( complaint_resolved, - np.clip(rng.exponential(12), 1, 60).astype(int), + np.clip(rng.exponential(12, n), 1, 60).astype(int), 0, ) diff --git a/generators/media_development.py b/generators/media_development.py index f529639..d9a93e2 100644 --- a/generators/media_development.py +++ b/generators/media_development.py @@ -115,7 +115,7 @@ def generate(n_individuals: int = 18000, seed: int = 812) -> pd.DataFrame: ) found_dev_content_useful = np.where( exposed_to_dev_content, - rng.binomial(1, 0.65), + rng.binomial(1, 0.65, n), 0 ) @@ -132,7 +132,7 @@ def generate(n_individuals: int = 18000, seed: int = 812) -> pd.DataFrame: )) corrected_by_others = np.where( shared_misinformation, - rng.binomial(1, 0.30), + rng.binomial(1, 0.30, n), 0 ) @@ -154,7 +154,7 @@ def generate(n_individuals: int = 18000, seed: int = 812) -> pd.DataFrame: in_media_literacy_programme = rng.binomial(1, 0.12, n) programme_improved_skills = np.where( in_media_literacy_programme, - rng.binomial(1, 0.70), + rng.binomial(1, 0.70, n), 0 ) diff --git a/generators/public_health.py b/generators/public_health.py index 988956b..d7173cd 100644 --- a/generators/public_health.py +++ b/generators/public_health.py @@ -158,15 +158,30 @@ def generate(n_individuals: int = 20000, seed: int = 501) -> pd.DataFrame: # ------------------------------------------------------------------ # # Mental health — PHQ-9-like score (0-27) # ------------------------------------------------------------------ # - # 9 items scored 0-3, correlated with poverty, female, shocks + # Built the way the instrument is: nine items each scored 0-3, summed to 0-27. + # The previous version mapped one latent normal through 9 * logistic(...) * 3, + # a bounded transform with no item-level variance. It could not reach either end + # of the scale: the floor sat at 2 and `depression_severe` (>= 20) was identically + # zero in every draw, so the severe-screening column carried no information at all. + # + # Item severities are spaced so a person on the latent mean endorses the common + # items (low mood, low energy) and not the rare ones (psychomotor change, + # self-harm ideation), which is what produces the strong floor real PHQ-9 data has. + # These are design targets for a synthetic teaching dataset, not estimates from a + # named survey: ~14.5% at or above the >= 10 moderate threshold and ~1.5% at or + # above the >= 20 severe threshold, within the broad range general-population + # screening reports. Verified by tests/test_calibration.py. phq9_latent = ( - -0.3 - - 0.25 * (wealth_quintile - 3) / 2 - + 0.3 * female - + 0.01 * np.maximum(age - 50, 0) - + rng.normal(0, 0.8, n) + -0.35 * (wealth_quintile - 3) / 2 + + 0.45 * female + + 0.015 * np.maximum(age - 50, 0) + + rng.normal(0, 1.45, n) ) - phq9_score = np.clip(np.round(9 * _logistic(0.33, phq9_latent, 0.6) * 3), 0, 27).astype(int) + # Item severity thresholds, easiest (most endorsed) first. + item_severity = np.array([1.5, 1.7, 1.9, 2.1, 2.4, 2.7, 3.0, 3.4, 4.2]) + item_p = 1.0 / (1.0 + np.exp(-(phq9_latent[:, None] - item_severity[None, :]))) + phq9_items = rng.binomial(3, item_p) # n x 9, each item 0-3 + phq9_score = phq9_items.sum(axis=1).astype(int) depression_moderate = (phq9_score >= 10).astype(int) depression_severe = (phq9_score >= 20).astype(int) diff --git a/generators/rct_experiment.py b/generators/rct_experiment.py index 5db0cfa..2022e4c 100644 --- a/generators/rct_experiment.py +++ b/generators/rct_experiment.py @@ -5,7 +5,9 @@ (e.g., cash transfer, school feeding, deworming). Realistic features: - • Stratified randomization by district and gender + • Two-level design: villages are randomised into treatment or pure-control, + then individuals within treatment villages are stratified-randomised by arm + • Stratified randomization by village and gender • Baseline and endline observations • Partial compliance (take-up < 100%) • Attrition correlated with treatment arm and baseline characteristics @@ -43,19 +45,38 @@ def generate(n_individuals: int = 25000, seed: int = 123) -> pd.DataFrame: 0, 27 ) - # --- Treatment assignment (stratified by district × gender) --- + # --- Villages --- + # Individuals are clustered in villages within districts. This level exists so + # spillover exposure can vary: without it every district contained treated units + # by construction, and the spillover flag below was an exact alias of the control + # dummy (see tests/test_no_degenerate_columns.py, which now pins that down). + villages_per_district = 8 + village_idx = rng.integers(0, villages_per_district, n) + village_id = np.array([f"{d}-V{v}" for d, v in zip(districts, village_idx)]) + + # --- Treatment assignment (two-level) --- + # 70% of villages are treatment villages; the remaining 30% are pure controls, + # where nobody is offered anything. Within a treatment village, individuals are + # stratified-randomised across the four arms by gender. arms = ["control", "cash_transfer", "cash_plus_training", "training_only"] - treatment = np.empty(n, dtype=" pd.DataFrame: het_female = 0.04 * female * actually_treated het_poor = 0.06 * (baseline_consumption < np.median(baseline_consumption)).astype(float) * actually_treated + # --- Spillover onto untreated neighbours --- + # Cash landing in a village lifts local demand, so untreated households in a + # treatment village gain a little even though they were offered nothing. The + # true spillover is +3% on consumption; see TRUTH.md. It is why an ITT computed + # against *all* controls is biased toward zero, and why the clean comparison is + # against pure-control villages only. + spillover_multiplier = np.where(exposed_control, 1.03, 1.0) + # --- Endline outcome --- time_trend = 1.03 # 3% general improvement noise = np.exp(rng.normal(0, 0.15, n)) endline_consumption = np.round( - baseline_consumption * time_trend * te_multiplier * (1 + het_female + het_poor) * noise, 2 + baseline_consumption * time_trend * te_multiplier * spillover_multiplier + * (1 + het_female + het_poor) * noise, 2 ) # Endline food insecurity (should improve with treatment) @@ -99,19 +129,21 @@ def generate(n_individuals: int = 25000, seed: int = 123) -> pd.DataFrame: attrition_prob -= 0.01 * (educ_years / 18) attrited = rng.binomial(1, np.clip(attrition_prob, 0.02, 0.25), n).astype(bool) - # --- Spillover flag (10% of control units in treated villages) --- - spillover_risk = np.zeros(n, dtype=int) - for d in np.unique(districts): - d_mask = districts == d - has_treated = (treatment[d_mask] != "control").any() - if has_treated: - ctrl = d_mask & (treatment == "control") - spillover_risk[ctrl] = 1 + # --- Spillover exposure --- + # Share of each village actually offered treatment, and the exposure flag for + # control units sitting inside a treatment village. Controls in pure-control + # villages score 0 and are the clean comparison group; controls in treatment + # villages are the ones a spillover analysis has to worry about. + assigned = (treatment != "control").astype(float) + vs = pd.Series(assigned).groupby(pd.Series(village_id)).transform("mean") + village_treated_share = np.round(vs.to_numpy(), 4) + spillover_risk = exposed_control.astype(int) # Build DataFrame df = pd.DataFrame({ "participant_id": ids, "district": districts, + "village_id": village_id, "urban": urban.astype(int), "female": female, "age": age, @@ -124,6 +156,7 @@ def generate(n_individuals: int = 25000, seed: int = 123) -> pd.DataFrame: "endline_consumption_usd": np.where(attrited, np.nan, endline_consumption), "endline_food_insecurity": np.where(attrited, np.nan, endline_food_insecurity).astype(float), "attrited": attrited.astype(int), + "village_treated_share": village_treated_share, "spillover_risk": spillover_risk, }) diff --git a/generators/social_protection.py b/generators/social_protection.py index 0f313e4..fea3f41 100644 --- a/generators/social_protection.py +++ b/generators/social_protection.py @@ -72,7 +72,7 @@ def generate(n_households: int = 20000, seed: int = 710) -> pd.DataFrame: has_conditionality = np.isin(programme_type, ["conditional_cash", "cash_plus", "school_feeding"]) cond_health_visits = np.where(has_conditionality, rng.binomial(1, 0.72 + 0.05 * head_educ / 18), 0) cond_school_attendance = np.where(has_conditionality & (n_children > 0), - rng.binomial(1, 0.78), 0) + rng.binomial(1, 0.78, n), 0) compliant = np.where(has_conditionality, cond_health_visits | cond_school_attendance, 1) diff --git a/generators/targeting.py b/generators/targeting.py index 1ee55e3..cde2397 100644 --- a/generators/targeting.py +++ b/generators/targeting.py @@ -32,7 +32,7 @@ def generate(n_households: int = 20000, seed: int = 505) -> pd.DataFrame: hh_size = rng.choice(range(1, 11), n, p=[0.03, 0.06, 0.10, 0.16, 0.20, 0.18, 0.13, 0.08, 0.04, 0.02]) n_children = np.clip(rng.poisson(hh_size * 0.35), 0, hh_size - 1) - n_elderly = np.clip(rng.poisson(0.3), 0, min(3, max(0, hh_size.max()))) + n_elderly = np.clip(rng.poisson(0.3, n), 0, min(3, max(0, hh_size.max()))) n_elderly = np.minimum(n_elderly, hh_size - n_children) head_female = rng.binomial(1, 0.28, n) head_age = rng.integers(20, 75, n) @@ -43,16 +43,16 @@ def generate(n_households: int = 20000, seed: int = 505) -> pd.DataFrame: # Housing wall_permanent = rng.binomial(1, 0.35 + 0.15 * urban.astype(float)) roof_permanent = rng.binomial(1, 0.40 + 0.15 * urban.astype(float)) - rooms = np.clip(rng.poisson(2), 1, 8) + rooms = np.clip(rng.poisson(2, n), 1, 8) has_electricity = rng.binomial(1, 0.30 + 0.30 * urban.astype(float)) has_piped_water = rng.binomial(1, 0.20 + 0.25 * urban.astype(float)) has_flush_toilet = rng.binomial(1, 0.15 + 0.20 * urban.astype(float)) # Assets - owns_radio = rng.binomial(1, 0.55) + owns_radio = rng.binomial(1, 0.55, n) owns_tv = rng.binomial(1, 0.25 + 0.15 * urban.astype(float)) - owns_mobile = rng.binomial(1, 0.70) - owns_bicycle = rng.binomial(1, 0.35) + owns_mobile = rng.binomial(1, 0.70, n) + owns_bicycle = rng.binomial(1, 0.35, n) owns_motorcycle = rng.binomial(1, 0.10 + 0.05 * urban.astype(float)) owns_land = rng.binomial(1, 0.55 - 0.20 * urban.astype(float)) land_acres = np.where(owns_land, np.clip(rng.lognormal(0.5, 0.8, n), 0.1, 20), 0) diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..44dca33 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,2 @@ +-r requirements.txt +pytest==8.4.2 diff --git a/requirements.txt b/requirements.txt index 450e12c..b00e7d1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,13 @@ -numpy>=1.24 -pandas>=2.0 -scipy>=1.10 -pyarrow>=12.0 +# Pinned deliberately. The maintenance policy asks that a clone still run years +# from now, and an unpinned set on a repository nobody watches is a CI failure +# waiting for a quiet week. These versions were resolved together and verified on +# Python 3.11 with the full suite green. +# +# Raise them deliberately and together, then re-run `pytest tests/`. Do not let a +# resolver do it silently: `pandas>=2.0` was how this repository would have picked +# up pandas 3, where text columns get a `str` dtype rather than `object` and any +# check keyed on `dtype == object` skips every text column without saying so. +numpy==2.4.6 +pandas==3.0.6 +scipy==1.17.1 +pyarrow==25.0.1 diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..1328897 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,40 @@ +"""Shared fixtures. Tests import the package from the repository root.""" +import sys +from functools import lru_cache +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +import pytest # noqa: E402 +from generate import GENERATORS # noqa: E402 + +# Small enough that the whole suite runs in CI in well under a minute, large +# enough that a prevalence assertion is not dominated by sampling noise. +SMOKE_N = 2000 + + +@lru_cache(maxsize=None) +def _build_cached(name, n, seed): + import importlib + info = GENERATORS[name] + kwargs = {"seed": seed} + if info["size_param"]: + kwargs[info["size_param"]] = n + return importlib.import_module(info["module"]).generate(**kwargs) + + +def build(name, n=SMOKE_N, seed=11): + """Generate one dataset by registry name. + + Cached: the suite asks for the same (name, n, seed) from several modules, and + regenerating 40,000 rows each time made the run take minutes. Callers must not + mutate what they get back: copy first if you need to. + """ + return _build_cached(name, n, seed) + + +@pytest.fixture(scope="session") +def generator_names(): + return sorted(GENERATORS) diff --git a/tests/test_calibration.py b/tests/test_calibration.py new file mode 100644 index 0000000..e090aa8 --- /dev/null +++ b/tests/test_calibration.py @@ -0,0 +1,71 @@ +"""Distributional targets from TRUTH.md. + +These are design targets for a synthetic teaching dataset, not estimates from any +named survey. They exist so that a future edit cannot silently move a prevalence +into a range that would teach the wrong thing. +""" +import pytest + +from conftest import build + + +@pytest.fixture(scope="module") +def ph(): + return build("public_health", n=30_000, seed=4) + + +def test_phq9_uses_the_full_scale(ph): + """The old construction could not reach either end: the floor sat at 2.""" + s = ph.phq9_score + assert s.min() == 0, f"nobody scores 0 (min={s.min()}): the floor is wrong" + assert s.max() >= 20, f"nobody reaches the severe range (max={s.max()})" + assert s.max() <= 27, "PHQ-9 is bounded at 27" + + +def test_phq9_prevalence_bands(ph): + s = ph.phq9_score + minimal = (s < 5).mean() + moderate = (s >= 10).mean() + severe = (s >= 20).mean() + assert 0.45 <= minimal <= 0.65, f"minimal band {minimal:.3f} outside 0.45-0.65" + assert 0.12 <= moderate <= 0.18, f"moderate {moderate:.3f} outside 0.12-0.18" + assert 0.008 <= severe <= 0.025, f"severe {severe:.3f} outside 0.008-0.025" + + +def test_depression_severe_carries_information(ph): + """It was identically zero in every draw before the generator was rebuilt.""" + assert ph.depression_severe.nunique() == 2 + + +def test_phq9_gradients_run_the_right_way(ph): + by_q = ph.groupby("wealth_quintile").depression_moderate.mean() + assert by_q.loc[1] > by_q.loc[5], "poorer quintiles should screen higher, not lower" + by_sex = ph.groupby("female").depression_severe.mean() + assert by_sex.loc[1] > by_sex.loc[0], "expected a higher rate among women" + + +def test_girls_school_distance_splits_urban_and_rural(): + d = build("girls_education", n=20_000, seed=3) + rural = d[d.urban == 0].distance_to_school_km.mean() + urban = d[d.urban == 1].distance_to_school_km.mean() + assert 3.5 <= rural <= 4.5, f"rural mean {rural:.2f} km outside 3.5-4.5" + assert 1.2 <= urban <= 1.8, f"urban mean {urban:.2f} km outside 1.2-1.8" + assert rural > urban + + +@pytest.mark.parametrize("col,rate", [("owns_radio", 0.55), ("owns_mobile", 0.70), + ("owns_bicycle", 0.35)]) +def test_targeting_assets_match_their_stated_rates(col, rate): + """All three were constants before the size-less draws were fixed.""" + d = build("targeting", n=20_000, seed=3) + got = d[col].mean() + assert abs(got - rate) < 0.05, f"{col} mean {got:.3f} vs stated {rate}" + + +def test_rct_takeup_is_partial_in_every_arm(): + d = build("rct_experiment", n=20_000, seed=3) + for arm, v in d.groupby("treatment_arm").actually_treated.mean().items(): + if arm == "control": + assert v == 0, "control arm must have zero take-up by definition" + else: + assert 0.60 <= v <= 0.90, f"{arm} take-up {v:.3f} outside 0.60-0.90" diff --git a/tests/test_generators_run.py b/tests/test_generators_run.py new file mode 100644 index 0000000..81d98e4 --- /dev/null +++ b/tests/test_generators_run.py @@ -0,0 +1,43 @@ +"""Every registered generator must run, and the registry must match the disk. + +CI previously ran only `generate.py --list`, which imports no generator at all. +Thirty-six modules were therefore never executed by any automated check. +""" +import pathlib + +import pytest + +from conftest import build +from generate import GENERATORS + + +def test_registry_matches_disk(): + """A module on disk but not in GENERATORS is invisible; the reverse crashes.""" + disk = {p.stem for p in (pathlib.Path(__file__).parent.parent / "generators").glob("*.py")} + disk -= {"__init__", "utils"} + assert disk == set(GENERATORS), ( + f"on disk but unregistered: {sorted(disk - set(GENERATORS))}; " + f"registered but missing: {sorted(set(GENERATORS) - disk)}" + ) + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_generator_runs_and_is_non_trivial(name): + df = build(name) + assert len(df) > 0, f"{name} produced no rows" + assert df.shape[1] >= 5, f"{name} produced only {df.shape[1]} columns" + assert not df.columns.duplicated().any(), f"{name} has duplicate column names" + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_same_seed_same_data(name): + """Reproducibility is the whole point of shipping a seed.""" + assert build(name, seed=5).equals(build(name, seed=5)), f"{name} is not reproducible" + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_different_seed_different_data(name): + """A seed that changes nothing means the seed is not wired through.""" + assert not build(name, seed=5).equals(build(name, seed=6)), ( + f"{name} ignores its seed argument" + ) diff --git a/tests/test_no_degenerate_columns.py b/tests/test_no_degenerate_columns.py new file mode 100644 index 0000000..a859d65 --- /dev/null +++ b/tests/test_no_degenerate_columns.py @@ -0,0 +1,108 @@ +"""The guard for the bug that made this repository's data quietly useless. + +`rng.binomial(1, 0.55)` without a size argument returns a *scalar*, which numpy +then broadcasts across the whole column. Nothing errors, the file writes, the row +count is right, and the variable is a constant. It reached the repository in two +shapes: + + owns_radio = rng.binomial(1, 0.55) # every household: 1 + barrier_marriage = np.where(cond, rng.binomial(1, .25), 0) # every girl: 0 + +Nineteen call sites across nine generators were affected, producing fifteen +constant columns. Four of them were assets in `targeting`, which exists to teach +proxy means testing, and a PMT whose asset predictors do not vary is not a PMT. + +The tell that confirmed the diagnosis is still in `girls_education`: `barrier_cost` +on the neighbouring line takes an *array* probability, so numpy returned a vector +and that column was always fine. + +This test also catches the second shape of the same failure: a column that varies +but is a perfect alias of another column. +""" +import itertools + +import numpy as np +import pandas as pd +import pytest + +from conftest import build +from generate import GENERATORS + +# Legitimately constant, with the reason. +EXPECTED_CONSTANT = { + # Item parameters are fixed by construction in a wide-format IRT table: every + # respondent answers the same 30 items, so difficulty and discrimination do + # not vary by row. They are shipped as columns so the frame is self-describing. + ("irt_assessment", "item_{}_difficulty"), + ("irt_assessment", "item_{}_discrimination"), + # A poverty line is a single threshold applied to everyone. That is what it is. + ("targeting", "poverty_line_usd"), +} +_CONST_EXACT = {(g, c) for g, c in EXPECTED_CONSTANT if "{}" not in c} +_CONST_ITEM = {(g, c) for g, c in EXPECTED_CONSTANT if "{}" in c} + + +def _is_expected_constant(gen, col): + if (gen, col) in _CONST_EXACT: + return True + return any( + g == gen and col == c.format(i) + for g, c in _CONST_ITEM + for i in range(1, 51) + ) + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_no_unexpected_constant_columns(name): + df = build(name, n=3000) + offenders = [ + c for c in df.columns + if df[c].nunique(dropna=True) <= 1 and not _is_expected_constant(name, c) + ] + assert not offenders, ( + f"{name}: columns with no variation {offenders}. Usually a size-less RNG " + f"draw. See this module's docstring." + ) + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_no_all_null_columns(name): + df = build(name, n=3000) + empty = [c for c in df.columns if df[c].isna().all()] + assert not empty, f"{name}: columns that are entirely null {empty}" + + +# Pairs where one column is defined as the other, by design rather than by accident. +EXPECTED_ALIASES = { + # Categorical eligibility for the female-headed-household window *is* the + # female-head indicator; the dataset ships both so the targeting rule is legible. + ("targeting", frozenset({"head_female", "categorical_eligible_fhh"})), + ("targeting", frozenset({"head_disabled", "categorical_eligible_disabled"})), +} + + +@pytest.mark.parametrize("name", sorted(GENERATORS)) +def test_no_accidental_binary_aliases(name): + """A binary column that is an exact function of another carries no information. + + `rct_experiment.spillover_risk` was precisely this: 1 for every control and 0 + for every treated unit, because randomisation was stratified within district so + every district always contained treated units. It is now defined at village + level, and pure-control villages give it real variation. + """ + df = build(name, n=3000) + binary = [c for c in df.columns if df[c].nunique(dropna=True) == 2] + offenders = [] + for a, b in itertools.combinations(binary, 2): + if (name, frozenset({a, b})) in EXPECTED_ALIASES: + continue + tab = pd.crosstab(df[a].astype(str), df[b].astype(str)) + if tab.shape != (2, 2): + continue + v = tab.to_numpy() + if (np.diag(v) == 0).all() or (np.diag(np.fliplr(v)) == 0).all(): + offenders.append((a, b)) + assert not offenders, ( + f"{name}: perfectly collinear binary pairs {offenders}. One is an alias of " + f"the other, so a model including both drops a term." + ) diff --git a/tests/test_rct_truth.py b/tests/test_rct_truth.py new file mode 100644 index 0000000..95bf129 --- /dev/null +++ b/tests/test_rct_truth.py @@ -0,0 +1,89 @@ +"""The RCT generator must give back the parameters TRUTH.md says it uses. + +If these fail, either the generator changed or TRUTH.md is now lying to learners. +Fix whichever is wrong; do not loosen the tolerance to make the failure go away. +""" +import numpy as np +import pytest + +from conftest import build + +SEEDS = (1, 42, 123, 777, 2026) +N = 40_000 +TOL = 0.01 # log points + +# log(multiplier) + mean heterogeneity loading among compliers; see TRUTH.md. +HET = np.log(1 + 0.04 * 0.52 + 0.06 * 0.50) +TRUE_LATE = { + "cash_transfer": np.log(1.15) + HET, + "cash_plus_training": np.log(1.22) + HET, + "training_only": np.log(1.08) + HET, +} +TRUE_SPILLOVER = np.log(1.03) + + +def _panel(seed): + df = build("rct_experiment", n=N, seed=seed) + d = df.dropna(subset=["endline_consumption_usd", "baseline_consumption_usd"]).copy() + d["lr"] = np.log(d.endline_consumption_usd) - np.log(d.baseline_consumption_usd) + return d + + +def _pure_control_mean(d): + return d[(d.treatment_arm == "control") & (d.spillover_risk == 0)].lr.mean() + + +@pytest.mark.parametrize("arm", sorted(TRUE_LATE)) +def test_wald_late_recovers_true_complier_effect(arm): + got = [] + for seed in SEEDS: + d = _panel(seed) + a = d[d.treatment_arm == arm] + got.append((a.lr.mean() - _pure_control_mean(d)) / a.actually_treated.mean()) + mean = float(np.mean(got)) + assert abs(mean - TRUE_LATE[arm]) < TOL, ( + f"{arm}: Wald LATE {mean:+.4f} vs TRUTH.md {TRUE_LATE[arm]:+.4f} " + f"(per-seed: {[round(x, 4) for x in got]})" + ) + + +def test_spillover_is_recoverable_from_pure_control_villages(): + got = [] + for seed in SEEDS: + d = _panel(seed) + exposed = d[(d.treatment_arm == "control") & (d.spillover_risk == 1)].lr.mean() + got.append(exposed - _pure_control_mean(d)) + mean = float(np.mean(got)) + assert abs(mean - TRUE_SPILLOVER) < TOL, ( + f"spillover {mean:+.4f} vs TRUTH.md {TRUE_SPILLOVER:+.4f}" + ) + + +def test_contaminated_control_group_biases_itt_toward_zero(): + """The lesson the design exists to teach, asserted so it cannot be lost.""" + d = _panel(123) + pure = _pure_control_mean(d) + allc = d[d.treatment_arm == "control"].lr.mean() + assert allc > pure, "controls in treatment villages should do better than pure controls" + for arm in TRUE_LATE: + m = d[d.treatment_arm == arm].lr.mean() + assert (m - allc) < (m - pure), f"{arm}: contaminated ITT should be smaller" + + +def test_spillover_risk_is_not_an_alias_of_the_control_dummy(): + """The original defect, pinned directly.""" + d = build("rct_experiment", n=20_000, seed=7) + ctrl = d[d.treatment_arm == "control"] + assert ctrl.spillover_risk.nunique() == 2, ( + "spillover_risk must vary among controls: some sit in pure-control villages" + ) + assert (d[d.treatment_arm != "control"].spillover_risk == 0).all(), ( + "treated units are not spillover-exposed by this definition" + ) + + +def test_village_structure_has_pure_control_villages(): + d = build("rct_experiment", n=20_000, seed=7) + share = d.groupby("village_id").village_treated_share.first() + assert (share == 0).any(), "no pure-control villages: spillover is unidentified" + assert (share > 0).any(), "no treatment villages"