Skip to content

feat(calibration): add lab-result intake module + synthetic example data (v0.5.19) - #133

Merged
cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-20260704d
Jul 4, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
agent/oamp-loop-20260704d

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

The wet-lab feedback loop had lab-result ingestion (data/lab_results.py, lab-result-report CLI) but no way to JOIN pilot panel predictions with lab-result actuals. Without that join, any future recalibration has no intake valve and reviewers cannot see prediction-vs-actual alignment.

This PR adds the missing join layer.

Changes

New module: openamp_foundry.calibration.intake

  • build_calibration_intake_report(panel_csv, results_dir) joins a pilot panel CSV (computational predictions) with a directory of validated lab result JSON files (experimental actuals) on candidate_id.
  • write_calibration_intake_json / write_calibration_intake_markdown produce machine-readable and human-readable review artifacts.
  • Honest minimum-cohort-size gate (MIN_COHORT_SIZE=5): aggregate cohort metrics are NOT reported when n < 5. Below the gate the metric is marked insufficient_data: True and no point estimate is produced. This prevents small-sample theater.
  • Two pilot cohort metrics: activity_vs_active_mic (MIC ≤ 32 µg/mL active) and rich_selectivity_vs_high_hemolysis (hemolysis ≥ 10% high-risk).
  • Per-candidate join rows expose every prediction column plus aggregated actual outcomes. Control failures and orphan lab results (results with no matching candidate) surfaced as separate audit fields.

New CLI: openamp-foundry calibration-intake

openamp-foundry calibration-intake \
  --panel <pilot_panel.csv> \
  --results-dir <lab_results_dir> \
  --out-json <out.json> \
  --out-md <out.md>

Makefile targets

  • make lab-result-intake PANEL=<panel.csv> RESULTS_DIR=<dir> (real data)
  • make lab-result-intake-example (synthetic demo, no args needed)

Synthetic example data (clearly labeled SYNTHETIC)

  • examples/lab_results/RES-SYN-001..005.json — 5 synthetic JSONs covering active hit, inactive candidate, low hemolysis, high hemolysis, control failure
  • examples/lab_results_panel.csv — 8-candidate synthetic pilot panel
  • examples/lab_results/README.md — prominent SYNTHETIC disclaimer

Every JSON file's disclaimer field explicitly states it is a synthetic example. The README repeats the warning prominently.

Tests

29 new tests in tests/test_calibration_intake.py. Total: 1614 passing (was 1585). Coverage of:

  • Empty panels, missing columns tolerated
  • Orphan lab result detection
  • Control failure surfacing
  • Cohort metric gating (below/equal/above MIN_COHORT_SIZE)
  • Hemolysis direction logic (rich_selectivity above threshold means LOWER predicted hemolysis)
  • Synthetic example data schema validation
  • Threshold constants match documented values

Documentation updates

  • docs/ROADMAP.md — new v0.5.19 entry
  • docs/METRICS_CURRENT.md — header updated, change log entry added
  • AGENTS.md — Priority 7 marked as having v0.5.19 progress
  • CHANGELOG.md — v0.5.19 section added
  • Makefile — test count updated from 1287 to 1614

Honest Limitations (documented in module docstring, CLI help, README)

  • Module is descriptive only; does NOT trigger recalibration. It is the intake valve that produces a review artifact for a separate, human-reviewed recalibration workflow.
  • Cohort metrics are gated by MIN_COHORT_SIZE=5. Below the gate, no point estimate is reported. This prevents small-sample theater where 2-3 MIC readouts could swing an "AUROC" by 0.30.
  • Synthetic example data is clearly labeled in every JSON's disclaimer field. It exists solely to exercise the intake workflow end-to-end. When real validated lab results arrive, replace the example directory — the pipeline itself does not change.
  • Recalibration requires a separate human-reviewed decision log (docs/DECISION_RULES.md, docs/WAVE2_PLAN.md).

Verification

$ make lab-result-intake-example
{
  "status": "ok",
  "n_panel_candidates": 8,
  "n_lab_results": 5,
  "n_matched_candidates": 4,
  "n_orphan_lab_results": 0,
  "cohort_metrics": {
    "activity_vs_active_mic": {"n": 3, "insufficient_data": true},
    "rich_selectivity_vs_high_hemolysis": {"n": 2, "insufficient_data": true}
  }
}

$ python -m pytest tests/ -q
1614 passed, 1 skipped in 28.55s

$ ruff check src/openamp_foundry/calibration/ tests/test_calibration_intake.py
All checks passed!

The cohort metrics correctly report insufficient_data: true because the synthetic example only has 3 MIC results and 2 hemolysis results (both below the 5-sample gate). This is exactly the honest behavior we want — a single MIC readout cannot swing an aggregate metric.

Future loop benefit

  • When real wet-lab data arrives, replacing the examples/lab_results/ directory with real JSON is the ONLY change needed to get a real prediction-vs-actual report.
  • Reviewers can now see every panel candidate side-by-side with its predictions and any actual outcomes in one document.
  • The minimum-cohort-size gate prevents the next agent from accidentally reporting a "30% hit rate" based on 2 lucky MIC values.
  • The synthetic example data + README + 29 tests make the workflow impossible to misinterpret.

…ata (v0.5.19)

Lore: add calibration intake module to join pilot panel predictions with
lab result actuals. Honest minimum-cohort-size gate prevents small-sample
theater. Synthetic example data clearly labeled SYNTHETIC in every file.

Co-authored-by: OmX <omx@openamp.local>
@cschanhniem
cschanhniem merged commit f79d1f4 into main Jul 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant