diff --git a/AGENTS.md b/AGENTS.md index 886218da..60cc2aff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -515,6 +515,8 @@ Prepare: After qualified assay data exists, build: +**v0.5.19 progress:** `openamp-foundry calibration-intake` joins pilot panel predictions with validated lab result actuals and produces a per-candidate report. Cohort metrics are gated by `MIN_COHORT_SIZE=5` to prevent small-sample theater. The module is descriptive only — recalibration requires a separate human-reviewed decision log (`docs/DECISION_RULES.md`). Synthetic-but-clearly-labeled example data is in `examples/lab_results/`. 29 new tests, total 1614 passing. + * lab-result ingestion; * hit/failure calibration; * retrospective analysis of why the model was right or wrong; diff --git a/CHANGELOG.md b/CHANGELOG.md index 401db47a..7d1b8ce4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -235,3 +235,52 @@ improvements made before committing the ~$10k wet-lab synthesis budget. - Ensemble scoring with configurable weights - JSONL output with per-candidate `selected` field - `pipeline.yaml` and `phase3.yaml` configs + +--- + +## [Unreleased — Calibration Intake Module] — 2026-07-04 + +### v0.5.19 — Calibration Intake Module + +- **`openamp_foundry/calibration/intake.py`**: new module + - `build_calibration_intake_report(panel_csv, results_dir)` joins a pilot + panel CSV (computational predictions) with a directory of validated lab + result JSON files (experimental actuals) on `candidate_id`. + - `write_calibration_intake_json` and `write_calibration_intake_markdown` + produce machine-readable and human-readable review artifacts. + - **Honest minimum-cohort-size gate (`MIN_COHORT_SIZE=5`)**: aggregate + cohort metrics are NOT reported when `n < 5`. Below the gate the metric + is marked `insufficient_data: True` and no point estimate is produced. + This prevents small-sample theater. + - Two pilot cohort metrics: `activity_vs_active_mic` (MIC ≤ 32 µg/mL active) + and `rich_selectivity_vs_high_hemolysis` (hemolysis ≥ 10% high-risk). + - Per-candidate join rows expose every prediction column plus aggregated + actual outcomes. Control failures and orphan lab results surfaced. +- **`cli/commands/reports.py`**: new `_run_calibration_intake` command. +- **`cli/main.py`**: new `calibration-intake` subcommand registered. +- **`examples/lab_results/`**: 5 SYNTHETIC JSON files demonstrating the + workflow (active hit, inactive candidate, low hemolysis, high hemolysis, + control failure). All files include a `disclaimer` field that explicitly + states they are synthetic examples. README in the directory repeats the + warning prominently. +- **`examples/lab_results_panel.csv`**: 8-candidate synthetic pilot panel. +- **`tests/test_calibration_intake.py`**: 29 tests covering empty panels, + missing columns, orphan detection, control failure surfacing, cohort + metric gating (below/equal/above `MIN_COHORT_SIZE`), hemolysis direction + logic, synthetic-example schema validation, threshold constants. +- **`Makefile`**: `lab-result-intake` and `lab-result-intake-example` + targets added. Test-count help text updated. +- **Total tests:** 1614 passing (was 1585). + +#### Honest limitations + +- This module does **NOT** trigger recalibration. It is the intake valve that + produces a review artifact for a separate, human-reviewed recalibration + workflow (see `docs/DECISION_RULES.md` and `docs/WAVE2_PLAN.md`). +- Cohort metrics are descriptive only; they do not validate the pipeline, do + not control for selection bias from the pre-registered shortlist, and must + not be used to rewrite scoring weights after the fact. +- Synthetic example data is clearly labeled in every JSON file's `disclaimer` + field and in `examples/lab_results/README.md`. It exists solely to exercise + the intake workflow end-to-end. When real validated lab results arrive, + replace the example directory — the pipeline itself does not change. diff --git a/Makefile b/Makefile index e29b559b..827b4605 100644 --- a/Makefile +++ b/Makefile @@ -41,7 +41,9 @@ help: @echo " make wave0-5-evidence Re-generate evidence certificates" @echo " make wave0-5b-generate Generate Wave 0.5b candidates (safety-optimized, no aromatics)" @echo " make wave0-5b-filter Filter Wave 0.5b shortlist (depends on wave0-5b-generate)" - @echo " make test Run full test suite (1287 tests, ≥80% coverage)" + @echo " make lab-result-intake Join a pilot panel CSV with lab result JSON files" + @echo " make lab-result-intake-example Run intake on the synthetic example data" + @echo " make test Run full test suite (1614 tests, >=80% coverage)" @echo " make coverage Test suite with per-module coverage report" @echo " make lint Ruff lint check on src/ tests/ scripts/" @echo " make typecheck mypy type check on src/" @@ -242,6 +244,27 @@ novelty-broad: pilot --references-csv examples/known_reference/amp_curated_references.csv \ --out outputs/novelty_broad_report.md +lab-result-intake-example: + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli calibration-intake \ + --panel examples/lab_results_panel.csv \ + --results-dir examples/lab_results \ + --out-json outputs/calibration_intake_example.json \ + --out-md outputs/calibration_intake_example.md + +lab-result-intake: + @if [ -z "$(PANEL)" ] || [ -z "$(RESULTS_DIR)" ]; then \ + echo "Usage: make lab-result-intake PANEL= RESULTS_DIR="; \ + echo ""; \ + echo "Or run the synthetic example without arguments:"; \ + echo " make lab-result-intake-example"; \ + exit 1; \ + fi + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli calibration-intake \ + --panel "$(PANEL)" \ + --results-dir "$(RESULTS_DIR)" \ + --out-json outputs/calibration_intake.json \ + --out-md outputs/calibration_intake.md + clean: rm -rf outputs/*.jsonl outputs/*.md outputs/*.json outputs/evidence outputs/phase3_evidence .pytest_cache .ruff_cache diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 6a983af0..66d0e06f 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-03 (rich selectivity integrated into production pipeline) +> **Last updated:** 2026-07-04 (calibration intake module added) +> **New in v0.5.19:** `openamp-foundry calibration-intake` joins pilot panel predictions with lab result actuals. Descriptive only; does NOT trigger recalibration. > **Pipeline version:** v0.5.x > **Branch:** main @@ -713,4 +714,5 @@ Decoys score low on activity. Selective AMPs score moderately on both. | 2026-07-03 | **Rich selectivity integrated into production pipeline:** rich_selectivity_score now computed in score_candidates() (pipeline.py), replaces hemolysis_safety as the expert composite hemolysis-risk component (weight 0.10), used in pilot_priority formula, displayed in pilot panel report, and included in evidence certificates. Expert AUROC drops 0.7119→0.7097 (−0.0022) — acceptable tradeoff: the expert now includes a significant hemolysis detector (CI excludes 0.5) instead of the old non-significant one. | OpenAMP loop | | 2026-07-03 | **Two-gate triage composite added:** gate_triage = activity × rich_selectivity, added to triage benchmark. First scorer to pass all three standard triage conditions with strong selective_vs_hemolytic separation (0.666). Top-20: 16 selective / 1 hemolytic / 3 decoy — best distribution. Does NOT pass strict triage (hem_vs_dec 0.489) — honest limitation. Must not replace ensemble activity gate. | OpenAMP loop | | 2026-07-03 | **Feature decomposition benchmark added:** per-feature selective_vs_hemolytic AUROC for all 30 scalar physicochemical features. hydrophobic_fraction is the strongest single discriminative feature (0.6745, CI 0.58-0.77) but is NOT used by the selectivity proxy. 8/30 features have significant signal; 6 of those are unused. Provides actionable diagnostic for why composite scorers fail selective_vs_hemolytic discrimination. | OpenAMP loop | +| 2026-07-04 | **Calibration intake module added:** `openamp-foundry calibration-intake` joins a pilot panel CSV with a directory of validated lab result JSON files, produces a per-candidate prediction-vs-actual report with cohort metrics gated by `MIN_COHORT_SIZE=5`. Descriptive only — does NOT trigger recalibration, weight updates, or selection-rule changes. Synthetic example data in `examples/lab_results/` is clearly labeled in every file and in `examples/lab_results/README.md`. 29 new tests; total 1614 passing. | OpenAMP loop | | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 2c4b37d5..5ff79acd 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -433,3 +433,62 @@ review (2026-06-28). Progress on these would materially raise breakthrough proba | Wet-lab result integration (active-learning round 2) | Required to move from 15–30% to 50%+ credible probability | Requires wet-lab | | ~~Pre-registration of assay protocol before synthesis~~ | ~~Strengthens causal inference; reduces reporting bias~~ | **Done** (docs/ASSAY_PREREGISTRATION.md — PRs #83; includes MRSA USA300, serum stability, Gate P3 aligned) | | Public benchmark paper (replicable, cluster-split, open datasets) | Sets community standard; enables external validation | Large | + +## v0.5.19 — Calibration Intake Module ✓ (2026-07-04) + +The lab-result ingestion layer (`data/lab_results.py`, `reports/lab_result_report.py`, +`lab-result-report` CLI) was already in place, but there was no way to **join the +pilot panel predictions with the lab-result actuals** to see whether the pipeline's +predictions matched reality. Without that join, the wet-lab feedback loop has no +intake valve, and any future recalibration would be operating on raw assay data +without the prediction-vs-actual contrast it needs. + +This version adds the first step of the closed wet-lab feedback loop: a +calibration-intake report that joins a pilot panel CSV with a directory of +validated lab result JSON files. It is descriptive only — it does not rewrite +selection rules, does not modify scoring weights, and does not upgrade assay +readouts into biological claims. + +Changes: +- `src/openamp_foundry/calibration/__init__.py`: new package +- `src/openamp_foundry/calibration/intake.py`: new module + - `build_calibration_intake_report(panel_csv, results_dir)` — joins predictions + with actuals on `candidate_id`, produces a machine-readable report + - `write_calibration_intake_json(report, out_path)` and + `write_calibration_intake_markdown(report, out_path)` — output writers + - **Honest minimum-cohort-size gate**: aggregate cohort metrics are NOT + reported when `n < 5`. Below the gate the metric is marked + `insufficient_data: True` and no point estimate is produced. This prevents + small-sample theater where 2-3 MIC readouts could swing an "AUROC" by 0.30. + - Two pilot cohort metrics: `activity_vs_active_mic` (MIC ≤ 32 µg/mL active) + and `rich_selectivity_vs_high_hemolysis` (hemolysis ≥ 10% high-risk). + - Per-candidate join rows expose every prediction column plus the + aggregated actual outcomes. + - Control failures and orphan lab results (results with no matching + candidate) are surfaced as separate audit fields. +- `src/openamp_foundry/cli/commands/reports.py`: `_run_calibration_intake` +- `src/openamp_foundry/cli/main.py`: `calibration-intake` subcommand registered +- `examples/lab_results/`: 5 clearly-labeled SYNTHETIC JSON files demonstrating + the workflow (active hit, inactive candidate, low hemolysis, high hemolysis, + control failure). Disclaimer required in every file's `disclaimer` field. +- `examples/lab_results/README.md`: explicitly disclaims real-world use +- `examples/lab_results_panel.csv`: 8-candidate synthetic pilot panel +- `tests/test_calibration_intake.py`: 29 tests covering empty panels, missing + columns, orphan detection, control failure surfacing, cohort metric gating + (below/equal/above `MIN_COHORT_SIZE`), hemolysis direction logic, + synthetic-example schema validation, and threshold constants +- `Makefile`: `lab-result-intake` and `lab-result-intake-example` targets +- Total tests: 1614 passing (was 1585) + +Key honest limitation: this module does NOT trigger recalibration. It is the +intake valve that produces a review artifact for a separate, human-reviewed +recalibration workflow (see `docs/DECISION_RULES.md` and `docs/WAVE2_PLAN.md`). +Cohort metrics are descriptive only; they do not validate the pipeline, do not +control for selection bias from the pre-registered shortlist, and must not be +used to rewrite scoring weights after the fact. + +The synthetic example data is clearly labeled in every JSON file's `disclaimer` +field and in `examples/lab_results/README.md`. It exists solely to exercise the +intake workflow end-to-end so future agents and reviewers can verify the code +path without real wet-lab data. When real validated lab results arrive, replace +the example directory — the pipeline itself does not change. diff --git a/examples/lab_results/README.md b/examples/lab_results/README.md new file mode 100644 index 00000000..c3b9c717 --- /dev/null +++ b/examples/lab_results/README.md @@ -0,0 +1,40 @@ +# Lab Result Examples (SYNTHETIC) + +> **ALL FILES IN THIS DIRECTORY ARE SYNTHETIC EXAMPLE DATA.** +> +> These JSON files demonstrate the calibration intake workflow. They are +> **not** real wet-lab measurements, do not correspond to any actual +> peptide synthesis, and must not be cited as experimental evidence of +> activity, safety, or any other biological property. They exist solely +> to exercise `openamp-foundry calibration-intake` end-to-end so future +> agents and reviewers can verify the workflow without real data. + +## Files + +| File | Assay | Purpose | +|------|-------|---------| +| `RES-SYN-001.json` | MIC (E. coli) | Synthetic active hit, controls pass | +| `RES-SYN-002.json` | hemolysis_RBC | Synthetic low hemolysis | +| `RES-SYN-003.json` | MIC (E. coli) | Synthetic inactive candidate | +| `RES-SYN-004.json` | hemolysis_RBC | Synthetic high hemolysis | +| `RES-SYN-005.json` | MIC (E. coli) | Synthetic active hit, controls fail | + +## How to use + +```bash +make lab-result-intake-example +``` + +This runs `openamp-foundry calibration-intake` against the synthetic panel +`examples/lab_results_panel.csv` and the synthetic lab results in this +directory. The output is written to `outputs/calibration_intake_example/`. + +## When real lab data exists + +Replace this directory with the real validated lab result JSON files, +where each file matches `schemas/lab_result.schema.json`. The pipeline +itself does not need to change — only the input data does. + +See `docs/WET_LAB_HANDOFF.md`, `docs/DECISION_RULES.md`, and +`docs/WAVE2_PLAN.md` for the workflow that converts these inputs into +recalibration decisions. diff --git a/examples/lab_results/RES-SYN-001.json b/examples/lab_results/RES-SYN-001.json new file mode 100644 index 00000000..34a1d239 --- /dev/null +++ b/examples/lab_results/RES-SYN-001.json @@ -0,0 +1,20 @@ +{ + "result_id": "RES-SYN-001", + "candidate_id": "SEED-005_VAR_049", + "assay_type": "MIC", + "organism_or_cell_line": "SYNTHETIC EXAMPLE - E. coli ATCC 25922", + "result_value": 8.0, + "result_unit": "µg/mL", + "result_qualitative": "active", + "positive_control_passed": true, + "negative_control_passed": true, + "positive_control_id": "ciprofloxacin 0.25 µg/mL", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 3, + "performed_by_lab": "SYNTHETIC EXAMPLE - University Test Lab", + "raw_data_sha256": null, + "computational_candidate_certificate_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "notes": "SYNTHETIC DATA: not a real wet-lab result. Included only to exercise the calibration intake workflow.", + "disclaimer": "SYNTHETIC EXAMPLE. This is not a real experimental result on a computationally nominated candidate and does not constitute a drug or clinical claim." +} diff --git a/examples/lab_results/RES-SYN-002.json b/examples/lab_results/RES-SYN-002.json new file mode 100644 index 00000000..9d6d01db --- /dev/null +++ b/examples/lab_results/RES-SYN-002.json @@ -0,0 +1,20 @@ +{ + "result_id": "RES-SYN-002", + "candidate_id": "SEED-005_VAR_049", + "assay_type": "hemolysis_RBC", + "organism_or_cell_line": "SYNTHETIC EXAMPLE - hRBC", + "result_value": 3.5, + "result_unit": "%", + "result_qualitative": "inactive", + "positive_control_passed": true, + "negative_control_passed": true, + "positive_control_id": "melittin 10 µM", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 3, + "performed_by_lab": "SYNTHETIC EXAMPLE - University Test Lab", + "raw_data_sha256": null, + "computational_candidate_certificate_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "notes": "SYNTHETIC DATA: not a real wet-lab result. Included only to exercise the calibration intake workflow.", + "disclaimer": "SYNTHETIC EXAMPLE. This is not a real experimental result on a computationally nominated candidate and does not constitute a drug or clinical claim." +} diff --git a/examples/lab_results/RES-SYN-003.json b/examples/lab_results/RES-SYN-003.json new file mode 100644 index 00000000..ca5763a4 --- /dev/null +++ b/examples/lab_results/RES-SYN-003.json @@ -0,0 +1,20 @@ +{ + "result_id": "RES-SYN-003", + "candidate_id": "SEED-007_VAR_022", + "assay_type": "MIC", + "organism_or_cell_line": "SYNTHETIC EXAMPLE - E. coli ATCC 25922", + "result_value": 128.0, + "result_unit": "µg/mL", + "result_qualitative": "inactive", + "positive_control_passed": true, + "negative_control_passed": true, + "positive_control_id": "ciprofloxacin 0.25 µg/mL", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 3, + "performed_by_lab": "SYNTHETIC EXAMPLE - University Test Lab", + "raw_data_sha256": null, + "computational_candidate_certificate_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "notes": "SYNTHETIC DATA: not a real wet-lab result. Included only to exercise the calibration intake workflow.", + "disclaimer": "SYNTHETIC EXAMPLE. This is not a real experimental result on a computationally nominated candidate and does not constitute a drug or clinical claim." +} diff --git a/examples/lab_results/RES-SYN-004.json b/examples/lab_results/RES-SYN-004.json new file mode 100644 index 00000000..249bb6b4 --- /dev/null +++ b/examples/lab_results/RES-SYN-004.json @@ -0,0 +1,20 @@ +{ + "result_id": "RES-SYN-004", + "candidate_id": "SEED-008_VAR_007", + "assay_type": "hemolysis_RBC", + "organism_or_cell_line": "SYNTHETIC EXAMPLE - hRBC", + "result_value": 42.0, + "result_unit": "%", + "result_qualitative": "toxic", + "positive_control_passed": true, + "negative_control_passed": true, + "positive_control_id": "melittin 10 µM", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 3, + "performed_by_lab": "SYNTHETIC EXAMPLE - University Test Lab", + "raw_data_sha256": null, + "computational_candidate_certificate_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "notes": "SYNTHETIC DATA: not a real wet-lab result. Included only to exercise the calibration intake workflow.", + "disclaimer": "SYNTHETIC EXAMPLE. This is not a real experimental result on a computationally nominated candidate and does not constitute a drug or clinical claim." +} diff --git a/examples/lab_results/RES-SYN-005.json b/examples/lab_results/RES-SYN-005.json new file mode 100644 index 00000000..2388fcc7 --- /dev/null +++ b/examples/lab_results/RES-SYN-005.json @@ -0,0 +1,20 @@ +{ + "result_id": "RES-SYN-005", + "candidate_id": "SEED-001_VAR_064", + "assay_type": "MIC", + "organism_or_cell_line": "SYNTHETIC EXAMPLE - S. aureus ATCC 25923", + "result_value": 16.0, + "result_unit": "µg/mL", + "result_qualitative": "active", + "positive_control_passed": false, + "negative_control_passed": true, + "positive_control_id": "vancomycin 1.0 µg/mL", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 2, + "performed_by_lab": "SYNTHETIC EXAMPLE - University Test Lab", + "raw_data_sha256": null, + "computational_candidate_certificate_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "notes": "SYNTHETIC DATA: not a real wet-lab result. Positive control did NOT pass expected range; result must NOT be used for activity claims until retested with valid controls.", + "disclaimer": "SYNTHETIC EXAMPLE. This is not a real experimental result on a computationally nominated candidate and does not constitute a drug or clinical claim." +} diff --git a/examples/lab_results_panel.csv b/examples/lab_results_panel.csv new file mode 100644 index 00000000..f2b19da9 --- /dev/null +++ b/examples/lab_results_panel.csv @@ -0,0 +1,9 @@ +pilot_rank,candidate_id,sequence,length,seed,ensemble,activity,boman_activity,disagreement,safety,synthesis,novelty,serum_stability,selectivity_proxy,rich_selectivity,pilot_priority,amphipathic_score,charge_ph74 +1,SEED-005_VAR_049,GIGKFLKKAKKFGKAFVKILKK,21,SEED-005,0.82,0.85,0.78,0.07,0.92,0.88,0.71,0.65,0.62,0.74,0.81,1.85,4.2 +2,SEED-007_VAR_022,IKTKNLLISGLKNLGKALQQA,21,SEED-007,0.79,0.81,0.74,0.07,0.89,0.85,0.68,0.62,0.58,0.71,0.78,1.72,3.8 +3,SEED-008_VAR_007,KWFWAWPGVKVGLF,14,SEED-008,0.78,0.88,0.71,0.17,0.65,0.42,0.74,0.55,0.45,0.32,0.62,1.45,2.8 +4,SEED-001_VAR_064,GIGKWLHAAFKIGKKIF,18,SEED-001,0.80,0.83,0.79,0.04,0.95,0.92,0.42,0.72,0.55,0.78,0.80,1.65,4.5 +5,SEED-019_VAR_004,RVRIRLVKRLLK,12,SEED-019,0.76,0.82,0.73,0.09,0.91,0.90,0.62,0.68,0.71,0.82,0.77,1.92,4.7 +6,SEED-020_VAR_002,KVRIRVLKRLLK,12,SEED-020,0.74,0.80,0.71,0.09,0.93,0.93,0.60,0.70,0.73,0.81,0.75,1.95,4.8 +7,SEED-010_VAR_001,KRHHGYKRKFH,11,SEED-010,0.72,0.78,0.72,0.06,0.94,0.95,0.45,0.71,0.60,0.74,0.73,1.50,5.0 +8,SEED-003_VAR_001,RRWQWRMKKLG,11,SEED-003,0.78,0.86,0.76,0.10,0.85,0.55,0.35,0.60,0.52,0.66,0.72,1.70,4.1 diff --git a/src/openamp_foundry/calibration/__init__.py b/src/openamp_foundry/calibration/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/src/openamp_foundry/calibration/intake.py b/src/openamp_foundry/calibration/intake.py new file mode 100644 index 00000000..feef54f6 --- /dev/null +++ b/src/openamp_foundry/calibration/intake.py @@ -0,0 +1,493 @@ +"""Lab-result calibration intake for the wet-lab feedback loop. + +Joins a pilot panel CSV (computational predictions) with validated lab +result JSON files (experimental outcomes) to produce a decision-grade +calibration intake report. + +This module is the *first* step of the wet-lab feedback loop: + + pilot panel (predictions) + + + lab results (actuals) + | + v + build_calibration_intake_report + | + v + per-candidate prediction-vs-actual rows + | + v + cohort metrics gated by minimum sample size + | + v + review artifact (NOT a recalibration decision) + +The output is descriptive and audit-only. It does NOT: + - rewrite selection rules; + - modify scoring weights; + - claim efficacy, safety, or clinical utility; + - recompute evidence certificates; + - upgrade assay readouts into biological claims. + +Recalibration decisions must follow a separate, human-reviewed workflow +that explicitly compares pre-registered selection rules against observed +outcomes. See docs/WAVE2_PLAN.md and docs/DECISION_RULES.md. +""" +from __future__ import annotations + +import csv +import json +from pathlib import Path + +from openamp_foundry.data.lab_results import ( + load_lab_results_dir, + summarise_candidate_outcomes, + summarise_lab_results, +) + +# Minimum sample size required before any aggregate cohort metric is reported. +# Below this threshold the cohort is marked ``insufficient_data: True`` and +# no point estimate is produced. This protects against small-sample theater +# where a single MIC readout could swing an "AUROC" by 0.30. +MIN_COHORT_SIZE = 5 + +# Activity-vs-MIC binary threshold for retrospective triage comparison. +# Candidates with MIC <= this value are labelled "predicted_active" if their +# pipeline activity score was above ACTIVITY_THRESHOLD, else "predicted_inactive". +# Standard "active" cutoff from CLSI / EUCAST broth microdilution guidance. +MIC_ACTIVE_CUTOFF_UG_ML = 32.0 +ACTIVITY_THRESHOLD = 0.70 + +# Hemolysis "high risk" cutoff for retrospective triage comparison. +# Reported as % hemolysis at the highest tested concentration. +# Candidates above this are flagged "hemolysis_high". +HEMOLYSIS_HIGH_PCT = 10.0 + +_DISCLAIMER = ( + "Calibration intake report. Descriptive join of computational predictions " + "and validated wet-lab results only. This is NOT proof of efficacy, safety, " + "or clinical utility. It does NOT trigger recalibration, weight updates, " + "or changes to the pre-registered selection rule. Recalibration requires a " + "separate human-reviewed decision log and must not rewrite success " + "definitions after the fact. See docs/DECISION_RULES.md and " + "docs/WAVE2_PLAN.md for the pre-registered Wave 2 workflow." +) + + +def _load_panel_csv(panel_csv): + """Load a pilot panel CSV into a list of row dicts. + + Accepts both legacy ``pilot_panel.csv`` and the wider ``wave1_final_panel.csv`` + shape. Missing optional columns are tolerated. + """ + p = Path(panel_csv) + with p.open("r", newline="", encoding="utf-8") as f: + return list(csv.DictReader(f)) + + +def _to_float(value): + """Parse a CSV cell into float, returning None on failure or empty.""" + if value is None or value == "": + return None + try: + return float(value) + except (TypeError, ValueError): + return None + + +def _candidate_predictions(row): + """Extract the prediction columns we will compare against lab outcomes.""" + return { + "ensemble": _to_float(row.get("ensemble")), + "activity": _to_float(row.get("activity")), + "safety": _to_float(row.get("safety")), + "synthesis": _to_float(row.get("synthesis")), + "novelty": _to_float(row.get("novelty")), + "selectivity_proxy": _to_float(row.get("selectivity_proxy")), + "rich_selectivity": _to_float(row.get("rich_selectivity")), + "pilot_priority": _to_float(row.get("pilot_priority")), + } + + +def _is_active_mic(result): + """Heuristic: classify a MIC result as active / inactive. + + Returns None when the result is qualitative-only or the assay is not MIC. + Conservative cutoff at MIC <= 32 ug/mL matches the Wave 2 plan's "active" + definition (docs/WAVE2_PLAN.md, Scenario B). + """ + if result.get("assay_type") != "MIC": + return None + value = result.get("result_value") + if value is None: + return None + try: + return float(value) <= MIC_ACTIVE_CUTOFF_UG_ML + except (TypeError, ValueError): + return None + + +def _is_high_hemolysis(result): + """Heuristic: classify a hemolysis result as high risk. + + Returns None when the result is qualitative-only or the assay is not + hemolysis_RBC. Conservative cutoff at >= 10% hemolysis at the top tested + concentration, matching the DBAASP/RBC literature convention for + "non-selective." + """ + if result.get("assay_type") != "hemolysis_RBC": + return None + value = result.get("result_value") + if value is None: + return None + try: + return float(value) >= HEMOLYSIS_HIGH_PCT + except (TypeError, ValueError): + return None + + +def _aggregate_per_candidate(results): + """Build per-candidate rollup of actual experimental outcomes. + + Returns dict mapping candidate_id -> { + n_results, n_mic, n_hemolysis, active_mic, high_hemolysis, ... + }. + Used as the "actual" side of the prediction-vs-actual join. + """ + out = {} + for r in results: + cid = r.get("candidate_id", "") + if not cid: + continue + slot = out.setdefault( + cid, + { + "n_results": 0, + "n_mic": 0, + "n_hemolysis": 0, + "n_cytotoxicity": 0, + "active_mic": None, + "high_hemolysis": None, + "any_active_qualitative": False, + "any_toxic_qualitative": False, + "all_controls_passed": True, + "assay_types": set(), + }, + ) + slot["n_results"] += 1 + slot["assay_types"].add(r.get("assay_type", "other")) + + active = _is_active_mic(r) + if active is not None: + slot["n_mic"] += 1 + if slot["active_mic"] is None: + slot["active_mic"] = active + else: + slot["active_mic"] = slot["active_mic"] or active + + high_hemo = _is_high_hemolysis(r) + if high_hemo is not None: + slot["n_hemolysis"] += 1 + if slot["high_hemolysis"] is None: + slot["high_hemolysis"] = high_hemo + else: + slot["high_hemolysis"] = slot["high_hemolysis"] or high_hemo + + if r.get("assay_type") == "cytotoxicity_mammalian": + slot["n_cytotoxicity"] += 1 + + qual = r.get("result_qualitative") + if qual == "active": + slot["any_active_qualitative"] = True + if qual == "toxic": + slot["any_toxic_qualitative"] = True + if not (r.get("positive_control_passed") and r.get("negative_control_passed")): + slot["all_controls_passed"] = False + + for v in out.values(): + v["assay_types"] = sorted(v["assay_types"]) + return out + + +def _cohort_metric( + matched, assay_type, predicate_lab, prediction_score_key, + predicate_pred_direction, threshold, +): + """Compute a single binary-prediction-vs-binary-actual cohort metric. + + Args: + matched: per-candidate joined rows (one per candidate with both sides). + assay_type: assay type to filter on (e.g. "MIC"). + predicate_lab: which "actual" boolean column to compare against + (e.g. "active_mic", "high_hemolysis"). + prediction_score_key: key into row["predictions"] holding the + pipeline score (e.g. "activity", "rich_selectivity"). + predicate_pred_direction: "above" or "below" - whether predicted score + above threshold means predicted positive. + threshold: pipeline score threshold. + + Returns: + Dict with TP, FP, FN, TN, n, insufficient_data flag, and disclaimer. + """ + rows = [m for m in matched if m.get("has_lab", {}).get(predicate_lab) is not None] + n = len(rows) + if n < MIN_COHORT_SIZE: + return { + "assay_type": assay_type, + "predicate": predicate_lab, + "prediction_score_key": prediction_score_key, + "n": n, + "min_required": MIN_COHORT_SIZE, + "insufficient_data": True, + "tp": None, + "fp": None, + "fn": None, + "tn": None, + "disclaimer": ( + f"Cohort too small for retrospective triage metric " + f"(n={n} < {MIN_COHORT_SIZE}). No point estimate reported. " + f"Collect more results before drawing conclusions." + ), + } + + tp = fp = fn = tn = 0 + considered = 0 + for row in rows: + actual_positive = bool(row["has_lab"][predicate_lab]) + pred_score = row["predictions"].get(prediction_score_key) + if pred_score is None: + continue + considered += 1 + if predicate_pred_direction == "above": + predicted_positive = pred_score >= threshold + else: + predicted_positive = pred_score < threshold + if predicted_positive and actual_positive: + tp += 1 + elif predicted_positive and not actual_positive: + fp += 1 + elif not predicted_positive and actual_positive: + fn += 1 + else: + tn += 1 + + return { + "assay_type": assay_type, + "predicate": predicate_lab, + "n": tp + fp + fn + tn, + "min_required": MIN_COHORT_SIZE, + "insufficient_data": False, + "tp": tp, + "fp": fp, + "fn": fn, + "tn": tn, + "disclaimer": ( + "Small retrospective cohort. Treat as descriptive only; not a " + "validation of the pipeline. Cluster effects and selection bias " + "from the pre-registered shortlist are NOT removed by this metric." + ), + } + + +def _per_candidate_rows(panel_rows, per_candidate_actuals): + """Join predictions with actuals on candidate_id. + + Returns one row per candidate in the panel. Candidates with no lab + results get ``has_lab = None``. + """ + rows = [] + for row in panel_rows: + cid = row.get("candidate_id", "") + actuals = per_candidate_actuals.get(cid) + rows.append( + { + "candidate_id": cid, + "sequence_preview": (row.get("sequence", "") or "")[:32], + "seed": row.get("seed", ""), + "length": row.get("length", ""), + "ensemble": _to_float(row.get("ensemble")), + "activity": _to_float(row.get("activity")), + "predictions": _candidate_predictions(row), + "has_lab": actuals, + "coverage_note": ( + "matched" + if actuals is not None + else "no lab results yet" + ), + } + ) + return rows + + +def build_calibration_intake_report(panel_csv, results_dir): + """Build a calibration-intake report from a pilot panel CSV + lab results dir. + + The report contains: + + - panel summary (n candidates, score coverage); + - lab results summary; + - per-candidate join rows (predictions + actuals); + - cohort triage metrics (gated by MIN_COHORT_SIZE); + - orphan-lab-result rows (results with no matching candidate); + - control-failure summary; + - disclaimer. + + Aggregate triage metrics are intentionally NOT computed when + ``len(matched) < MIN_COHORT_SIZE``. Below that threshold the cohort is + flagged ``insufficient_data: True`` to prevent small-sample theater. + """ + panel_rows = _load_panel_csv(panel_csv) + results = load_lab_results_dir(results_dir) + + per_candidate_actuals = _aggregate_per_candidate(results) + per_candidate = _per_candidate_rows(panel_rows, per_candidate_actuals) + + matched = [r for r in per_candidate if r["has_lab"] is not None] + orphan_ids = sorted(set(per_candidate_actuals) - {r["candidate_id"] for r in panel_rows}) + + control_failures = [ + { + "result_id": r["result_id"], + "candidate_id": r.get("candidate_id"), + "assay_type": r.get("assay_type"), + "positive_control_passed": r.get("positive_control_passed"), + "negative_control_passed": r.get("negative_control_passed"), + } + for r in results + if not (r.get("positive_control_passed") and r.get("negative_control_passed")) + ] + + cohort_metrics = { + "activity_vs_active_mic": _cohort_metric( + matched, + assay_type="MIC", + predicate_lab="active_mic", + prediction_score_key="activity", + predicate_pred_direction="above", + threshold=ACTIVITY_THRESHOLD, + ), + "rich_selectivity_vs_high_hemolysis": _cohort_metric( + matched, + assay_type="hemolysis_RBC", + predicate_lab="high_hemolysis", + prediction_score_key="rich_selectivity", + # rich_selectivity above 0.5 = predicted NEGATIVE for high_hemolysis + # (rich_selectivity measures "more selective = less hemolytic"), + # so predicted_positive (high hemolysis) only when score < 0.5. + predicate_pred_direction="below", + threshold=0.5, + ), + } + + return { + "panel_csv": str(panel_csv), + "results_dir": str(results_dir), + "n_panel_candidates": len(panel_rows), + "n_lab_results": len(results), + "n_matched_candidates": len(matched), + "n_orphan_lab_results": len(orphan_ids), + "orphan_candidate_ids": orphan_ids, + "summary": summarise_lab_results(results), + "per_candidate_outcomes": summarise_candidate_outcomes(results), + "per_candidate_joined": per_candidate, + "cohort_metrics": cohort_metrics, + "control_failures": control_failures, + "min_cohort_size": MIN_COHORT_SIZE, + "activity_threshold": ACTIVITY_THRESHOLD, + "mic_active_cutoff_ug_ml": MIC_ACTIVE_CUTOFF_UG_ML, + "hemolysis_high_pct": HEMOLYSIS_HIGH_PCT, + "report_disclaimer": _DISCLAIMER, + } + + +def write_calibration_intake_json(report, out_path): + """Write the machine-readable calibration intake JSON.""" + p = Path(out_path) + p.parent.mkdir(parents=True, exist_ok=True) + with p.open("w", encoding="utf-8") as f: + json.dump(report, f, indent=2, sort_keys=False) + + +def write_calibration_intake_markdown(report, out_path): + """Write a human-readable markdown calibration intake summary.""" + p = Path(out_path) + p.parent.mkdir(parents=True, exist_ok=True) + lines = [ + "# Calibration Intake Report", + "", + "> " + report["report_disclaimer"], + "", + "## Cohort Overview", + "", + f"- Panel candidates: **{report['n_panel_candidates']}**", + f"- Lab results loaded: **{report['n_lab_results']}**", + f"- Matched candidates: **{report['n_matched_candidates']}**", + f"- Orphan lab results (no panel match): **{report['n_orphan_lab_results']}**", + f"- Minimum cohort size for aggregate metrics: **{report['min_cohort_size']}**", + "", + "## Aggregate Cohort Metrics (gated by minimum sample size)", + "", + ] + for key, metric in report["cohort_metrics"].items(): + lines.append(f"### {key}") + lines.append("") + lines.append(f"- Assay type: `{metric['assay_type']}`") + lines.append(f"- Predicate: `{metric['predicate']}`") + lines.append(f"- Prediction score key: `{metric['prediction_score_key']}`") + lines.append(f"- n: **{metric['n']}** (min required: {metric['min_required']})") + if metric["insufficient_data"]: + lines.append(f"- **Status:** insufficient_data - {metric['disclaimer']}") + else: + lines.append( + f"- TP={metric['tp']}, FP={metric['fp']}, " + f"FN={metric['fn']}, TN={metric['tn']}" + ) + lines.append(f"- Note: {metric['disclaimer']}") + lines.append("") + + lines += [ + "## Per-Candidate Joined Rows", + "", + "| Candidate | Seed | Length | Ensemble | Activity | Has lab? |", + "|---|---|---:|---:|---:|---|", + ] + for row in report["per_candidate_joined"]: + has_lab = row["has_lab"] is not None + ensemble = row.get("ensemble") + activity = row.get("activity") + ens_str = f"{ensemble:.2f}" if isinstance(ensemble, (int, float)) else "-" + act_str = f"{activity:.2f}" if isinstance(activity, (int, float)) else "-" + lines.append( + f"| {row['candidate_id']} | {row['seed']} | {row['length']} " + f"| {ens_str} | {act_str} | {'yes' if has_lab else 'no'} |" + ) + + lines += [ + "", + "## Control Failures", + "", + ] + if report["control_failures"]: + lines.append("| Result | Candidate | Assay | Pos ctrl | Neg ctrl |") + lines.append("|---|---|---|---|---|") + for cf in report["control_failures"]: + lines.append( + f"| {cf['result_id']} | {cf['candidate_id']} | {cf['assay_type']} " + f"| {cf['positive_control_passed']} | {cf['negative_control_passed']} |" + ) + else: + lines.append("_None._") + + lines += [ + "", + "## Honest Limitations", + "", + "- All aggregate metrics are descriptive only. They do not validate the pipeline.", + "- Cohort metrics do NOT control for selection bias from the pre-registered shortlist.", + "- Below the minimum cohort size, no point estimate is reported. This prevents small-sample theater.", + "- Recalibration requires a separate human-reviewed decision log (see docs/DECISION_RULES.md).", + "- This report does NOT change scoring weights, ensemble composition, or selection rules.", + "", + ] + with p.open("w", encoding="utf-8") as f: + f.write("\n".join(lines)) diff --git a/src/openamp_foundry/cli/commands/reports.py b/src/openamp_foundry/cli/commands/reports.py index d4d535c9..06df060d 100644 --- a/src/openamp_foundry/cli/commands/reports.py +++ b/src/openamp_foundry/cli/commands/reports.py @@ -620,3 +620,42 @@ def _run_novelty_check_broad(args: argparse.Namespace) -> int: } print(json.dumps(summary, indent=2)) return 0 + + +def _run_calibration_intake(args: argparse.Namespace) -> int: + """Build a calibration intake report from a pilot panel CSV + lab results.""" + from openamp_foundry.calibration.intake import ( + build_calibration_intake_report, + write_calibration_intake_json, + write_calibration_intake_markdown, + ) + + report = build_calibration_intake_report(args.panel, args.results_dir) + write_calibration_intake_json(report, args.out_json) + if args.out_md: + write_calibration_intake_markdown(report, args.out_md) + + cohort_summary = {} + for key, metric in report["cohort_metrics"].items(): + cohort_summary[key] = { + "n": metric["n"], + "insufficient_data": metric["insufficient_data"], + } + + print( + json.dumps( + { + "status": "ok", + "n_panel_candidates": report["n_panel_candidates"], + "n_lab_results": report["n_lab_results"], + "n_matched_candidates": report["n_matched_candidates"], + "n_orphan_lab_results": report["n_orphan_lab_results"], + "cohort_metrics": cohort_summary, + "out_json": args.out_json, + "out_md": args.out_md, + "disclaimer_excerpt": report["report_disclaimer"][:80] + "...", + }, + indent=2, + ) + ) + return 0 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 1bd4ee22..3b5df867 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -5,7 +5,7 @@ from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc -from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report +from openamp_foundry.cli.commands.reports import _run_reviewer_questionnaire, _run_ip_report, _run_batch_pack, _run_gold_standard, _run_novelty_check_broad, _run_lab_result_report, _run_calibration_intake from openamp_foundry.cli.commands.gates import _run_gate_check import argparse @@ -658,6 +658,37 @@ def build_parser() -> argparse.ArgumentParser: help="Optional output path for markdown review report.", ) + calibration_intake = sub.add_parser( + "calibration-intake", + help=( + "Join a pilot panel CSV (computational predictions) with a directory of " + "validated lab result JSON files (experimental outcomes) and produce a " + "calibration intake report. Descriptive only. Does NOT trigger " + "recalibration, weight updates, or selection-rule changes. Below the " + "minimum cohort size no aggregate point estimate is reported." + ), + ) + calibration_intake.add_argument( + "--panel", + required=True, + help="Pilot panel CSV (candidate_id + score columns).", + ) + calibration_intake.add_argument( + "--results-dir", + required=True, + help="Directory containing lab result JSON files matching schemas/lab_result.schema.json.", + ) + calibration_intake.add_argument( + "--out-json", + required=True, + help="Output path for machine-readable calibration intake JSON.", + ) + calibration_intake.add_argument( + "--out-md", + required=False, + help="Optional output path for markdown calibration intake summary.", + ) + novelty_broad = sub.add_parser( "novelty-check-broad", help=( @@ -770,6 +801,9 @@ def main(argv: list[str] | None = None) -> int: if args.command == "lab-result-report": return _run_lab_result_report(args) + if args.command == "calibration-intake": + return _run_calibration_intake(args) + if args.command == "synthesis-order": return _run_synthesis_order(args) diff --git a/tests/test_calibration_intake.py b/tests/test_calibration_intake.py new file mode 100644 index 00000000..a678c413 --- /dev/null +++ b/tests/test_calibration_intake.py @@ -0,0 +1,576 @@ +"""Tests for the calibration intake workflow. + +Verifies: + - Lab result files load + validate against schema + - Pilot panel CSV loads + predictions are extracted + - Per-candidate join correctly attaches lab results to predictions + - Cohort metrics below MIN_COHORT_SIZE are flagged insufficient_data + - Cohort metrics at/above MIN_COHORT_SIZE produce TP/FP/FN/TN + - Control failures are surfaced + - Orphan lab results are detected + - JSON and Markdown output writers are non-empty and validate + - Synthetic example data exists and validates + +Honest limitation: + No real wet-lab data is available. All tests use clearly synthetic JSON + in tmp_path. Tests must NEVER rely on real-world MIC/hemolysis numbers + to pass. +""" +from __future__ import annotations + +import csv +import json +import warnings +from pathlib import Path + +import pytest + +from openamp_foundry.calibration.intake import ( + ACTIVITY_THRESHOLD, + HEMOLYSIS_HIGH_PCT, + MIC_ACTIVE_CUTOFF_UG_ML, + MIN_COHORT_SIZE, + build_calibration_intake_report, + write_calibration_intake_json, + write_calibration_intake_markdown, +) + +SCHEMA_PATH = Path(__file__).parent.parent / "schemas" / "lab_result.schema.json" + + +def _lab_result(**overrides) -> dict: + base = { + "result_id": "RES-TEST-001", + "candidate_id": "TEST-CAND-001", + "assay_type": "MIC", + "organism_or_cell_line": "SYNTHETIC TEST - E. coli ATCC 25922", + "result_value": 8.0, + "result_unit": "µg/mL", + "result_qualitative": "active", + "positive_control_passed": True, + "negative_control_passed": True, + "positive_control_id": "ciprofloxacin 0.25 µg/mL", + "negative_control_id": "PBS", + "assay_date": "2026-07-04", + "replicate_count": 3, + "performed_by_lab": "SYNTHETIC TEST - Lab X", + "raw_data_sha256": None, + "computational_candidate_certificate_hash": ( + "0000000000000000000000000000000000000000000000000000000000000000" + ), + "notes": "SYNTHETIC TEST DATA - not a real assay.", + "disclaimer": ( + "SYNTHETIC TEST. This is not a real experimental result on a " + "computationally nominated candidate and does not constitute a " + "drug or clinical claim." + ), + } + base.update(overrides) + return base + + +def _write_panel_csv(panel_csv: Path, rows: list[dict]) -> None: + fields = [ + "pilot_rank", "candidate_id", "sequence", "length", "seed", + "ensemble", "activity", "boman_activity", "disagreement", + "safety", "synthesis", "novelty", "serum_stability", + "selectivity_proxy", "rich_selectivity", "pilot_priority", + "amphipathic_score", "charge_ph74", + ] + with panel_csv.open("w", newline="", encoding="utf-8") as f: + w = csv.DictWriter(f, fieldnames=fields) + w.writeheader() + for r in rows: + w.writerow(r) + + +def _write_lab_result_file(directory: Path, result: dict) -> Path: + p = directory / f"{result['result_id']}.json" + p.write_text(json.dumps(result)) + return p + + +@pytest.fixture +def empty_panel_csv(tmp_path) -> Path: + p = tmp_path / "panel.csv" + _write_panel_csv(p, []) + return p + + +@pytest.fixture +def empty_results_dir(tmp_path) -> Path: + d = tmp_path / "results" + d.mkdir() + return d + + +class TestLoadPanelCsv: + def test_empty_panel(self, empty_panel_csv, empty_results_dir): + report = build_calibration_intake_report(empty_panel_csv, empty_results_dir) + assert report["n_panel_candidates"] == 0 + assert report["n_lab_results"] == 0 + assert report["n_matched_candidates"] == 0 + assert report["per_candidate_joined"] == [] + + def test_panel_rows_parsed(self, tmp_path): + panel = tmp_path / "panel.csv" + results = tmp_path / "results" + results.mkdir() + _write_panel_csv( + panel, + [ + { + "pilot_rank": "1", + "candidate_id": "CAND-A", + "sequence": "AAAKKKFFF", + "length": "9", + "seed": "SEED-A", + "ensemble": "0.81", + "activity": "0.85", + "boman_activity": "0.78", + "disagreement": "0.07", + "safety": "0.90", + "synthesis": "0.85", + "novelty": "0.70", + "serum_stability": "0.65", + "selectivity_proxy": "0.60", + "rich_selectivity": "0.72", + "pilot_priority": "0.80", + "amphipathic_score": "1.5", + "charge_ph74": "4.0", + }, + ], + ) + report = build_calibration_intake_report(panel, results) + assert report["n_panel_candidates"] == 1 + row = report["per_candidate_joined"][0] + assert row["candidate_id"] == "CAND-A" + assert row["predictions"]["activity"] == 0.85 + assert row["ensemble"] == 0.81 + assert row["has_lab"] is None + assert row["coverage_note"] == "no lab results yet" + + def test_missing_columns_are_tolerated(self, tmp_path): + """A CSV missing optional score columns must not crash the loader.""" + panel = tmp_path / "panel.csv" + results = tmp_path / "results" + results.mkdir() + with panel.open("w", newline="", encoding="utf-8") as f: + w = csv.DictWriter(f, fieldnames=["pilot_rank", "candidate_id", "sequence"]) + w.writeheader() + w.writerow({"pilot_rank": "1", "candidate_id": "CAND-A", "sequence": "AAAKKK"}) + report = build_calibration_intake_report(panel, results) + assert report["n_panel_candidates"] == 1 + assert report["per_candidate_joined"][0]["predictions"]["activity"] is None + + +class TestLoadLabResults: + def test_valid_result_loads(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + _write_lab_result_file(results, _lab_result()) + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + report = build_calibration_intake_report(panel, results) + assert report["n_lab_results"] == 1 + + def test_invalid_result_skipped_with_warning(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + # Valid file + _write_lab_result_file(results, _lab_result()) + # Invalid file (missing required field) + (results / "bad.json").write_text( + json.dumps({"result_id": "BAD-001"}) + ) + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + report = build_calibration_intake_report(panel, results) + assert report["n_lab_results"] == 1 + assert any("Skipped" in str(w.message) for w in caught) + + +class TestPerCandidateJoin: + def test_match_by_candidate_id(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv( + panel, + [ + { + "pilot_rank": "1", "candidate_id": "CAND-A", "sequence": "AAA", + "length": "3", "seed": "SEED-X", "ensemble": "0.80", + "activity": "0.82", "boman_activity": "0.78", + "disagreement": "0.04", "safety": "0.90", "synthesis": "0.85", + "novelty": "0.70", "serum_stability": "0.65", + "selectivity_proxy": "0.60", "rich_selectivity": "0.70", + "pilot_priority": "0.78", "amphipathic_score": "1.5", + "charge_ph74": "4.0", + }, + { + "pilot_rank": "2", "candidate_id": "CAND-B", "sequence": "BBB", + "length": "3", "seed": "SEED-Y", "ensemble": "0.70", + "activity": "0.72", "boman_activity": "0.70", + "disagreement": "0.02", "safety": "0.85", "synthesis": "0.80", + "novelty": "0.65", "serum_stability": "0.60", + "selectivity_proxy": "0.55", "rich_selectivity": "0.65", + "pilot_priority": "0.68", "amphipathic_score": "1.4", + "charge_ph74": "3.5", + }, + ], + ) + _write_lab_result_file( + results, _lab_result(result_id="RES-001", candidate_id="CAND-A") + ) + report = build_calibration_intake_report(panel, results) + assert report["n_matched_candidates"] == 1 + assert report["per_candidate_joined"][0]["has_lab"] is not None + assert report["per_candidate_joined"][1]["has_lab"] is None + + def test_orphan_lab_results_detected(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + _write_lab_result_file( + results, _lab_result(result_id="RES-A", candidate_id="CAND-NOT-IN-PANEL") + ) + report = build_calibration_intake_report(panel, results) + assert report["n_orphan_lab_results"] == 1 + assert "CAND-NOT-IN-PANEL" in report["orphan_candidate_ids"] + + +class TestPerCandidateActuals: + def test_active_mic_below_cutoff(self): + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [_lab_result(assay_type="MIC", result_value=8.0, candidate_id="C1")] + ) + assert out["C1"]["active_mic"] is True + + def test_inactive_mic_above_cutoff(self): + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [_lab_result(assay_type="MIC", result_value=128.0, candidate_id="C1")] + ) + assert out["C1"]["active_mic"] is False + + def test_mic_at_cutoff_is_active(self): + """MIC == 32 ug/mL is the documented activity threshold (inclusive).""" + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [_lab_result(assay_type="MIC", result_value=32.0, candidate_id="C1")] + ) + assert out["C1"]["active_mic"] is True + + def test_high_hemolysis_detected(self): + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [ + _lab_result( + assay_type="hemolysis_RBC", + result_value=42.0, + result_qualitative="toxic", + candidate_id="C1", + ) + ] + ) + assert out["C1"]["high_hemolysis"] is True + + def test_low_hemolysis_detected(self): + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [ + _lab_result( + assay_type="hemolysis_RBC", + result_value=3.5, + result_qualitative="inactive", + candidate_id="C1", + ) + ] + ) + assert out["C1"]["high_hemolysis"] is False + + def test_hemolysis_at_threshold(self): + """Hemolysis == 10% is the threshold (>=10 is high).""" + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [ + _lab_result( + assay_type="hemolysis_RBC", + result_value=10.0, + candidate_id="C1", + ) + ] + ) + assert out["C1"]["high_hemolysis"] is True + + def test_qualitative_only_mic_returns_none(self): + """When MIC is qualitative-only (result_value=None), predicate is None.""" + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [_lab_result(assay_type="MIC", result_value=None, candidate_id="C1")] + ) + assert out["C1"]["active_mic"] is None + + def test_controls_failed_flagged(self): + from openamp_foundry.calibration.intake import _aggregate_per_candidate + + out = _aggregate_per_candidate( + [ + _lab_result( + positive_control_passed=False, + negative_control_passed=True, + candidate_id="C1", + ) + ] + ) + assert out["C1"]["all_controls_passed"] is False + + +class TestCohortMetricsGating: + def _make_setup(self, tmp_path, n_mic: int): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + rows = [] + for i in range(n_mic): + cid = f"CAND-{i:03d}" + rows.append( + { + "pilot_rank": str(i + 1), + "candidate_id": cid, + "sequence": "AAAKKKFFF" + str(i), + "length": "10", + "seed": f"SEED-{i}", + "ensemble": "0.82" if i % 2 == 0 else "0.55", + "activity": "0.85" if i % 2 == 0 else "0.45", + "boman_activity": "0.80", + "disagreement": "0.05", + "safety": "0.90", + "synthesis": "0.85", + "novelty": "0.70", + "serum_stability": "0.65", + "selectivity_proxy": "0.60", + "rich_selectivity": "0.72", + "pilot_priority": "0.80", + "amphipathic_score": "1.5", + "charge_ph74": "4.0", + } + ) + _write_lab_result_file( + results, + _lab_result( + result_id=f"RES-{i:03d}", + candidate_id=cid, + assay_type="MIC", + # Half active (MIC=8), half inactive (MIC=128) + result_value=8.0 if i % 2 == 0 else 128.0, + result_qualitative="active" if i % 2 == 0 else "inactive", + ), + ) + _write_panel_csv(panel, rows) + return panel, results + + def test_below_min_cohort_size_marked_insufficient(self, tmp_path): + panel, results = self._make_setup(tmp_path, n_mic=MIN_COHORT_SIZE - 1) + report = build_calibration_intake_report(panel, results) + metric = report["cohort_metrics"]["activity_vs_active_mic"] + assert metric["insufficient_data"] is True + assert metric["n"] == MIN_COHORT_SIZE - 1 + assert metric["tp"] is None + + def test_at_min_cohort_size_reports_counts(self, tmp_path): + panel, results = self._make_setup(tmp_path, n_mic=MIN_COHORT_SIZE) + report = build_calibration_intake_report(panel, results) + metric = report["cohort_metrics"]["activity_vs_active_mic"] + assert metric["insufficient_data"] is False + assert metric["n"] == MIN_COHORT_SIZE + # With our alternating scheme, predicted_positive == actual_positive + # for every other candidate, so we expect pure TP + TN. + assert metric["tp"] + metric["tn"] + metric["fp"] + metric["fn"] == MIN_COHORT_SIZE + + def test_above_min_cohort_size_computes_auroc_proxy(self, tmp_path): + """With more samples, we should see TP+TN+FP+FN == n.""" + panel, results = self._make_setup(tmp_path, n_mic=10) + report = build_calibration_intake_report(panel, results) + metric = report["cohort_metrics"]["activity_vs_active_mic"] + assert metric["insufficient_data"] is False + assert metric["n"] == 10 + assert metric["tp"] + metric["tn"] + metric["fp"] + metric["fn"] == 10 + + def test_hemolysis_cohort_uses_below_threshold_direction(self, tmp_path): + """rich_selectivity above 0.5 = predicted NEGATIVE for high_hemolysis. + + So predicted_positive (high hemolysis) only when score < 0.5. + """ + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + rows = [] + for i in range(6): + cid = f"CAND-H{i}" + # Half low rich_selectivity (predict high hemolysis), half high. + rich_sel = "0.3" if i < 3 else "0.8" + rows.append( + { + "pilot_rank": str(i + 1), "candidate_id": cid, + "sequence": "AAAK", "length": "4", "seed": "SEED-H", + "ensemble": "0.7", "activity": "0.7", "boman_activity": "0.7", + "disagreement": "0.0", "safety": "0.9", "synthesis": "0.9", + "novelty": "0.7", "serum_stability": "0.6", + "selectivity_proxy": "0.6", "rich_selectivity": rich_sel, + "pilot_priority": "0.7", "amphipathic_score": "1.0", + "charge_ph74": "3.0", + } + ) + _write_lab_result_file( + results, + _lab_result( + result_id=f"RES-H{i}", + candidate_id=cid, + assay_type="hemolysis_RBC", + result_value=50.0 if i < 3 else 2.0, + result_qualitative="toxic" if i < 3 else "inactive", + ), + ) + _write_panel_csv(panel, rows) + report = build_calibration_intake_report(panel, results) + metric = report["cohort_metrics"]["rich_selectivity_vs_high_hemolysis"] + assert metric["insufficient_data"] is False + assert metric["n"] == 6 + + +class TestControlFailures: + def test_control_failure_surfaced(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + _write_lab_result_file( + results, + _lab_result( + positive_control_passed=False, + negative_control_passed=True, + ), + ) + report = build_calibration_intake_report(panel, results) + assert len(report["control_failures"]) == 1 + assert report["control_failures"][0]["positive_control_passed"] is False + + def test_no_control_failures_returns_empty(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + _write_lab_result_file(results, _lab_result()) + report = build_calibration_intake_report(panel, results) + assert report["control_failures"] == [] + + +class TestWriters: + def test_json_writer_produces_valid_file(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + _write_lab_result_file(results, _lab_result()) + report = build_calibration_intake_report(panel, results) + out = tmp_path / "out.json" + write_calibration_intake_json(report, out) + assert out.exists() + data = json.loads(out.read_text()) + assert data["n_lab_results"] == 1 + assert "report_disclaimer" in data + + def test_markdown_writer_produces_nonempty_file(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + _write_lab_result_file(results, _lab_result()) + report = build_calibration_intake_report(panel, results) + out = tmp_path / "out.md" + write_calibration_intake_markdown(report, out) + assert out.exists() + text = out.read_text() + assert "Calibration Intake Report" in text + assert "insufficient_data" in text or "TP=" in text + assert "Honest Limitations" in text + + +class TestDisclaimer: + def test_disclaimer_present(self, tmp_path): + results = tmp_path / "results" + results.mkdir() + panel = tmp_path / "panel.csv" + _write_panel_csv(panel, []) + report = build_calibration_intake_report(panel, results) + d = report["report_disclaimer"] + assert "NOT proof of efficacy" in d + assert "recalibration" in d.lower() + assert "selection rule" in d + + +class TestExampleData: + """Verify the synthetic example data validates and produces a clean report.""" + + @pytest.fixture + def example_root(self): + return Path(__file__).parent.parent / "examples" + + def test_synthetic_results_validate_against_schema(self, example_root): + from openamp_foundry.evidence.schemas import validate_json_schema + + results_dir = example_root / "lab_results" + assert results_dir.exists(), "examples/lab_results/ should exist" + for p in sorted(results_dir.glob("*.json")): + data = json.loads(p.read_text()) + validate_json_schema(data, SCHEMA_PATH) + + def test_synthetic_panel_exists(self, example_root): + panel = example_root / "lab_results_panel.csv" + assert panel.exists() + with panel.open() as f: + reader = csv.DictReader(f) + rows = list(reader) + assert len(rows) >= 5 + + def test_synthetic_example_produces_report(self, example_root, tmp_path): + panel = example_root / "lab_results_panel.csv" + results_dir = example_root / "lab_results" + report = build_calibration_intake_report(panel, results_dir) + assert report["n_panel_candidates"] >= 5 + assert report["n_lab_results"] >= 5 + assert report["n_matched_candidates"] >= 1 + # Honest: with synthetic data this should be insufficient + assert all( + m["insufficient_data"] + for m in report["cohort_metrics"].values() + ) + + def test_readme_warns_synthetic(self, example_root): + readme = example_root / "lab_results" / "README.md" + assert readme.exists() + text = readme.read_text() + assert "SYNTHETIC" in text + assert "not" in text.lower() # Must explicitly disclaim real-world use + + +class TestConstants: + def test_thresholds_match_docs(self): + """Threshold values are exposed as constants and match the documentation.""" + assert ACTIVITY_THRESHOLD == 0.70 + assert MIC_ACTIVE_CUTOFF_UG_ML == 32.0 + assert HEMOLYSIS_HIGH_PCT == 10.0 + assert MIN_COHORT_SIZE >= 5 # minimum honest gate