diff --git a/Makefile b/Makefile index 2cf76853..a1b44eda 100644 --- a/Makefile +++ b/Makefile @@ -154,6 +154,10 @@ bench-selectivity: --hemolysis-csv examples/validation/hemolysis_reference.csv --out outputs/selectivity_benchmark_report.json + +bench-triage: + PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench triage \ + --out outputs/triage_benchmark_report.json validate-scoring-strict: PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli validate-scoring \ --amp-csv examples/validation/known_amps.csv \ diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index bcbfcef6..06906174 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -3,7 +3,7 @@ > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-01 (hemolysis benchmark expanded to 238 peptides) +> **Last updated:** 2026-07-01 (triage benchmark added) > **Pipeline version:** v0.5.x > **Branch:** main @@ -375,6 +375,70 @@ hemolysis mechanism is not fully captured by 1D features. --- +## Multi-Class Triage Benchmark (v0.5.12 — added 2026-07-01) + +> Tests the v1.1 ROADMAP item: "benchmark candidate triage against a reference +> panel that includes selective AMPs, hemolytic positives, inactive peptides, +> and random controls." Prior benchmarks tested two separate 2-class problems +> (AMP vs decoy, hemolytic vs selective). This benchmark tests the combined +> triage task the virtual assay layer must solve: rank selective AMPs above +> hemolytic AMPs above random decoys in a single panel. +> +> Run: `make bench-triage` + +**Dataset:** 125 selective AMPs (HC50 >= 100 µg/mL) + 45 hemolytic AMPs (HC50 < 25 µg/mL) ++ 96 random background decoys = 266 total. + +### Per-scorer pairwise AUROCs + +A scorer that triages correctly should have all three AUROCs > 0.5: + - selective > decoy (identifies AMPs) + - hemolytic > decoy (identifies AMPs) + - selective > hemolytic (prefers safe AMPs) + +| Scorer | sel > decoy | hem > decoy | sel > hem | Triages correctly? | +|--------|:-----------:|:-----------:|:---------:|:------------------:| +| ensemble | 0.848 | 0.891 | 0.466 | **NO** (anti-selective) | +| activity | 0.885 | 0.934 | 0.430 | NO | +| selectivity_proxy | 0.782 | 0.795 | 0.610 | **YES** | +| triage_score (activity × (1 - hemo_risk)) | 0.863 | 0.902 | 0.462 | NO | +| safe_weighted_ensemble | 0.849 | 0.890 | 0.483 | NO | +| safety | 0.344 | 0.300 | 0.538 | NO | +| synthesis | 0.590 | 0.634 | 0.469 | NO | +| hemolysis_risk (inverted) | 0.485 | 0.492 | 0.488 | NO | +| serum_stability | 0.217 | 0.160 | 0.569 | NO | + +**Key findings:** + +1. **The ensemble does NOT triage correctly.** It ranks hemolytic AMPs above + selective AMPs (sel_vs_hem AUROC = 0.466 < 0.5). This is the anti-selective + bias documented in the selectivity benchmark, now confirmed in the combined + triage context. + +2. **Only selectivity_proxy triages correctly** (all three AUROCs > 0.5). It + sacrifices some decoy discrimination (0.782 vs ensemble's 0.848) but is the + only scorer that prefers selective AMPs over hemolytic ones. + +3. **The naive triage_score (activity × (1 - hemolysis_risk)) does NOT fix the + anti-selective bias** (sel_vs_hem = 0.462). This is because hemolysis_risk + is too weak (detection AUROC 0.565, not significant on expanded benchmark). + A naive virtual-assay composite does not outperform the ensemble. + +4. **Top-20 distribution shift:** The triage_score moves 2 more selective AMPs + into the top-20 (16 vs 14 for ensemble), removing 2 hemolytic AMPs (4 vs 6). + The shift is in the right direction but modest — the hemolysis_risk penalty + is weak. + +**Implication for the virtual assay layer:** Any future virtual assay module +must beat this triage benchmark baseline. The minimum bar is: triage correctly +(all three AUROCs > 0.5) while maintaining ensemble-level decoy discrimination +(sel_vs_decoy > 0.80). The selectivity_proxy achieves correct triage but at +the cost of decoy discrimination. A successful virtual assay must achieve both. + +**Honest limitation:** The benchmark uses literature HC50 values with high +inter-assay variability. The binary thresholds (25 / 100 µg/mL) are coarse. +The MODERATE class (HC50 25-100, n=68) is excluded from the binary task. + ## Test Suite | Metric | Value | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 48435aa6..01a1bb58 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -185,6 +185,30 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - Blind spots: 14 hemolytic AMPs with safety >= 0.8 (melittin, indolicidin, protegrins, tachyplesins, piscidin-1, arenicin, polyphemusin, gramicidin-S-like, BMAP fragment) - 18 new tests in `test_selectivity_benchmark.py`; 1453 total tests +## v0.5.12 — Multi-Class Triage Benchmark ✓ (2026-07-01) + +- **Multi-class triage benchmark** (`benchmark/triage.py`): tests whether the + pipeline can rank selective AMPs > hemolytic AMPs > random decoys in a single + combined panel (125 selective + 45 hemolytic + 96 decoys = 266 total). +- Addresses the v1.1 ROADMAP item: "benchmark candidate triage against a + reference panel that includes selective AMPs, hemolytic positives, inactive + peptides, and random controls." +- **Key finding:** The ensemble does NOT triage correctly. It ranks hemolytic + AMPs above selective AMPs (AUROC 0.466 < 0.5) due to the activity scorer's + anti-selective bias. Hemolytic AMPs have stronger amphipathic helices, higher + hydrophobic moment, and higher charge — exactly the features the activity + scorer rewards. +- **Key finding:** Only `selectivity_proxy` triages correctly (all three pairwise + AUROCs > 0.5), but at the cost of decoy discrimination (0.782 vs ensemble 0.848). +- **Key finding:** The naive `triage_score` (activity × (1 - hemolysis_risk)) + does NOT fix the anti-selective bias because hemolysis_risk is too weak + (detection AUROC 0.565, not significant on expanded benchmark). +- The `triage_score` does shift the top-20 distribution favorably (16 selective + vs 14 for ensemble; 4 hemolytic vs 6), but the pairwise AUROC remains < 0.5. +- CLI: `openamp-foundry bench triage --hemolysis-csv ... --decoy-csv ...` +- Makefile: `make bench-triage` +- 16 new tests (15 triage benchmark + 1 CLI integration); 1487 tests total. + ## v1.0 — Validated dry-lab-to-wet-lab loop - independently reviewed assay batch (expert_review.yml GitHub issue template); @@ -197,7 +221,7 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - ~~write the first explicit simulator scope document: what OpenAMP will model and what it will not;~~ - ~~add membrane/selectivity/stability proxy interfaces with uncertainty fields;~~ -- benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls; +- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ (v0.5.12: triage benchmark added — ensemble fails, selectivity_proxy only correct triager) - require every proxy module to justify itself against cheap heuristic baselines before it affects selection. ## v2.x — Wet-lab compression engine diff --git a/src/openamp_foundry/benchmark/triage.py b/src/openamp_foundry/benchmark/triage.py new file mode 100644 index 00000000..c9c39772 --- /dev/null +++ b/src/openamp_foundry/benchmark/triage.py @@ -0,0 +1,300 @@ +"""Multi-class triage benchmark: can the pipeline rank selective AMPs above +hemolytic AMPs above random decoys in a single combined panel? + +This benchmark directly addresses the v1.1 ROADMAP item: + "benchmark candidate triage against a reference panel that includes + selective AMPs, hemolytic positives, inactive peptides, and random controls." + +The existing benchmarks test two separate 2-class problems: + - AMP vs decoy (retrospective AUROC) + - Hemolytic vs selective (within-AMP selectivity) + +But the virtual assay layer must solve a *combined* triage task: given a mixed +panel, rank candidates that are both active AND selective above those that are +active but hemolytic, and both above non-AMP random controls. No existing +benchmark tests this. + +The benchmark assembles three classes from existing datasets: + 1. SELECTIVE — AMPs with HC50 >= 100 µg/mL (from hemolysis_reference.csv) + 2. HEMOLYTIC — AMPs with HC50 < 25 µg/mL (from hemolysis_reference.csv) + 3. DECOY — random background peptides (from random_background.csv) + +For each pair of classes, we compute AUROC. The key question is whether the +pipeline ranks SELECTIVE > HEMOLYTIC (currently it does NOT — the ensemble has +an anti-selective bias documented in METRICS_CURRENT.md). + +We also compute a "triage score" = activity * (1 - hemolysis_risk), which +combines activity with hemolysis penalty. This is the simplest possible +virtual-assay-style composite, and the benchmark tests whether it improves +triage over the raw ensemble. + +All results are computational. No biological activity is implied. +""" +from __future__ import annotations + +import csv +from pathlib import Path +from typing import Any + +from openamp_foundry.benchmark.retrospective import _auc_wilcoxon, _bootstrap_auroc_ci +from openamp_foundry.config import load_config +from openamp_foundry.features.physchem import compute_features +from openamp_foundry.scoring.activity import activity_likeness_score +from openamp_foundry.scoring.boman import boman_activity_score, model_disagreement +from openamp_foundry.scoring.ensemble import ensemble_score +from openamp_foundry.scoring.hemolysis import hemolysis_risk_score +from openamp_foundry.scoring.novelty import novelty_score +from openamp_foundry.scoring.safety import safety_score +from openamp_foundry.scoring.stability import serum_stability_score +from openamp_foundry.scoring.synthesis import synthesis_feasibility_score + + +def _load_hemolysis_reference(path: str | Path) -> list[dict[str, Any]]: + """Load hemolysis_reference.csv and return rows with parsed hc50 and class.""" + rows = [] + with open(path, newline="", encoding="utf-8") as f: + for row in csv.DictReader(f): + seq = row["sequence"].strip().upper() + if not all(aa in "ACDEFGHIKLMNPQRSTVWY" for aa in seq): + continue + hc50 = float(row["hc50_ugml"]) + hemo_class = row["hemolysis_class"].strip().upper() + rows.append({ + "id": row["id"], + "sequence": seq, + "family": row.get("family", ""), + "hc50_ugml": hc50, + "hemolysis_class": hemo_class, + "reference": row.get("reference", ""), + }) + return rows + + +def _load_random_background(path: str | Path) -> list[dict[str, Any]]: + """Load random_background.csv.""" + rows = [] + with open(path, newline="", encoding="utf-8") as f: + for row in csv.DictReader(f): + seq = row["sequence"].strip().upper() + if not all(aa in "ACDEFGHIKLMNPQRSTVWY" for aa in seq): + continue + rows.append({ + "id": row["id"], + "sequence": seq, + "family": row.get("family", "background"), + "source": row.get("source", ""), + }) + return rows + + +def _score_all(rows: list[dict[str, Any]], weights: dict[str, float]) -> list[dict[str, Any]]: + """Score all rows with the full pipeline + triage composite.""" + scored = [] + for row in rows: + seq = row["sequence"] + features = compute_features(seq) + act = activity_likeness_score(features) + safe = safety_score(features) + synth = synthesis_feasibility_score(features, valid_sequence=True) + nov, _ = novelty_score(seq, []) + boman_act = boman_activity_score(seq) + stability = serum_stability_score(features) + sel_proxy = float(features.get("selectivity_proxy", 0.0)) + hemo_risk = hemolysis_risk_score(features) + raw = { + "activity": act, "safety": safe, + "synthesis": synth, "novelty": nov, + "boman_activity": boman_act, + "disagreement": model_disagreement(act, boman_act), + "serum_stability": stability, + "selectivity_proxy": sel_proxy, + } + ens = ensemble_score(raw, weights) + # Triage composite: activity penalized by hemolysis risk. + # This is the simplest virtual-assay-style score: it rewards activity + # but penalizes candidates with high hemolysis risk. + triage_score = round(act * (1.0 - hemo_risk), 4) + # Safety-weighted ensemble: same weights but safety replaced by + # safety * (1 - hemolysis_risk) to penalize hemolytic candidates. + safe_adjusted = round(safe * (1.0 - hemo_risk), 4) + raw_adjusted = dict(raw) + raw_adjusted["safety"] = safe_adjusted + safe_weighted_ens = ensemble_score(raw_adjusted, weights) + scored.append({ + **row, + "ensemble": round(ens, 4), + "activity": round(act, 4), + "safety": round(safe, 4), + "synthesis": round(synth, 4), + "boman_activity": round(boman_act, 4), + "serum_stability": round(stability, 4), + "selectivity_proxy": round(sel_proxy, 4), + "hemolysis_risk": round(hemo_risk, 4), + "triage_score": triage_score, + "safe_weighted_ensemble": round(safe_weighted_ens, 4), + }) + return scored + + +def _pairwise_auroc( + pos_scores: list[float], neg_scores: list[float], n_bootstrap: int = 2000, +) -> dict[str, Any]: + """Compute AUROC and bootstrap CI for a pair of score distributions.""" + auroc = round(_auc_wilcoxon(pos_scores, neg_scores), 4) + ci = _bootstrap_auroc_ci(pos_scores, neg_scores, n_bootstrap=n_bootstrap) + return { + "auroc": auroc, + "ci_lo": ci["ci_lo"], + "ci_hi": ci["ci_hi"], + "n_pos": len(pos_scores), + "n_neg": len(neg_scores), + } + + +def run_triage_benchmark( + hemolysis_csv: str | Path, + decoy_csv: str | Path, + config_path: str | Path = "configs/pipeline.yaml", + n_bootstrap: int = 2000, +) -> dict[str, Any]: + """Run the multi-class triage benchmark. + + Tests whether the pipeline can rank: + SELECTIVE AMPs > HEMOLYTIC AMPs > DECOY random peptides + + For each scorer, computes pairwise AUROC for all three class pairs. + A scorer that triages correctly should have: + - AUROC(selective vs decoy) > 0.5 (selective rank above decoy) + - AUROC(hemolytic vs decoy) > 0.5 (hemolytic rank above decoy) + - AUROC(selective vs hemolytic) > 0.5 (selective rank above hemolytic) + + The last condition is the hardest: the existing ensemble has an anti-selective + bias (it ranks hemolytic AMPs above selective AMPs because hemolytic AMPs have + stronger amphipathic helices). + + Returns a dict with per-scorer pairwise AUROCs, class distributions, and + an honest verdict. + """ + config = load_config(config_path) + weights = config["weights"] + + hemo_rows = _load_hemolysis_reference(hemolysis_csv) + decoy_rows = _load_random_background(decoy_csv) + + selective = [r for r in hemo_rows if r["hemolysis_class"] == "SELECTIVE"] + hemolytic = [r for r in hemo_rows if r["hemolysis_class"] == "HEMOLYTIC"] + decoys = decoy_rows + + selective_scored = _score_all(selective, weights) + hemolytic_scored = _score_all(hemolytic, weights) + decoy_scored = _score_all(decoys, weights) + + scorers = [ + "ensemble", "activity", "safety", "synthesis", + "selectivity_proxy", "hemolysis_risk", "serum_stability", + "triage_score", "safe_weighted_ensemble", + ] + + per_scorer = {} + for scorer in scorers: + sel_vals = [s[scorer] for s in selective_scored] + hemo_vals = [s[scorer] for s in hemolytic_scored] + decoy_vals = [s[scorer] for s in decoy_scored] + + # For hemolysis_risk, lower is better (selective should be lower) + # For all others, higher is better + if scorer == "hemolysis_risk": + # Invert: 1 - score so higher = better (lower risk) + sel_vals = [1.0 - v for v in sel_vals] + hemo_vals = [1.0 - v for v in hemo_vals] + decoy_vals = [1.0 - v for v in decoy_vals] + + sel_vs_decoy = _pairwise_auroc(sel_vals, decoy_vals, n_bootstrap) + hemo_vs_decoy = _pairwise_auroc(hemo_vals, decoy_vals, n_bootstrap) + sel_vs_hemo = _pairwise_auroc(sel_vals, hemo_vals, n_bootstrap) + + # Triage success: all three AUROCs > 0.5 + all_above = all( + r["auroc"] > 0.5 + for r in [sel_vs_decoy, hemo_vs_decoy, sel_vs_hemo] + ) + # Strong triage: all three > 0.60 + all_strong = all( + r["auroc"] > 0.60 + for r in [sel_vs_decoy, hemo_vs_decoy, sel_vs_hemo] + ) + + per_scorer[scorer] = { + "selective_vs_decoy": sel_vs_decoy, + "hemolytic_vs_decoy": hemo_vs_decoy, + "selective_vs_hemolytic": sel_vs_hemo, + "triages_correctly": all_above, + "strong_triage": all_strong, + } + + # Identify the best triage scorer + best_scorer = max( + per_scorer, + key=lambda k: ( + per_scorer[k]["selective_vs_hemolytic"]["auroc"] + + per_scorer[k]["selective_vs_decoy"]["auroc"] + ) / 2, + ) + + # Per-sequence detail (top/bottom from each class) + all_scored = selective_scored + hemolytic_scored + decoy_scored + for s in all_scored: + s["triage_class"] = ( + "SELECTIVE" if s.get("hemolysis_class") == "SELECTIVE" + else "HEMOLYTIC" if s.get("hemolysis_class") == "HEMOLYTIC" + else "DECOY" + ) + + # Rank by ensemble and show class distribution in top-k + by_ensemble = sorted(all_scored, key=lambda x: x["ensemble"], reverse=True) + top_20_classes = [r["triage_class"] for r in by_ensemble[:20]] + top_20_dist = { + "SELECTIVE": top_20_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_classes.count("HEMOLYTIC"), + "DECOY": top_20_classes.count("DECOY"), + } + + by_triage = sorted(all_scored, key=lambda x: x["triage_score"], reverse=True) + top_20_triage_classes = [r["triage_class"] for r in by_triage[:20]] + top_20_triage_dist = { + "SELECTIVE": top_20_triage_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_triage_classes.count("HEMOLYTIC"), + "DECOY": top_20_triage_classes.count("DECOY"), + } + + return { + "benchmark": "multi_class_triage", + "n_selective": len(selective_scored), + "n_hemolytic": len(hemolytic_scored), + "n_decoy": len(decoy_scored), + "n_total": len(all_scored), + "per_scorer": per_scorer, + "best_scorer": best_scorer, + "top_20_by_ensemble": top_20_dist, + "top_20_by_triage_score": top_20_triage_dist, + "known_blind_spots": [ + "The ensemble ranks hemolytic AMPs above selective AMPs " + "(anti-selective bias) because hemolytic AMPs have stronger " + "amphipathic helices, higher hydrophobic moment, and higher charge " + "— exactly the features the activity scorer rewards.", + "The safety scorer cannot distinguish hemolytic from selective AMPs " + "(detection AUROC ~0.51 on expanded benchmark).", + "The hemolysis_risk scorer has only weak directional signal " + "(detection AUROC ~0.57, CI includes 0.5 on expanded benchmark).", + "The triage_score (activity * (1 - hemolysis_risk)) is a simple " + "composite that may not outperform the ensemble on selective_vs_decoy " + "because hemolysis_risk is not a significant detector.", + ], + "disclaimer": ( + "These are computational benchmark results on literature-curated " + "AMP sequences and random decoy peptides. They do not prove " + "biological activity, safety, or hemolysis. They measure whether " + "the pipeline's scorers can triage a mixed panel correctly. " + "Wet-lab validation remains mandatory for all candidates." + ), + } diff --git a/src/openamp_foundry/cli/commands/benchmark.py b/src/openamp_foundry/cli/commands/benchmark.py index 7721eefd..e7445a3d 100644 --- a/src/openamp_foundry/cli/commands/benchmark.py +++ b/src/openamp_foundry/cli/commands/benchmark.py @@ -182,3 +182,40 @@ def _run_selectivity_bench(args: argparse.Namespace) -> int: } print(_json.dumps(summary, indent=2)) return 0 + + +def _run_triage(args: argparse.Namespace) -> int: + import json + from openamp_foundry.benchmark.triage import run_triage_benchmark + from openamp_foundry.utils.io import write_json + + result = run_triage_benchmark( + hemolysis_csv=args.hemolysis_csv, + decoy_csv=args.decoy_csv, + config_path=args.config, + n_bootstrap=args.n_bootstrap, + ) + if args.out: + write_json(args.out, result) + summary = { + "status": "ok", + "benchmark": result["benchmark"], + "n_selective": result["n_selective"], + "n_hemolytic": result["n_hemolytic"], + "n_decoy": result["n_decoy"], + "best_scorer": result["best_scorer"], + "per_scorer": { + s: { + "selective_vs_decoy": info["selective_vs_decoy"]["auroc"], + "hemolytic_vs_decoy": info["hemolytic_vs_decoy"]["auroc"], + "selective_vs_hemolytic": info["selective_vs_hemolytic"]["auroc"], + "triages_correctly": info["triages_correctly"], + } + for s, info in result["per_scorer"].items() + }, + "top_20_by_ensemble": result["top_20_by_ensemble"], + "top_20_by_triage_score": result["top_20_by_triage_score"], + "out": args.out, + } + print(json.dumps(summary, indent=2)) + return 0 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index c80cac2c..baa7ecb7 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -1,7 +1,7 @@ from __future__ import annotations from openamp_foundry.cli.commands.core import _run_generate_batch -from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench, _run_expert_ablation_bench, _run_selectivity_bench +from openamp_foundry.cli.commands.benchmark import _run_bench, _run_validate_scoring, _run_cluster_split_bench, _run_expert_ablation_bench, _run_selectivity_bench, _run_triage from openamp_foundry.cli.commands.selection import _run_pilot_panel, _run_pilot_confident, _run_diversity_check from openamp_foundry.cli.commands.external import _run_external_predict, _run_external_consensus from openamp_foundry.cli.commands.qc import _run_synthesis_order, _run_presynth_qc @@ -170,6 +170,32 @@ def build_parser() -> argparse.ArgumentParser: help="Optional JSON output path.", ) + triage = bench_sub.add_parser( + "triage", + help=( + "Multi-class triage benchmark: can the pipeline rank " + "selective AMPs > hemolytic AMPs > random decoys in a combined panel? " + "Tests the v1.1 ROADMAP triage benchmark requirement." + ), + ) + triage.add_argument( + "--hemolysis-csv", + default="examples/validation/hemolysis_reference.csv", + help="Hemolysis reference CSV (id,sequence,family,hc50_ugml,hemolysis_class,reference).", + ) + triage.add_argument( + "--decoy-csv", + default="examples/validation/random_background.csv", + help="Random background decoy CSV (id,sequence,family,source,label).", + ) + triage.add_argument("--config", default="configs/pipeline.yaml") + triage.add_argument("--n-bootstrap", type=int, default=2000) + triage.add_argument( + "--out", + required=False, + help="Optional JSON output path.", + ) + generate = sub.add_parser( "generate-batch", help=( @@ -619,6 +645,8 @@ def main(argv: list[str] | None = None) -> int: return _run_expert_ablation_bench(args) if args.bench_command == "selectivity": return _run_selectivity_bench(args) + if args.bench_command == "triage": + return _run_triage(args) return _run_bench(args) if args.command == "generate-batch": diff --git a/tests/test_cli_triage.py b/tests/test_cli_triage.py new file mode 100644 index 00000000..a939d9ba --- /dev/null +++ b/tests/test_cli_triage.py @@ -0,0 +1,35 @@ +"""CLI integration test for the triage benchmark command.""" + +import json +from pathlib import Path + +import pytest + +from openamp_foundry.cli.main import main + + +def test_bench_triage_cli(tmp_path, capsys): + """bench triage CLI must return per-scorer pairwise AUROCs.""" + hemo_csv = Path("examples/validation/hemolysis_reference.csv") + decoy_csv = Path("examples/validation/random_background.csv") + if not hemo_csv.exists() or not decoy_csv.exists(): + pytest.skip("Triage benchmark data not found — run from project root") + out = str(tmp_path / "triage_report.json") + rc = main([ + "bench", "triage", + "--hemolysis-csv", str(hemo_csv), + "--decoy-csv", str(decoy_csv), + "--n-bootstrap", "100", + "--out", out, + ]) + assert rc == 0 + captured = capsys.readouterr() + data = json.loads(captured.out) + assert data["status"] == "ok" + assert data["benchmark"] == "multi_class_triage" + assert data["n_selective"] > 0 + assert data["n_hemolytic"] > 0 + assert data["n_decoy"] > 0 + assert "ensemble" in data["per_scorer"] + assert "triage_score" in data["per_scorer"] + assert "selective_vs_hemolytic" in data["per_scorer"]["ensemble"] diff --git a/tests/test_triage_benchmark.py b/tests/test_triage_benchmark.py new file mode 100644 index 00000000..a357cb17 --- /dev/null +++ b/tests/test_triage_benchmark.py @@ -0,0 +1,180 @@ +"""Tests for the multi-class triage benchmark. + +Tests the v1.1 ROADMAP item: "benchmark candidate triage against a reference +panel that includes selective AMPs, hemolytic positives, inactive peptides, +and random controls." + +The benchmark assembles three classes: + 1. SELECTIVE — AMPs with HC50 >= 100 µg/mL (from hemolysis_reference.csv) + 2. HEMOLYTIC — AMPs with HC50 < 25 µg/mL (from hemolysis_reference.csv) + 3. DECOY — random background peptides (from random_background.csv) + +A scorer that triages correctly should rank: + SELECTIVE > HEMOLYTIC > DECOY + +All results are computational. No biological activity is implied. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openamp_foundry.benchmark.triage import run_triage_benchmark + + +HEMOLYSIS_CSV = "examples/validation/hemolysis_reference.csv" +DECOY_CSV = "examples/validation/random_background.csv" + + +@pytest.fixture(scope="class") +def result(): + """Run the triage benchmark once for all tests in this class.""" + hemo = Path(HEMOLYSIS_CSV) + decoy = Path(DECOY_CSV) + if not hemo.exists() or not decoy.exists(): + pytest.skip("Triage benchmark data not found — run from project root") + return run_triage_benchmark(hemo, decoy, n_bootstrap=200) + + +class TestTriageBenchmarkStructure: + """Verify the benchmark returns well-formed results.""" + + def test_returns_required_keys(self, result): + for key in ["benchmark", "n_selective", "n_hemolytic", "n_decoy", + "n_total", "per_scorer", "best_scorer", + "top_20_by_ensemble", "top_20_by_triage_score", + "known_blind_spots", "disclaimer"]: + assert key in result, f"Missing key: {key}" + + def test_benchmark_name(self, result): + assert result["benchmark"] == "multi_class_triage" + + def test_class_counts_positive(self, result): + assert result["n_selective"] > 0 + assert result["n_hemolytic"] > 0 + assert result["n_decoy"] > 0 + assert result["n_total"] == result["n_selective"] + result["n_hemolytic"] + result["n_decoy"] + + def test_per_scorer_has_all_scorers(self, result): + expected = { + "ensemble", "activity", "safety", "synthesis", + "selectivity_proxy", "hemolysis_risk", "serum_stability", + "triage_score", "safe_weighted_ensemble", + } + assert expected.issubset(set(result["per_scorer"].keys())) + + def test_each_scorer_has_pairwise_aurocs(self, result): + for scorer, info in result["per_scorer"].items(): + for pair in ["selective_vs_decoy", "hemolytic_vs_decoy", "selective_vs_hemolytic"]: + assert pair in info, f"{scorer} missing {pair}" + assert 0.0 <= info[pair]["auroc"] <= 1.0 + assert info[pair]["n_pos"] > 0 + assert info[pair]["n_neg"] > 0 + + def test_disclaimer_present(self, result): + assert "computational" in result["disclaimer"].lower() + assert "wet-lab" in result["disclaimer"].lower() or "wet lab" in result["disclaimer"].lower() + + def test_known_blind_spots_documented(self, result): + assert len(result["known_blind_spots"]) >= 3 + + +class TestTriageBenchmarkFindings: + """Test the key honest findings from the benchmark.""" + + def test_ensemble_fails_selective_vs_hemolytic(self, result): + """The ensemble should NOT rank selective above hemolytic. + + This is the documented anti-selective bias: the activity scorer + rewards hemolytic AMPs for their strong amphipathic helices. + The ensemble inherits this bias. + """ + sel_vs_hemo = result["per_scorer"]["ensemble"]["selective_vs_hemolytic"]["auroc"] + assert sel_vs_hemo < 0.55, ( + f"ensemble selective_vs_hemolytic AUROC={sel_vs_hemo:.4f}: " + "expected < 0.55 (ensemble has anti-selective bias). " + "If this has improved, update this test." + ) + + def test_ensemble_beats_decoy(self, result): + """The ensemble should still rank both AMP classes above decoys.""" + sel_vs_dec = result["per_scorer"]["ensemble"]["selective_vs_decoy"]["auroc"] + hemo_vs_dec = result["per_scorer"]["ensemble"]["hemolytic_vs_decoy"]["auroc"] + assert sel_vs_dec > 0.70, ( + f"ensemble selective_vs_decoy AUROC={sel_vs_dec:.4f}: should be > 0.70" + ) + assert hemo_vs_dec > 0.70, ( + f"ensemble hemolytic_vs_decoy AUROC={hemo_vs_dec:.4f}: should be > 0.70" + ) + + def test_ensemble_does_not_triage_correctly(self, result): + """The ensemble should NOT triage all three classes correctly. + + It fails because selective_vs_hemolytic < 0.5. + This is the key honest finding that motivates the virtual assay layer. + """ + assert result["per_scorer"]["ensemble"]["triages_correctly"] is False, ( + "Ensemble should not triage correctly (anti-selective bias). " + "If this has changed, the pipeline has improved and this test should be updated." + ) + + def test_selectivity_proxy_triages_correctly(self, result): + """The selectivity_proxy should be the only scorer that triages correctly. + + It ranks selective > hemolytic because it penalizes high-GRAVY peptides. + """ + triaging = [ + s for s, info in result["per_scorer"].items() + if info["triages_correctly"] + ] + assert "selectivity_proxy" in triaging, ( + "selectivity_proxy should triage correctly (selective > hemolytic > decoy)" + ) + + def test_triage_score_does_not_fix_anti_selective_bias(self, result): + """The simple triage_score (activity * (1 - hemolysis_risk)) should NOT fix + the anti-selective bias because hemolysis_risk is too weak. + + This is an honest negative result: a naive virtual-assay composite + does not outperform the ensemble on the hardest triage condition. + """ + triage_sel_vs_hemo = result["per_scorer"]["triage_score"]["selective_vs_hemolytic"]["auroc"] + ens_sel_vs_hemo = result["per_scorer"]["ensemble"]["selective_vs_hemolytic"]["auroc"] + # triage_score may be slightly better or worse, but should not dramatically fix the bias + assert triage_sel_vs_hemo < 0.55, ( + f"triage_score selective_vs_hemolytic AUROC={triage_sel_vs_hemo:.4f}: " + "expected < 0.55 (hemolysis_risk too weak to fix anti-selective bias). " + f"ensemble was {ens_sel_vs_hemo:.4f}. " + "If this has improved, the triage composite is working and this test should be updated." + ) + + def test_top_20_by_ensemble_contains_no_decoys(self, result): + """The ensemble top-20 should contain only AMPs (no decoys). + + This confirms the ensemble can distinguish AMPs from decoys even if + it cannot distinguish selective from hemolytic. + """ + assert result["top_20_by_ensemble"]["DECOY"] == 0, ( + "Ensemble top-20 should contain no decoys" + ) + + def test_top_20_by_triage_contains_no_decoys(self, result): + """The triage_score top-20 should also contain no decoys.""" + assert result["top_20_by_triage_score"]["DECOY"] == 0, ( + "Triage score top-20 should contain no decoys" + ) + + def test_top_20_by_triage_has_more_selective_than_ensemble(self, result): + """The triage_score should rank MORE selective AMPs in top-20 than the ensemble. + + This is the value proposition of the triage composite: even if the + pairwise AUROC doesn't dramatically improve, the top-k distribution + should shift toward selective candidates. + """ + ens_sel = result["top_20_by_ensemble"]["SELECTIVE"] + tri_sel = result["top_20_by_triage_score"]["SELECTIVE"] + assert tri_sel >= ens_sel, ( + f"triage_score top-20 selective={tri_sel} should be >= " + f"ensemble top-20 selective={ens_sel}" + )