feat: selectivity benchmark — safety_score AUROC=0.54 (FAILED) reveals critical gap

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
+.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict validate-selectivity external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
 
 PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
 PYTEST  := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
@@ -26,6 +26,7 @@ help:
 	@echo "  make validate-scoring         AUROC on 95 AMP + 96 decoy (pipeline config)"
 	@echo "  make validate-scoring-phase3  AUROC with phase3.yaml config"
 	@echo "  make validate-scoring-strict  AUROC with scrambled-decoy strict benchmark"
+	@echo "  make validate-selectivity    Selectivity benchmark: safety vs hemolytic AMPs"
 	@echo "  make bench-leakage            Check for near-duplicates between candidates and refs"
 	@echo "  make bench-baseline           Hidden-positive recovery benchmark (demo set)"
 	@echo "  make bench-hidden-active      Hidden-positive recovery on mixed benchmark set"
@@ -141,6 +142,9 @@ validate-scoring-strict:
 		--benchmark-type strict \
 		--out outputs/validate_scoring_strict_report.json
 
+validate-selectivity:
+	PYTHONPATH=src $(PYTHON) -c "from openamp_foundry.cli import main; main(['validate-selectivity', '--out', 'outputs/selectivity_report.json'])"
+
 external-predict: pilot
 	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli external-predict \
 		--pilot-csv outputs/pilot_panel.csv \

docs/ARCHITECTURE.md

@@ -39,7 +39,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic
 | `openamp_foundry.scoring` | activity, safety, novelty, synthesis, ensemble scoring |
 | `openamp_foundry.selection` | ranking and diversity selection |
 | `openamp_foundry.evidence` | JSON certificate generation and validation |
-| `openamp_foundry.benchmark` | leakage checks and evaluation scaffolding |
+| `openamp_foundry.benchmark` | leakage checks, retrospective AUROC, selectivity benchmark, evaluation scaffolding |
 | `openamp_foundry.generators` | safe, bounded toy candidate generation |
 | `openamp_foundry.simulation` | membrane/selectivity/stability proxy modeling (scaffolded) |
 

docs/BENCHMARK_CARD.md

@@ -1,7 +1,7 @@
 # Benchmark Card — OpenAMP Foundry v0.5.x
 
 > **Purpose:** Single-page summary of benchmark methodology, data, and metrics.
-> **Last updated:** 2026-06-29 (PR #110 expanded benchmark)
+> **Last updated:** 2026-07-01 (selectivity benchmark added)
 
 ---
 
@@ -50,14 +50,32 @@ Confidence intervals: percentile bootstrap (2000 resamples, seed=0).
 | Bias | Impact | Mitigation |
 |------|--------|------------|
 | Helical-centric scorer | β-sheet AMPs (defensins) score below panel | Noted in METRICS_CURRENT.md |
-| Melittin safety blind spot | Safety=1.0 despite strong hemolysis | Hemolysis assay mandatory for all |
+| Melittin safety blind spot | Safety=1.0 despite strong hemolysis; 5/8 hemolytic AMPs score safety=1.0 (selectivity AUROC=0.54) | Hemolysis assay mandatory for all; safety_score redesign identified as improvement target |
 | Composition-matched scrambled AUROC < 0.5 | Model relies on composition > order; fails strict order-sensitivity test | OpenAMP is an **evidence-ranking tool**, not a validated sequence-order activity predictor. Composition-scrambled test confirms the model captures AMP-like composition, not sequence-order features. |
 | Near-seed generation only | Novel sequence space not explored | Acknowledged limitation in all docs |
 
 ## Classification
 
 OpenAMP performs well on standard decoys (AUROC 0.7832) but **fails the strict composition-scrambled benchmark (AUROC 0.4335, below random)**. This means the model primarily detects AMP-like amino acid composition rather than sequence-order-dependent antimicrobial features. This is expected for a physicochemical heuristic scorer and is appropriate for its intended use: **triage and ranking**, not deep biological prediction.
 
+## Selectivity Benchmark
+
+> Tests whether safety features distinguish selective from hemolytic AMPs.
+> Panel: 10 selective + 8 hemolytic AMPs (literature hemolysis classifications).
+> Run: `make validate-selectivity`
+
+| Feature | AUROC | Verdict |
+|---------|:-----:|---------|
+| safety_score | 0.54 | **FAILED** — near random |
+| selectivity_proxy | 0.61 | Weak |
+| hydrophobic_fraction | 0.81 | Strong naive baseline |
+| GRAVY | 0.76 | Strong naive baseline |
+| hydrophobic_moment | 0.61 | Weak |
+
+**Finding:** safety_score does NOT discriminate selective from hemolytic AMPs.
+5/8 hemolytic AMPs score safety=1.0 (melittin, mastoparan-X, PMAP-23, bombolitin-II, polybia-MP1).
+hydrophobic_fraction is the best single discriminator. Improvement target identified.
+
 ## Historical Baseline
 
 | Point | Set | Pipeline AUROC | Phase3 AUROC |
@@ -71,4 +89,5 @@ OpenAMP performs well on standard decoys (AUROC 0.7832) but **fails the strict c
 
 | Date | Change | Author |
 |------|--------|--------|
+| 2026-07-01 | Selectivity benchmark added — safety_score AUROC=0.54 | OpenAMP CI |
 | 2026-06-29 | Initial card — expanded benchmark (PR #110) | OpenAMP CI |

docs/METRICS_CURRENT.md

@@ -3,7 +3,7 @@
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-06-29 (PR #110 expanded benchmark)
+> **Last updated:** 2026-07-01 (selectivity benchmark added)
 > **Pipeline version:** v0.5.x
 > **Branch:** main
 
@@ -27,6 +27,43 @@
 | Recall@43 | 0.4211 | 40/95 positives in top 43 |
 | Interpretation | **STRONG** | AUROC > 0.70 gate passed |
 
+## Selectivity Benchmark Metrics
+
+> Tests whether pipeline safety features can distinguish selective AMPs (low hemolysis)
+> from hemolytic AMPs. Reference panel: 10 selective + 8 hemolytic AMPs with literature
+> hemolysis classifications. See `examples/validation/selectivity_panel.csv`.
+> Run: `make validate-selectivity`
+
+| Feature | AUROC | Direction | Notes |
+|---------|-------|-----------|-------|
+| **safety_score** | **0.5437** | Higher = safer | **FAILED** — near random; does not discriminate selective from hemolytic |
+| selectivity_proxy | 0.6125 | Higher = more selective | Weak signal above random |
+| hydrophobic_fraction | 0.8063 | Higher = more hemolytic | **Best discriminator** — naive baseline outperforms designed safety score |
+| hydrophobic_moment (μH) | 0.6125 | Higher = more hemolytic | Weak signal |
+| charge_density_ph74 | 0.5437 | Higher = more hemolytic | Near random |
+| GRAVY | 0.7625 | Higher = more hemolytic | Strong naive baseline |
+
+### Key findings
+
+1. **safety_score does NOT discriminate selective from hemolytic AMPs (AUROC=0.54, near random).**
+   Five of eight hemolytic AMPs receive safety_score=1.0 (maximum safety), including melittin
+   (HC50 ~ 5 ug/mL). The safety filter's thresholds for hydrophobic moment, charge density, and
+   hydrophobic fraction are too permissive for short helical peptides.
+
+2. **hydrophobic_fraction is the strongest single discriminator (AUROC=0.81).** Hemolytic AMPs
+   have higher hydrophobic fraction (mean 0.55 vs 0.35 for selective). The safety_score's
+   threshold (> 0.65) is set too high to catch this difference.
+
+3. **5 known blind spots identified:** melittin, mastoparan-X, PMAP-23, bombolitin-II,
+   polybia-MP1 all score safety=1.0 despite being literature-hemolytic. These are short
+   helical peptides where the hemolytic mechanism (oligomeric pore formation, toroidal
+   insertion) is not captured by 1D hydrophobic moment analysis.
+
+4. **Implication for candidate selection:** The current safety filter should NOT be trusted
+   as a hemolysis proxy. Wet-lab hemolysis assay remains mandatory for all candidates.
+   Improvement target: incorporate hydrophobic_fraction threshold lowering or add a
+   multi-residue hydrophobic face analysis to the safety score.
+
 ### Historical baselines
 
 | Point | Benchmark | AUROC | Phase3 AUROC | Source |

docs/ROADMAP.md

@@ -148,9 +148,21 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 
 - ~~write the first explicit simulator scope document: what OpenAMP will model and what it will not;~~
 - ~~add membrane/selectivity/stability proxy interfaces with uncertainty fields;~~
-- benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;
+- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ **Done (2026-07-01):** `benchmark/selectivity.py` + `examples/validation/selectivity_panel.csv` — 10 selective + 8 hemolytic AMPs. Key finding: safety_score AUROC=0.54 (FAILED), hydrophobic_fraction AUROC=0.81 (best baseline). 5 blind spots identified.
 - require every proxy module to justify itself against cheap heuristic baselines before it affects selection.
 
+## v1.1.x — Safety score improvement (identified by selectivity benchmark)
+
+The selectivity benchmark revealed that safety_score does not discriminate selective from
+hemolytic AMPs (AUROC=0.54, near random). Improvement targets:
+
+- Lower hydrophobic_fraction threshold in safety_score from 0.65 to ~0.50 (captures the
+  selective/hemolytic boundary at hydrophobic_fraction ~0.45-0.55)
+- Add a multi-residue hydrophobic face coverage metric (current μH threshold misses
+  short helical peptides where hemolysis arises from oligomeric pore formation)
+- Re-run selectivity benchmark after any safety_score change to verify improvement
+- Target: safety_score AUROC >= 0.70 on the selectivity panel
+
 ## v2.x — Wet-lab compression engine
 
 - calibrate virtual assay layers on qualified assay outcomes;

examples/validation/selectivity_panel.csv

@@ -0,0 +1,19 @@
+id,sequence,family,hemolysis_class,reference
+SEL-MAG2,GIGKFLHSAKKFGKAFVGEIMNS,magainin,selective,Zasloff_1987_PNAS_HC50_gt_1mg_mL
+SEL-CECA,KWKLFKKIEKVGQNIRDGIIKAGPAV,cecropin,selective,Steiner_1981_PNAS_low_hemolysis
+SEL-BUF2,RAGLQFPVGRVHRLLRK,buforin,selective,Park_2000_JBC_non_lytic
+SEL-HIS5,DSHAKRHHGYKRKFHEKHHSHRGY,histatin,selective,Oppenheim_1988_JBC_low_hemolysis
+SEL-APIA,GNNRPVYIPQPRPPHPRI,apidaecin,selective,Casteels_1989_EMBO_low_hemolysis
+SEL-DROS,GKPRPYSPRPTSHPRPIRV,drosocin,selective,Bulet_1993_JBC_low_hemolysis
+SEL-AUR1,GLFDIIKKIAESF,aurein,selective,Rozek_2000_low_hemolysis
+SEL-IND,ILPWKWPWWPWRR,indolicidin,selective,Selsted_1992_JBC_moderate_selectivity
+SEL-PLEC,GFGCNGPWNEDDLRCHNHCKSIKGYKGGYCAKGGFVCKCY,plectasin,selective,Mygind_2005_Nature_low_hemolysis
+SEL-TMPG,ILSKIFSLLKKVLP,temporin_G,selective,Simmaco_1996_low_hemolysis
+HLY-MEL,GIGAVLKVLTTGLPALISWIKRKRQQ,melittin,hemolytic,Habermann_1972_HC50_5_ug_mL
+HLY-MPX,INWKGIAAMAKKLL,mastoparan_x,hemolytic,Higashijima_1990_hemolytic
+HLY-BMP28,GGLRSLGRKILRAWKKYGPIIVPIIRI,bmap28,hemolytic,Skerlavaj_1996_HC50_5_uM
+HLY-PMAP23,RIIDLLWRVRRPQKPKFVTVWV,pmap23,hemolytic,Zanetti_1994_hemolytic
+HLY-BOMB2,IKAILIKKILKGV,bombolitin_ii,hemolytic,Argiolas_1985_hemolytic
+HLY-PMP1,IDWKKLLDAAKQIL,polybia_mp1,hemolytic,Souza_2005_hemolytic
+HLY-LL37,LLGDFFRKSKEKIGKEFKRIVQRIKDFLRNLVPRTES,ll37,hemolytic,Turner_1998_hemolytic_high_conc
+HLY-MPN,INLKALAALAKKLL,mastoparan,hemolytic,Hirai_1979_hemolytic

src/openamp_foundry/benchmark/__init__.py

@@ -0,0 +1 @@
+"""Benchmark module: leakage checks, retrospective AUROC, selectivity benchmark, evaluation."""