feat(bench): within-AMP selectivity benchmark — hemolytic vs selective AMP discrimination

CHANGELOG.md

@@ -4,6 +4,37 @@ All notable changes to OpenAMP Foundry are documented here.
 
 ---
 
+## [Unreleased — Within-AMP Selectivity Benchmark] — 2026-07-01
+
+### v0.5.9 — Within-AMP Selectivity Benchmark
+
+- **Hemolysis reference dataset**: 42 known AMPs with literature HC50 values
+  (`examples/validation/hemolysis_reference.csv`): 14 HEMOLYTIC (HC50 < 25 µg/mL),
+  21 SELECTIVE (HC50 >= 100 µg/mL), 7 BORDER (25-100 µg/mL)
+- **`run_selectivity_benchmark()`** in `benchmark/retrospective.py`: tests whether
+  pipeline scorers can distinguish hemolytic from selective AMPs — the within-AMP
+  ranking task that the expert ablation identified as the correct test for
+  safety/selectivity/synthesis components
+- **`bench selectivity` CLI command** + `make bench-selectivity` Makefile target
+- **18 new tests** in `test_selectivity_benchmark.py`; 1453 total tests
+
+**Key findings:**
+
+1. Safety scorer FAILS hemolysis detection (detection AUROC=0.3844, CI lo=0.26).
+   All 14 hemolytic AMPs score safety >= 0.8. Melittin blind spot confirmed
+   quantitatively.
+2. Selectivity proxy FAILS hemolysis detection (detection AUROC=0.4133, CI lo=0.28).
+   Charge/GRAVY heuristic insufficient.
+3. Synthesis feasibility is the only significant hemolysis risk detector
+   (detection AUROC=0.8027, CI lo=0.63) — incidental correlation via cysteine
+   content and repeat runs in hemolytic AMPs.
+4. Expert composite is better than ensemble on hemolysis detection (0.5119 vs
+   0.3486) but not significant (CI includes 0.5) at n=14 vs n=21.
+5. Activity scorer is anti-selective (detection AUROC=0.34): ranks hemolytic
+   AMPs higher due to stronger amphipathic helices. Ensemble inherits this bias.
+
+---
+
 ## [Unreleased — Pre-synthesis Quality Sprint] — 2026-06
 
 **Status:** Pipeline frozen for synthesis batch ordering. All changes below were quality

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
+.PHONY: help demo test lint ci clean bench-leakage bench-baseline bench-hidden-active bench-cluster-split bench-expert-ablation bench-selectivity generate phase3 pilot validate-scoring validate-scoring-phase3 validate-scoring-strict external-predict pilot-confident presynth-qc gold-standard diversity synthesis-order novelty-broad external-consensus questionnaire gate-check ip-report benchmark-card wave0-5-gate-check wave0-5-novelty-audit wave0-5-novelty-audit-v2 wave0-5-panel wave0-5-evidence wave0-5-fill-external wave0-5b-generate wave0-5b-filter
 
 PYTHON := $(shell [ -f .venv/bin/python ] && echo .venv/bin/python || echo python3)
 PYTEST  := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pytest)
@@ -30,6 +30,7 @@ help:
 	@echo "  make bench-baseline           Hidden-positive recovery benchmark (demo set)"
 	@echo "  make bench-cluster-split      Cluster-aware bootstrap CI (de-inflates near-duplicates)"
 	@echo "  make bench-expert-ablation    Expert composite vs ensemble ablation (honesty check)"
+	@echo "  make bench-selectivity        Within-AMP selectivity (hemolytic vs selective)"
 	@echo "  make bench-hidden-active      Hidden-positive recovery on mixed benchmark set"
 	@echo ""
 	@echo "  make wave0-5-gate-check     Run Wave 0.5 gates W0.5-1 through W0.5-7"
@@ -148,6 +149,11 @@ bench-expert-ablation:
 		--decoy-csv examples/validation/random_background.csv 
 		--out outputs/expert_ablation_report.json
 
+bench-selectivity:
+	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench selectivity 
+		--hemolysis-csv examples/validation/hemolysis_reference.csv 
+		--out outputs/selectivity_benchmark_report.json
+
 validate-scoring-strict:
 	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli validate-scoring \
 		--amp-csv examples/validation/known_amps.csv \

docs/ARCHITECTURE.md

@@ -39,7 +39,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic
 | `openamp_foundry.scoring` | activity, safety, novelty, synthesis, ensemble scoring |
 | `openamp_foundry.selection` | ranking and diversity selection |
 | `openamp_foundry.evidence` | JSON certificate generation and validation |
-| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, expert ablation benchmark, and evaluation scaffolding |
+| `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, expert ablation benchmark, within-AMP selectivity benchmark, and evaluation scaffolding |
 | `openamp_foundry.generators` | safe, bounded toy candidate generation |
 | `openamp_foundry.simulation` | membrane/selectivity/stability proxy modeling (scaffolded) |
 

docs/BENCHMARKING.md

@@ -25,6 +25,7 @@ Common artifacts:
 | Hidden-positive recovery | Check whether known actives are ranked high |
 | Toxicity down-ranking | Ensure predicted risky candidates are penalized |
 | Novelty stress test | Avoid near-clones of references |
+| Within-AMP selectivity | Test whether scorers distinguish hemolytic from selective AMPs |
 
 ## Minimum report fields
 

docs/EXPERT_REVIEW_PACK.md

@@ -295,7 +295,7 @@ The expert reviewer must be aware of these limitations:
 |-----------|--------|
 | Level 0–2 evidence only | Scores are physicochemical heuristics, not validated ML predictions |
 | AUROC = 0.7832 on n=191 (expanded) | CI₉₅: 0.72–0.84; 52 new AMPs from 12 taxonomic classes now included (defensins, proline-rich, lantibiotics) — more honest estimate but broader AMP classes still challenge the helic-centric scorer |
-| Safety model blind spot | Melittin scores Safety=1.0; curvature-mediated hemolysis not captured; hemolysis assay mandatory |
+| Safety model blind spot | Melittin scores Safety=1.0; confirmed quantitatively in v0.5.9 selectivity benchmark: safety scorer detection AUROC=0.3844 (14 hemolytic vs 21 selective AMPs); all 14 hemolytic AMPs score safety >= 0.8; hemolysis assay mandatory |
 | SEED-008 high disagreement (0.41–0.44) | Boman index penalises Trp; activity_likeness more appropriate for Trp-rich sequences |
 | Serum stability model calibrated for 18–30 AA | SEED-003 (11 AA), SEED-007 (11 AA), and SEED-008 (13 AA) scores may underestimate actual stability (short-peptide steric/flexibility effects) |
 | Pro-rich serum stability not in model | SEED-009 model scores (0.572) expected to underestimate actual stability; Pro-X bond resistance is mechanistic |

docs/METRICS_CURRENT.md

@@ -224,16 +224,93 @@ frequencies tend to have more moderate biophysical properties.
 only. The expert composite's selectivity, safety, and synthesis components are
 designed for within-AMP candidate differentiation, not for separating AMPs from
 non-AMPs. A within-AMP ranking benchmark (comparing selective vs hemolytic AMPs)
-would be a more appropriate test for those components and is a candidate for the
-next loop.
+has been added in v0.5.9 (see Within-AMP Selectivity Benchmark section below).
 
 ---
 
+
+## Within-AMP Selectivity Benchmark (v0.5.x — added 2026-07-01)
+
+> The expert ablation benchmark found that safety, synthesis, and serum stability are
+> anti-signal for AMP-vs-decoy discrimination. But those scorers were designed for
+> *within-AMP ranking*: distinguishing hemolytic AMPs from selective AMPs. This benchmark
+> tests them on that intended task.
+>
+> Run: `make bench-selectivity`
+
+**Dataset:** 42 known AMPs with literature HC50 values (hemolysis_reference.csv)
+
+| Class | HC50 threshold | Count |
+|-------|:--------------:|:-----:|
+| HEMOLYTIC | < 25 µg/mL | 14 |
+| SELECTIVE | >= 100 µg/mL | 21 |
+| BORDER (excluded from AUROC) | 25-100 µg/mL | 7 |
+
+**Task:** Can pipeline scorers distinguish hemolytic AMPs from selective AMPs?
+For safety/selectivity scorers, the correct direction is: hemolytic AMPs score *lower*
+(less safe, less selective). We report "hemolysis detection AUROC" where higher = better
+risk detection (1 - raw AUROC for safety-type scorers).
+
+### Per-score hemolysis detection AUROC
+
+| Score | Detection AUROC | CI₉₅ | Significant? | Verdict |
+|-------|:--------------:|:----:|:------------:|---------|
+| synthesis | 0.8027 | 0.63-0.95 | **YES** | Synthesis difficulty correlates with hemolysis — hemolytic AMPs are harder to synthesize |
+| boman_activity | 0.6837 | 0.49-0.85 | No (CI lo < 0.5) | Weak trend: hemolytic AMPs have lower Boman activity |
+| serum_stability | 0.6020 | 0.40-0.80 | No (CI lo < 0.5) | Weak trend: hemolytic AMPs less serum-stable |
+| expert_composite | 0.5119 | 0.31-0.71 | No (CI lo < 0.5) | Better than ensemble but not significant |
+| hinge_selectivity | 0.4456 | 0.24-0.64 | No | No selectivity signal from hinge detection |
+| selectivity_proxy | 0.4133 | 0.28-0.55 | No | **FAILS** — charge/GRAVY does not capture hemolysis |
+| safety | 0.3844 | 0.26-0.52 | No | **FAILS** — confirms melittin blind spot |
+| activity | 0.3401 | 0.16-0.52 | No | Activity scorer ranks hemolytic AMPs *higher* (anti-selective) |
+| ensemble | 0.3486 | 0.17-0.54 | No | Ensemble inherits activity scorer's anti-selective bias |
+
+**Key findings:**
+
+1. **The safety scorer does NOT detect hemolysis** (detection AUROC = 0.3844, CI lo = 0.26).
+   This confirms the expert ablation's prediction and the previously documented melittin
+   blind spot. All 14 hemolytic AMPs in the reference set score safety >= 0.8 — the scorer
+   cannot distinguish them from selective AMPs.
+
+2. **The selectivity proxy does NOT detect hemolysis** (detection AUROC = 0.4133, CI lo = 0.28).
+   The charge/GRAVY heuristic is insufficient for capturing hemolysis risk. Hemolytic AMPs
+   like melittin and protegrin have optimal charge (+2 to +7) and moderate GRAVY, so the
+   proxy assigns them high selectivity scores.
+
+3. **Synthesis feasibility is the only significant risk detector** (detection AUROC = 0.8027,
+   CI lo = 0.63). Hemolytic AMPs tend to be harder to synthesize: they have more cysteines
+   (protegrins, tachyplesins), repeat runs, and hydrophobic segments. This is an incidental
+   correlation, not a designed safety feature — but it means the synthesis gate provides
+   partial hemolysis filtering as a side effect.
+
+4. **The activity scorer is anti-selective** (detection AUROC = 0.34): it ranks hemolytic
+   AMPs *higher* than selective AMPs. This is expected: hemolytic AMPs like melittin have
+   strong amphipathic helices, high hydrophobic moment, and high charge — exactly the
+   features the activity scorer rewards. The ensemble inherits this bias.
+
+5. **The expert composite is better than the ensemble** on hemolysis detection (0.5119 vs
+   0.3486) but not significantly so (CI includes 0.5). The added selectivity and safety
+   components partially offset the activity scorer's anti-selective bias, but not enough
+   to reach significance at n=14 vs n=21.
+
+**Honest limitation:** HC50 values are approximate literature values with high inter-assay
+variability (RBC source, buffer, incubation time, concentration range). The binary
+thresholds (25 / 100 µg/mL) are coarse. A larger reference set with standardized HC50
+measurements would tighten the CIs and might flip some near-zero results to significant.
+The current sample size (14 vs 21) is too small for confident conclusions on any score
+with CI lower bound below 0.5.
+
+**Implication for the pipeline:** Hemolysis remains unpredictable by the current
+physicochemical scorers. The melittin blind spot is confirmed quantitatively. Hemolysis
+must be assayed experimentally for every candidate regardless of safety or selectivity
+score. The synthesis gate provides partial indirect filtering but should not be relied
+upon as a hemolysis predictor.
+
 ## Test Suite
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 1435 |
+| Total tests | 1453 |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 
@@ -262,4 +339,5 @@ next loop.
 | 2026-06-29 | External predictor results filled from wave05_combined_consensus.csv; all 7 gates PASS | OpenAMP Wave 0.5 |
 | 2026-06-29 | Wave 0.5 scaffold diversification — 24-candidate Wave 1 panel across 14 families | OpenAMP Wave 0.5 |
 | 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.736 vs ensemble 0.7832 (delta −0.0472); 3 components anti-signal; ensemble remains primary gate | OpenAMP loop |
+| 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop |
 | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI |

docs/REVIEWER_SUMMARY.md

@@ -77,7 +77,7 @@ assay is mandatory for all candidates.
 | AUROC 0.7832 — 22% of benchmark pairs misranked | ~1 in 5 candidate ranking decisions are wrong; wet-lab is the judge |
 | No 3D structure or docking | Helical assumption may misclassify non-helical mechanisms |
 | Near-seed generation only | Novel sequence space not explored by de novo generation |
-| Melittin safety blind spot | Hemolysis must be assayed, cannot be predicted |
+| Melittin safety blind spot | Hemolysis must be assayed — confirmed quantitatively: safety scorer detection AUROC=0.3844 on 14 hemolytic vs 21 selective AMPs (v0.5.9 selectivity benchmark) |
 | No external predictor ensemble yet | CAMPR4/AMPScanner/dbAMP/AntiCP2/Macrel web submission checklist exists at `outputs/external_predict_checklist.md` but results not yet obtained |
 | Probability table uncalibrated | All "P(≥1 active) ≈ 92–97%" statements are upper bounds; honest estimate 55–80% for any hit, 35–60% for novel family |
 

docs/ROADMAP.md

@@ -142,10 +142,24 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 - `bench expert-ablation` CLI command + `make bench-expert-ablation` target
 - Per-component AUROC attribution: identifies signal-bearing (activity 0.814, selectivity 0.773), near-zero (hinge, novelty, boman, motif), and anti-signal (safety 0.349, serum_stability 0.223, synthesis 0.423) components
 - **Key finding:** Expert composite AUROC 0.736 < ensemble AUROC 0.783 (delta −0.047). The added complexity does NOT improve AMP-vs-decoy discrimination. Three components are anti-signal because real AMPs have extreme biophysical properties that safety/stability/synthesis scorers penalise.
-- Ensemble remains primary synthesis gate. Expert composite may add value for within-AMP ranking (not yet demonstrated).
+- Ensemble remains primary synthesis gate. Expert composite may add value for within-AMP ranking (tested in v0.5.9 — see selectivity benchmark; expert composite detection AUROC 0.5119 vs ensemble 0.3486, but not significant at n=14 vs n=21).
 - `make bench-cluster-split` Makefile target added (was missing despite CLI command existing)
 - 14 new tests in `test_expert_ablation.py`; 1435 total tests
 
+## v0.5.9 — Within-AMP Selectivity Benchmark ✓ (2026-07-01)
+
+- `run_selectivity_benchmark()` in `benchmark/retrospective.py` — tests whether pipeline scorers can distinguish hemolytic AMPs (HC50 < 25 µg/mL) from selective AMPs (HC50 >= 100 µg/mL)
+- `bench selectivity` CLI command + `make bench-selectivity` target
+- Hemolysis reference dataset: 42 known AMPs with literature HC50 values (`examples/validation/hemolysis_reference.csv`)
+- Per-score hemolysis detection AUROC: identifies the only significant risk detector (synthesis, AUROC=0.8027, CI: 0.63-0.95) and confirms the safety scorer blind spot (detection AUROC=0.3844)
+- **Key finding:** Safety scorer FAILS to detect hemolysis (detection AUROC=0.3844, CI lo=0.26). All 14 hemolytic AMPs score safety >= 0.8. The melittin blind spot is confirmed quantitatively — 1D hydrophobic moment cannot capture hemolysis mechanisms like curvature-mediated lysis or beta-sheet pore formation.
+- **Key finding:** Selectivity proxy FAILS to detect hemolysis (detection AUROC=0.4133, CI lo=0.28). The charge/GRAVY heuristic is insufficient — hemolytic AMPs like melittin have optimal charge and moderate GRAVY.
+- **Key finding:** Synthesis feasibility is the only significant hemolysis risk detector (detection AUROC=0.8027, CI lo=0.63). This is an incidental correlation: hemolytic AMPs tend to have more cysteines, repeat runs, and hydrophobic segments that make them harder to synthesize. The synthesis gate provides partial indirect hemolysis filtering.
+- **Key finding:** Expert composite is better than ensemble on hemolysis detection (0.5119 vs 0.3486) but not significant (CI includes 0.5). The added selectivity/safety components partially offset the activity scorer's anti-selective bias.
+- **Key finding:** Activity scorer is anti-selective (detection AUROC=0.34): it ranks hemolytic AMPs *higher* because they have stronger amphipathic helices. The ensemble inherits this bias.
+- Blind spots: 14 hemolytic AMPs with safety >= 0.8 (melittin, indolicidin, protegrins, tachyplesins, piscidin-1, arenicin, polyphemusin, gramicidin-S-like, BMAP fragment)
+- 18 new tests in `test_selectivity_benchmark.py`; 1453 total tests
+
 ## v1.0 — Validated dry-lab-to-wet-lab loop
 
 - independently reviewed assay batch (expert_review.yml GitHub issue template);