cschanhniem · Jun 30, 2026 · Jun 30, 2026 · Jun 30, 2026
diff --git a/docs/BENCHMARK_CARD.md b/docs/BENCHMARK_CARD.md
@@ -1,7 +1,7 @@
 # Benchmark Card — OpenAMP Foundry v0.5.x
 
 > **Purpose:** Single-page summary of benchmark methodology, data, and metrics.
-> **Last updated:** 2026-07-01 (cluster-split benchmark added)
+> **Last updated:** 2026-07-01 (hemolysis benchmark expanded to 238 peptides)
 
 ---
 
@@ -39,6 +39,32 @@
 | Bootstrap | 2000 resamples | 2000 resamples | — |
 | **Interpretation** | **STRONG** | **STRONG** | Below random (expected) |
 
+## Within-AMP Selectivity Benchmark
+
+> Tests whether scorers can distinguish hemolytic AMPs (HC50 < 25 µg/ml) from
+> selective AMPs (HC50 >= 100 µg/ml). Run: `make bench-selectivity`
+
+| Set | Count | Description |
+|-----|:-----:|-------------|
+| Hemolytic | **54** | HC50 < 25 µg/ml (literature + DBAASP v3) |
+| Selective | **125** | HC50 >= 100 µg/ml |
+| Border | **59** | 25 <= HC50 < 100 (excluded from binary AUROC) |
+| **Total** | **238** | Expanded from 42 in v0.5.11 using DBAASP data |
+
+| Score | Detection AUROC | CI₉₅ | Significant? |
+|-------|:--------------:|:----:|:------------:|
+| hemolysis_risk | 0.5650 | 0.47–0.66 | No (direction correct) |
+| safety | 0.5116 | 0.43–0.60 | No |
+| selectivity_proxy | 0.5744 | 0.50–0.66 | Borderline |
+| ensemble | 0.4201 | 0.33–0.51 | No (wrong direction) |
+| expert_composite | 0.5459 | 0.46–0.63 | No |
+| serum_stability | 0.5873 | 0.51–0.66 | Yes (risk detector) |
+
+**Key finding:** The hemolysis risk scorer's original AUROC=0.9218 (CI 0.82-0.99)
+on n=35 was small-sample inflation. On the expanded n=179, no scorer achieves
+strong hemolysis detection. Hemolysis remains unpredictable by 1D
+physicochemical features and must be assayed experimentally.
+
 ## Method
 
 AUROC computed via Wilcoxon-Mann-Whitney statistic (concordant-pair enumeration).

diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md
@@ -3,7 +3,7 @@
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-01 (cluster-split benchmark added)
+> **Last updated:** 2026-07-01 (hemolysis benchmark expanded to 238 peptides)
 > **Pipeline version:** v0.5.x
 > **Branch:** main
 
@@ -312,64 +312,74 @@ upon as a hemolysis predictor.
 > hemolysis detection (AUROC=0.3844). A dedicated hemolysis risk scorer was
 > built from empirically-validated components identified in that benchmark.
 >
+> **v0.5.11 correction:** The original 42-peptide reference set (14 hemolytic
+> vs 21 selective, n=35) produced detection AUROC=0.9218 (CI 0.82-0.99). This
+> was **small-sample inflation**. Expansion to 238 peptides using DBAASP human
+> erythrocyte data (54 hemolytic vs 125 selective, n=179) dropped the detection
+> AUROC to 0.5650 (CI 0.47-0.66) — direction correct but NOT statistically
+> significant. The scorer retains weak directional signal but should not be
+> trusted as a standalone hemolysis detector.
+>
 > Run: `make bench-selectivity` (hemolysis_risk column in the output)
 
 **Module:** `src/openamp_foundry/scoring/hemolysis.py`
 
-**Components** (detection AUROC from leave-one-out on n=14 hemolytic vs n=21 selective):
+**Components** (individual AUROC from original n=14 vs n=21; may not replicate on expanded set):
 
-| Component | Individual AUROC | Weight | Signal source |
-|-----------|:----------------:|:------:|---------------|
+| Component | Individual AUROC (n=35) | Weight | Signal source |
+|-----------|:-----------------------:|:------:|---------------|
 | Synthesis difficulty (1 - synth_feasibility) | 0.8027 | 0.30 | Incidental: hemolytic AMPs harder to synthesize |
 | Aromatic fraction (F/W/Y density) | 0.8299 | 0.30 | Trp/Phe intercalation in both membrane types |
 | Cationic-on-hydrophobic-face fraction | 0.7585 | 0.20 | Poor amphipathic face segregation |
 | Cysteine fraction | 0.7500 | 0.20 | Beta-sheet defensin/protegrin class |
 
-**Combined performance:**
+**Combined performance (expanded n=179):**
 
-| Metric | Value | Notes |
-|--------|-------|-------|
-| **Detection AUROC** | **0.9218** | Bootstrap CI₉₅: 0.82-0.99 (n_bootstrap=2000) |
-| Mean hemolytic risk | 0.4064 | n=14 |
-| Mean selective risk | 0.1501 | n=21 |
-| Risk threshold > 0.40 | Enriched in hemolytic | 10/14 hemolytic vs 2/21 selective |
-| Safety scorer (comparison) | 0.3844 | FAILS — all 14 hemolytic AMPs score safety >= 0.94 |
+| Metric | Original (n=35) | Expanded (n=179) | Notes |
+|--------|:---------------:|:-----------------:|-------|
+| **Detection AUROC** | **0.9218** | **0.5650** | Small-sample inflation corrected |
+| CI₉₅ lower bound | 0.82 | 0.47 | No longer > 0.5 — not significant |
+| CI₉₅ upper bound | 0.99 | 0.66 | |
+| Mean hemolytic risk | 0.4064 (n=14) | 0.2042 (n=54) | Direction still correct |
+| Mean selective risk | 0.1501 (n=21) | 0.1535 (n=125) | |
+| Safety scorer detection | 0.3844 | 0.5116 | Safety also improves slightly with more data |
 
-**Expert composite integration:**
+**Expert composite integration (expanded n=179):**
 
-| Metric | Before (v0.5.9) | After (v0.5.10) |
-|--------|:---------------:|:----------------:|
-| Expert composite detection AUROC | 0.5119 | 0.6429 |
-| Expert composite CI lo | 0.3129 | 0.4490 |
-| Ensemble detection AUROC | 0.3486 | 0.3486 (unchanged) |
+| Metric | Before (v0.5.9, n=35) | After (v0.5.10, n=35) | Expanded (n=179) |
+|--------|:---------------------:|:---------------------:|:-----------------:|
+| Expert composite detection AUROC | 0.5119 | 0.6429 | 0.5459 |
+| Expert composite CI lo | 0.3129 | 0.4490 | 0.4562 |
+| Ensemble detection AUROC | 0.3486 | 0.3486 | 0.4201 |
 
-**Expert ablation (AMP-vs-decoy):**
+**Expert ablation (AMP-vs-decoy, unchanged):**
 
 | Metric | Value | Classification |
 |--------|:-----:|:--------------:|
 | hemolysis_safety AUROC | 0.3285 | **Anti-signal** (above_random = -0.1715) |
 | Expert composite AUROC | 0.7119 | Down from 0.7360 (delta -0.0713 vs ensemble) |
 
-**Key finding:** The dedicated hemolysis risk scorer is the first pipeline score
-with a statistically significant hemolysis detection signal (CI lo = 0.82 > 0.5).
-It complements (not replaces) the safety scorer, which retains its role for
-AMP-vs-decoy discrimination. The hemolysis_safety component is anti-signal on
-AMP-vs-decoy (AUROC=0.3285) — expected, since real AMPs have higher hemolysis
-risk than random decoys. This confirms the component measures a within-AMP
-property, not an AMP-vs-non-AMP property.
+**Key finding (corrected):** The hemolysis risk scorer's original detection
+AUROC=0.9218 on n=35 was small-sample inflation. On the expanded n=179
+reference set, detection AUROC=0.5650 (CI 0.47-0.66) — direction is correct
+(hemolytic > selective on average) but not statistically significant. The
+scorer should NOT be described as a "statistically significant hemolysis
+detector." It provides weak directional signal that may be useful as one
+factor in a composite but cannot be relied upon for hemolysis triage. Hemolysis
+must still be assayed experimentally for every candidate.
 
-**Honest limitation:** The reference set is small (n=35). The CI is wide.
-Melittin's risk score (0.13) remains modest because its bent-helix hemolysis
-mechanism is not fully captured by 1D features. Hemolysis must still be assayed
-experimentally for every candidate.
+**Honest limitation:** The expanded reference set (n=179) provides a more honest
+estimate, but HC50 values are approximate literature values with high inter-assay
+variability. Melittin's risk score (0.13) remains modest because its bent-helix
+hemolysis mechanism is not fully captured by 1D features.
 
 ---
 
 ## Test Suite
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 1471 |
+| Total tests | 1471+ |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 
@@ -398,6 +408,7 @@ experimentally for every candidate.
 | 2026-06-29 | External predictor results filled from wave05_combined_consensus.csv; all 7 gates PASS | OpenAMP Wave 0.5 |
 | 2026-06-29 | Wave 0.5 scaffold diversification — 24-candidate Wave 1 panel across 14 families | OpenAMP Wave 0.5 |
 | 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.736 vs ensemble 0.7832 (delta −0.0472); 3 components anti-signal; ensemble remains primary gate | OpenAMP loop |
+| 2026-07-01 | **Hemolysis benchmark expanded:** 42 -> 238 peptides using DBAASP human erythrocyte data (54 hemolytic vs 125 selective, n=179 binary). Hemolysis risk scorer detection AUROC drops 0.9218 -> 0.5650 (CI 0.47-0.66) — original performance was small-sample inflation. Direction correct, not significant. Safety scorer detection improves 0.3844 -> 0.5116 (still not significant). 196 new peptides from DBAASP v3. | OpenAMP loop |
 | 2026-07-01 | Dedicated hemolysis risk scorer: 4-component score (synth+aromatic+face+cys) achieves detection AUROC=0.9218 (CI: 0.82-0.99); integrated into expert composite (detection 0.5119→0.6429); safety scorer unchanged; 1471 tests | OpenAMP loop |
 | 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop |
 | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI |
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -146,12 +146,27 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 - `make bench-cluster-split` Makefile target added (was missing despite CLI command existing)
 - 14 new tests in `test_expert_ablation.py`; 1435 total tests
 
+## v0.5.11 — Hemolysis Benchmark Expansion (DBAASP) ✓ (2026-07-01)
+
+- Expanded hemolysis reference from 42 to 238 peptides using DBAASP v3 human erythrocyte data
+- 196 new peptides with 50% hemolysis (HC50) values, converted to µg/ml using peptide MW
+- Binary task: 54 hemolytic (HC50 < 25 µg/ml) vs 125 selective (HC50 >= 100 µg/ml) = 179 total (was 14 vs 21 = 35)
+- **Critical honesty correction:** hemolysis risk scorer detection AUROC drops from 0.9218 (CI 0.82-0.99) to 0.5650 (CI 0.47-0.66)
+  - Original n=35 performance was **small-sample inflation**
+  - Direction still correct (hemolytic mean=0.204 > selective mean=0.154) but NOT statistically significant
+  - Safety scorer detection improves slightly: 0.3844 → 0.5116 (still not significant)
+  - Selectivity proxy: detection AUROC=0.5744 (CI 0.50-0.66) — borderline significant
+- Extraction script: `scripts/expand_hemolysis_benchmark.py`
+- Tests updated: `test_selectivity_benchmark.py` test renamed from `test_hemolysis_risk_is_significant_detector` to `test_hemolysis_risk_direction_correct`
+- Docs updated: METRICS_CURRENT.md, hemolysis.py, expert.py comments corrected to reflect expanded-set reality
+- **Lesson:** A 0.92 AUROC on n=35 is not robust. The project's adversarial benchmark philosophy caught this.
+
 ## v0.5.10 — Dedicated Hemolysis Risk Scorer ✓ (2026-07-01)
 
 - `scoring/hemolysis.py` — standalone 4-component hemolysis risk score (synth difficulty + aromatic density + face cationic leakage + cysteine content)
-- Combined detection AUROC=0.9218 (CI₉₅: 0.82-0.99) — first statistically significant hemolysis detector
+- Combined detection AUROC=0.9218 (CI₉₅: 0.82-0.99) on original 42-peptide reference — **CORRECTED in v0.5.11: small-sample inflation; expanded n=179 gives 0.5650 (CI 0.47-0.66)**
 - Integrated into selectivity benchmark as risk-direction score; integrated into expert composite as `hemolysis_safety` (weight 0.10)
-- Expert composite hemolysis detection improved: 0.5119 → 0.6429 (not significant, but trend correct)
+- Expert composite hemolysis detection improved: 0.5119 → 0.6429 on n=35 — **expanded n=179 gives 0.5459 (CI 0.46-0.63)**
 - Safety scorer UNCHANGED — standard AUROC remains 0.7832; hemolysis risk is complementary, not replacement
 - Expert ablation: hemolysis_safety is anti-signal on AMP-vs-decoy (AUROC=0.3285) — confirms it measures within-AMP property
 - 15 new tests in `test_hemolysis_risk.py`; 3 new tests in `test_selectivity_benchmark.py`; 1471 total tests