feat(bench): strict triage benchmark with composition-matched decoys

docs/METRICS_CURRENT.md

@@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-01 (triage benchmark added)
+> **Last updated:** 2026-07-02 (strict triage benchmark added)
 > **Pipeline version:** v0.5.x
 > **Branch:** main
 
@@ -452,11 +452,61 @@ top-20. A successful virtual assay must avoid both failures.
 inter-assay variability. The binary thresholds (25 / 100 µg/mL) are coarse.
 The MODERATE class (HC50 25-100, n=68) is excluded from the binary task.
 
+### Strict Triage: Composition-Matched Decoys (v0.5.14 — added 2026-07-02)
+
+> The standard triage benchmark uses random background peptides as decoys.
+> These are trivially distinguishable from AMPs because their composition is
+> protein-like, not AMP-like. This inflates selective_vs_decoy and
+> hemolytic_vs_decoy AUROCs, making scorers appear to triage well.
+>
+> The strict triage benchmark replaces random decoys with **composition-matched
+> scrambled versions** of the selective AMPs — same amino acids, permuted order.
+> This destroys amphipathic helical phase, hydrophobic moment, and charge
+> distribution patterns while preserving all composition-based features.
+
+**Key finding: standard triage success was partly an illusion.**
+
+| Scorer | Std sel_vs_dec | Strict sel_vs_dec | Std sel_vs_hemo | Strict sel_vs_hemo | Std correct | Strict correct |
+|-------|-----------------|-------------------|------------------|---------------------|--------------|----------------|
+| ensemble | 0.848 | **0.572** | 0.466 | 0.466 | NO | NO |
+| activity | 0.885 | **0.617** | 0.430 | 0.430 | NO | NO |
+| selectivity_proxy | 0.782 | **0.500** | 0.610 | 0.610 | YES | **NO** |
+| expert_composite | 0.757 | **0.510** | 0.545 | 0.545 | YES | **NO** |
+| triage_score | 0.863 | **0.674** | 0.462 | 0.462 | NO | NO |
+| hemolysis_risk | 0.485 | 0.617 | 0.488 | 0.488 | NO | NO |
+
+**What this reveals:**
+
+1. **selectivity_proxy collapses to exactly 0.5000** on selective_vs_decoy —
+   confirming it is purely composition-driven (charge and GRAVY are identical
+   between a sequence and its scrambled version).
+
+2. **The ensemble drops from 0.848 to 0.572** — most of its apparent triage
+   power was composition-based, not order-based.
+
+3. **No scorer triages correctly** with composition-matched decoys. The standard
+   triage "success" of selectivity_proxy and expert_composite was an artifact
+   of trivially distinguishable decoys.
+
+4. **selective_vs_hemolytic is stable** across both benchmarks (identical AUROCs)
+   — as expected, since both classes are real AMP sequences and only the decoy
+   class changes.
+
+5. **The ensemble admits 7 scrambled decoys into top-20** (vs 0 with random
+   decoys) — it cannot distinguish real AMPs from scrambled versions of themselves.
+
+**Implication:** The pipeline's triage signal is almost entirely composition-driven.
+The real bottleneck is selective-vs-hemolytic discrimination, which requires
+structural or contextual features beyond what current 1D physicochemical scorers
+can capture. Any future virtual assay layer must demonstrate order-dependent
+triage signal on this strict benchmark before claiming to improve candidate
+selection.
+
 ## Test Suite
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 1471+ |
+| Total tests | 1515+ |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 
@@ -489,4 +539,5 @@ The MODERATE class (HC50 25-100, n=68) is excluded from the binary task.
 | 2026-07-01 | Dedicated hemolysis risk scorer: 4-component score (synth+aromatic+face+cys) achieves detection AUROC=0.9218 (CI: 0.82-0.99); integrated into expert composite (detection 0.5119→0.6429); safety scorer unchanged; 1471 tests | OpenAMP loop |
 | 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop |
 | 2026-07-01 | Expert composite ranking integration: `score_candidates()` now computes `expert_composite` and `hemolysis_risk`; `--ranking-mode expert` CLI flag; expert-ranked top-5 have lower mean hemolysis_risk than ensemble | OpenAMP loop |
+| 2026-07-02 | **Strict triage benchmark added:** composition-matched scrambled decoys replace random background. No scorer triages correctly — standard triage "success" of selectivity_proxy (0.782 sel_vs_dec) and expert_composite (0.757) was inflated by trivially distinguishable decoys. selectivity_proxy collapses to 0.500 (purely composition-driven), ensemble drops to 0.572. Real bottleneck (selective_vs_hemolytic) unchanged. | OpenAMP loop |
 | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI |

docs/ROADMAP.md

@@ -242,6 +242,40 @@ Expert ranking is a safety-aware alternative, not a validated safety guarantee o
 a replacement for activity-gated ensemble selection. Wet-lab hemolysis assay remains
 mandatory.
 
+## v0.5.14 — Strict Triage Benchmark: Composition-Matched Decoys ✓ (2026-07-02)
+
+The standard triage benchmark (v0.5.12) uses random background peptides as
+decoys. These are trivially distinguishable from AMPs because their amino acid
+composition is protein-like, not AMP-like. This inflates selective_vs_decoy and
+hemolytic_vs_decoy AUROCs, making scorers appear to triage well when they are
+really only solving the easy "AMP vs random" problem.
+
+The strict triage benchmark replaces random decoys with **composition-matched
+scrambled versions** of the selective AMPs — same amino acids, permuted order.
+This destroys amphipathic helical phase, hydrophobic moment, and charge
+distribution patterns while preserving all composition-based features.
+
+Changes:
+- `benchmark/triage.py`: `run_strict_triage_benchmark()` — generates scrambled
+  decoys, runs the same 10-scorer pairwise triage comparison
+- `benchmark/metrics_snapshot.py`: strict_triage section added to snapshot
+- `tests/test_triage_benchmark.py`: 12 new tests for strict triage (structure
+  + findings)
+- `outputs/metrics_snapshot.json`: regenerated with strict triage results
+- `docs/METRICS_CURRENT.md`: strict triage results table and interpretation
+
+Key honest finding: **No scorer triages correctly with composition-matched
+decoys.** The standard triage "success" of selectivity_proxy and
+expert_composite was an artifact of trivially distinguishable decoys.
+selectivity_proxy collapses to exactly 0.5000 on selective_vs_decoy (confirming
+it is purely composition-driven), and the ensemble drops from 0.848 to 0.572.
+
+The selective_vs_hemolytic AUROC is identical between standard and strict
+(expected: both classes are real AMPs, only the decoy class changes). This
+confirms the real bottleneck — selective-vs-hemolytic discrimination — is
+unchanged and requires structural or contextual features beyond current 1D
+physicochemical scorers.
+
 ## v1.0 — Validated dry-lab-to-wet-lab loop
 
 - independently reviewed assay batch (expert_review.yml GitHub issue template);
@@ -255,7 +289,7 @@ mandatory.
 
 - ~~write the first explicit simulator scope document: what OpenAMP will model and what it will not;~~
 - ~~add membrane/selectivity/stability proxy interfaces with uncertainty fields;~~
-- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ (v0.5.12: triage benchmark added — ensemble fails, selectivity_proxy only correct triager)
+- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ (v0.5.12: standard triage added — selectivity_proxy and expert_composite pass; v0.5.14: strict triage with composition-matched decoys added — **no scorer passes**, standard triage success was inflated by trivially distinguishable random decoys)
 - require every proxy module to justify itself against cheap heuristic baselines before it affects selection.
 
 ## v2.x — Wet-lab compression engine

outputs/metrics_snapshot.json

@@ -90,6 +90,90 @@
     "recall_at_20": 0.2105,
     "recall_at_43": 0.4211
   },
+  "strict_triage": {
+    "best_scorer": "triage_score",
+    "decoy_type": "composition_matched_scrambled",
+    "n_decoy": 125,
+    "n_hemolytic": 45,
+    "n_selective": 125,
+    "per_scorer": {
+      "activity": {
+        "hemolytic_vs_decoy": 0.6974,
+        "selective_vs_decoy": 0.6167,
+        "selective_vs_hemolytic": 0.43,
+        "triages_correctly": false
+      },
+      "ensemble": {
+        "hemolytic_vs_decoy": 0.6156,
+        "selective_vs_decoy": 0.5716,
+        "selective_vs_hemolytic": 0.4661,
+        "triages_correctly": false
+      },
+      "expert_composite": {
+        "hemolytic_vs_decoy": 0.4755,
+        "selective_vs_decoy": 0.5102,
+        "selective_vs_hemolytic": 0.5449,
+        "triages_correctly": false
+      },
+      "hemolysis_risk": {
+        "hemolytic_vs_decoy": 0.6199,
+        "selective_vs_decoy": 0.6173,
+        "selective_vs_hemolytic": 0.4876,
+        "triages_correctly": false
+      },
+      "safe_weighted_ensemble": {
+        "hemolytic_vs_decoy": 0.6857,
+        "selective_vs_decoy": 0.6405,
+        "selective_vs_hemolytic": 0.483,
+        "triages_correctly": false
+      },
+      "safety": {
+        "hemolytic_vs_decoy": 0.348,
+        "selective_vs_decoy": 0.3882,
+        "selective_vs_hemolytic": 0.5384,
+        "triages_correctly": false
+      },
+      "selectivity_proxy": {
+        "hemolytic_vs_decoy": 0.3905,
+        "selective_vs_decoy": 0.5,
+        "selective_vs_hemolytic": 0.6095,
+        "triages_correctly": false
+      },
+      "serum_stability": {
+        "hemolytic_vs_decoy": 0.4281,
+        "selective_vs_decoy": 0.4908,
+        "selective_vs_hemolytic": 0.5686,
+        "triages_correctly": false
+      },
+      "synthesis": {
+        "hemolytic_vs_decoy": 0.6096,
+        "selective_vs_decoy": 0.5672,
+        "selective_vs_hemolytic": 0.4692,
+        "triages_correctly": false
+      },
+      "triage_score": {
+        "hemolytic_vs_decoy": 0.7346,
+        "selective_vs_decoy": 0.6737,
+        "selective_vs_hemolytic": 0.4622,
+        "triages_correctly": false
+      }
+    },
+    "top_20_by_ensemble": {
+      "DECOY": 7,
+      "HEMOLYTIC": 5,
+      "SELECTIVE": 8
+    },
+    "top_20_by_expert_composite": {
+      "DECOY": 7,
+      "HEMOLYTIC": 0,
+      "SELECTIVE": 13
+    },
+    "top_20_by_triage_score": {
+      "DECOY": 2,
+      "HEMOLYTIC": 4,
+      "SELECTIVE": 14
+    }
+  },
   "triage": {
     "best_scorer": "selectivity_proxy",
     "per_scorer": {

src/openamp_foundry/benchmark/metrics_snapshot.py

@@ -10,7 +10,7 @@
     run_retrospective_benchmark,
     run_selectivity_benchmark,
 )
-from openamp_foundry.benchmark.triage import run_triage_benchmark
+from openamp_foundry.benchmark.triage import run_strict_triage_benchmark, run_triage_benchmark
 
 
 def build_metrics_snapshot(
@@ -61,6 +61,10 @@ def build_metrics_snapshot(
         config_path=standard_config,
         n_bootstrap=n_bootstrap,
     )
+    strict_triage = run_strict_triage_benchmark(
+        hemolysis_csv=hemolysis_csv,
+        n_bootstrap=n_bootstrap,
+    )
 
     return {
         "schema_version": 1,
@@ -151,4 +155,23 @@ def build_metrics_snapshot(
                 for scorer, info in triage["per_scorer"].items()
             },
         },
+        "strict_triage": {
+            "decoy_type": strict_triage["decoy_type"],
+            "n_selective": strict_triage["n_selective"],
+            "n_hemolytic": strict_triage["n_hemolytic"],
+            "n_decoy": strict_triage["n_decoy"],
+            "best_scorer": strict_triage["best_scorer"],
+            "top_20_by_ensemble": strict_triage["top_20_by_ensemble"],
+            "top_20_by_triage_score": strict_triage["top_20_by_triage_score"],
+            "top_20_by_expert_composite": strict_triage["top_20_by_expert_composite"],
+            "per_scorer": {
+                scorer: {
+                    "selective_vs_decoy": info["selective_vs_decoy"]["auroc"],
+                    "hemolytic_vs_decoy": info["hemolytic_vs_decoy"]["auroc"],
+                    "selective_vs_hemolytic": info["selective_vs_hemolytic"]["auroc"],
+                    "triages_correctly": info["triages_correctly"],
+                }
+                for scorer, info in strict_triage["per_scorer"].items()
+            },
+        },
     }