test: benchmark expert ranking in triage

docs/METRICS_CURRENT.md

@@ -390,8 +390,8 @@ hemolysis mechanism is not fully captured by 1D features.
 >
 > Run: `make bench-triage`
 
-**Dataset:** 125 selective AMPs (HC50 >= 100 µg/mL) + 45 hemolytic AMPs (HC50 < 25 µg/mL)
-+ 96 random background decoys = 266 total.
+**Dataset:** 125 selective AMPs (HC50 >= 100 µg/mL) + 54 hemolytic AMPs (HC50 < 25 µg/mL)
++ 96 random background decoys = 275 total.
 
 ### Per-scorer pairwise AUROCs
 
@@ -405,6 +405,7 @@ A scorer that triages correctly should have all three AUROCs > 0.5:
 | ensemble | 0.848 | 0.891 | 0.466 | **NO** (anti-selective) |
 | activity | 0.885 | 0.934 | 0.430 | NO |
 | selectivity_proxy | 0.782 | 0.795 | 0.610 | **YES** |
+| expert_composite | 0.757 | 0.746 | 0.545 | **YES** |
 | triage_score (activity × (1 - hemo_risk)) | 0.863 | 0.902 | 0.462 | NO |
 | safe_weighted_ensemble | 0.849 | 0.890 | 0.483 | NO |
 | safety | 0.344 | 0.300 | 0.538 | NO |
@@ -419,9 +420,10 @@ A scorer that triages correctly should have all three AUROCs > 0.5:
    bias documented in the selectivity benchmark, now confirmed in the combined
    triage context.
 
-2. **Only selectivity_proxy triages correctly** (all three AUROCs > 0.5). It
-   sacrifices some decoy discrimination (0.782 vs ensemble's 0.848) but is the
-   only scorer that prefers selective AMPs over hemolytic ones.
+2. **selectivity_proxy and expert_composite triage correctly by pairwise AUROC**
+   (all three AUROCs > 0.5). selectivity_proxy remains the best scorer because it
+   has stronger selective-vs-hemolytic separation (0.610 vs expert_composite 0.545)
+   while keeping slightly better selective-vs-decoy discrimination (0.782 vs 0.757).
 
 3. **The naive triage_score (activity × (1 - hemolysis_risk)) does NOT fix the
    anti-selective bias** (sel_vs_hem = 0.462). This is because hemolysis_risk
@@ -433,11 +435,18 @@ A scorer that triages correctly should have all three AUROCs > 0.5:
    The shift is in the right direction but modest — the hemolysis_risk penalty
    is weak.
 
+5. **Expert-composite top-k failure:** The expert_composite removes hemolytic
+   AMPs from its top-20 (15 selective / 0 hemolytic), but admits 5 random decoys.
+   That is a useful negative result: expert ranking is not a replacement for the
+   ensemble activity gate, even when its pairwise AUROCs clear 0.5.
+
 **Implication for the virtual assay layer:** Any future virtual assay module
 must beat this triage benchmark baseline. The minimum bar is: triage correctly
-(all three AUROCs > 0.5) while maintaining ensemble-level decoy discrimination
-(sel_vs_decoy > 0.80). The selectivity_proxy achieves correct triage but at
-the cost of decoy discrimination. A successful virtual assay must achieve both.
+(all three AUROCs > 0.5), keep decoys out of the top-k selection surface, and
+maintain near-ensemble decoy discrimination (sel_vs_decoy > 0.80). The
+selectivity_proxy achieves correct triage but loses decoy-discrimination margin.
+The expert_composite achieves correct pairwise triage but admits decoys into its
+top-20. A successful virtual assay must avoid both failures.
 
 **Honest limitation:** The benchmark uses literature HC50 values with high
 inter-assay variability. The binary thresholds (25 / 100 µg/mL) are coarse.

docs/ROADMAP.md

@@ -189,7 +189,7 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 
 - **Multi-class triage benchmark** (`benchmark/triage.py`): tests whether the
   pipeline can rank selective AMPs > hemolytic AMPs > random decoys in a single
-  combined panel (125 selective + 45 hemolytic + 96 decoys = 266 total).
+  combined panel (125 selective + 54 hemolytic + 96 decoys = 275 total).
 - Addresses the v1.1 ROADMAP item: "benchmark candidate triage against a
   reference panel that includes selective AMPs, hemolytic positives, inactive
   peptides, and random controls."
@@ -198,8 +198,12 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
   anti-selective bias. Hemolytic AMPs have stronger amphipathic helices, higher
   hydrophobic moment, and higher charge — exactly the features the activity
   scorer rewards.
-- **Key finding:** Only `selectivity_proxy` triages correctly (all three pairwise
-  AUROCs > 0.5), but at the cost of decoy discrimination (0.782 vs ensemble 0.848).
+- **Key finding:** `selectivity_proxy` triages best (all three pairwise AUROCs > 0.5),
+  but at the cost of decoy discrimination (0.782 vs ensemble 0.848).
+- **Key finding:** `expert_composite` now faces the same mixed-panel benchmark:
+  it clears the pairwise triage rule (sel>decoy 0.757, hem>decoy 0.746,
+  sel>hem 0.545), but its top-20 includes 5 decoys. It must not replace the
+  ensemble activity gate.
 - **Key finding:** The naive `triage_score` (activity × (1 - hemolysis_risk))
   does NOT fix the anti-selective bias because hemolysis_risk is too weak
   (detection AUROC 0.565, not significant on expanded benchmark).
@@ -226,14 +230,17 @@ Changes:
 Why this matters:
 The triage benchmark (v0.5.12) showed the ensemble has an anti-selective bias
 (sel_vs_hem AUROC=0.466). The expert composite incorporates selectivity_proxy,
-hemolysis risk, helix-hinge analysis, and motif novelty — partially correcting
-this bias. Expert-ranked top-5 candidates have lower mean hemolysis_risk than
-ensemble-ranked top-5 on the mixed benchmark set (verified by test).
+hemolysis risk, helix-hinge analysis, and motif novelty. It partially improves
+selective-vs-hemolytic ordering (triage benchmark: 0.545 vs ensemble 0.466), but
+it also weakens top-k decoy exclusion (top-20 includes 5 decoys vs 0 for ensemble).
+Expert-ranked top-5 candidates have lower mean hemolysis_risk than ensemble-ranked
+top-5 on the mixed benchmark set (verified by test).
 
 Honest limitation: The expert composite's hemolysis components are not statistically
 significant detectors on the expanded n=179 benchmark (AUROC=0.565, CI 0.47-0.66).
-Expert ranking is a safety-aware alternative, not a validated safety guarantee.
-Wet-lab hemolysis assay remains mandatory.
+Expert ranking is a safety-aware alternative, not a validated safety guarantee or
+a replacement for activity-gated ensemble selection. Wet-lab hemolysis assay remains
+mandatory.
 
 ## v1.0 — Validated dry-lab-to-wet-lab loop
 

outputs/metrics_snapshot.json

@@ -105,6 +105,12 @@
         "selective_vs_hemolytic": 0.4661,
         "triages_correctly": false
       },
+      "expert_composite": {
+        "hemolytic_vs_decoy": 0.7461,
+        "selective_vs_decoy": 0.7567,
+        "selective_vs_hemolytic": 0.5449,
+        "triages_correctly": true
+      },
       "hemolysis_risk": {
         "hemolytic_vs_decoy": 0.4924,
         "selective_vs_decoy": 0.485,
@@ -153,6 +159,11 @@
       "HEMOLYTIC": 6,
       "SELECTIVE": 14
     },
+    "top_20_by_expert_composite": {
+      "DECOY": 5,
+      "HEMOLYTIC": 0,
+      "SELECTIVE": 15
+    },
     "top_20_by_triage_score": {
       "DECOY": 0,
       "HEMOLYTIC": 4,

src/openamp_foundry/benchmark/metrics_snapshot.py

@@ -140,6 +140,7 @@ def build_metrics_snapshot(
             "best_scorer": triage["best_scorer"],
             "top_20_by_ensemble": triage["top_20_by_ensemble"],
             "top_20_by_triage_score": triage["top_20_by_triage_score"],
+            "top_20_by_expert_composite": triage["top_20_by_expert_composite"],
             "per_scorer": {
                 scorer: {
                     "selective_vs_decoy": info["selective_vs_decoy"]["auroc"],

src/openamp_foundry/benchmark/triage.py

@@ -42,6 +42,7 @@
 from openamp_foundry.scoring.activity import activity_likeness_score
 from openamp_foundry.scoring.boman import boman_activity_score, model_disagreement
 from openamp_foundry.scoring.ensemble import ensemble_score
+from openamp_foundry.scoring.expert import expert_score
 from openamp_foundry.scoring.hemolysis import hemolysis_risk_score
 from openamp_foundry.scoring.novelty import novelty_score
 from openamp_foundry.scoring.safety import safety_score
@@ -101,6 +102,7 @@ def _score_all(rows: list[dict[str, Any]], weights: dict[str, float]) -> list[di
         stability = serum_stability_score(features)
         sel_proxy = float(features.get("selectivity_proxy", 0.0))
         hemo_risk = hemolysis_risk_score(features)
+        exp = expert_score(seq, features=features)
         raw = {
             "activity": act, "safety": safe,
             "synthesis": synth, "novelty": nov,
@@ -130,6 +132,7 @@ def _score_all(rows: list[dict[str, Any]], weights: dict[str, float]) -> list[di
             "serum_stability": round(stability, 4),
             "selectivity_proxy": round(sel_proxy, 4),
             "hemolysis_risk": round(hemo_risk, 4),
+            "expert_composite": exp.composite,
             "triage_score": triage_score,
             "safe_weighted_ensemble": round(safe_weighted_ens, 4),
         })
@@ -192,7 +195,7 @@ def run_triage_benchmark(
     scorers = [
         "ensemble", "activity", "safety", "synthesis",
         "selectivity_proxy", "hemolysis_risk", "serum_stability",
-        "triage_score", "safe_weighted_ensemble",
+        "expert_composite", "triage_score", "safe_weighted_ensemble",
     ]
 
     per_scorer = {}
@@ -267,6 +270,14 @@ def run_triage_benchmark(
         "DECOY": top_20_triage_classes.count("DECOY"),
     }
 
+    by_expert = sorted(all_scored, key=lambda x: x["expert_composite"], reverse=True)
+    top_20_expert_classes = [r["triage_class"] for r in by_expert[:20]]
+    top_20_expert_dist = {
+        "SELECTIVE": top_20_expert_classes.count("SELECTIVE"),
+        "HEMOLYTIC": top_20_expert_classes.count("HEMOLYTIC"),
+        "DECOY": top_20_expert_classes.count("DECOY"),
+    }
+
     return {
         "benchmark": "multi_class_triage",
         "n_selective": len(selective_scored),
@@ -277,6 +288,7 @@ def run_triage_benchmark(
         "best_scorer": best_scorer,
         "top_20_by_ensemble": top_20_dist,
         "top_20_by_triage_score": top_20_triage_dist,
+        "top_20_by_expert_composite": top_20_expert_dist,
         "known_blind_spots": [
             "The ensemble ranks hemolytic AMPs above selective AMPs "
             "(anti-selective bias) because hemolytic AMPs have stronger "
@@ -289,6 +301,9 @@ def run_triage_benchmark(
             "The triage_score (activity * (1 - hemolysis_risk)) is a simple "
             "composite that may not outperform the ensemble on selective_vs_decoy "
             "because hemolysis_risk is not a significant detector.",
+            "The expert_composite is now included in this benchmark because it is "
+            "available as a ranking mode. It must earn selection authority on the "
+            "same mixed-panel task as simpler scorers.",
         ],
         "disclaimer": (
             "These are computational benchmark results on literature-curated "

src/openamp_foundry/selection/pareto.py

@@ -13,9 +13,9 @@ def rank_candidates(
         candidates: List of ScoredCandidate objects.
         ranking_mode: "ensemble" (default) or "expert". The expert mode
             ranks by the expert composite score, which incorporates
-            selectivity, hemolysis risk, and helix-hinge analysis —
-            partially correcting the ensemble's anti-selective bias
-            (triage benchmark: ensemble ranks hemolytic > selective).
+            selectivity, hemolysis risk, and helix-hinge analysis. It is a
+            safety-aware alternative ranking, not a validated safety predictor;
+            benchmark evidence must be checked before giving it selection authority.
 
     Returns:
         Candidates sorted by the chosen score, highest first.

tests/test_cli_triage.py

@@ -31,5 +31,6 @@ def test_bench_triage_cli(tmp_path, capsys):
     assert data["n_hemolytic"] > 0
     assert data["n_decoy"] > 0
     assert "ensemble" in data["per_scorer"]
+    assert "expert_composite" in data["per_scorer"]
     assert "triage_score" in data["per_scorer"]
     assert "selective_vs_hemolytic" in data["per_scorer"]["ensemble"]

tests/test_metrics_snapshot.py

@@ -18,6 +18,9 @@ def test_metrics_snapshot_matches_current_benchmark_truth(tmp_path):
     assert snapshot["expert_ablation"]["expert_auroc"] == 0.7119
     assert snapshot["selectivity"]["n_hemolytic"] == 54
     assert snapshot["triage"]["best_scorer"] == "selectivity_proxy"
+    assert "expert_composite" in snapshot["triage"]["per_scorer"]
+    assert snapshot["triage"]["per_scorer"]["expert_composite"]["triages_correctly"] is True
+    assert snapshot["triage"]["top_20_by_expert_composite"]["DECOY"] > 0
 
     out = tmp_path / "metrics_snapshot.json"
     out.write_text(json.dumps(snapshot, indent=2, sort_keys=True), encoding="utf-8")

tests/test_triage_benchmark.py

@@ -44,6 +44,7 @@ def test_returns_required_keys(self, result):
         for key in ["benchmark", "n_selective", "n_hemolytic", "n_decoy",
                      "n_total", "per_scorer", "best_scorer",
                      "top_20_by_ensemble", "top_20_by_triage_score",
+                     "top_20_by_expert_composite",
                      "known_blind_spots", "disclaimer"]:
             assert key in result, f"Missing key: {key}"
 
@@ -60,7 +61,7 @@ def test_per_scorer_has_all_scorers(self, result):
         expected = {
             "ensemble", "activity", "safety", "synthesis",
             "selectivity_proxy", "hemolysis_risk", "serum_stability",
-            "triage_score", "safe_weighted_ensemble",
+            "expert_composite", "triage_score", "safe_weighted_ensemble",
         }
         assert expected.issubset(set(result["per_scorer"].keys()))
 
@@ -165,6 +166,18 @@ def test_top_20_by_triage_contains_no_decoys(self, result):
             "Triage score top-20 should contain no decoys"
         )
 
+    def test_expert_composite_is_benchmarked_before_selection_authority(self, result):
+        """The selectable expert ranking mode must face the mixed-panel benchmark."""
+        expert = result["per_scorer"]["expert_composite"]
+        assert "selective_vs_decoy" in expert
+        assert "hemolytic_vs_decoy" in expert
+        assert "selective_vs_hemolytic" in expert
+        assert expert["triages_correctly"] is True
+        assert result["top_20_by_expert_composite"]["DECOY"] > 0, (
+            "Expert composite passes pairwise triage but admits decoys into top-k; "
+            "do not treat it as a replacement for the ensemble activity gate."
+        )
+
     def test_top_20_by_triage_has_more_selective_than_ensemble(self, result):
         """The triage_score should rank MORE selective AMPs in top-20 than the ensemble.