diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 5a485056..8b79c9fc 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -390,8 +390,8 @@ hemolysis mechanism is not fully captured by 1D features. > > Run: `make bench-triage` -**Dataset:** 125 selective AMPs (HC50 >= 100 µg/mL) + 45 hemolytic AMPs (HC50 < 25 µg/mL) -+ 96 random background decoys = 266 total. +**Dataset:** 125 selective AMPs (HC50 >= 100 µg/mL) + 54 hemolytic AMPs (HC50 < 25 µg/mL) ++ 96 random background decoys = 275 total. ### Per-scorer pairwise AUROCs @@ -405,6 +405,7 @@ A scorer that triages correctly should have all three AUROCs > 0.5: | ensemble | 0.848 | 0.891 | 0.466 | **NO** (anti-selective) | | activity | 0.885 | 0.934 | 0.430 | NO | | selectivity_proxy | 0.782 | 0.795 | 0.610 | **YES** | +| expert_composite | 0.757 | 0.746 | 0.545 | **YES** | | triage_score (activity × (1 - hemo_risk)) | 0.863 | 0.902 | 0.462 | NO | | safe_weighted_ensemble | 0.849 | 0.890 | 0.483 | NO | | safety | 0.344 | 0.300 | 0.538 | NO | @@ -419,9 +420,10 @@ A scorer that triages correctly should have all three AUROCs > 0.5: bias documented in the selectivity benchmark, now confirmed in the combined triage context. -2. **Only selectivity_proxy triages correctly** (all three AUROCs > 0.5). It - sacrifices some decoy discrimination (0.782 vs ensemble's 0.848) but is the - only scorer that prefers selective AMPs over hemolytic ones. +2. **selectivity_proxy and expert_composite triage correctly by pairwise AUROC** + (all three AUROCs > 0.5). selectivity_proxy remains the best scorer because it + has stronger selective-vs-hemolytic separation (0.610 vs expert_composite 0.545) + while keeping slightly better selective-vs-decoy discrimination (0.782 vs 0.757). 3. **The naive triage_score (activity × (1 - hemolysis_risk)) does NOT fix the anti-selective bias** (sel_vs_hem = 0.462). This is because hemolysis_risk @@ -433,11 +435,18 @@ A scorer that triages correctly should have all three AUROCs > 0.5: The shift is in the right direction but modest — the hemolysis_risk penalty is weak. +5. **Expert-composite top-k failure:** The expert_composite removes hemolytic + AMPs from its top-20 (15 selective / 0 hemolytic), but admits 5 random decoys. + That is a useful negative result: expert ranking is not a replacement for the + ensemble activity gate, even when its pairwise AUROCs clear 0.5. + **Implication for the virtual assay layer:** Any future virtual assay module must beat this triage benchmark baseline. The minimum bar is: triage correctly -(all three AUROCs > 0.5) while maintaining ensemble-level decoy discrimination -(sel_vs_decoy > 0.80). The selectivity_proxy achieves correct triage but at -the cost of decoy discrimination. A successful virtual assay must achieve both. +(all three AUROCs > 0.5), keep decoys out of the top-k selection surface, and +maintain near-ensemble decoy discrimination (sel_vs_decoy > 0.80). The +selectivity_proxy achieves correct triage but loses decoy-discrimination margin. +The expert_composite achieves correct pairwise triage but admits decoys into its +top-20. A successful virtual assay must avoid both failures. **Honest limitation:** The benchmark uses literature HC50 values with high inter-assay variability. The binary thresholds (25 / 100 µg/mL) are coarse. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8b191f34..0a65cfcf 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -189,7 +189,7 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - **Multi-class triage benchmark** (`benchmark/triage.py`): tests whether the pipeline can rank selective AMPs > hemolytic AMPs > random decoys in a single - combined panel (125 selective + 45 hemolytic + 96 decoys = 266 total). + combined panel (125 selective + 54 hemolytic + 96 decoys = 275 total). - Addresses the v1.1 ROADMAP item: "benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls." @@ -198,8 +198,12 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): anti-selective bias. Hemolytic AMPs have stronger amphipathic helices, higher hydrophobic moment, and higher charge — exactly the features the activity scorer rewards. -- **Key finding:** Only `selectivity_proxy` triages correctly (all three pairwise - AUROCs > 0.5), but at the cost of decoy discrimination (0.782 vs ensemble 0.848). +- **Key finding:** `selectivity_proxy` triages best (all three pairwise AUROCs > 0.5), + but at the cost of decoy discrimination (0.782 vs ensemble 0.848). +- **Key finding:** `expert_composite` now faces the same mixed-panel benchmark: + it clears the pairwise triage rule (sel>decoy 0.757, hem>decoy 0.746, + sel>hem 0.545), but its top-20 includes 5 decoys. It must not replace the + ensemble activity gate. - **Key finding:** The naive `triage_score` (activity × (1 - hemolysis_risk)) does NOT fix the anti-selective bias because hemolysis_risk is too weak (detection AUROC 0.565, not significant on expanded benchmark). @@ -226,14 +230,17 @@ Changes: Why this matters: The triage benchmark (v0.5.12) showed the ensemble has an anti-selective bias (sel_vs_hem AUROC=0.466). The expert composite incorporates selectivity_proxy, -hemolysis risk, helix-hinge analysis, and motif novelty — partially correcting -this bias. Expert-ranked top-5 candidates have lower mean hemolysis_risk than -ensemble-ranked top-5 on the mixed benchmark set (verified by test). +hemolysis risk, helix-hinge analysis, and motif novelty. It partially improves +selective-vs-hemolytic ordering (triage benchmark: 0.545 vs ensemble 0.466), but +it also weakens top-k decoy exclusion (top-20 includes 5 decoys vs 0 for ensemble). +Expert-ranked top-5 candidates have lower mean hemolysis_risk than ensemble-ranked +top-5 on the mixed benchmark set (verified by test). Honest limitation: The expert composite's hemolysis components are not statistically significant detectors on the expanded n=179 benchmark (AUROC=0.565, CI 0.47-0.66). -Expert ranking is a safety-aware alternative, not a validated safety guarantee. -Wet-lab hemolysis assay remains mandatory. +Expert ranking is a safety-aware alternative, not a validated safety guarantee or +a replacement for activity-gated ensemble selection. Wet-lab hemolysis assay remains +mandatory. ## v1.0 — Validated dry-lab-to-wet-lab loop diff --git a/outputs/metrics_snapshot.json b/outputs/metrics_snapshot.json index 497bc291..c458076b 100644 --- a/outputs/metrics_snapshot.json +++ b/outputs/metrics_snapshot.json @@ -105,6 +105,12 @@ "selective_vs_hemolytic": 0.4661, "triages_correctly": false }, + "expert_composite": { + "hemolytic_vs_decoy": 0.7461, + "selective_vs_decoy": 0.7567, + "selective_vs_hemolytic": 0.5449, + "triages_correctly": true + }, "hemolysis_risk": { "hemolytic_vs_decoy": 0.4924, "selective_vs_decoy": 0.485, @@ -153,6 +159,11 @@ "HEMOLYTIC": 6, "SELECTIVE": 14 }, + "top_20_by_expert_composite": { + "DECOY": 5, + "HEMOLYTIC": 0, + "SELECTIVE": 15 + }, "top_20_by_triage_score": { "DECOY": 0, "HEMOLYTIC": 4, diff --git a/src/openamp_foundry/benchmark/metrics_snapshot.py b/src/openamp_foundry/benchmark/metrics_snapshot.py index bbd431dc..30ea8b56 100644 --- a/src/openamp_foundry/benchmark/metrics_snapshot.py +++ b/src/openamp_foundry/benchmark/metrics_snapshot.py @@ -140,6 +140,7 @@ def build_metrics_snapshot( "best_scorer": triage["best_scorer"], "top_20_by_ensemble": triage["top_20_by_ensemble"], "top_20_by_triage_score": triage["top_20_by_triage_score"], + "top_20_by_expert_composite": triage["top_20_by_expert_composite"], "per_scorer": { scorer: { "selective_vs_decoy": info["selective_vs_decoy"]["auroc"], diff --git a/src/openamp_foundry/benchmark/triage.py b/src/openamp_foundry/benchmark/triage.py index c9c39772..ffbd7b59 100644 --- a/src/openamp_foundry/benchmark/triage.py +++ b/src/openamp_foundry/benchmark/triage.py @@ -42,6 +42,7 @@ from openamp_foundry.scoring.activity import activity_likeness_score from openamp_foundry.scoring.boman import boman_activity_score, model_disagreement from openamp_foundry.scoring.ensemble import ensemble_score +from openamp_foundry.scoring.expert import expert_score from openamp_foundry.scoring.hemolysis import hemolysis_risk_score from openamp_foundry.scoring.novelty import novelty_score from openamp_foundry.scoring.safety import safety_score @@ -101,6 +102,7 @@ def _score_all(rows: list[dict[str, Any]], weights: dict[str, float]) -> list[di stability = serum_stability_score(features) sel_proxy = float(features.get("selectivity_proxy", 0.0)) hemo_risk = hemolysis_risk_score(features) + exp = expert_score(seq, features=features) raw = { "activity": act, "safety": safe, "synthesis": synth, "novelty": nov, @@ -130,6 +132,7 @@ def _score_all(rows: list[dict[str, Any]], weights: dict[str, float]) -> list[di "serum_stability": round(stability, 4), "selectivity_proxy": round(sel_proxy, 4), "hemolysis_risk": round(hemo_risk, 4), + "expert_composite": exp.composite, "triage_score": triage_score, "safe_weighted_ensemble": round(safe_weighted_ens, 4), }) @@ -192,7 +195,7 @@ def run_triage_benchmark( scorers = [ "ensemble", "activity", "safety", "synthesis", "selectivity_proxy", "hemolysis_risk", "serum_stability", - "triage_score", "safe_weighted_ensemble", + "expert_composite", "triage_score", "safe_weighted_ensemble", ] per_scorer = {} @@ -267,6 +270,14 @@ def run_triage_benchmark( "DECOY": top_20_triage_classes.count("DECOY"), } + by_expert = sorted(all_scored, key=lambda x: x["expert_composite"], reverse=True) + top_20_expert_classes = [r["triage_class"] for r in by_expert[:20]] + top_20_expert_dist = { + "SELECTIVE": top_20_expert_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_expert_classes.count("HEMOLYTIC"), + "DECOY": top_20_expert_classes.count("DECOY"), + } + return { "benchmark": "multi_class_triage", "n_selective": len(selective_scored), @@ -277,6 +288,7 @@ def run_triage_benchmark( "best_scorer": best_scorer, "top_20_by_ensemble": top_20_dist, "top_20_by_triage_score": top_20_triage_dist, + "top_20_by_expert_composite": top_20_expert_dist, "known_blind_spots": [ "The ensemble ranks hemolytic AMPs above selective AMPs " "(anti-selective bias) because hemolytic AMPs have stronger " @@ -289,6 +301,9 @@ def run_triage_benchmark( "The triage_score (activity * (1 - hemolysis_risk)) is a simple " "composite that may not outperform the ensemble on selective_vs_decoy " "because hemolysis_risk is not a significant detector.", + "The expert_composite is now included in this benchmark because it is " + "available as a ranking mode. It must earn selection authority on the " + "same mixed-panel task as simpler scorers.", ], "disclaimer": ( "These are computational benchmark results on literature-curated " diff --git a/src/openamp_foundry/selection/pareto.py b/src/openamp_foundry/selection/pareto.py index 50f9caf8..f675a343 100644 --- a/src/openamp_foundry/selection/pareto.py +++ b/src/openamp_foundry/selection/pareto.py @@ -13,9 +13,9 @@ def rank_candidates( candidates: List of ScoredCandidate objects. ranking_mode: "ensemble" (default) or "expert". The expert mode ranks by the expert composite score, which incorporates - selectivity, hemolysis risk, and helix-hinge analysis — - partially correcting the ensemble's anti-selective bias - (triage benchmark: ensemble ranks hemolytic > selective). + selectivity, hemolysis risk, and helix-hinge analysis. It is a + safety-aware alternative ranking, not a validated safety predictor; + benchmark evidence must be checked before giving it selection authority. Returns: Candidates sorted by the chosen score, highest first. diff --git a/tests/test_cli_triage.py b/tests/test_cli_triage.py index a939d9ba..cc04b98a 100644 --- a/tests/test_cli_triage.py +++ b/tests/test_cli_triage.py @@ -31,5 +31,6 @@ def test_bench_triage_cli(tmp_path, capsys): assert data["n_hemolytic"] > 0 assert data["n_decoy"] > 0 assert "ensemble" in data["per_scorer"] + assert "expert_composite" in data["per_scorer"] assert "triage_score" in data["per_scorer"] assert "selective_vs_hemolytic" in data["per_scorer"]["ensemble"] diff --git a/tests/test_metrics_snapshot.py b/tests/test_metrics_snapshot.py index cf3a5c80..db2ea63a 100644 --- a/tests/test_metrics_snapshot.py +++ b/tests/test_metrics_snapshot.py @@ -18,6 +18,9 @@ def test_metrics_snapshot_matches_current_benchmark_truth(tmp_path): assert snapshot["expert_ablation"]["expert_auroc"] == 0.7119 assert snapshot["selectivity"]["n_hemolytic"] == 54 assert snapshot["triage"]["best_scorer"] == "selectivity_proxy" + assert "expert_composite" in snapshot["triage"]["per_scorer"] + assert snapshot["triage"]["per_scorer"]["expert_composite"]["triages_correctly"] is True + assert snapshot["triage"]["top_20_by_expert_composite"]["DECOY"] > 0 out = tmp_path / "metrics_snapshot.json" out.write_text(json.dumps(snapshot, indent=2, sort_keys=True), encoding="utf-8") diff --git a/tests/test_triage_benchmark.py b/tests/test_triage_benchmark.py index a357cb17..454e8fef 100644 --- a/tests/test_triage_benchmark.py +++ b/tests/test_triage_benchmark.py @@ -44,6 +44,7 @@ def test_returns_required_keys(self, result): for key in ["benchmark", "n_selective", "n_hemolytic", "n_decoy", "n_total", "per_scorer", "best_scorer", "top_20_by_ensemble", "top_20_by_triage_score", + "top_20_by_expert_composite", "known_blind_spots", "disclaimer"]: assert key in result, f"Missing key: {key}" @@ -60,7 +61,7 @@ def test_per_scorer_has_all_scorers(self, result): expected = { "ensemble", "activity", "safety", "synthesis", "selectivity_proxy", "hemolysis_risk", "serum_stability", - "triage_score", "safe_weighted_ensemble", + "expert_composite", "triage_score", "safe_weighted_ensemble", } assert expected.issubset(set(result["per_scorer"].keys())) @@ -165,6 +166,18 @@ def test_top_20_by_triage_contains_no_decoys(self, result): "Triage score top-20 should contain no decoys" ) + def test_expert_composite_is_benchmarked_before_selection_authority(self, result): + """The selectable expert ranking mode must face the mixed-panel benchmark.""" + expert = result["per_scorer"]["expert_composite"] + assert "selective_vs_decoy" in expert + assert "hemolytic_vs_decoy" in expert + assert "selective_vs_hemolytic" in expert + assert expert["triages_correctly"] is True + assert result["top_20_by_expert_composite"]["DECOY"] > 0, ( + "Expert composite passes pairwise triage but admits decoys into top-k; " + "do not treat it as a replacement for the ensemble activity gate." + ) + def test_top_20_by_triage_has_more_selective_than_ensemble(self, result): """The triage_score should rank MORE selective AMPs in top-20 than the ensemble.