diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 8b79c9fc..ad1b5bc3 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-01 (triage benchmark added) +> **Last updated:** 2026-07-02 (strict triage benchmark added) > **Pipeline version:** v0.5.x > **Branch:** main @@ -452,11 +452,61 @@ top-20. A successful virtual assay must avoid both failures. inter-assay variability. The binary thresholds (25 / 100 µg/mL) are coarse. The MODERATE class (HC50 25-100, n=68) is excluded from the binary task. +### Strict Triage: Composition-Matched Decoys (v0.5.14 — added 2026-07-02) + +> The standard triage benchmark uses random background peptides as decoys. +> These are trivially distinguishable from AMPs because their composition is +> protein-like, not AMP-like. This inflates selective_vs_decoy and +> hemolytic_vs_decoy AUROCs, making scorers appear to triage well. +> +> The strict triage benchmark replaces random decoys with **composition-matched +> scrambled versions** of the selective AMPs — same amino acids, permuted order. +> This destroys amphipathic helical phase, hydrophobic moment, and charge +> distribution patterns while preserving all composition-based features. + +**Key finding: standard triage success was partly an illusion.** + +| Scorer | Std sel_vs_dec | Strict sel_vs_dec | Std sel_vs_hemo | Strict sel_vs_hemo | Std correct | Strict correct | +|-------|-----------------|-------------------|------------------|---------------------|--------------|----------------| +| ensemble | 0.848 | **0.572** | 0.466 | 0.466 | NO | NO | +| activity | 0.885 | **0.617** | 0.430 | 0.430 | NO | NO | +| selectivity_proxy | 0.782 | **0.500** | 0.610 | 0.610 | YES | **NO** | +| expert_composite | 0.757 | **0.510** | 0.545 | 0.545 | YES | **NO** | +| triage_score | 0.863 | **0.674** | 0.462 | 0.462 | NO | NO | +| hemolysis_risk | 0.485 | 0.617 | 0.488 | 0.488 | NO | NO | + +**What this reveals:** + +1. **selectivity_proxy collapses to exactly 0.5000** on selective_vs_decoy — + confirming it is purely composition-driven (charge and GRAVY are identical + between a sequence and its scrambled version). + +2. **The ensemble drops from 0.848 to 0.572** — most of its apparent triage + power was composition-based, not order-based. + +3. **No scorer triages correctly** with composition-matched decoys. The standard + triage "success" of selectivity_proxy and expert_composite was an artifact + of trivially distinguishable decoys. + +4. **selective_vs_hemolytic is stable** across both benchmarks (identical AUROCs) + — as expected, since both classes are real AMP sequences and only the decoy + class changes. + +5. **The ensemble admits 7 scrambled decoys into top-20** (vs 0 with random + decoys) — it cannot distinguish real AMPs from scrambled versions of themselves. + +**Implication:** The pipeline's triage signal is almost entirely composition-driven. +The real bottleneck is selective-vs-hemolytic discrimination, which requires +structural or contextual features beyond what current 1D physicochemical scorers +can capture. Any future virtual assay layer must demonstrate order-dependent +triage signal on this strict benchmark before claiming to improve candidate +selection. + ## Test Suite | Metric | Value | |--------|-------| -| Total tests | 1471+ | +| Total tests | 1515+ | | Coverage (branch) | 99% (6 CLI guard lines only) | | Source modules at 100% | All pipeline, QC, scoring modules | @@ -489,4 +539,5 @@ The MODERATE class (HC50 25-100, n=68) is excluded from the binary task. | 2026-07-01 | Dedicated hemolysis risk scorer: 4-component score (synth+aromatic+face+cys) achieves detection AUROC=0.9218 (CI: 0.82-0.99); integrated into expert composite (detection 0.5119→0.6429); safety scorer unchanged; 1471 tests | OpenAMP loop | | 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop | | 2026-07-01 | Expert composite ranking integration: `score_candidates()` now computes `expert_composite` and `hemolysis_risk`; `--ranking-mode expert` CLI flag; expert-ranked top-5 have lower mean hemolysis_risk than ensemble | OpenAMP loop | +| 2026-07-02 | **Strict triage benchmark added:** composition-matched scrambled decoys replace random background. No scorer triages correctly — standard triage "success" of selectivity_proxy (0.782 sel_vs_dec) and expert_composite (0.757) was inflated by trivially distinguishable decoys. selectivity_proxy collapses to 0.500 (purely composition-driven), ensemble drops to 0.572. Real bottleneck (selective_vs_hemolytic) unchanged. | OpenAMP loop | | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 0a65cfcf..7b664338 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -242,6 +242,40 @@ Expert ranking is a safety-aware alternative, not a validated safety guarantee o a replacement for activity-gated ensemble selection. Wet-lab hemolysis assay remains mandatory. +## v0.5.14 — Strict Triage Benchmark: Composition-Matched Decoys ✓ (2026-07-02) + +The standard triage benchmark (v0.5.12) uses random background peptides as +decoys. These are trivially distinguishable from AMPs because their amino acid +composition is protein-like, not AMP-like. This inflates selective_vs_decoy and +hemolytic_vs_decoy AUROCs, making scorers appear to triage well when they are +really only solving the easy "AMP vs random" problem. + +The strict triage benchmark replaces random decoys with **composition-matched +scrambled versions** of the selective AMPs — same amino acids, permuted order. +This destroys amphipathic helical phase, hydrophobic moment, and charge +distribution patterns while preserving all composition-based features. + +Changes: +- `benchmark/triage.py`: `run_strict_triage_benchmark()` — generates scrambled + decoys, runs the same 10-scorer pairwise triage comparison +- `benchmark/metrics_snapshot.py`: strict_triage section added to snapshot +- `tests/test_triage_benchmark.py`: 12 new tests for strict triage (structure + + findings) +- `outputs/metrics_snapshot.json`: regenerated with strict triage results +- `docs/METRICS_CURRENT.md`: strict triage results table and interpretation + +Key honest finding: **No scorer triages correctly with composition-matched +decoys.** The standard triage "success" of selectivity_proxy and +expert_composite was an artifact of trivially distinguishable decoys. +selectivity_proxy collapses to exactly 0.5000 on selective_vs_decoy (confirming +it is purely composition-driven), and the ensemble drops from 0.848 to 0.572. + +The selective_vs_hemolytic AUROC is identical between standard and strict +(expected: both classes are real AMPs, only the decoy class changes). This +confirms the real bottleneck — selective-vs-hemolytic discrimination — is +unchanged and requires structural or contextual features beyond current 1D +physicochemical scorers. + ## v1.0 — Validated dry-lab-to-wet-lab loop - independently reviewed assay batch (expert_review.yml GitHub issue template); @@ -255,7 +289,7 @@ mandatory. - ~~write the first explicit simulator scope document: what OpenAMP will model and what it will not;~~ - ~~add membrane/selectivity/stability proxy interfaces with uncertainty fields;~~ -- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ (v0.5.12: triage benchmark added — ensemble fails, selectivity_proxy only correct triager) +- ~~benchmark candidate triage against a reference panel that includes selective AMPs, hemolytic positives, inactive peptides, and random controls;~~ (v0.5.12: standard triage added — selectivity_proxy and expert_composite pass; v0.5.14: strict triage with composition-matched decoys added — **no scorer passes**, standard triage success was inflated by trivially distinguishable random decoys) - require every proxy module to justify itself against cheap heuristic baselines before it affects selection. ## v2.x — Wet-lab compression engine diff --git a/outputs/metrics_snapshot.json b/outputs/metrics_snapshot.json index c458076b..a7047a52 100644 --- a/outputs/metrics_snapshot.json +++ b/outputs/metrics_snapshot.json @@ -90,6 +90,90 @@ "recall_at_20": 0.2105, "recall_at_43": 0.4211 }, + "strict_triage": { + "best_scorer": "triage_score", + "decoy_type": "composition_matched_scrambled", + "n_decoy": 125, + "n_hemolytic": 45, + "n_selective": 125, + "per_scorer": { + "activity": { + "hemolytic_vs_decoy": 0.6974, + "selective_vs_decoy": 0.6167, + "selective_vs_hemolytic": 0.43, + "triages_correctly": false + }, + "ensemble": { + "hemolytic_vs_decoy": 0.6156, + "selective_vs_decoy": 0.5716, + "selective_vs_hemolytic": 0.4661, + "triages_correctly": false + }, + "expert_composite": { + "hemolytic_vs_decoy": 0.4755, + "selective_vs_decoy": 0.5102, + "selective_vs_hemolytic": 0.5449, + "triages_correctly": false + }, + "hemolysis_risk": { + "hemolytic_vs_decoy": 0.6199, + "selective_vs_decoy": 0.6173, + "selective_vs_hemolytic": 0.4876, + "triages_correctly": false + }, + "safe_weighted_ensemble": { + "hemolytic_vs_decoy": 0.6857, + "selective_vs_decoy": 0.6405, + "selective_vs_hemolytic": 0.483, + "triages_correctly": false + }, + "safety": { + "hemolytic_vs_decoy": 0.348, + "selective_vs_decoy": 0.3882, + "selective_vs_hemolytic": 0.5384, + "triages_correctly": false + }, + "selectivity_proxy": { + "hemolytic_vs_decoy": 0.3905, + "selective_vs_decoy": 0.5, + "selective_vs_hemolytic": 0.6095, + "triages_correctly": false + }, + "serum_stability": { + "hemolytic_vs_decoy": 0.4281, + "selective_vs_decoy": 0.4908, + "selective_vs_hemolytic": 0.5686, + "triages_correctly": false + }, + "synthesis": { + "hemolytic_vs_decoy": 0.6096, + "selective_vs_decoy": 0.5672, + "selective_vs_hemolytic": 0.4692, + "triages_correctly": false + }, + "triage_score": { + "hemolytic_vs_decoy": 0.7346, + "selective_vs_decoy": 0.6737, + "selective_vs_hemolytic": 0.4622, + "triages_correctly": false + } + }, + "top_20_by_ensemble": { + "DECOY": 7, + "HEMOLYTIC": 5, + "SELECTIVE": 8 + }, + "top_20_by_expert_composite": { + "DECOY": 7, + "HEMOLYTIC": 0, + "SELECTIVE": 13 + }, + "top_20_by_triage_score": { + "DECOY": 2, + "HEMOLYTIC": 4, + "SELECTIVE": 14 + } + }, "triage": { "best_scorer": "selectivity_proxy", "per_scorer": { diff --git a/src/openamp_foundry/benchmark/metrics_snapshot.py b/src/openamp_foundry/benchmark/metrics_snapshot.py index 30ea8b56..6eda652f 100644 --- a/src/openamp_foundry/benchmark/metrics_snapshot.py +++ b/src/openamp_foundry/benchmark/metrics_snapshot.py @@ -10,7 +10,7 @@ run_retrospective_benchmark, run_selectivity_benchmark, ) -from openamp_foundry.benchmark.triage import run_triage_benchmark +from openamp_foundry.benchmark.triage import run_strict_triage_benchmark, run_triage_benchmark def build_metrics_snapshot( @@ -61,6 +61,10 @@ def build_metrics_snapshot( config_path=standard_config, n_bootstrap=n_bootstrap, ) + strict_triage = run_strict_triage_benchmark( + hemolysis_csv=hemolysis_csv, + n_bootstrap=n_bootstrap, + ) return { "schema_version": 1, @@ -151,4 +155,23 @@ def build_metrics_snapshot( for scorer, info in triage["per_scorer"].items() }, }, + "strict_triage": { + "decoy_type": strict_triage["decoy_type"], + "n_selective": strict_triage["n_selective"], + "n_hemolytic": strict_triage["n_hemolytic"], + "n_decoy": strict_triage["n_decoy"], + "best_scorer": strict_triage["best_scorer"], + "top_20_by_ensemble": strict_triage["top_20_by_ensemble"], + "top_20_by_triage_score": strict_triage["top_20_by_triage_score"], + "top_20_by_expert_composite": strict_triage["top_20_by_expert_composite"], + "per_scorer": { + scorer: { + "selective_vs_decoy": info["selective_vs_decoy"]["auroc"], + "hemolytic_vs_decoy": info["hemolytic_vs_decoy"]["auroc"], + "selective_vs_hemolytic": info["selective_vs_hemolytic"]["auroc"], + "triages_correctly": info["triages_correctly"], + } + for scorer, info in strict_triage["per_scorer"].items() + }, + }, } diff --git a/src/openamp_foundry/benchmark/triage.py b/src/openamp_foundry/benchmark/triage.py index ffbd7b59..2877ef68 100644 --- a/src/openamp_foundry/benchmark/triage.py +++ b/src/openamp_foundry/benchmark/triage.py @@ -186,11 +186,10 @@ def run_triage_benchmark( selective = [r for r in hemo_rows if r["hemolysis_class"] == "SELECTIVE"] hemolytic = [r for r in hemo_rows if r["hemolysis_class"] == "HEMOLYTIC"] - decoys = decoy_rows selective_scored = _score_all(selective, weights) hemolytic_scored = _score_all(hemolytic, weights) - decoy_scored = _score_all(decoys, weights) + decoy_scored = _score_all(decoy_rows, weights) scorers = [ "ensemble", "activity", "safety", "synthesis", @@ -313,3 +312,238 @@ def run_triage_benchmark( "Wet-lab validation remains mandatory for all candidates." ), } + + +def _scramble_sequence(seq: str, seed: int = 42) -> str: + """Scramble a sequence while preserving amino acid composition. + + Uses a deterministic RNG seed so results are reproducible. + Returns a permutation of the input sequence's residues. + If the sequence is too short to scramble meaningfully (< 4 residues), + returns the original sequence. + """ + import random as _random + if len(seq) < 4: + return seq + rng = _random.Random(seed) + chars = list(seq) + rng.shuffle(chars) + result = "".join(chars) + # If the scramble accidentally equals the original (rare for short seqs), + # try again with a different seed + if result == seq: + rng2 = _random.Random(seed + 1) + rng2.shuffle(chars) + result = "".join(chars) + return result + + +def _generate_scrambled_decoys( + selective_rows: list[dict[str, Any]], + seed: int = 42, +) -> list[dict[str, Any]]: + """Create composition-matched decoys by scrambling each selective AMP. + + Each decoy has the exact same amino acid composition as its parent + selective AMP, but with a random permutation of residue order. This + destroys amphipathic helical phase, hydrophobic moment, and charge + distribution patterns while preserving all composition-based features + (charge, hydrophobicity, aromatic content, length). + + The resulting triage benchmark tests whether the pipeline has any + ORDER-DEPENDENT signal: can it distinguish real selective AMPs from + their composition-matched scrambled versions? + + For most composition-heavy scorers (activity, ensemble, safety), the + answer is expected to be NO — they will collapse to ~0.5 AUROC on + selective vs scrambled decoy. This is the honest finding that the + standard triage benchmark masks by using random protein-like decoys. + """ + decoys = [] + for i, row in enumerate(selective_rows): + parent_seq = row["sequence"] + scrambled = _scramble_sequence(parent_seq, seed=seed + i) + decoys.append({ + "id": f"SCR-{row['id']}", + "sequence": scrambled, + "family": f"scrambled_{row.get('family', 'amp')}", + "source": f"composition_matched_seed{seed + i}", + "parent_id": row["id"], + "parent_sequence": parent_seq, + }) + return decoys + + +def run_strict_triage_benchmark( + hemolysis_csv: str | Path, + seed: int = 42, + n_bootstrap: int = 2000, +) -> dict[str, Any]: + """Run the strict triage benchmark with composition-matched decoys. + + This benchmark replaces the random background decoys from + run_triage_benchmark with composition-matched scrambled versions of + the selective AMPs. It tests whether the pipeline's scorers have any + order-dependent triage signal — beyond what raw composition provides. + + The standard triage benchmark uses random background peptides that are + trivially distinguishable from AMPs (composition is protein-like, not + AMP-like). This inflates selective_vs_decoy and hemolytic_vs_decoy + AUROCs, making scorers appear to triage well when they are really only + solving the easy "AMP vs random" problem. + + The strict triage benchmark reveals: + - Which scorers have genuine order-dependent signal (hydrophobic + moment, amphipathic phase, charge distribution) + - Which scorers are purely composition-driven and collapse when + composition is controlled for + - Whether the selective_vs_hemolytic signal is composition-driven + (should be similar in strict and standard) or order-driven + + Expected honest findings: + - activity, ensemble: selective_vs_decoy collapses toward 0.5 because + they are composition-based scorers + - hydrophobic_moment-driven components: some signal may remain + - selectivity_proxy: may retain weak signal because it uses GRAVY + (a composition metric) but also charge distribution + - selective_vs_hemolytic should be SIMILAR to standard triage because + both selective and hemolytic AMPs are real sequences; only the + decoy class changes + + All results are computational. No biological activity is implied. + """ + config = load_config("configs/pipeline.yaml") + weights = config["weights"] + + hemo_rows = _load_hemolysis_reference(hemolysis_csv) + + selective = [r for r in hemo_rows if r["hemolysis_class"] == "SELECTIVE"] + hemolytic = [r for r in hemo_rows if r["hemolysis_class"] == "HEMOLYTIC"] + + # Generate composition-matched decoys from selective AMPs + scrambled_decoys = _generate_scrambled_decoys(selective, seed=seed) + decoy_rows = scrambled_decoys + + selective_scored = _score_all(selective, weights) + hemolytic_scored = _score_all(hemolytic, weights) + decoy_scored = _score_all(decoy_rows, weights) + + scorers = [ + "ensemble", "activity", "safety", "synthesis", + "selectivity_proxy", "hemolysis_risk", "serum_stability", + "expert_composite", "triage_score", "safe_weighted_ensemble", + ] + + per_scorer = {} + for scorer in scorers: + sel_vals = [s[scorer] for s in selective_scored] + hemo_vals = [s[scorer] for s in hemolytic_scored] + decoy_vals = [s[scorer] for s in decoy_scored] + + if scorer == "hemolysis_risk": + sel_vals = [1.0 - v for v in sel_vals] + hemo_vals = [1.0 - v for v in hemo_vals] + decoy_vals = [1.0 - v for v in decoy_vals] + + sel_vs_decoy = _pairwise_auroc(sel_vals, decoy_vals, n_bootstrap) + hemo_vs_decoy = _pairwise_auroc(hemo_vals, decoy_vals, n_bootstrap) + sel_vs_hemo = _pairwise_auroc(sel_vals, hemo_vals, n_bootstrap) + + all_above = all( + r["auroc"] > 0.5 + for r in [sel_vs_decoy, hemo_vs_decoy, sel_vs_hemo] + ) + all_strong = all( + r["auroc"] > 0.60 + for r in [sel_vs_decoy, hemo_vs_decoy, sel_vs_hemo] + ) + + per_scorer[scorer] = { + "selective_vs_decoy": sel_vs_decoy, + "hemolytic_vs_decoy": hemo_vs_decoy, + "selective_vs_hemolytic": sel_vs_hemo, + "triages_correctly": all_above, + "strong_triage": all_strong, + } + + best_scorer = max( + per_scorer, + key=lambda k: ( + per_scorer[k]["selective_vs_hemolytic"]["auroc"] + + per_scorer[k]["selective_vs_decoy"]["auroc"] + ) / 2, + ) + + # Class distribution in top-k + all_scored = selective_scored + hemolytic_scored + decoy_scored + for s in all_scored: + s["triage_class"] = ( + "SELECTIVE" if s.get("hemolysis_class") == "SELECTIVE" + else "HEMOLYTIC" if s.get("hemolysis_class") == "HEMOLYTIC" + else "DECOY" + ) + + by_ensemble = sorted(all_scored, key=lambda x: x["ensemble"], reverse=True) + top_20_classes = [r["triage_class"] for r in by_ensemble[:20]] + top_20_dist = { + "SELECTIVE": top_20_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_classes.count("HEMOLYTIC"), + "DECOY": top_20_classes.count("DECOY"), + } + + by_triage = sorted(all_scored, key=lambda x: x["triage_score"], reverse=True) + top_20_triage_classes = [r["triage_class"] for r in by_triage[:20]] + top_20_triage_dist = { + "SELECTIVE": top_20_triage_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_triage_classes.count("HEMOLYTIC"), + "DECOY": top_20_triage_classes.count("DECOY"), + } + + by_expert = sorted(all_scored, key=lambda x: x["expert_composite"], reverse=True) + top_20_expert_classes = [r["triage_class"] for r in by_expert[:20]] + top_20_expert_dist = { + "SELECTIVE": top_20_expert_classes.count("SELECTIVE"), + "HEMOLYTIC": top_20_expert_classes.count("HEMOLYTIC"), + "DECOY": top_20_expert_classes.count("DECOY"), + } + + return { + "benchmark": "strict_multi_class_triage", + "decoy_type": "composition_matched_scrambled", + "decoy_source": "scrambled_selective_amps", + "decoy_seed": seed, + "n_selective": len(selective_scored), + "n_hemolytic": len(hemolytic_scored), + "n_decoy": len(decoy_scored), + "n_total": len(all_scored), + "per_scorer": per_scorer, + "best_scorer": best_scorer, + "top_20_by_ensemble": top_20_dist, + "top_20_by_triage_score": top_20_triage_dist, + "top_20_by_expert_composite": top_20_expert_dist, + "known_blind_spots": [ + "Composition-matched decoys control for amino acid composition, " + "so only order-dependent features (hydrophobic moment, amphipathic " + "phase, charge distribution) can distinguish selective AMPs from " + "their scrambled versions.", + "Scorers that are purely composition-driven (charge density, GRAVY, " + "hydrophobic fraction) will collapse to ~0.5 AUROC on " + "selective_vs_decoy because the scrambled decoy has identical " + "composition to the parent selective AMP.", + "The selective_vs_hemolytic AUROC should be similar to the standard " + "triage benchmark because both classes are real AMP sequences; only " + "the decoy class changes.", + "If NO scorer achieves selective_vs_decoy > 0.55, the pipeline has " + "no order-dependent triage signal beyond composition, which means " + "the current selection is entirely composition-driven and cannot " + "distinguish a real AMP from a scrambled version of itself.", + ], + "disclaimer": ( + "These are computational benchmark results on literature-curated " + "AMP sequences and their composition-matched scrambled versions. " + "They do not prove biological activity, safety, or hemolysis. " + "They measure whether the pipeline's scorers have order-dependent " + "triage signal beyond composition. Wet-lab validation remains " + "mandatory for all candidates." + ), + } diff --git a/tests/test_triage_benchmark.py b/tests/test_triage_benchmark.py index 454e8fef..18e217c0 100644 --- a/tests/test_triage_benchmark.py +++ b/tests/test_triage_benchmark.py @@ -20,7 +20,7 @@ import pytest -from openamp_foundry.benchmark.triage import run_triage_benchmark +from openamp_foundry.benchmark.triage import run_strict_triage_benchmark, run_triage_benchmark HEMOLYSIS_CSV = "examples/validation/hemolysis_reference.csv" @@ -191,3 +191,134 @@ def test_top_20_by_triage_has_more_selective_than_ensemble(self, result): f"triage_score top-20 selective={tri_sel} should be >= " f"ensemble top-20 selective={ens_sel}" ) + + +# --------------------------------------------------------------------------- +# Strict triage benchmark: composition-matched decoys +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="class") +def strict_result(): + """Run the strict triage benchmark once for all tests in this class.""" + hemo = Path(HEMOLYSIS_CSV) + if not hemo.exists(): + pytest.skip("Hemolysis benchmark data not found — run from project root") + return run_strict_triage_benchmark(hemo, n_bootstrap=200) + + +class TestStrictTriageBenchmarkStructure: + """Verify the strict triage benchmark returns well-formed results.""" + + def test_returns_required_keys(self, strict_result): + for key in ["benchmark", "decoy_type", "n_selective", "n_hemolytic", + "n_decoy", "n_total", "per_scorer", "best_scorer", + "top_20_by_ensemble", "top_20_by_triage_score", + "top_20_by_expert_composite", + "known_blind_spots", "disclaimer"]: + assert key in strict_result, f"Missing key: {key}" + + def test_benchmark_name(self, strict_result): + assert strict_result["benchmark"] == "strict_multi_class_triage" + + def test_decoy_type(self, strict_result): + assert strict_result["decoy_type"] == "composition_matched_scrambled" + + def test_class_counts_positive(self, strict_result): + assert strict_result["n_selective"] > 0 + assert strict_result["n_hemolytic"] > 0 + assert strict_result["n_decoy"] > 0 + assert strict_result["n_total"] == ( + strict_result["n_selective"] + + strict_result["n_hemolytic"] + + strict_result["n_decoy"] + ) + + def test_decoy_count_equals_selective_count(self, strict_result): + """Each selective AMP generates exactly one scrambled decoy.""" + assert strict_result["n_decoy"] == strict_result["n_selective"] + + +class TestStrictTriageBenchmarkFindings: + """Test the key honest findings from the strict triage benchmark.""" + + def test_no_scorer_triages_correctly(self, strict_result): + """With composition-matched decoys, no scorer should triage correctly. + + This is the central honest finding: the standard triage benchmark + appears to show that selectivity_proxy and expert_composite triage + correctly, but this is an illusion created by trivially distinguishable + random decoys. When decoys have the same composition as selective AMPs, + the apparent triage signal vanishes. + """ + triaging = [ + s for s, info in strict_result["per_scorer"].items() + if info["triages_correctly"] + ] + assert len(triaging) == 0, ( + f"Scorers that triage correctly with composition-matched decoys: {triaging}. " + "Expected none — if any scorer now triages correctly, the pipeline " + "has genuine order-dependent triage signal and this test should be updated." + ) + + def test_selectivity_proxy_collapses_on_selective_vs_decoy(self, strict_result): + """The selectivity_proxy should be ~0.5 on selective vs scrambled decoys. + + This confirms it is purely composition-driven: charge and GRAVY are + identical between a sequence and its scrambled version. + """ + sel_vs_dec = strict_result["per_scorer"]["selectivity_proxy"]["selective_vs_decoy"]["auroc"] + assert abs(sel_vs_dec - 0.5) < 0.02, ( + f"selectivity_proxy selective_vs_decoy={sel_vs_dec:.4f}: " + "expected ~0.5 (composition-matched decoy has identical charge/GRAVY). " + "If this has changed, selectivity_proxy may have order-dependent signal." + ) + + def test_ensemble_drops_on_selective_vs_decoy(self, strict_result): + """The ensemble should drop dramatically vs standard triage on selective_vs_decoy. + + Standard triage: ensemble sel_vs_decoy ~0.85 (trivially distinguishes AMPs + from random protein sequences). + Strict triage: ensemble sel_vs_decoy should be near 0.5 because most + ensemble signal is composition-based. + """ + ens_sel_vs_dec = strict_result["per_scorer"]["ensemble"]["selective_vs_decoy"]["auroc"] + assert ens_sel_vs_dec < 0.65, ( + f"ensemble selective_vs_decoy={ens_sel_vs_dec:.4f}: " + "expected < 0.65 (composition-matched decoys remove most signal). " + "If this has improved, the ensemble has order-dependent signal." + ) + + def test_selective_vs_hemolytic_unchanged(self, strict_result, result): + """The selective_vs_hemolytic AUROC should be identical between standard and strict. + + Both selective and hemolytic are real AMP sequences in both benchmarks. + Only the decoy class changes, so this pairwise comparison should be stable. + """ + std_sv = result["per_scorer"]["ensemble"]["selective_vs_hemolytic"]["auroc"] + strict_sv = strict_result["per_scorer"]["ensemble"]["selective_vs_hemolytic"]["auroc"] + assert abs(std_sv - strict_sv) < 0.001, ( + f"standard sel_vs_hemo={std_sv:.4f} vs strict={strict_sv:.4f}: " + "expected identical (same AMP sequences, only decoy class changes)." + ) + + def test_ensemble_admits_decoys_into_top_20(self, strict_result): + """The ensemble should admit scrambled decoys into top-20. + + In standard triage, ensemble admits 0 decoys into top-20 because + random background peptides are compositionally unlike AMPs. With + composition-matched decoys, the ensemble cannot distinguish real AMPs + from scrambled versions, so decoys leak into top-k. + """ + n_decoys = strict_result["top_20_by_ensemble"]["DECOY"] + assert n_decoys > 0, ( + "Ensemble should admit composition-matched decoys into top-20. " + "If 0, the ensemble has strong order-dependent signal." + ) + + def test_disclaimer_present(self, strict_result): + assert "composition-matched" in strict_result["disclaimer"].lower() + assert "wet-lab" in strict_result["disclaimer"].lower() + + def test_known_blind_spots_documented(self, strict_result): + assert len(strict_result["known_blind_spots"]) >= 3