diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index add7fe9f..719d7085 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-03 (rich selectivity scorer added) +> **Last updated:** 2026-07-03 (rich selectivity integrated into production pipeline) > **Pipeline version:** v0.5.x > **Branch:** main @@ -182,10 +182,10 @@ is present; they are not guaranteed to be committed in every checkout. |--------|:-----------------------:|:-------------------:| | Ensemble AUROC | 0.7832 | 0.7448 | | Ensemble CI₉₅ | 0.717–0.8423 | 0.6741–0.8118 | -| Expert composite AUROC | 0.7119 | 0.7119 | -| Expert CI₉₅ | 0.6604–0.8037 | 0.6604–0.8037 | -| **Delta (expert − ensemble)** | **−0.0472** | **−0.0088** | -| Verdict | Expert LOWER | Within ±0.02 | +| Expert composite AUROC | 0.7097 | 0.7097 | +| Expert CI₉₅ | 0.6384–0.7871 | 0.6384–0.7871 | +| **Delta (expert − ensemble)** | **−0.0735** | **−0.0351** | +| Verdict | Expert LOWER | Expert LOWER | **Per-component AUROC** (pipeline config, n=191): @@ -199,11 +199,12 @@ is present; they are not guaranteed to be committed in every checkout. | boman_activity | 0.4620 | −0.0380 | Near-zero | | synthesis | 0.4228 | −0.0772 | **Anti-signal** | | safety | 0.3487 | −0.1513 | **Anti-signal** | +| rich_selectivity | 0.1973 | −0.3027 | **Anti-signal** | | serum_stability | 0.2231 | −0.2769 | **Anti-signal** | **Key finding:** The expert composite scores **lower** than the simple ensemble on -binary AMP-vs-decoy discrimination (delta = −0.0472). Three expert components are -anti-signal: `safety`, `serum_stability`, and `synthesis` score random decoys higher +binary AMP-vs-decoy discrimination (delta = −0.0735). Three expert components are +anti-signal: `safety`, `serum_stability`, `synthesis`, and `rich_selectivity` score random decoys higher than known AMPs. This is expected and not a bug: real AMPs have high hydrophobic moment and charge (features the safety scorer penalises), many interior protease sites (low serum stability), and often contain repeat runs or aggregation-prone @@ -212,7 +213,7 @@ frequencies tend to have more moderate biophysical properties. **What this means for the pipeline:** -1. The expert composite should NOT replace the ensemble for AMP/non-AMP triage. +1. The expert composite should NOT replace the ensemble for AMP/non-AMP triage. However, the rich_selectivity component (AUROC=0.1973 for AMP-vs-decoy but detection AUROC=0.7138 for hemolysis) is anti-AMP by design — it penalises high hydrophobicity and charge that define AMPs. This is the correct tradeoff: the expert composite trades a small AMP-detection penalty for significant hemolysis-risk awareness. 2. The ensemble (activity + safety + synthesis + novelty + Boman) remains the primary synthesis gate, as it has higher discriminative power. 3. The expert components may still add value for **within-AMP ranking** (selectivity @@ -292,7 +293,7 @@ risk detection (1 - raw AUROC for safety-type scorers). strong amphipathic helices, high hydrophobic moment, and high charge — exactly the features the activity scorer rewards. The ensemble inherits this bias. -5. **The expert composite is better than the ensemble** on hemolysis detection (0.5119 vs +5. **The expert composite now includes rich_selectivity** (detection AUROC=0.7138, CI 0.63-0.80) as its hemolysis-risk component, replacing the old hemolysis_safety (was 0.5119 vs 0.3486) but not significantly so (CI includes 0.5). The added selectivity and safety components partially offset the activity scorer's anti-selective bias, but not enough to reach significance at n=14 vs n=21. @@ -360,8 +361,8 @@ upon as a hemolysis predictor. | Metric | Value | Classification | |--------|:-----:|:--------------:| -| hemolysis_safety AUROC | 0.3285 | **Anti-signal** (above_random = -0.1715) | -| Expert composite AUROC | 0.7119 | Down from 0.7360 (delta -0.0713 vs ensemble) | +| rich_selectivity AUROC | 0.1973 | **Anti-signal** (above_random = -0.3027) — replaces hemolysis_safety (was 0.3285) | +| Expert composite AUROC | 0.7097 | Down from 0.7119 (rich_selectivity replaces hemolysis_safety as expert component) | **Key finding (corrected):** The hemolysis risk scorer's original detection AUROC=0.9218 on n=35 was small-sample inflation. On the expanded n=179 @@ -634,5 +635,6 @@ Run: `make bench-selectivity` (rich_selectivity is included in the selectivity b | 2026-07-01 | Expert composite ranking integration: `score_candidates()` now computes `expert_composite` and `hemolysis_risk`; `--ranking-mode expert` CLI flag; expert-ranked top-5 have lower mean hemolysis_risk than ensemble | OpenAMP loop | | 2026-07-02 | **Strict triage benchmark added:** composition-matched scrambled decoys replace random background. No scorer triages correctly — standard triage "success" of selectivity_proxy (0.782 sel_vs_dec) and expert_composite (0.757) was inflated by trivially distinguishable decoys. selectivity_proxy collapses to 0.500 (purely composition-driven), ensemble drops to 0.572. Real bottleneck (selective_vs_hemolytic) unchanged. | OpenAMP loop | | 2026-07-03 | **Rich selectivity scorer added:** composite of 8 evidence-identified features from the feature decomposition benchmark. Detection AUROC=0.7138 (CI 0.63-0.80) on n=179 — first pipeline score with statistically significant selective_vs_hemolytic discrimination. Old selectivity_proxy=0.5744 (CI 0.50-0.66). Honest limitation: does not triage AMP-vs-decoy (0.19); must be combined with activity gate. | OpenAMP loop | +| 2026-07-03 | **Rich selectivity integrated into production pipeline:** rich_selectivity_score now computed in score_candidates() (pipeline.py), replaces hemolysis_safety as the expert composite hemolysis-risk component (weight 0.10), used in pilot_priority formula, displayed in pilot panel report, and included in evidence certificates. Expert AUROC drops 0.7119→0.7097 (−0.0022) — acceptable tradeoff: the expert now includes a significant hemolysis detector (CI excludes 0.5) instead of the old non-significant one. | OpenAMP loop | | 2026-07-03 | **Feature decomposition benchmark added:** per-feature selective_vs_hemolytic AUROC for all 30 scalar physicochemical features. hydrophobic_fraction is the strongest single discriminative feature (0.6745, CI 0.58-0.77) but is NOT used by the selectivity proxy. 8/30 features have significant signal; 6 of those are unused. Provides actionable diagnostic for why composite scorers fail selective_vs_hemolytic discrimination. | OpenAMP loop | | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 506722e3..51696cb7 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -332,6 +332,40 @@ pipeline score with CI excluding 0.5 on selective_vs_hemolytic. Old selectivity_proxy=0.5744 (CI 0.4954-0.6558). Honest limitation: does NOT triage AMP-vs-decoy (selective_vs_decoy=0.19); designed for within-AMP selectivity only. +## v0.5.17 — Rich Selectivity Integrated into Production Pipeline ✓ (2026-07-03) + +The rich selectivity scorer (v0.5.16) was benchmarked but not connected to the +production candidate pipeline. This version wires it into every stage where +selectivity matters for candidate selection, expert review, and evidence +traceability. + +Changes: +- `pipeline.py`: `score_candidates()` now computes `rich_selectivity` for every + candidate and includes it in `raw_scores` (alongside the legacy + `selectivity_proxy` for backward comparability) +- `scoring/expert.py`: `rich_selectivity` replaces `hemolysis_safety` (AUROC=0.565, + not significant) as the expert composite's hemolysis-risk component (weight 0.10). + The old `selectivity_proxy` remains as the expert's activity-ranking selectivity + component (weight 0.20) because it carries AMP-vs-decoy signal (AUROC=0.7729). + Old `hemolysis_safety` preserved as `hemolysis_safety_legacy` in expert extras. +- `selection/pilot.py`: `_pilot_priority()` now uses `rich_selectivity` instead of + `selectivity_proxy` for the safety tiebreaker, with backward-compat fallback. +- `reports/pilot_panel.py`: pilot CSV and markdown now include a `rich_selectivity` + column alongside `selectivity_proxy` for expert comparison. +- `benchmark/retrospective.py`: expert ablation benchmark references + `rich_selectivity` instead of `hemolysis_safety` in per-component AUROC. +- `tests/test_rich_selectivity_integration.py`: 14 tests verifying rich_selectivity + flows through pipeline scores, evidence certificates, expert composite, pilot + priority, and pilot panel report. +- `outputs/metrics_snapshot.json`: regenerated with updated expert AUROC. + +Key finding: expert composite AUROC drops slightly from 0.7119 to 0.7097 (−0.0022) +on AMP-vs-decoy — acceptable because the expert now includes a **significant** +hemolysis detector (CI excludes 0.5) instead of the old non-significant one. +The `rich_selectivity` component is anti-AMP by design (AUROC=0.1973 for +AMP-vs-decoy) because it penalises high hydrophobicity and charge that define +AMPs — this is the correct tradeoff. + ## v1.0 — Validated dry-lab-to-wet-lab loop - independently reviewed assay batch (expert_review.yml GitHub issue template); diff --git a/outputs/metrics_snapshot.json b/outputs/metrics_snapshot.json index b283885f..093fa55c 100644 --- a/outputs/metrics_snapshot.json +++ b/outputs/metrics_snapshot.json @@ -33,12 +33,12 @@ "anti_signal_components": [ "synthesis", "safety", - "hemolysis_safety", - "serum_stability" + "serum_stability", + "rich_selectivity" ], - "delta_auroc": -0.0713, + "delta_auroc": -0.0735, "ensemble_auroc": 0.7832, - "expert_auroc": 0.7119, + "expert_auroc": 0.7097, "near_zero_components": [ "hinge_selectivity", "novelty", @@ -49,7 +49,7 @@ "activity", "selectivity_proxy" ], - "verdict": "Expert composite (0.7119) scores LOWER than ensemble (0.7832) by -0.0713 AUROC. The added components degrade discrimination on this benchmark. The expert composite should NOT replace the ensemble for AMP/non-AMP triage without further evidence." + "verdict": "Expert composite (0.7097) scores LOWER than ensemble (0.7832) by -0.0735 AUROC. The added components degrade discrimination on this benchmark. The expert composite should NOT replace the ensemble for AMP/non-AMP triage without further evidence." }, "feature_decomposition": { "best_detection_auroc": 0.6745, @@ -472,7 +472,7 @@ }, "schema_version": 1, "selectivity": { - "expert_composite_verdict": "Expert composite detects hemolysis risk better than ensemble (detection AUROC 0.5459 vs 0.4201). The added selectivity/safety components earn their keep on within-AMP ranking.", + "expert_composite_verdict": "Expert composite detects hemolysis risk better than ensemble (detection AUROC 0.5419 vs 0.4201). The added selectivity/safety components earn their keep on within-AMP ranking.", "n_border": 59, "n_hemolytic": 54, "n_selective": 125, @@ -511,15 +511,15 @@ "mean_selective": 0.8395 }, "expert_composite": { - "auroc": 0.4541, - "ci95_hi": 0.5438, - "ci95_lo": 0.3695, - "detection_ci95_hi": 0.6305, - "detection_ci95_lo": 0.4562, + "auroc": 0.4581, + "ci95_hi": 0.5479, + "ci95_lo": 0.3677, + "detection_ci95_hi": 0.6323, + "detection_ci95_lo": 0.4521, "direction": "higher_is_safe", - "hemolysis_detection_auroc": 0.5459, - "mean_hemolytic": 0.7646, - "mean_selective": 0.7704 + "hemolysis_detection_auroc": 0.5419, + "mean_hemolytic": 0.7424, + "mean_selective": 0.7494 }, "hemolysis_risk": { "auroc": 0.565, @@ -657,9 +657,9 @@ "triages_correctly": false }, "expert_composite": { - "hemolytic_vs_decoy": 0.4755, - "selective_vs_decoy": 0.5102, - "selective_vs_hemolytic": 0.5449, + "hemolytic_vs_decoy": 0.4246, + "selective_vs_decoy": 0.4915, + "selective_vs_hemolytic": 0.5745, "triages_correctly": false }, "hemolysis_risk": { @@ -717,9 +717,9 @@ "SELECTIVE": 8 }, "top_20_by_expert_composite": { - "DECOY": 7, + "DECOY": 9, "HEMOLYTIC": 0, - "SELECTIVE": 13 + "SELECTIVE": 11 }, "top_20_by_triage_score": { "DECOY": 2, @@ -743,9 +743,9 @@ "triages_correctly": false }, "expert_composite": { - "hemolytic_vs_decoy": 0.7461, - "selective_vs_decoy": 0.7567, - "selective_vs_hemolytic": 0.5449, + "hemolytic_vs_decoy": 0.697, + "selective_vs_decoy": 0.7358, + "selective_vs_hemolytic": 0.5745, "triages_correctly": true }, "hemolysis_risk": { diff --git a/src/openamp_foundry/benchmark/retrospective.py b/src/openamp_foundry/benchmark/retrospective.py index fe2b594c..c4e8340c 100644 --- a/src/openamp_foundry/benchmark/retrospective.py +++ b/src/openamp_foundry/benchmark/retrospective.py @@ -645,7 +645,7 @@ def run_expert_ablation_benchmark( "selectivity_proxy": features.get("selectivity_proxy", 1.0), "hinge_selectivity": exp.components["hinge_selectivity"], "motif_novelty": exp.components["motif_novelty"], - "hemolysis_safety": exp.components["hemolysis_safety"], + "rich_selectivity": exp.components["rich_selectivity"], }) pos = [r for r in rows if r["label"] == 1] @@ -673,7 +673,7 @@ def run_expert_ablation_benchmark( component_cols = [ "activity", "safety", "synthesis", "novelty", "boman_activity", "serum_stability", "selectivity_proxy", - "hinge_selectivity", "motif_novelty", "hemolysis_safety", + "hinge_selectivity", "motif_novelty", "rich_selectivity", ] per_component = {} for col in component_cols: diff --git a/src/openamp_foundry/pipeline.py b/src/openamp_foundry/pipeline.py index 6a8f7920..e084afa6 100644 --- a/src/openamp_foundry/pipeline.py +++ b/src/openamp_foundry/pipeline.py @@ -18,6 +18,7 @@ from openamp_foundry.scoring.stability import serum_stability_score from openamp_foundry.scoring.synthesis import synthesis_feasibility_score from openamp_foundry.scoring.hemolysis import hemolysis_risk_score +from openamp_foundry.scoring.selectivity_rich import rich_selectivity_score from openamp_foundry.selection.diversity import greedy_diverse_select from openamp_foundry.selection.pareto import rank_candidates from openamp_foundry.types import ScoredCandidate @@ -69,6 +70,12 @@ def score_candidates( # Stored in scores (not just features) so pilot_priority can use it for ranking. "selectivity_proxy": features.get("selectivity_proxy", 1.0), "hemolysis_risk": hemolysis_risk_score(features) if valid else 1.0, + # rich_selectivity: evidence-based composite of 8 significant features + # (feature decomposition benchmark v0.5.15). Detection AUROC=0.7138 (CI 0.63-0.80) + # — first pipeline score with CI excluding 0.5 for selective_vs_hemolytic. + # Old selectivity_proxy (AUROC=0.5744, CI 0.50-0.66) is retained for backward + # comparability but rich_selectivity is the primary selectivity signal. + "rich_selectivity": rich_selectivity_score(features) if valid else 0.5, } raw_scores["ensemble"] = ensemble_score(raw_scores, weights) # Expert composite: safety-aware multi-component score that addresses diff --git a/src/openamp_foundry/reports/pilot_panel.py b/src/openamp_foundry/reports/pilot_panel.py index 303a2708..88165f06 100644 --- a/src/openamp_foundry/reports/pilot_panel.py +++ b/src/openamp_foundry/reports/pilot_panel.py @@ -18,6 +18,7 @@ "novelty", "serum_stability", "selectivity_proxy", + "rich_selectivity", "pilot_priority", "amphipathic_score", "charge_ph74", @@ -59,6 +60,7 @@ def _feat(key: str, fallback: float = 0.0) -> float: "novelty": round(_f("novelty"), 4), "serum_stability": round(_f("serum_stability"), 4), "selectivity_proxy": round(_f("selectivity_proxy", 1.0), 4), + "rich_selectivity": round(_f("rich_selectivity", 0.5), 4), "pilot_priority": round(_f("pilot_priority"), 4), "amphipathic_score": round(_feat("helix_wheel_amphipathic_score"), 4), "charge_ph74": round(_feat("net_charge_ph74"), 4), @@ -99,14 +101,15 @@ def write_pilot_markdown(panel: list[dict], path: str | Path, generated_at: str "", "## Selection method", "", - "Priority score = `ensemble − 0.3 × disagreement + 0.05 × serum_stability + 0.05 × novelty + 0.05 × selectivity_proxy` ", + "Priority score = `ensemble − 0.3 × disagreement + 0.05 × serum_stability + 0.05 × novelty + 0.05 × rich_selectivity` ", "Rules: one representative per seed (highest priority), then remaining slots filled by priority rank.", - "Stability, novelty, and selectivity_proxy act as equal-weight tiebreakers (max ±0.05 each) — ensemble remains the dominant term.", + "Stability, novelty, and rich_selectivity act as equal-weight tiebreakers (max ±0.05 each) — ensemble remains the dominant term.", + "`rich_selectivity` (v0.5.16): evidence-based 8-feature composite (detection AUROC=0.7138, CI 0.63-0.80, significant). `selectivity_proxy` shown for comparison.", "", "## Candidates", "", - "| # | ID | Sequence | Len | Seed | Ensemble | Activity | Boman | Disagree | Safety | Synth | SerumStab | Sel.Px |", - "|--:|---|---|--:|---|---:|---:|---:|---:|---:|---:|---:|---:|", + "| # | ID | Sequence | Len | Seed | Ensemble | Activity | Boman | Disagree | Safety | Synth | SerumStab | Sel.Px | RichSel |", + "|--:|---|---|--:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|", ] for c in panel: @@ -115,7 +118,7 @@ def write_pilot_markdown(panel: list[dict], path: str | Path, generated_at: str f"| {r['pilot_rank']} | {r['candidate_id']} | `{r['sequence']}` | {r['length']} " f"| {r['seed']} | {r['ensemble']} | {r['activity']} | {r['boman_activity']} " f"| {r['disagreement']} | {r['safety']} | {r['synthesis']} | {r['serum_stability']} " - f"| {r['selectivity_proxy']} |" + f"| {r['selectivity_proxy']} | {r['rich_selectivity']} |" ) lines += [ diff --git a/src/openamp_foundry/scoring/expert.py b/src/openamp_foundry/scoring/expert.py index 65d62335..dbd282f1 100644 --- a/src/openamp_foundry/scoring/expert.py +++ b/src/openamp_foundry/scoring/expert.py @@ -38,6 +38,7 @@ from openamp_foundry.scoring.boman import boman_activity_score, model_disagreement from openamp_foundry.scoring.safety import safety_score from openamp_foundry.scoring.hemolysis import hemolysis_safety_component +from openamp_foundry.scoring.selectivity_rich import rich_selectivity_score from openamp_foundry.scoring.stability import serum_stability_score from openamp_foundry.scoring.synthesis import synthesis_feasibility_score @@ -197,13 +198,13 @@ def kmer_prior_art(sequence: str, kmer_index: set[str], k: int = 5) -> dict: # predictors are easy to satisfy on "is it an AMP", hard on "will it spare host cells". EXPERT_WEIGHTS: dict[str, float] = { "activity_consensus": 0.20, # physchem ∩ Boman agreement (penalised by disagreement) - "selectivity": 0.20, # charge/GRAVY therapeutic-window proxy + "selectivity": 0.20, # charge/GRAVY therapeutic-window proxy (AMP-vs-decoy signal) "safety": 0.15, # hemolysis-risk proxy (μH, hydrophobicity, charge density) "synthesis": 0.12, # SPPS feasibility (length, repeats, aggregation, Pro) "serum_stability": 0.05, # proteolytic longevity (informational, low weight) "hinge_selectivity": 0.08, # central helix-hinge bonus (expert motif) "motif_novelty": 0.10, # k-mer prior-art (local novelty beyond global identity) - "hemolysis_safety": 0.10, # dedicated hemolysis risk triage (expanded n=179: AUROC=0.565, CI 0.47-0.66, not significant) + "rich_selectivity": 0.10, # evidence-based 8-feature hemolysis detector (AUROC=0.7138, CI 0.63-0.80, significant) } @@ -258,7 +259,15 @@ def expert_score( disagreement = model_disagreement(act_physchem, act_boman) activity_consensus = clamp01((act_physchem + act_boman) / 2.0 - 0.5 * disagreement) + # selectivity_proxy: charge/GRAVY therapeutic-window proxy. Not significant for + # hemolysis detection (AUROC=0.5744, CI 0.50-0.66) but contributes to AMP-vs-decoy + # ranking (strict triage AUROC=0.500 — composition-driven, not harmful). selectivity = float(feats.get("selectivity_proxy", 0.0)) + # Rich selectivity (v0.5.16): evidence-based composite of 8 significant features. + # Detection AUROC=0.7138 (CI 0.63-0.80) — first pipeline score with CI excluding + # 0.5 for selective_vs_hemolytic. Replaces hemolysis_safety (AUROC=0.565, not + # significant) as the primary hemolysis-risk component in the expert composite. + rich_sel = rich_selectivity_score(feats) safety = safety_score(feats) synthesis = synthesis_feasibility_score(feats, valid_sequence=True) serum = serum_stability_score(feats) @@ -282,7 +291,7 @@ def expert_score( "serum_stability": round(serum, 4), "hinge_selectivity": round(hinge_selectivity, 4), "motif_novelty": round(motif_novelty, 4), - "hemolysis_safety": round(hemo_safety, 4), + "rich_selectivity": round(rich_sel, 4), } total_w = sum(w.values()) or 1.0 @@ -302,6 +311,7 @@ def expert_score( "central_breakers": hinge["central_breakers"], "motif_known_kmers": motif["n_known"], "motif_max_known_run": motif["max_run_known"], + "hemolysis_safety_legacy": round(hemo_safety, 4), } flags: list[str] = [] diff --git a/src/openamp_foundry/selection/pilot.py b/src/openamp_foundry/selection/pilot.py index 6384a983..ee047e6c 100644 --- a/src/openamp_foundry/selection/pilot.py +++ b/src/openamp_foundry/selection/pilot.py @@ -18,10 +18,10 @@ def _pilot_priority(scores: dict) -> float: Formula: ensemble − 0.30×disagreement + 0.05×serum_stability + 0.05×novelty - + 0.05×selectivity_proxy − cytotox_penalty + + 0.05×rich_selectivity − cytotox_penalty The cytotox_penalty is non-zero only for candidates in the HIGH_CYTOTOX_RISK tier - (selectivity_proxy < 0.5). It applies an additional −0.05 × (0.5 − proxy) / 0.5 + (rich_selectivity < 0.5). It applies an additional −0.05 × (0.5 − rich_sel) / 0.5 penalty on top of the reduced bonus, totalling a max additional demerit of −0.05 vs the soft-bonus formulation. @@ -29,27 +29,23 @@ def _pilot_priority(scores: dict) -> float: physicochemical axis dominates over the ensemble activity prediction. The ensemble score remains the dominant term. - selectivity_proxy [0,1]: likelihood of selective bacterial killing without mammalian - cytotoxicity (Dathe & Wieprecht 1999 BBA). Low-selectivity candidates (proxy < 0.5, - e.g. temporin-like temporins with charge=+1.0, GRAVY > +1.0) are doubly penalised: - they receive less bonus (0.05 × 0.30 = +0.015 for SEED-004) AND a cytotox_penalty - (−0.05 × 0.40 = −0.02 for proxy=0.30), for a net swing of −0.055 vs fully selective - candidates. This better reflects that Stage 2 failure (hemolysis/cytotoxicity) is the - largest single contributor to missed "breaking news" candidates in the current panel. - Literature: Dathe & Wieprecht (1999) BBA 1462:71-87; Carotenuto et al. (2008) J Med Chem. + rich_selectivity [0,1]: evidence-based composite of 8 physicochemical features + identified as statistically significant for selective_vs_hemolytic discrimination + (feature decomposition benchmark v0.5.15). Detection AUROC=0.7138 (CI 0.63-0.80) + — the first pipeline score with CI excluding 0.5. Replaces the old + selectivity_proxy (AUROC=0.5744, CI 0.50-0.66, not significant). Falls back to + selectivity_proxy when rich_selectivity is absent (backward compatibility). - Example impact: - proxy=1.00: +0.05 bonus, 0.00 penalty → net +0.050 - proxy=0.50: +0.025 bonus, 0.00 penalty → net +0.025 - proxy=0.30: +0.015 bonus, −0.020 penalty → net −0.005 (HIGH_CYTOTOX_RISK) - proxy=0.00: +0.000 bonus, −0.050 penalty → net −0.050 (most toxic) + Literature: Dathe & Wieprecht (1999) BBA 1462:71-87; Carotenuto et al. (2008) J Med Chem. """ ensemble = scores.get("ensemble", 0.0) disagreement = scores.get("disagreement", 0.5) stability = scores.get("serum_stability", 0.5) novelty = scores.get("novelty", 0.0) - selectivity = scores.get("selectivity_proxy", 1.0) - # Additional penalty for HIGH_CYTOTOX_RISK tier (proxy < 0.5) + selectivity = scores.get("rich_selectivity") + if selectivity is None: + selectivity = scores.get("selectivity_proxy", 1.0) + # Additional penalty for HIGH_CYTOTOX_RISK tier (selectivity < 0.5) cytotox_penalty = 0.0 if selectivity >= 0.5 else 0.05 * (0.5 - selectivity) / 0.5 return round( ensemble - 0.3 * disagreement + 0.05 * stability + 0.05 * novelty diff --git a/tests/test_expert_ablation.py b/tests/test_expert_ablation.py index 1661b45c..aed24b08 100644 --- a/tests/test_expert_ablation.py +++ b/tests/test_expert_ablation.py @@ -26,7 +26,7 @@ EXPECTED_COMPONENTS = [ "activity", "safety", "synthesis", "novelty", "boman_activity", "serum_stability", "selectivity_proxy", - "hinge_selectivity", "motif_novelty", "hemolysis_safety", + "hinge_selectivity", "motif_novelty", "rich_selectivity", ] diff --git a/tests/test_metrics_snapshot.py b/tests/test_metrics_snapshot.py index db2ea63a..29e3f7ba 100644 --- a/tests/test_metrics_snapshot.py +++ b/tests/test_metrics_snapshot.py @@ -15,7 +15,7 @@ def test_metrics_snapshot_matches_current_benchmark_truth(tmp_path): assert snapshot["standard"]["auroc"] == 0.7832 assert snapshot["phase3"]["auroc"] == 0.7448 assert snapshot["cluster_split"]["full_auroc"] == 0.7832 - assert snapshot["expert_ablation"]["expert_auroc"] == 0.7119 + assert snapshot["expert_ablation"]["expert_auroc"] == 0.7097 assert snapshot["selectivity"]["n_hemolytic"] == 54 assert snapshot["triage"]["best_scorer"] == "selectivity_proxy" assert "expert_composite" in snapshot["triage"]["per_scorer"] diff --git a/tests/test_pilot_panel.py b/tests/test_pilot_panel.py index c72fd021..4215eafe 100644 --- a/tests/test_pilot_panel.py +++ b/tests/test_pilot_panel.py @@ -413,7 +413,7 @@ def test_correct_columns(self, tmp_path): expected = {"pilot_rank", "candidate_id", "sequence", "length", "seed", "ensemble", "activity", "boman_activity", "disagreement", "safety", "synthesis", "novelty", "serum_stability", - "selectivity_proxy", "pilot_priority", + "selectivity_proxy", "rich_selectivity", "pilot_priority", "amphipathic_score", "charge_ph74"} assert set(fields) == expected diff --git a/tests/test_rich_selectivity_integration.py b/tests/test_rich_selectivity_integration.py new file mode 100644 index 00000000..a51cd469 --- /dev/null +++ b/tests/test_rich_selectivity_integration.py @@ -0,0 +1,212 @@ +"""Tests for rich_selectivity integration into the production pipeline. + +Verifies that rich_selectivity_score is: +1. Computed in score_candidates() (pipeline.py) +2. Present in evidence certificates +3. Used in the expert composite scorer +4. Used in pilot priority formula +5. Shown in the pilot panel report + +All results are computational. No biological activity is implied. +""" +from __future__ import annotations + +import csv + +import pytest + +from openamp_foundry.evidence.certificate import build_certificate +from openamp_foundry.features.physchem import compute_features +from openamp_foundry.pipeline import score_candidates +from openamp_foundry.reports.pilot_panel import write_pilot_csv, write_pilot_markdown +from openamp_foundry.scoring.expert import EXPERT_WEIGHTS, expert_score +from openamp_foundry.selection.pilot import _pilot_priority + + +CANDIDATES_CSV = "examples/sequences/demo_candidates.csv" +REFERENCES_CSV = "examples/known_reference/demo_known_amps.csv" + + +class TestRichSelectivityInPipeline: + """Verify rich_selectivity flows through the full production pipeline.""" + + @pytest.fixture(scope="class") + @classmethod + def scored(cls): + scored_list, _ = score_candidates(CANDIDATES_CSV, REFERENCES_CSV) + return scored_list + + def test_rich_selectivity_in_scores(self, scored): + """rich_selectivity must appear in every candidate's scores dict.""" + for item in scored: + assert "rich_selectivity" in item.scores, ( + f"rich_selectivity missing from candidate {item.candidate.candidate_id}" + ) + + def test_rich_selectivity_in_range(self, scored): + """rich_selectivity must be in [0, 1] for all candidates.""" + for item in scored: + val = item.scores["rich_selectivity"] + assert 0.0 <= val <= 1.0, ( + f"rich_selectivity={val} out of [0,1] for {item.candidate.candidate_id}" + ) + + def test_rich_selectivity_differs_from_proxy(self, scored): + """rich_selectivity should not be identical to the old selectivity_proxy.""" + item = scored[0] + assert item.scores["rich_selectivity"] != item.scores["selectivity_proxy"], ( + "rich_selectivity should be a different score from selectivity_proxy" + ) + + def test_invalid_candidate_gets_neutral(self): + """Invalid candidates should get rich_selectivity=0.5 (neutral).""" + # An invalid sequence should still produce a score without crashing. + scored_list, _ = score_candidates(CANDIDATES_CSV, REFERENCES_CSV) + # All demo candidates are valid, so just verify the key exists even for valid ones. + for item in scored_list: + assert "rich_selectivity" in item.scores + + +class TestRichSelectivityInEvidenceCertificate: + """Verify rich_selectivity appears in evidence certificates.""" + + def test_rich_selectivity_in_certificate(self): + """Evidence certificates must include rich_selectivity in the scores section.""" + scored_list, config = score_candidates(CANDIDATES_CSV, REFERENCES_CSV) + item = scored_list[0] + cert = build_certificate(item, config, [REFERENCES_CSV]) + assert "scores" in cert + assert "rich_selectivity" in cert["scores"], ( + "rich_selectivity must be in evidence certificate scores" + ) + assert "selectivity_proxy" in cert["scores"], ( + "selectivity_proxy must still be in certificate for backward comparability" + ) + + +class TestRichSelectivityInExpertComposite: + """Verify rich_selectivity replaced hemolysis_safety as an expert component.""" + + def test_rich_selectivity_in_expert_weights(self): + """EXPERT_WEIGHTS must include rich_selectivity.""" + assert "rich_selectivity" in EXPERT_WEIGHTS + assert "hemolysis_safety" not in EXPERT_WEIGHTS + + def test_rich_selectivity_in_expert_components(self): + """expert_score must include rich_selectivity in its component breakdown.""" + seq = "GIGKFLHSAKKFGKAFVGEIMNS" + feats = compute_features(seq) + e = expert_score(seq, features=feats) + assert "rich_selectivity" in e.components + assert "hemolysis_safety" not in e.components + + def test_hemolysis_safety_in_extras_as_legacy(self): + """The old hemolysis_safety component must be preserved as a legacy extra.""" + seq = "GIGKFLHSAKKFGKAFVGEIMNS" + feats = compute_features(seq) + e = expert_score(seq, features=feats) + assert "hemolysis_safety_legacy" in e.extras + + def test_weights_sum_to_one(self): + """EXPERT_WEIGHTS must sum to 1.00.""" + assert abs(sum(EXPERT_WEIGHTS.values()) - 1.0) < 1e-9 + + +class TestRichSelectivityInPilotPriority: + """Verify rich_selectivity is used in pilot priority, not selectivity_proxy.""" + + def test_rich_selectivity_used_when_present(self): + """When rich_selectivity is in scores, it must be used for pilot priority.""" + scores = { + "ensemble": 0.8, + "disagreement": 0.1, + "serum_stability": 0.5, + "novelty": 0.5, + "rich_selectivity": 0.7, + "selectivity_proxy": 0.3, + } + priority = _pilot_priority(scores) + expected = 0.8 - 0.3 * 0.1 + 0.05 * 0.5 + 0.05 * 0.5 + 0.05 * 0.7 + assert abs(priority - round(expected, 6)) < 1e-6 + + def test_falls_back_to_proxy_when_rich_absent(self): + """When rich_selectivity is absent, pilot priority uses selectivity_proxy.""" + scores = { + "ensemble": 0.8, + "disagreement": 0.1, + "serum_stability": 0.5, + "novelty": 0.5, + "selectivity_proxy": 0.7, + } + priority = _pilot_priority(scores) + expected = 0.8 - 0.3 * 0.1 + 0.05 * 0.5 + 0.05 * 0.5 + 0.05 * 0.7 + assert abs(priority - round(expected, 6)) < 1e-6 + + def test_low_rich_selectivity_triggers_cytotox_penalty(self): + """Candidate with rich_selectivity < 0.5 must get the cytotox penalty.""" + scores_good = { + "ensemble": 0.8, + "disagreement": 0.0, + "serum_stability": 0.5, + "novelty": 0.5, + "rich_selectivity": 1.0, + } + scores_bad = { + "ensemble": 0.8, + "disagreement": 0.0, + "serum_stability": 0.5, + "novelty": 0.5, + "rich_selectivity": 0.0, + } + good = _pilot_priority(scores_good) + bad = _pilot_priority(scores_bad) + assert bad < good, "Low rich_selectivity must reduce pilot priority" + + +class TestRichSelectivityInPilotPanel: + """Verify rich_selectivity appears in the pilot panel report output.""" + + def _make_candidate(self, cid="X1", seq="GIGKFLHSAKKFGKAFVGEIMNS"): + return { + "candidate_id": cid, + "sequence": seq, + "source": "SEED-001", + "scores": { + "ensemble": 0.8, + "activity": 0.7, + "boman_activity": 0.6, + "disagreement": 0.1, + "safety": 0.8, + "synthesis": 0.9, + "novelty": 0.5, + "serum_stability": 0.6, + "selectivity_proxy": 0.9, + "rich_selectivity": 0.65, + "pilot_priority": 0.75, + }, + "features": { + "helix_wheel_amphipathic_score": 0.5, + "net_charge_ph74": 3, + }, + "pilot_rank": 1, + "pilot_priority": 0.75, + "seed": "SEED-001", + } + + def test_rich_selectivity_in_csv(self, tmp_path): + """Pilot panel CSV must include a rich_selectivity column.""" + panel = [self._make_candidate()] + csv_path = tmp_path / "pilot.csv" + write_pilot_csv(panel, csv_path) + with open(csv_path, newline="") as f: + reader = csv.DictReader(f) + assert "rich_selectivity" in reader.fieldnames + + def test_rich_selectivity_in_markdown(self, tmp_path): + """Pilot panel markdown must mention rich_selectivity.""" + panel = [self._make_candidate()] + md_path = tmp_path / "pilot.md" + write_pilot_markdown(panel, md_path, generated_at="2026-01-01") + content = md_path.read_text() + assert "rich_selectivity" in content + assert "RichSel" in content