diff --git a/docs/DISCOVERY_PREDICTION.md b/docs/DISCOVERY_PREDICTION.md index 39bd7828..5990cb69 100644 --- a/docs/DISCOVERY_PREDICTION.md +++ b/docs/DISCOVERY_PREDICTION.md @@ -3,341 +3,4 @@ **Pipeline:** OpenAMP-Foundry v0.1.0 **Date:** 2026-06-28 (updated 2026-06-28) **Status:** Pre-synthesis scientific assessment — for expert review before ordering -**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge at pH 7.4 via Henderson-Hasselbalch (PR #34) - ---- - -## Executive Summary - -This document provides an honest, evidence-based probability assessment for the 20-member pilot -panel nominated by the computational pipeline. It covers the likelihood of wet-lab success at -each stage, identifies the key risk factors in the current nominee set, and lists concrete -improvements already implemented or recommended. - -**Bottom line:** The pilot panel has a ~55–65% probability of yielding at least one candidate -with MIC ≤ 32 μg/mL, and **~12–28%** probability of generating "breaking news" publication -material with the updated panel (up from 5–12% before computational improvements in PRs #31–#36). -The family diversity cap (max_per_seed=4) ensures 4 candidates per seed family including 4 SEED-004 -variants (serum_stability 0.85, C-terminal amidation recommended). New features (net charge pH 7.4, -Chou-Fasman helix propensity, C-amidation flag) improve pipeline accuracy and synthesis guidance. -The dominant remaining computational gap is scaffold novelty (all nominees remain near-seed variants). - ---- - -## What "Breaking News" Requires - -To be publishable as a significant advance in AMP discovery, a candidate must satisfy all of: - -| Criterion | Threshold | Probability (original) | Probability (updated panel) | -|-----------|-----------|------------------------|----------------------------| -| Synthesis success (HPLC ≥ 90% purity) | ≥ 90% purity | ~90% | ~90% (unchanged) | -| MIC vs ATCC reference strains | ≤ 32 μg/mL | ~55–65% of panel | ~55–65% (unchanged) | -| Excellent selectivity | TI > 10 (MIC_bacteria / MIC_RBC) | ~35–50% | ~35–50% (unchanged) | -| Serum stability | t½ > 2 h in 50% human serum | **~10–20%** | **~25–40%** ✓ improved | -| Scaffold novelty | Not described in APD3/DRAMP | **~10–15%** | ~10–15% (unchanged) | -| Potency against MDR strains | MIC < 8 μg/mL vs MRSA/MDR-GNR | not tested | not tested | - -**Combined probability of satisfying all criteria simultaneously (original panel):** ~5–12% -**Combined probability of satisfying all criteria simultaneously (updated panel, PRs #31–#36):** ~12–28% - -**Serum stability improvement rationale:** Before PR #31, 16/20 pilot panel candidates were SEED-003 -variants with serum_stability ≈ 0.27 (≥4 interior K/R sites → t½ < 30 min). After family -diversity cap (max_per_seed=4), the updated panel includes 4 SEED-004 variants (stability 0.85, -only 1 interior K/R site) and 4 SEED-002 variants (stability 0.62). Expected per-family -stability estimates: SEED-001=45%, SEED-002=60%, SEED-003=10%, SEED-004=85%, SEED-005=50%. - -This is not a failure of the pipeline. It reflects the reality that: -- Computational AMP prediction with physicochemical heuristics typically achieves AUROC ~0.75–0.85 - and yields wet-lab hit rates of 30–60%, but "first-in-class" hits are far rarer -- Serum stability requires protease-resistance engineering (D-amino acids, N-methylation, cyclic - backbone) that no computational heuristic can substitute for -- True scaffold novelty requires departure from the 5 seed templates used in this pipeline - ---- - -## Stage-by-Stage Probability Breakdown - -### Stage 0: Synthesis Success - -**Probability: ~90%** (18–19 / 20 candidates synthesized at ≥ 90% purity) - -Evidence: -- All 20 candidates rated MODERATE SPPS difficulty except SEED-002_VAR_084 (HIGH, 23-mer) -- No cysteines in any candidate (disulfide bond risk = 0) -- Aggregation risk flag: none triggered -- Methionine present in 18/20 candidates — oxidation risk during storage, managed by −80°C -- Longest repeat run: all ≤ 2 (no difficult homopolymer stretches) - -Action: Order from two synthesis vendors to hedge against batch failures. For SEED-002_VAR_084 -(23-mer), request Fmoc SPPS with acetonitrile/water gradient; verify by MALDI-TOF pre-assay. - ---- - -### Stage 1: MIC Activity vs Reference Strains - -**Probability: ~55–65%** (11–13 / 20 candidates with MIC ≤ 32 μg/mL) - -Basis: -- Pipeline AUROC = 0.8037 (bootstrap CI₉₅: 0.71–0.89) vs composition-matched UniProt decoys -- Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates) -- SEED-003 family (16/20): parent sequence RRWQWRMKKLG is a curated known AMP; variants with - conservative substitutions at positions 2, 4, 5, 8 expected ~65–75% hit rate -- SEED-005_VAR_009 (KRFFKKIGSALKFA): similar to buforin/magainin family; ~50% estimate -- SEED-001_VAR_071 (KWKMFKKIGAVLKAL): LL-37 analogue; ~45% estimate -- SEED-002_VAR_084 (23-mer): melittin-like; ~30% (hemolytic risk may mask selectivity) -- SEED-004_VAR_001 (ALPFIGRVLSGIL): low charge (0.8 at pH 7.4), relies purely on hydrophobicity; - ~20–25% estimate — the riskiest nomination - -**Risk to this estimate:** Our activity scorer does not capture proline-rich AMPs (intracellular -targets), beta-sheet AMPs (tachyplesin family), or lipopeptides. These blind spots are documented -in the validate_scoring_report.json. - ---- - -### Stage 2: Selectivity (Therapeutic Index) - -**Probability: ~35–50%** (7–10 / 20 candidates with TI > 10) - -Basis: -- High cationic charge (+3.8 to +5.8 at pH 7.4) in SEED-003 family is protective vs hemolysis -- Hydrophobic moment (μH): 11/20 have μH > 0.55 → moderate hemolytic risk flagged in QC report - These candidates are tested at MIC/3 concentration first (per WET_LAB_HANDOFF.md protocol) -- SEED-004_VAR_001 (ALPFIGRVLSGIL): charge = +0.8, μH = high → hemolysis-dominated profile expected - -Known systematic bias: The safety scorer penalizes hydrophobic moment and cysteine fraction but -does not predict lysis of specific cell types (erythrocytes vs epithelial vs immune cells). -Safety score > 0.70 reduces hemolysis probability but does not eliminate it. - ---- - -### Stage 3: Serum Stability - -**Probability: ~25–40%** (5–8 / 20 candidates with t½ > 2 h in 50% human serum) - -**Updated from original ~10–20%: family diversity cap now includes high-stability seed families.** - -Interior trypsin cleavage site analysis (computed by `serum_stability_score()`): - -| Seed family | Pilot panel count | serum_stability | Interior K/R density | Predicted t½ | -|-------------|------------------|-----------------|----------------------|--------------| -| SEED-001 | 4 | 0.47 | 0.27 | ~1–2 h (borderline) | -| SEED-002 | 4 | 0.62 | 0.20 | ~2–4 h (moderate) | -| SEED-003 | 4 | 0.27 | 0.45 | < 30 min (poor) | -| SEED-004 | 4 | 0.85 | 0.08 | > 4 h (good) | -| SEED-005 | 4 | 0.52 | 0.21 | ~1–3 h (borderline) | - -Literature basis: Hilpert et al. (2006), J Antimicrob Chemother; serum_stability_score ≥ 0.50 -corresponds to t½ > 1 h based on trypsin density calibration. Score ≥ 0.70 → t½ > 2 h likely. - -Candidates with best predicted stability (now prioritized by max_per_seed cap + stability bonus): -- SEED-004 variants (serum_stability 0.85): 4 candidates in panel — primary stability bet -- SEED-002 variants (serum_stability 0.62): 4 candidates — moderate stability -- SEED-005 variants (serum_stability 0.52): 4 candidates — borderline - -**For translational significance, serum stability must be assayed** (CLSI-standardized serum -stability assay recommended before claiming therapeutic relevance in any publication). - -**Improvement path:** Variants with K→ε-K (lysine isoforms), N-terminal acetylation, or C-terminal -amidation can dramatically extend serum half-life. These modifications are outside the current -pipeline's scope but can be flagged for a follow-up synthesis round. - ---- - -### Stage 4: Scaffold Novelty - -**Probability: ~10–15%** of pilot panel meeting publication novelty threshold - -The 5 seed sequences were selected from known AMPs in the reference database. Most nominees are -1–3 mutation variants of these seeds: - -| Seed family | Nearest known sequence | Mean novelty in pilot | -|------------|----------------------|----------------------| -| SEED-003 (16/20) | RRWQWRMKKLG (REF-RRW-001, curated) | 0.091–0.256 | -| SEED-005 (1/20) | KFLHSAKKFGKAFV (buforin-like) | 0.467 | -| SEED-001 (1/20) | KWKLFKKIGAVLKVL (LL-37 fragment) | 0.133 | -| SEED-002 (1/20) | GIGKFLKSAKKFGKAVP (cecropin variant) | 0.087 | -| SEED-004 (1/20) | ALPFIGRVLSGIL (hydrophobic peptide) | 0.154 | - -A reviewer examining these sequences against APD3 (Antimicrobial Peptide Database) or DRAMP will -recognize the templates. High-impact publication requires either: -1. Demonstrating potency against MDR clinical isolates (novelty of application) -2. Elucidating a novel mechanism of action (novelty of biology) -3. Finding candidates with novelty score > 0.50 from the full 88-nominee set - -The highest-novelty nominee in the full phase3 set reaches 0.467. No candidate is in genuinely -uncharted sequence space. - ---- - -## Root Cause Analysis of the Probability Gap - -``` -Target: 100% breaking news probability -Current: ~10–25% (after computational improvements) -Gap: ~75–90% - -Root causes (ranked by remaining impact): - -1. No protease-resistance engineering (~-30 pp remaining): SEED-003 variants (4/20, reduced from - 16/20) still degrade within 30 min in serum. Cannot be fixed computationally — requires - D-amino acid substitution or backbone N-methylation in Wave 2. - [Addressed: PR #31 added serum_stability_score + PR #31 family diversity cap — SEED-003 - reduced from 16 to 4 of 20 slots. SEED-004 variants (stability=0.85) now fill 4 slots.] - -2. Low scaffold novelty (-15 pp): All nominees are near-seed variants (mean novelty 0.169). - Cannot improve without either new seed sequences from non-AMP-database sources, or - discarding the pipeline's seed-based generation approach. - [Partially addressed: PR #33 expanded reference set 44→73 sequences. Does not change current - nominee scores because seeds are already in references; improves future candidate comparisons.] - -3. No MDR strain testing (-10 pp): Testing only ATCC reference strains limits publication impact. - Fix: include ≥1 MDR clinical isolate (MRSA, E. coli ST131, K. pneumoniae KPC) in assay plan. - [Not addressable computationally — wet-lab expansion required.] - -4. Scoring model limitations (-5 pp): Heuristic AUROC 0.80 leaves 20% of activity unexplained. - Fix: the pipeline is transparent about this — it's the cost of interpretability. - [Not addressable without wet-lab training data.] -``` - -**What the improvements achieved (PRs #31–#36):** -- **PR #31/#32**: Family diversity cap (max_per_seed=4) + serum_stability_score; SEED-003 16→4 slots -- **PR #33**: Reference set 44→73 sequences; 14 structural validation tests -- **PR #34**: Net charge at pH 7.4 via Henderson-Hasselbalch; His now +0.114 not +1.0; 24 tests -- **PR #35**: Chou-Fasman helix propensity (Pα) in compute_features(); small activity bonus; 23 tests -- **PR #36**: C-terminal amidation recommendation in presynth QC; SEED-004 variants flagged; 11 tests -- **Overall "breaking news" probability: ~5–12% → ~12–28%** (incremental gains from accuracy improvements) - ---- - -## Current Nominee Quality: What the Pipeline Got Right - -Despite the probability gap identified above, the pipeline has done the following correctly: - -1. **No false ZwitterAMP trap nominations**: KDKDKDKD-like sequences (Boman-high, charge-zero) - are correctly rejected by the disagreement gate. This prevents a common failure mode. - -2. **Consistent synthesis feasibility**: All 20 pilot candidates are MODERATE SPPS difficulty - with synthesis score = 1.00. Zero synthesis-impossible candidates were nominated. - -3. **Strong AUROC**: 0.8037 (bootstrap CI₉₅: 0.71–0.89) means the ensemble scoring correctly - separates AMP-like sequences from random-sequence background ~80% of the time. - -4. **Dual-scorer consensus**: All phase3 nominees have disagreement ≤ 0.296, meaning the - physicochemical (activity_likeness) and biophysical (Boman) scorers agree on these candidates. - -5. **Safety-first selection**: Phase3 gate requires safety ≥ 0.60 (max_safety_risk = 0.40). - The 88 nominees have mean safety = 0.952. High-hemolysis-risk candidates were excluded. - -6. **Benchmark-verified**: Hidden-active benchmark (bench-hidden-active target) confirmed the - pipeline recovers known actives at above-random rates before any nominees were selected. - ---- - -## Highest-Probability Bets in the Pilot Panel - -Ranked by estimated wet-lab success probability (MIC ≤ 32 μg/mL AND TI > 10): - -**Updated panel (after PRs #31–#33) — all 5 families now represented:** - -| Rank | Candidate | Sequence | Serum Stab | Novelty | Estimated P(all gates) | Why | -|------|-----------|----------|-----------|---------|------------------------|-----| -| 1 | SEED-004_VAR_001 | ALPFIGRVLSGIL | **0.85** | 0.154 | ~15–25% | Best serum stability; only 1 K/R site | -| 2 | SEED-005_VAR_009 | KRFFKKIGSALKFA | 0.52 | **0.467** | ~10–18% | Highest novelty; FF motif aids insertion | -| 3 | SEED-002_VAR_xxx | (best SEED-002) | 0.62 | 0.087 | ~8–15% | Moderate stability; cecropin-like scaffold | -| 4 | SEED-001_VAR_xxx | (best SEED-001) | 0.47 | 0.133 | ~6–12% | LL-37 analogue; diverse mechanism bet | -| 5 | SEED-003_VAR_012 | RKWQYRMKKLG | 0.27 | 0.182 | ~3–8% | High ensemble but poor serum stability | - -For a focused first wave (budget-constrained), prioritize SEED-004 variants first (serum stability -bet), then SEED-005 (novelty bet), then SEED-002 (balanced). SEED-003 variants are the highest -ensemble scorers but the poorest serum stability — test them last or concurrently with a D-amino -acid protection plan ready. - ---- - -## Roadmap to Higher Discovery Probability - -### Wave 2 Improvements (before or alongside Wave 1 synthesis) - -**1. Serum stability score ✅ DONE (PR #31, #32)** -Added `trypsin_site_density`, `chymotrypsin_site_density` to `compute_features()` and -`serum_stability_score()` in `scoring/stability.py`. Stability bonus (+0.05 weight) now -included in pilot panel priority formula. All nominees show explicit `serum_stability` column. -Impact achieved: SEED-004 variants (stability=0.85) now ranked higher; SEED-003 (stability=0.27) -explicitly flagged. Changed Stage 3 probability from ~10–20% to ~25–40%. - -**2. Family diversity cap ✅ DONE (PR #31)** -`max_per_seed=4` enforced in pilot panel selection (Makefile `make pilot` target). -Result: pilot panel now has exactly 4 candidates from each of the 5 seed families (was 16/20 SEED-003). -Impact achieved: serum stability bet now spread across SEED-001/002/003/004/005. - -**3. Expand reference set ✅ DONE (PR #33)** -Current: 73 reference sequences (expanded from 44), covering LL-37 fragments, histatin, -clavanin, proline-rich AMPs (apidaecin, oncocin, pyrrhocoricin), pleurocidin, BMAP-27/28, -WLBU2, BP100, omiganan, and other well-characterized diverse families. -Note: does not change novelty scores for current nominees (seeds are still in references); -improves discrimination for future runs with non-seed-derived candidates. - -**4. Net charge at physiological pH ✅ DONE (PR #34)** -Replaced `net_charge_proxy` (counts His as +1.0) with `net_charge_at_ph74()` using -Henderson-Hasselbalch (His ≈ +0.114 at pH 7.4, pKa=6.5 in peptides). New fields -`net_charge_ph74` and `charge_density_ph74` in all feature dicts; `activity_likeness_score()` -now uses physiologically accurate charge density. -Impact: charge density corrected for 24/88 nominees with His; proxy overestimated by ~0.89 -per His residue, affecting scoring accuracy not final panel composition. - -**5. Chou-Fasman helix propensity (Recommended next)** -Add `helix_propensity` feature using Chou-Fasman (1974) residue parameters. Helical AMPs -(cecropin, magainin, LL-37 family — all 5 seeds in pilot panel) rely on amphipathic helix -formation for membrane disruption. Mean helix propensity above 1.05 distinguishes good -helical AMPs from unstructured sequences more accurately than hydrophobic_fraction alone. -Affects: all 88 nominees; expected to improve AUROC by ~0.01–0.02. - -**6. D-amino acid variants (Wave 2 synthesis)** -Replace all L-amino acids with D-amino acids in top 2 active Wave 1 hits. -Expected: serum t½ increases from <30 min to >8 h (literature: Wade et al. 1990). -Impact: crosses the serum stability gate that currently blocks ~80% of candidates. - -**7. MDR strain panel (Assay expansion)** -Add MRSA USA300, E. coli ST131, K. pneumoniae ATCC BAA-1705 (KPC) to MIC assay. -Impact: any hit against MDR strains is immediately publishable as clinically significant. - ---- - -## Summary Table: Probability by Gate - -| Stage | Gate | Before PRs #31–33 | After PRs #31–33 | Main limiting factor | -|-------|------|-------------------|------------------|---------------------| -| 0 | Synthesis success | ~90% | ~90% | 1 HIGH SPPS difficulty candidate | -| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | Heuristic scoring; AUROC 0.80 | -| 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | Hemolytic risk for high-μH candidates | -| 3 | t½ > 2h (serum) | ~10–20% | **~25–40%** ✓ | SEED-004 variants now prioritized | -| 4 | Scaffold novelty | ~10–15% | ~10–15% | Near-seed variants, irreducible | -| All gates | "Breaking news" hit | ~5–12% | **~10–25%** ✓ | Serum stability improvement | - -**Probability of ≥1 active AMP from pilot panel (Stage 1 only):** ~85–95% -(Probability of zero active from 20 candidates with ~60% individual hit rate) - -**Probability of ≥1 candidate satisfying ALL gates (updated after PRs #31–#36):** ~12–28% - ---- - -## Confidence Calibration - -This assessment is based on: -- Internal benchmark AUROC = 0.8037 (n=88, bootstrap n=2000) -- Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002) -- Published serum stability data for short cationic peptides (Hilpert et al. 2006) -- Database cross-reference with APD3 template sequences - -**Key uncertainty:** The pipeline has not been prospectively validated — these are predicted, -not observed, hit rates. The true hit rate may differ significantly. - -**Honest statement:** The computational work is complete and defensible. The "100% probability" -target is biologically impossible without wet-lab data. The correct goal is to maximise the -probability of finding at least one active, selective, stable, and novel candidate — and to -iterate rapidly on failures. - ---- - -*Generated by OpenAMP-Foundry v0.1.0. All scores are computational heuristics. -No biological activity has been demonstrated. The lab is the judge.* +**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37) \ No newline at end of file diff --git a/src/openamp_foundry/selection/pilot.py b/src/openamp_foundry/selection/pilot.py index e8e9ce10..413c20d1 100644 --- a/src/openamp_foundry/selection/pilot.py +++ b/src/openamp_foundry/selection/pilot.py @@ -14,15 +14,17 @@ def _seed_from_source(source: str) -> str: def _pilot_priority(scores: dict) -> float: - """Higher is better: reward ensemble, reward serum stability, penalise disagreement. + """Higher is better: reward ensemble, stability, and novelty; penalise disagreement. - Serum stability is weighted lightly (0.05) so it acts as a tiebreaker without - overriding the ensemble score. A candidate with stability=1.0 gains at most +0.05. + Within a seed family, higher-novelty variants are preferred as tiebreakers. + Novelty is weighted lightly (0.05) so the ensemble score remains dominant. + A candidate with novelty=0.467 (SEED-005 max) gains at most +0.023. """ ensemble = scores.get("ensemble", 0.0) disagreement = scores.get("disagreement", 0.5) stability = scores.get("serum_stability", 0.5) - return round(ensemble - 0.3 * disagreement + 0.05 * stability, 6) + novelty = scores.get("novelty", 0.0) + return round(ensemble - 0.3 * disagreement + 0.05 * stability + 0.05 * novelty, 6) def select_pilot_panel( diff --git a/tests/test_pilot_panel.py b/tests/test_pilot_panel.py index 89ddbbcc..be330ef1 100644 --- a/tests/test_pilot_panel.py +++ b/tests/test_pilot_panel.py @@ -90,6 +90,22 @@ def test_higher_stability_higher_priority(self): p2 = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10, "serum_stability": 0.0}) assert p1 > p2 + def test_higher_novelty_higher_priority(self): + p1 = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10, "novelty": 0.467}) + p2 = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10, "novelty": 0.10}) + assert p1 > p2 + + def test_missing_novelty_defaults_to_0(self): + p_absent = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10}) + p_zero = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10, "novelty": 0.0}) + assert p_absent == pytest.approx(p_zero, abs=1e-9) + + def test_novelty_formula_contribution(self): + # novelty bonus = 0.05 * 0.40 = 0.02; stability absent → 0.5 + p = _pilot_priority({"ensemble": 0.70, "disagreement": 0.10, "novelty": 0.40}) + expected = 0.70 - 0.3 * 0.10 + 0.05 * 0.5 + 0.05 * 0.40 + assert p == pytest.approx(expected, abs=1e-6) + class TestSelectPilotPanel: def test_empty_input_returns_empty(self):