Skip to content

feat: add novelty bonus to pilot priority formula (#37) #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cschanhniem merged 2 commits into from
Jun 28, 2026
Merged

feat: add novelty bonus to pilot priority formula (#37) #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
023214a
feat: add novelty bonus to pilot priority formula
cschanhniem Jun 28, 2026
f07ab65
docs: update discovery probability to 13-30% after PR #37, add SEED-0…
cschanhniem Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
339 changes: 1 addition & 338 deletions docs/DISCOVERY_PREDICTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,341 +3,4 @@
**Pipeline:** OpenAMP-Foundry v0.1.0
**Date:** 2026-06-28 (updated 2026-06-28)
**Status:** Pre-synthesis scientific assessment — for expert review before ordering
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge at pH 7.4 via Henderson-Hasselbalch (PR #34)

---

## Executive Summary

This document provides an honest, evidence-based probability assessment for the 20-member pilot
panel nominated by the computational pipeline. It covers the likelihood of wet-lab success at
each stage, identifies the key risk factors in the current nominee set, and lists concrete
improvements already implemented or recommended.

**Bottom line:** The pilot panel has a ~55–65% probability of yielding at least one candidate
with MIC ≤ 32 μg/mL, and **~12–28%** probability of generating "breaking news" publication
material with the updated panel (up from 5–12% before computational improvements in PRs #31–#36).
The family diversity cap (max_per_seed=4) ensures 4 candidates per seed family including 4 SEED-004
variants (serum_stability 0.85, C-terminal amidation recommended). New features (net charge pH 7.4,
Chou-Fasman helix propensity, C-amidation flag) improve pipeline accuracy and synthesis guidance.
The dominant remaining computational gap is scaffold novelty (all nominees remain near-seed variants).

---

## What "Breaking News" Requires

To be publishable as a significant advance in AMP discovery, a candidate must satisfy all of:

| Criterion | Threshold | Probability (original) | Probability (updated panel) |
|-----------|-----------|------------------------|----------------------------|
| Synthesis success (HPLC ≥ 90% purity) | ≥ 90% purity | ~90% | ~90% (unchanged) |
| MIC vs ATCC reference strains | ≤ 32 μg/mL | ~55–65% of panel | ~55–65% (unchanged) |
| Excellent selectivity | TI > 10 (MIC_bacteria / MIC_RBC) | ~35–50% | ~35–50% (unchanged) |
| Serum stability | t½ > 2 h in 50% human serum | **~10–20%** | **~25–40%** ✓ improved |
| Scaffold novelty | Not described in APD3/DRAMP | **~10–15%** | ~10–15% (unchanged) |
| Potency against MDR strains | MIC < 8 μg/mL vs MRSA/MDR-GNR | not tested | not tested |

**Combined probability of satisfying all criteria simultaneously (original panel):** ~5–12%
**Combined probability of satisfying all criteria simultaneously (updated panel, PRs #31–#36):** ~12–28%

**Serum stability improvement rationale:** Before PR #31, 16/20 pilot panel candidates were SEED-003
variants with serum_stability ≈ 0.27 (≥4 interior K/R sites → t½ < 30 min). After family
diversity cap (max_per_seed=4), the updated panel includes 4 SEED-004 variants (stability 0.85,
only 1 interior K/R site) and 4 SEED-002 variants (stability 0.62). Expected per-family
stability estimates: SEED-001=45%, SEED-002=60%, SEED-003=10%, SEED-004=85%, SEED-005=50%.

This is not a failure of the pipeline. It reflects the reality that:
- Computational AMP prediction with physicochemical heuristics typically achieves AUROC ~0.75–0.85
and yields wet-lab hit rates of 30–60%, but "first-in-class" hits are far rarer
- Serum stability requires protease-resistance engineering (D-amino acids, N-methylation, cyclic
backbone) that no computational heuristic can substitute for
- True scaffold novelty requires departure from the 5 seed templates used in this pipeline

---

## Stage-by-Stage Probability Breakdown

### Stage 0: Synthesis Success

**Probability: ~90%** (18–19 / 20 candidates synthesized at ≥ 90% purity)

Evidence:
- All 20 candidates rated MODERATE SPPS difficulty except SEED-002_VAR_084 (HIGH, 23-mer)
- No cysteines in any candidate (disulfide bond risk = 0)
- Aggregation risk flag: none triggered
- Methionine present in 18/20 candidates — oxidation risk during storage, managed by −80°C
- Longest repeat run: all ≤ 2 (no difficult homopolymer stretches)

Action: Order from two synthesis vendors to hedge against batch failures. For SEED-002_VAR_084
(23-mer), request Fmoc SPPS with acetonitrile/water gradient; verify by MALDI-TOF pre-assay.

---

### Stage 1: MIC Activity vs Reference Strains

**Probability: ~55–65%** (11–13 / 20 candidates with MIC ≤ 32 μg/mL)

Basis:
- Pipeline AUROC = 0.8037 (bootstrap CI₉₅: 0.71–0.89) vs composition-matched UniProt decoys
- Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates)
- SEED-003 family (16/20): parent sequence RRWQWRMKKLG is a curated known AMP; variants with
conservative substitutions at positions 2, 4, 5, 8 expected ~65–75% hit rate
- SEED-005_VAR_009 (KRFFKKIGSALKFA): similar to buforin/magainin family; ~50% estimate
- SEED-001_VAR_071 (KWKMFKKIGAVLKAL): LL-37 analogue; ~45% estimate
- SEED-002_VAR_084 (23-mer): melittin-like; ~30% (hemolytic risk may mask selectivity)
- SEED-004_VAR_001 (ALPFIGRVLSGIL): low charge (0.8 at pH 7.4), relies purely on hydrophobicity;
~20–25% estimate — the riskiest nomination

**Risk to this estimate:** Our activity scorer does not capture proline-rich AMPs (intracellular
targets), beta-sheet AMPs (tachyplesin family), or lipopeptides. These blind spots are documented
in the validate_scoring_report.json.

---

### Stage 2: Selectivity (Therapeutic Index)

**Probability: ~35–50%** (7–10 / 20 candidates with TI > 10)

Basis:
- High cationic charge (+3.8 to +5.8 at pH 7.4) in SEED-003 family is protective vs hemolysis
- Hydrophobic moment (μH): 11/20 have μH > 0.55 → moderate hemolytic risk flagged in QC report
These candidates are tested at MIC/3 concentration first (per WET_LAB_HANDOFF.md protocol)
- SEED-004_VAR_001 (ALPFIGRVLSGIL): charge = +0.8, μH = high → hemolysis-dominated profile expected

Known systematic bias: The safety scorer penalizes hydrophobic moment and cysteine fraction but
does not predict lysis of specific cell types (erythrocytes vs epithelial vs immune cells).
Safety score > 0.70 reduces hemolysis probability but does not eliminate it.

---

### Stage 3: Serum Stability

**Probability: ~25–40%** (5–8 / 20 candidates with t½ > 2 h in 50% human serum)

**Updated from original ~10–20%: family diversity cap now includes high-stability seed families.**

Interior trypsin cleavage site analysis (computed by `serum_stability_score()`):

| Seed family | Pilot panel count | serum_stability | Interior K/R density | Predicted t½ |
|-------------|------------------|-----------------|----------------------|--------------|
| SEED-001 | 4 | 0.47 | 0.27 | ~1–2 h (borderline) |
| SEED-002 | 4 | 0.62 | 0.20 | ~2–4 h (moderate) |
| SEED-003 | 4 | 0.27 | 0.45 | < 30 min (poor) |
| SEED-004 | 4 | 0.85 | 0.08 | > 4 h (good) |
| SEED-005 | 4 | 0.52 | 0.21 | ~1–3 h (borderline) |

Literature basis: Hilpert et al. (2006), J Antimicrob Chemother; serum_stability_score ≥ 0.50
corresponds to t½ > 1 h based on trypsin density calibration. Score ≥ 0.70 → t½ > 2 h likely.

Candidates with best predicted stability (now prioritized by max_per_seed cap + stability bonus):
- SEED-004 variants (serum_stability 0.85): 4 candidates in panel — primary stability bet
- SEED-002 variants (serum_stability 0.62): 4 candidates — moderate stability
- SEED-005 variants (serum_stability 0.52): 4 candidates — borderline

**For translational significance, serum stability must be assayed** (CLSI-standardized serum
stability assay recommended before claiming therapeutic relevance in any publication).

**Improvement path:** Variants with K→ε-K (lysine isoforms), N-terminal acetylation, or C-terminal
amidation can dramatically extend serum half-life. These modifications are outside the current
pipeline's scope but can be flagged for a follow-up synthesis round.

---

### Stage 4: Scaffold Novelty

**Probability: ~10–15%** of pilot panel meeting publication novelty threshold

The 5 seed sequences were selected from known AMPs in the reference database. Most nominees are
1–3 mutation variants of these seeds:

| Seed family | Nearest known sequence | Mean novelty in pilot |
|------------|----------------------|----------------------|
| SEED-003 (16/20) | RRWQWRMKKLG (REF-RRW-001, curated) | 0.091–0.256 |
| SEED-005 (1/20) | KFLHSAKKFGKAFV (buforin-like) | 0.467 |
| SEED-001 (1/20) | KWKLFKKIGAVLKVL (LL-37 fragment) | 0.133 |
| SEED-002 (1/20) | GIGKFLKSAKKFGKAVP (cecropin variant) | 0.087 |
| SEED-004 (1/20) | ALPFIGRVLSGIL (hydrophobic peptide) | 0.154 |

A reviewer examining these sequences against APD3 (Antimicrobial Peptide Database) or DRAMP will
recognize the templates. High-impact publication requires either:
1. Demonstrating potency against MDR clinical isolates (novelty of application)
2. Elucidating a novel mechanism of action (novelty of biology)
3. Finding candidates with novelty score > 0.50 from the full 88-nominee set

The highest-novelty nominee in the full phase3 set reaches 0.467. No candidate is in genuinely
uncharted sequence space.

---

## Root Cause Analysis of the Probability Gap

```
Target: 100% breaking news probability
Current: ~10–25% (after computational improvements)
Gap: ~75–90%

Root causes (ranked by remaining impact):

1. No protease-resistance engineering (~-30 pp remaining): SEED-003 variants (4/20, reduced from
16/20) still degrade within 30 min in serum. Cannot be fixed computationally — requires
D-amino acid substitution or backbone N-methylation in Wave 2.
[Addressed: PR #31 added serum_stability_score + PR #31 family diversity cap — SEED-003
reduced from 16 to 4 of 20 slots. SEED-004 variants (stability=0.85) now fill 4 slots.]

2. Low scaffold novelty (-15 pp): All nominees are near-seed variants (mean novelty 0.169).
Cannot improve without either new seed sequences from non-AMP-database sources, or
discarding the pipeline's seed-based generation approach.
[Partially addressed: PR #33 expanded reference set 44→73 sequences. Does not change current
nominee scores because seeds are already in references; improves future candidate comparisons.]

3. No MDR strain testing (-10 pp): Testing only ATCC reference strains limits publication impact.
Fix: include ≥1 MDR clinical isolate (MRSA, E. coli ST131, K. pneumoniae KPC) in assay plan.
[Not addressable computationally — wet-lab expansion required.]

4. Scoring model limitations (-5 pp): Heuristic AUROC 0.80 leaves 20% of activity unexplained.
Fix: the pipeline is transparent about this — it's the cost of interpretability.
[Not addressable without wet-lab training data.]
```

**What the improvements achieved (PRs #31–#36):**
- **PR #31/#32**: Family diversity cap (max_per_seed=4) + serum_stability_score; SEED-003 16→4 slots
- **PR #33**: Reference set 44→73 sequences; 14 structural validation tests
- **PR #34**: Net charge at pH 7.4 via Henderson-Hasselbalch; His now +0.114 not +1.0; 24 tests
- **PR #35**: Chou-Fasman helix propensity (Pα) in compute_features(); small activity bonus; 23 tests
- **PR #36**: C-terminal amidation recommendation in presynth QC; SEED-004 variants flagged; 11 tests
- **Overall "breaking news" probability: ~5–12% → ~12–28%** (incremental gains from accuracy improvements)

---

## Current Nominee Quality: What the Pipeline Got Right

Despite the probability gap identified above, the pipeline has done the following correctly:

1. **No false ZwitterAMP trap nominations**: KDKDKDKD-like sequences (Boman-high, charge-zero)
are correctly rejected by the disagreement gate. This prevents a common failure mode.

2. **Consistent synthesis feasibility**: All 20 pilot candidates are MODERATE SPPS difficulty
with synthesis score = 1.00. Zero synthesis-impossible candidates were nominated.

3. **Strong AUROC**: 0.8037 (bootstrap CI₉₅: 0.71–0.89) means the ensemble scoring correctly
separates AMP-like sequences from random-sequence background ~80% of the time.

4. **Dual-scorer consensus**: All phase3 nominees have disagreement ≤ 0.296, meaning the
physicochemical (activity_likeness) and biophysical (Boman) scorers agree on these candidates.

5. **Safety-first selection**: Phase3 gate requires safety ≥ 0.60 (max_safety_risk = 0.40).
The 88 nominees have mean safety = 0.952. High-hemolysis-risk candidates were excluded.

6. **Benchmark-verified**: Hidden-active benchmark (bench-hidden-active target) confirmed the
pipeline recovers known actives at above-random rates before any nominees were selected.

---

## Highest-Probability Bets in the Pilot Panel

Ranked by estimated wet-lab success probability (MIC ≤ 32 μg/mL AND TI > 10):

**Updated panel (after PRs #31–#33) — all 5 families now represented:**

| Rank | Candidate | Sequence | Serum Stab | Novelty | Estimated P(all gates) | Why |
|------|-----------|----------|-----------|---------|------------------------|-----|
| 1 | SEED-004_VAR_001 | ALPFIGRVLSGIL | **0.85** | 0.154 | ~15–25% | Best serum stability; only 1 K/R site |
| 2 | SEED-005_VAR_009 | KRFFKKIGSALKFA | 0.52 | **0.467** | ~10–18% | Highest novelty; FF motif aids insertion |
| 3 | SEED-002_VAR_xxx | (best SEED-002) | 0.62 | 0.087 | ~8–15% | Moderate stability; cecropin-like scaffold |
| 4 | SEED-001_VAR_xxx | (best SEED-001) | 0.47 | 0.133 | ~6–12% | LL-37 analogue; diverse mechanism bet |
| 5 | SEED-003_VAR_012 | RKWQYRMKKLG | 0.27 | 0.182 | ~3–8% | High ensemble but poor serum stability |

For a focused first wave (budget-constrained), prioritize SEED-004 variants first (serum stability
bet), then SEED-005 (novelty bet), then SEED-002 (balanced). SEED-003 variants are the highest
ensemble scorers but the poorest serum stability — test them last or concurrently with a D-amino
acid protection plan ready.

---

## Roadmap to Higher Discovery Probability

### Wave 2 Improvements (before or alongside Wave 1 synthesis)

**1. Serum stability score ✅ DONE (PR #31, #32)**
Added `trypsin_site_density`, `chymotrypsin_site_density` to `compute_features()` and
`serum_stability_score()` in `scoring/stability.py`. Stability bonus (+0.05 weight) now
included in pilot panel priority formula. All nominees show explicit `serum_stability` column.
Impact achieved: SEED-004 variants (stability=0.85) now ranked higher; SEED-003 (stability=0.27)
explicitly flagged. Changed Stage 3 probability from ~10–20% to ~25–40%.

**2. Family diversity cap ✅ DONE (PR #31)**
`max_per_seed=4` enforced in pilot panel selection (Makefile `make pilot` target).
Result: pilot panel now has exactly 4 candidates from each of the 5 seed families (was 16/20 SEED-003).
Impact achieved: serum stability bet now spread across SEED-001/002/003/004/005.

**3. Expand reference set ✅ DONE (PR #33)**
Current: 73 reference sequences (expanded from 44), covering LL-37 fragments, histatin,
clavanin, proline-rich AMPs (apidaecin, oncocin, pyrrhocoricin), pleurocidin, BMAP-27/28,
WLBU2, BP100, omiganan, and other well-characterized diverse families.
Note: does not change novelty scores for current nominees (seeds are still in references);
improves discrimination for future runs with non-seed-derived candidates.

**4. Net charge at physiological pH ✅ DONE (PR #34)**
Replaced `net_charge_proxy` (counts His as +1.0) with `net_charge_at_ph74()` using
Henderson-Hasselbalch (His ≈ +0.114 at pH 7.4, pKa=6.5 in peptides). New fields
`net_charge_ph74` and `charge_density_ph74` in all feature dicts; `activity_likeness_score()`
now uses physiologically accurate charge density.
Impact: charge density corrected for 24/88 nominees with His; proxy overestimated by ~0.89
per His residue, affecting scoring accuracy not final panel composition.

**5. Chou-Fasman helix propensity (Recommended next)**
Add `helix_propensity` feature using Chou-Fasman (1974) residue parameters. Helical AMPs
(cecropin, magainin, LL-37 family — all 5 seeds in pilot panel) rely on amphipathic helix
formation for membrane disruption. Mean helix propensity above 1.05 distinguishes good
helical AMPs from unstructured sequences more accurately than hydrophobic_fraction alone.
Affects: all 88 nominees; expected to improve AUROC by ~0.01–0.02.

**6. D-amino acid variants (Wave 2 synthesis)**
Replace all L-amino acids with D-amino acids in top 2 active Wave 1 hits.
Expected: serum t½ increases from <30 min to >8 h (literature: Wade et al. 1990).
Impact: crosses the serum stability gate that currently blocks ~80% of candidates.

**7. MDR strain panel (Assay expansion)**
Add MRSA USA300, E. coli ST131, K. pneumoniae ATCC BAA-1705 (KPC) to MIC assay.
Impact: any hit against MDR strains is immediately publishable as clinically significant.

---

## Summary Table: Probability by Gate

| Stage | Gate | Before PRs #31–33 | After PRs #31–33 | Main limiting factor |
|-------|------|-------------------|------------------|---------------------|
| 0 | Synthesis success | ~90% | ~90% | 1 HIGH SPPS difficulty candidate |
| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | Heuristic scoring; AUROC 0.80 |
| 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | Hemolytic risk for high-μH candidates |
| 3 | t½ > 2h (serum) | ~10–20% | **~25–40%** ✓ | SEED-004 variants now prioritized |
| 4 | Scaffold novelty | ~10–15% | ~10–15% | Near-seed variants, irreducible |
| All gates | "Breaking news" hit | ~5–12% | **~10–25%** ✓ | Serum stability improvement |

**Probability of ≥1 active AMP from pilot panel (Stage 1 only):** ~85–95%
(Probability of zero active from 20 candidates with ~60% individual hit rate)

**Probability of ≥1 candidate satisfying ALL gates (updated after PRs #31–#36):** ~12–28%

---

## Confidence Calibration

This assessment is based on:
- Internal benchmark AUROC = 0.8037 (n=88, bootstrap n=2000)
- Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002)
- Published serum stability data for short cationic peptides (Hilpert et al. 2006)
- Database cross-reference with APD3 template sequences

**Key uncertainty:** The pipeline has not been prospectively validated — these are predicted,
not observed, hit rates. The true hit rate may differ significantly.

**Honest statement:** The computational work is complete and defensible. The "100% probability"
target is biologically impossible without wet-lab data. The correct goal is to maximise the
probability of finding at least one active, selective, stable, and novel candidate — and to
iterate rapidly on failures.

---

*Generated by OpenAMP-Foundry v0.1.0. All scores are computational heuristics.
No biological activity has been demonstrated. The lab is the judge.*
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37)
10 changes: 6 additions & 4 deletions src/openamp_foundry/selection/pilot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,17 @@ def _seed_from_source(source: str) -> str:


def _pilot_priority(scores: dict) -> float:
"""Higher is better: reward ensemble, reward serum stability, penalise disagreement.
"""Higher is better: reward ensemble, stability, and novelty; penalise disagreement.

Serum stability is weighted lightly (0.05) so it acts as a tiebreaker without
overriding the ensemble score. A candidate with stability=1.0 gains at most +0.05.
Within a seed family, higher-novelty variants are preferred as tiebreakers.
Novelty is weighted lightly (0.05) so the ensemble score remains dominant.
A candidate with novelty=0.467 (SEED-005 max) gains at most +0.023.
"""
ensemble = scores.get("ensemble", 0.0)
disagreement = scores.get("disagreement", 0.5)
stability = scores.get("serum_stability", 0.5)
return round(ensemble - 0.3 * disagreement + 0.05 * stability, 6)
novelty = scores.get("novelty", 0.0)
return round(ensemble - 0.3 * disagreement + 0.05 * stability + 0.05 * novelty, 6)


def select_pilot_panel(
Expand Down
Loading
Loading