diff --git a/docs/BENCHMARK_CARD.md b/docs/BENCHMARK_CARD.md index fc486497..a7d2c36d 100644 --- a/docs/BENCHMARK_CARD.md +++ b/docs/BENCHMARK_CARD.md @@ -1,7 +1,7 @@ # Benchmark Card — OpenAMP Foundry v0.5.x > **Purpose:** Single-page summary of benchmark methodology, data, and metrics. -> **Last updated:** 2026-07-01 (cluster-split benchmark added) +> **Last updated:** 2026-07-01 (hemolysis benchmark expanded to 238 peptides) --- @@ -39,6 +39,32 @@ | Bootstrap | 2000 resamples | 2000 resamples | — | | **Interpretation** | **STRONG** | **STRONG** | Below random (expected) | +## Within-AMP Selectivity Benchmark + +> Tests whether scorers can distinguish hemolytic AMPs (HC50 < 25 µg/ml) from +> selective AMPs (HC50 >= 100 µg/ml). Run: `make bench-selectivity` + +| Set | Count | Description | +|-----|:-----:|-------------| +| Hemolytic | **54** | HC50 < 25 µg/ml (literature + DBAASP v3) | +| Selective | **125** | HC50 >= 100 µg/ml | +| Border | **59** | 25 <= HC50 < 100 (excluded from binary AUROC) | +| **Total** | **238** | Expanded from 42 in v0.5.11 using DBAASP data | + +| Score | Detection AUROC | CI₉₅ | Significant? | +|-------|:--------------:|:----:|:------------:| +| hemolysis_risk | 0.5650 | 0.47–0.66 | No (direction correct) | +| safety | 0.5116 | 0.43–0.60 | No | +| selectivity_proxy | 0.5744 | 0.50–0.66 | Borderline | +| ensemble | 0.4201 | 0.33–0.51 | No (wrong direction) | +| expert_composite | 0.5459 | 0.46–0.63 | No | +| serum_stability | 0.5873 | 0.51–0.66 | Yes (risk detector) | + +**Key finding:** The hemolysis risk scorer's original AUROC=0.9218 (CI 0.82-0.99) +on n=35 was small-sample inflation. On the expanded n=179, no scorer achieves +strong hemolysis detection. Hemolysis remains unpredictable by 1D +physicochemical features and must be assayed experimentally. + ## Method AUROC computed via Wilcoxon-Mann-Whitney statistic (concordant-pair enumeration). diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index f95d41e4..bcbfcef6 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -3,7 +3,7 @@ > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-01 (cluster-split benchmark added) +> **Last updated:** 2026-07-01 (hemolysis benchmark expanded to 238 peptides) > **Pipeline version:** v0.5.x > **Branch:** main @@ -312,56 +312,66 @@ upon as a hemolysis predictor. > hemolysis detection (AUROC=0.3844). A dedicated hemolysis risk scorer was > built from empirically-validated components identified in that benchmark. > +> **v0.5.11 correction:** The original 42-peptide reference set (14 hemolytic +> vs 21 selective, n=35) produced detection AUROC=0.9218 (CI 0.82-0.99). This +> was **small-sample inflation**. Expansion to 238 peptides using DBAASP human +> erythrocyte data (54 hemolytic vs 125 selective, n=179) dropped the detection +> AUROC to 0.5650 (CI 0.47-0.66) — direction correct but NOT statistically +> significant. The scorer retains weak directional signal but should not be +> trusted as a standalone hemolysis detector. +> > Run: `make bench-selectivity` (hemolysis_risk column in the output) **Module:** `src/openamp_foundry/scoring/hemolysis.py` -**Components** (detection AUROC from leave-one-out on n=14 hemolytic vs n=21 selective): +**Components** (individual AUROC from original n=14 vs n=21; may not replicate on expanded set): -| Component | Individual AUROC | Weight | Signal source | -|-----------|:----------------:|:------:|---------------| +| Component | Individual AUROC (n=35) | Weight | Signal source | +|-----------|:-----------------------:|:------:|---------------| | Synthesis difficulty (1 - synth_feasibility) | 0.8027 | 0.30 | Incidental: hemolytic AMPs harder to synthesize | | Aromatic fraction (F/W/Y density) | 0.8299 | 0.30 | Trp/Phe intercalation in both membrane types | | Cationic-on-hydrophobic-face fraction | 0.7585 | 0.20 | Poor amphipathic face segregation | | Cysteine fraction | 0.7500 | 0.20 | Beta-sheet defensin/protegrin class | -**Combined performance:** +**Combined performance (expanded n=179):** -| Metric | Value | Notes | -|--------|-------|-------| -| **Detection AUROC** | **0.9218** | Bootstrap CI₉₅: 0.82-0.99 (n_bootstrap=2000) | -| Mean hemolytic risk | 0.4064 | n=14 | -| Mean selective risk | 0.1501 | n=21 | -| Risk threshold > 0.40 | Enriched in hemolytic | 10/14 hemolytic vs 2/21 selective | -| Safety scorer (comparison) | 0.3844 | FAILS — all 14 hemolytic AMPs score safety >= 0.94 | +| Metric | Original (n=35) | Expanded (n=179) | Notes | +|--------|:---------------:|:-----------------:|-------| +| **Detection AUROC** | **0.9218** | **0.5650** | Small-sample inflation corrected | +| CI₉₅ lower bound | 0.82 | 0.47 | No longer > 0.5 — not significant | +| CI₉₅ upper bound | 0.99 | 0.66 | | +| Mean hemolytic risk | 0.4064 (n=14) | 0.2042 (n=54) | Direction still correct | +| Mean selective risk | 0.1501 (n=21) | 0.1535 (n=125) | | +| Safety scorer detection | 0.3844 | 0.5116 | Safety also improves slightly with more data | -**Expert composite integration:** +**Expert composite integration (expanded n=179):** -| Metric | Before (v0.5.9) | After (v0.5.10) | -|--------|:---------------:|:----------------:| -| Expert composite detection AUROC | 0.5119 | 0.6429 | -| Expert composite CI lo | 0.3129 | 0.4490 | -| Ensemble detection AUROC | 0.3486 | 0.3486 (unchanged) | +| Metric | Before (v0.5.9, n=35) | After (v0.5.10, n=35) | Expanded (n=179) | +|--------|:---------------------:|:---------------------:|:-----------------:| +| Expert composite detection AUROC | 0.5119 | 0.6429 | 0.5459 | +| Expert composite CI lo | 0.3129 | 0.4490 | 0.4562 | +| Ensemble detection AUROC | 0.3486 | 0.3486 | 0.4201 | -**Expert ablation (AMP-vs-decoy):** +**Expert ablation (AMP-vs-decoy, unchanged):** | Metric | Value | Classification | |--------|:-----:|:--------------:| | hemolysis_safety AUROC | 0.3285 | **Anti-signal** (above_random = -0.1715) | | Expert composite AUROC | 0.7119 | Down from 0.7360 (delta -0.0713 vs ensemble) | -**Key finding:** The dedicated hemolysis risk scorer is the first pipeline score -with a statistically significant hemolysis detection signal (CI lo = 0.82 > 0.5). -It complements (not replaces) the safety scorer, which retains its role for -AMP-vs-decoy discrimination. The hemolysis_safety component is anti-signal on -AMP-vs-decoy (AUROC=0.3285) — expected, since real AMPs have higher hemolysis -risk than random decoys. This confirms the component measures a within-AMP -property, not an AMP-vs-non-AMP property. +**Key finding (corrected):** The hemolysis risk scorer's original detection +AUROC=0.9218 on n=35 was small-sample inflation. On the expanded n=179 +reference set, detection AUROC=0.5650 (CI 0.47-0.66) — direction is correct +(hemolytic > selective on average) but not statistically significant. The +scorer should NOT be described as a "statistically significant hemolysis +detector." It provides weak directional signal that may be useful as one +factor in a composite but cannot be relied upon for hemolysis triage. Hemolysis +must still be assayed experimentally for every candidate. -**Honest limitation:** The reference set is small (n=35). The CI is wide. -Melittin's risk score (0.13) remains modest because its bent-helix hemolysis -mechanism is not fully captured by 1D features. Hemolysis must still be assayed -experimentally for every candidate. +**Honest limitation:** The expanded reference set (n=179) provides a more honest +estimate, but HC50 values are approximate literature values with high inter-assay +variability. Melittin's risk score (0.13) remains modest because its bent-helix +hemolysis mechanism is not fully captured by 1D features. --- @@ -369,7 +379,7 @@ experimentally for every candidate. | Metric | Value | |--------|-------| -| Total tests | 1471 | +| Total tests | 1471+ | | Coverage (branch) | 99% (6 CLI guard lines only) | | Source modules at 100% | All pipeline, QC, scoring modules | @@ -398,6 +408,7 @@ experimentally for every candidate. | 2026-06-29 | External predictor results filled from wave05_combined_consensus.csv; all 7 gates PASS | OpenAMP Wave 0.5 | | 2026-06-29 | Wave 0.5 scaffold diversification — 24-candidate Wave 1 panel across 14 families | OpenAMP Wave 0.5 | | 2026-07-01 | Expert ablation benchmark added: expert composite AUROC 0.736 vs ensemble 0.7832 (delta −0.0472); 3 components anti-signal; ensemble remains primary gate | OpenAMP loop | +| 2026-07-01 | **Hemolysis benchmark expanded:** 42 -> 238 peptides using DBAASP human erythrocyte data (54 hemolytic vs 125 selective, n=179 binary). Hemolysis risk scorer detection AUROC drops 0.9218 -> 0.5650 (CI 0.47-0.66) — original performance was small-sample inflation. Direction correct, not significant. Safety scorer detection improves 0.3844 -> 0.5116 (still not significant). 196 new peptides from DBAASP v3. | OpenAMP loop | | 2026-07-01 | Dedicated hemolysis risk scorer: 4-component score (synth+aromatic+face+cys) achieves detection AUROC=0.9218 (CI: 0.82-0.99); integrated into expert composite (detection 0.5119→0.6429); safety scorer unchanged; 1471 tests | OpenAMP loop | | 2026-07-01 | Within-AMP selectivity benchmark added: safety scorer FAILS hemolysis detection (AUROC=0.3844); synthesis is only significant risk detector (AUROC=0.8027); expert composite better than ensemble but not significant (0.5119 vs 0.3486) | OpenAMP loop | | 2026-06-29 | Initial — expanded benchmark (PR #110) | OpenAMP CI | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index ec916ad9..48435aa6 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -146,12 +146,27 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - `make bench-cluster-split` Makefile target added (was missing despite CLI command existing) - 14 new tests in `test_expert_ablation.py`; 1435 total tests +## v0.5.11 — Hemolysis Benchmark Expansion (DBAASP) ✓ (2026-07-01) + +- Expanded hemolysis reference from 42 to 238 peptides using DBAASP v3 human erythrocyte data +- 196 new peptides with 50% hemolysis (HC50) values, converted to µg/ml using peptide MW +- Binary task: 54 hemolytic (HC50 < 25 µg/ml) vs 125 selective (HC50 >= 100 µg/ml) = 179 total (was 14 vs 21 = 35) +- **Critical honesty correction:** hemolysis risk scorer detection AUROC drops from 0.9218 (CI 0.82-0.99) to 0.5650 (CI 0.47-0.66) + - Original n=35 performance was **small-sample inflation** + - Direction still correct (hemolytic mean=0.204 > selective mean=0.154) but NOT statistically significant + - Safety scorer detection improves slightly: 0.3844 → 0.5116 (still not significant) + - Selectivity proxy: detection AUROC=0.5744 (CI 0.50-0.66) — borderline significant +- Extraction script: `scripts/expand_hemolysis_benchmark.py` +- Tests updated: `test_selectivity_benchmark.py` test renamed from `test_hemolysis_risk_is_significant_detector` to `test_hemolysis_risk_direction_correct` +- Docs updated: METRICS_CURRENT.md, hemolysis.py, expert.py comments corrected to reflect expanded-set reality +- **Lesson:** A 0.92 AUROC on n=35 is not robust. The project's adversarial benchmark philosophy caught this. + ## v0.5.10 — Dedicated Hemolysis Risk Scorer ✓ (2026-07-01) - `scoring/hemolysis.py` — standalone 4-component hemolysis risk score (synth difficulty + aromatic density + face cationic leakage + cysteine content) -- Combined detection AUROC=0.9218 (CI₉₅: 0.82-0.99) — first statistically significant hemolysis detector +- Combined detection AUROC=0.9218 (CI₉₅: 0.82-0.99) on original 42-peptide reference — **CORRECTED in v0.5.11: small-sample inflation; expanded n=179 gives 0.5650 (CI 0.47-0.66)** - Integrated into selectivity benchmark as risk-direction score; integrated into expert composite as `hemolysis_safety` (weight 0.10) -- Expert composite hemolysis detection improved: 0.5119 → 0.6429 (not significant, but trend correct) +- Expert composite hemolysis detection improved: 0.5119 → 0.6429 on n=35 — **expanded n=179 gives 0.5459 (CI 0.46-0.63)** - Safety scorer UNCHANGED — standard AUROC remains 0.7832; hemolysis risk is complementary, not replacement - Expert ablation: hemolysis_safety is anti-signal on AMP-vs-decoy (AUROC=0.3285) — confirms it measures within-AMP property - 15 new tests in `test_hemolysis_risk.py`; 3 new tests in `test_selectivity_benchmark.py`; 1471 total tests diff --git a/examples/validation/hemolysis_reference.csv b/examples/validation/hemolysis_reference.csv index eed25f60..66f96f13 100644 --- a/examples/validation/hemolysis_reference.csv +++ b/examples/validation/hemolysis_reference.csv @@ -41,3 +41,199 @@ REF-ESC-001,GIFSKLAGKKIKNLLISGLKG,esculentin,100,SELECTIVE,Simmaco_1993 REF-BOM-001,IIGPVLGLVGSALGGLLKKI,bombinin_H,100,SELECTIVE,Halverson_1990 REF-TMPG-001,ILSKIFSLLKKVLP,temporin_G,100,SELECTIVE,Simmaco_1996 REF-POLY1-001,RRWCFRVCYRGFCYRKCR,polyphemusin_I,10,MODERATE,Miyata_1989 +DBAASP-37,GMASKAGAIAGKIAKVALKAL,dbaasp_hemolysis,1.2,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-102,ALWKTLLKKVLKAAAKAALKAVLVGANA,dbaasp_hemolysis,1.4,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1797,GIGAVLWVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,2.6,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3173,FFGWLIKGAIHAGKAIHGLIHRRRH,dbaasp_hemolysis,2.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3174,FFGWLIRGAIHAGKAIHGLIHRRRH,dbaasp_hemolysis,2.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1790,GIGAVLAVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,3.0,HEMOLYTIC,DBAASP_v3_2meas +DBAASP-113,KLLAKAALKWLLKALKAA,dbaasp_hemolysis,3.3,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1793,GIGAVLGVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,3.4,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-111,KAIAKSILKWIKSIAKAI,dbaasp_hemolysis,3.6,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-115,KALKKLLAKWLAAAKALL,dbaasp_hemolysis,3.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1792,GIGAVLEVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,4.4,HEMOLYTIC,DBAASP_v3_2meas +DBAASP-112,KALAALLKKWAKLLAALK,dbaasp_hemolysis,4.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-2718,YKLLKKLLKKLKKLLKKL,dbaasp_hemolysis,6.1,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-2719,YKLLKWLLLKLKALLEKL,dbaasp_hemolysis,6.3,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1999,KWARLWRWFRITRWLWYIK,dbaasp_hemolysis,7.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1791,GIGAVLDVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,7.3,HEMOLYTIC,DBAASP_v3_2meas +DBAASP-948,FWGALAKGALKLIPSLFSSFSKKD,dbaasp_hemolysis,8.6,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-970,GLFGKLIKKFLRKAISYAVKKARGKH,dbaasp_hemolysis,8.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1418,FALALKALKKLLKKLKKLAKKAL,dbaasp_hemolysis,10.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1468,FLSLALAALPKFLCLVFKKC,dbaasp_hemolysis,11.1,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1470,FLPLLLAGLPKLLCLFFKKC,dbaasp_hemolysis,11.4,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-2201,FLPILAGLAAKIVPKLFCLATKKC,dbaasp_hemolysis,12.8,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3136,GLLKLIKTLL,dbaasp_hemolysis,13.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1469,FLSLALAALPKLFCLIFKKC,dbaasp_hemolysis,15.7,HEMOLYTIC,DBAASP_v3_3meas +DBAASP-1001,FLGLLFHGVHHVGKWIHGLIHGHH,dbaasp_hemolysis,16.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1465,FIGAIARLLSKIF,dbaasp_hemolysis,17.1,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-536,GLFGKLIKKFGRKAISYAVKKARGKH,dbaasp_hemolysis,17.4,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1240,INWLKLGKAIIDAL,dbaasp_hemolysis,18.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-117,KALAKALAKLWKALAKAA,dbaasp_hemolysis,18.7,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1030,KLALKLALKALKAALKLA,dbaasp_hemolysis,18.8,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-11,RVKRVWPLVIRTVIAGYNLYRAIKKK,dbaasp_hemolysis,18.9,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-2156,AIGSILGALAKGLPTLISWIKNR,dbaasp_hemolysis,19.1,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1421,FAKLLAKLAKKLL,dbaasp_hemolysis,19.2,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-116,KITLKLAIKAWKLALKAA,dbaasp_hemolysis,19.8,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-1422,FAKLLAKALKKLL,dbaasp_hemolysis,20.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-103,KLALKLALKAWKAALKLA,dbaasp_hemolysis,21.5,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-2453,GLPVCGETCFGGTCNTPGCSCETWPVCSRN,dbaasp_hemolysis,23.2,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3140,GLLKWIKTLL,dbaasp_hemolysis,23.7,HEMOLYTIC,DBAASP_v3_2meas +DBAASP-2712,YKLLKLLLPKLKGLLFKL,dbaasp_hemolysis,24.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3102,LLKKLLKKLLKKLLKK,dbaasp_hemolysis,24.0,HEMOLYTIC,DBAASP_v3_1meas +DBAASP-3100,FFRRFFRRFFRRFFRR,dbaasp_hemolysis,26.0,MODERATE,DBAASP_v3_1meas +DBAASP-107,ALWKTLLKKVLKAAAK,dbaasp_hemolysis,26.7,MODERATE,DBAASP_v3_2meas +DBAASP-2209,FLGSIVGALASALPSLISKIRN,dbaasp_hemolysis,26.7,MODERATE,DBAASP_v3_1meas +DBAASP-2714,YKLLKLLLPKLKGLLIKL,dbaasp_hemolysis,28.0,MODERATE,DBAASP_v3_1meas +DBAASP-2452,GIPCGESCVWIPCITSAIGCSCKSKVCYRN,dbaasp_hemolysis,28.3,MODERATE,DBAASP_v3_1meas +DBAASP-3137,GLLKIIKTLL,dbaasp_hemolysis,29.0,MODERATE,DBAASP_v3_1meas +DBAASP-1106,GEILCNLCTGLINTLENLLTTKRKRQQ,dbaasp_hemolysis,30.7,MODERATE,DBAASP_v3_1meas +DBAASP-3150,GLLKVIKVLL,dbaasp_hemolysis,32.0,MODERATE,DBAASP_v3_1meas +DBAASP-2204,FLSGIVGMLGKLF,dbaasp_hemolysis,34.5,MODERATE,DBAASP_v3_1meas +DBAASP-2370,FFHHIFRGIVHVGKTIHRLVTG,dbaasp_hemolysis,38.6,MODERATE,DBAASP_v3_2meas +DBAASP-911,WKLFKKILKWL,dbaasp_hemolysis,39.1,MODERATE,DBAASP_v3_1meas +DBAASP-3326,ILGIITSLLKSL,dbaasp_hemolysis,39.4,MODERATE,DBAASP_v3_1meas +DBAASP-2255,FFGKVLKLIRKIF,dbaasp_hemolysis,40.2,MODERATE,DBAASP_v3_2meas +DBAASP-701,GIWDTIKSMGKVFAGLILQNL,dbaasp_hemolysis,41.5,MODERATE,DBAASP_v3_1meas +DBAASP-691,KIAGKIAKIAGKIAKIAGKIA,dbaasp_hemolysis,42.0,MODERATE,DBAASP_v3_1meas +DBAASP-2000,KWAKKWKWFAKAAWKWYKK,dbaasp_hemolysis,42.0,MODERATE,DBAASP_v3_1meas +DBAASP-3104,FLSIIAKVLGSLF,dbaasp_hemolysis,42.2,MODERATE,DBAASP_v3_1meas +DBAASP-2159,ILGKIWEGIKSLF,dbaasp_hemolysis,43.6,MODERATE,DBAASP_v3_1meas +DBAASP-3105,FLPLVTMLLGKLF,dbaasp_hemolysis,44.8,MODERATE,DBAASP_v3_1meas +DBAASP-700,GIWDTIKSMGKVFAGAILQNL,dbaasp_hemolysis,45.2,MODERATE,DBAASP_v3_1meas +DBAASP-912,WKLFKKILKFL,dbaasp_hemolysis,46.8,MODERATE,DBAASP_v3_1meas +DBAASP-1889,FLPIVAGLAANFLPKIVCKITKKC,dbaasp_hemolysis,49.2,MODERATE,DBAASP_v3_1meas +DBAASP-1890,FLSTLLNVASNVVPTLICKITKKC,dbaasp_hemolysis,49.5,MODERATE,DBAASP_v3_1meas +DBAASP-1892,FLPMLAGLAANFLPKIVCKITKKC,dbaasp_hemolysis,49.8,MODERATE,DBAASP_v3_1meas +DBAASP-167,KWKLFKKIGIGAVLKVLTTGLPALKLTK,dbaasp_hemolysis,50.0,MODERATE,DBAASP_v3_1meas +DBAASP-2298,GLLGLLGSVVSHVVPAIVGHF,dbaasp_hemolysis,51.8,MODERATE,DBAASP_v3_1meas +DBAASP-1978,FLAGLIGGLAKML,dbaasp_hemolysis,52.1,MODERATE,DBAASP_v3_1meas +DBAASP-2713,YKLLKLLLPKLKPLLFKL,dbaasp_hemolysis,55.0,MODERATE,DBAASP_v3_1meas +DBAASP-2508,GFMKYIGPLIPHAVKAIKKLI,dbaasp_hemolysis,58.4,MODERATE,DBAASP_v3_1meas +DBAASP-3135,GLLKVIKTLL,dbaasp_hemolysis,60.0,MODERATE,DBAASP_v3_1meas +DBAASP-913,WKLFKKILKYL,dbaasp_hemolysis,62.2,MODERATE,DBAASP_v3_1meas +DBAASP-999,GFWGKLFKLGLHGIGLLHLHL,dbaasp_hemolysis,64.0,MODERATE,DBAASP_v3_1meas +DBAASP-1320,GLFDVIKKVASVIGGL,dbaasp_hemolysis,64.6,MODERATE,DBAASP_v3_1meas +DBAASP-2715,YKLLKLLLPKLKPLLIKL,dbaasp_hemolysis,69.0,MODERATE,DBAASP_v3_1meas +DBAASP-2264,VSRRRRRRGGRRRR,dbaasp_hemolysis,69.9,MODERATE,DBAASP_v3_1meas +DBAASP-3138,GLLKFIKTLL,dbaasp_hemolysis,77.0,MODERATE,DBAASP_v3_1meas +DBAASP-2203,NFLGTLVNLAKKIL,dbaasp_hemolysis,77.2,MODERATE,DBAASP_v3_1meas +DBAASP-121,GIGKFIHAAKKFGKLFIGEIMNS,dbaasp_hemolysis,80.2,MODERATE,DBAASP_v3_1meas +DBAASP-3143,GLLKRIKLLL,dbaasp_hemolysis,81.0,MODERATE,DBAASP_v3_1meas +DBAASP-2198,GLFGKILGVGKKVLCGLSGMC,dbaasp_hemolysis,83.2,MODERATE,DBAASP_v3_1meas +DBAASP-108,ALWKTLLKKVLKA,dbaasp_hemolysis,86.2,MODERATE,DBAASP_v3_2meas +DBAASP-3379,LNWGALLKHLLK,dbaasp_hemolysis,88.6,MODERATE,DBAASP_v3_1meas +DBAASP-1460,FLPVILPVIGKLLSGIL,dbaasp_hemolysis,89.6,MODERATE,DBAASP_v3_1meas +DBAASP-125,GIGKFLHTLKTFGKKWVGEIMNS,dbaasp_hemolysis,90.7,MODERATE,DBAASP_v3_1meas +DBAASP-704,GIWKTIKSMGKVFAGAILQNL,dbaasp_hemolysis,91.0,MODERATE,DBAASP_v3_1meas +DBAASP-1477,FLIGMTQGLICLITRKC,dbaasp_hemolysis,91.7,MODERATE,DBAASP_v3_1meas +DBAASP-2162,ILGKIWKGIKSLF,dbaasp_hemolysis,92.2,MODERATE,DBAASP_v3_3meas +DBAASP-497,KRLFKKLKFSLRKY,dbaasp_hemolysis,94.6,MODERATE,DBAASP_v3_4meas +DBAASP-3124,GLLKRIKALL,dbaasp_hemolysis,97.0,MODERATE,DBAASP_v3_1meas +DBAASP-3145,GLLKRIKFLL,dbaasp_hemolysis,98.0,MODERATE,DBAASP_v3_1meas +DBAASP-3147,GLLKRIKWLL,dbaasp_hemolysis,98.0,MODERATE,DBAASP_v3_1meas +DBAASP-1891,FLSTLLNVASKVVPTLFCKITKKC,dbaasp_hemolysis,99.5,MODERATE,DBAASP_v3_1meas +DBAASP-646,FLWALFKVASK,dbaasp_hemolysis,100.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3177,SLLSLIRKLIT,dbaasp_hemolysis,100.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-2335,HFLGTLVNLAKKIL,dbaasp_hemolysis,101.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-1127,FLPIVGKLLSGLL,dbaasp_hemolysis,102.7,SELECTIVE,DBAASP_v3_1meas +DBAASP-2300,GFKDLLKGAAKALVKTVLF,dbaasp_hemolysis,111.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-3287,GKWMSLLKHILK,dbaasp_hemolysis,113.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-3144,GLLKRIKILL,dbaasp_hemolysis,119.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-932,GLLALISWIKRKRQQ,dbaasp_hemolysis,120.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-498,LLLFLLKKRKKRKY,dbaasp_hemolysis,120.1,SELECTIVE,DBAASP_v3_4meas +DBAASP-3377,LNWGAFLKHFFK,dbaasp_hemolysis,122.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-2253,IKWKKLLRAAKRIL,dbaasp_hemolysis,130.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-1129,FLPIIGKLLSGLL,dbaasp_hemolysis,131.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-3123,GLLKRIATLL,dbaasp_hemolysis,132.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-496,KRLFKELLFSLRKY,dbaasp_hemolysis,134.4,SELECTIVE,DBAASP_v3_3meas +DBAASP-2292,GFLGPLLKLGLKGAAKLLPQLLPSRQQ,dbaasp_hemolysis,142.9,SELECTIVE,DBAASP_v3_1meas +DBAASP-2291,GFLGPLLKLGLKGVAKVIPHLIPSRQQ,dbaasp_hemolysis,144.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-123,GIGKFIHSVKKWGKTFIGEIMNS,dbaasp_hemolysis,144.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-904,WKLFKKILKVL,dbaasp_hemolysis,147.2,SELECTIVE,DBAASP_v3_1meas +DBAASP-2504,GFMKYIGPLIPHAVKAISDLI,dbaasp_hemolysis,148.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-2290,GVWTTILGGLKKFAKGGLEALTNPK,dbaasp_hemolysis,156.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2294,FLGALLGPLMNLLQ,dbaasp_hemolysis,157.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-3121,GLLKAIKTLL,dbaasp_hemolysis,161.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3365,LNWKAILKHIIK,dbaasp_hemolysis,162.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-3103,FLSSIGKILGNLL,dbaasp_hemolysis,164.9,SELECTIVE,DBAASP_v3_1meas +DBAASP-3360,LNWGAILKHIIK,dbaasp_hemolysis,167.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-1783,GIGKFLHAAKKFAKAFVAEIMNS,dbaasp_hemolysis,175.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2304,GFRDVLKGAAKAFVKTVAGHIAN,dbaasp_hemolysis,189.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-2001,KFAKKFKWFAKAAFKFFKK,dbaasp_hemolysis,195.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-109,KLALKAALKAWKAAAKLA,dbaasp_hemolysis,199.7,SELECTIVE,DBAASP_v3_1meas +DBAASP-1419,FAKKLAKLAKKLAKLAL,dbaasp_hemolysis,208.7,SELECTIVE,DBAASP_v3_1meas +DBAASP-14,GIWDTIKSMGKVFAGKILQNL,dbaasp_hemolysis,208.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-1883,FLPIVGKLLSGLF,dbaasp_hemolysis,210.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-126,GIGHFLHKVKSFGKSWIGEIMNS,dbaasp_hemolysis,211.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2401,SMLSVLKNLGKVGLGFVACKINKKC,dbaasp_hemolysis,212.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-2407,ILGKLLSTAAGLLSNL,dbaasp_hemolysis,213.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-3382,LNWGAWLKHWWK,dbaasp_hemolysis,214.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-3232,FFFLSRIF,dbaasp_hemolysis,215.2,SELECTIVE,DBAASP_v3_1meas +DBAASP-1882,FFPIVGKLLSGLF,dbaasp_hemolysis,215.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-1478,GLFDVVKGVLKGVGKNVAGSLLEQLKCKLSGGC,dbaasp_hemolysis,216.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-1795,GIGAVLLVLTTGLPALISWIKRKRQQ,dbaasp_hemolysis,219.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3346,ILPILSLIGGLL,dbaasp_hemolysis,219.9,SELECTIVE,DBAASP_v3_1meas +DBAASP-699,GIWKTIKSMGKVFAGKILQNL,dbaasp_hemolysis,221.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-3151,KLLKFIKTLL,dbaasp_hemolysis,231.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3153,KLLKVIKKLL,dbaasp_hemolysis,231.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2593,GIGKFLKKAKKFGKAFVKMKK,dbaasp_hemolysis,238.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-2400,SMLSVLKNLGKVGLGFVACKIKKQC,dbaasp_hemolysis,239.9,SELECTIVE,DBAASP_v3_1meas +DBAASP-952,RRFPWWWPFRR,dbaasp_hemolysis,240.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-1887,GLWNTIKEAGKKFAINVLDKIRCGIAGGCKT,dbaasp_hemolysis,247.9,SELECTIVE,DBAASP_v3_1meas +DBAASP-3157,GLLGPLLKIAAKVGSNLL,dbaasp_hemolysis,248.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-914,LKLFKKILKVL,dbaasp_hemolysis,255.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-1893,GILDKLKEFGISAARGVAQSLLNTASCKLAKTC,dbaasp_hemolysis,255.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-2289,GFWTTAAEGLKKFAKAGLASILNPK,dbaasp_hemolysis,275.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-2507,GFMKYIGPLIPHAVKAISKLI,dbaasp_hemolysis,275.6,SELECTIVE,DBAASP_v3_2meas +DBAASP-915,KKLFKKILKFL,dbaasp_hemolysis,285.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-2398,SMLSVLKKLGKVGLGFVACKINKQC,dbaasp_hemolysis,293.2,SELECTIVE,DBAASP_v3_1meas +DBAASP-1886,SLFSIFKTAAKFVGKNLLKQAGKAGLETLACKAKNEC,dbaasp_hemolysis,294.7,SELECTIVE,DBAASP_v3_1meas +DBAASP-3142,GLLKRIKVLL,dbaasp_hemolysis,310.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3149,KLLKRIKTLL,dbaasp_hemolysis,312.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2509,GFMKYIKPLIPHAVKAISKLI,dbaasp_hemolysis,319.7,SELECTIVE,DBAASP_v3_2meas +DBAASP-2395,SMLKVLKNLGKVGLGFVACKINKQC,dbaasp_hemolysis,323.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-2399,SMLSVLKNLGKVGLGFVKCKINKQC,dbaasp_hemolysis,325.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2404,GLKEIFKAGLGSLVKGIAAHVAS,dbaasp_hemolysis,328.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-2210,GIMSLFKGVLKTAGKHVAGSLVDQLKCKITGGC,dbaasp_hemolysis,336.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-2367,IWLTALKFLGKHAAKHLAKQQLSKL,dbaasp_hemolysis,355.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-3065,ILPWKWKWWPWRR,dbaasp_hemolysis,358.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3152,KLLKRIKKLL,dbaasp_hemolysis,369.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-1010,SMLSVLKNLGKVGLGFVACKINKQC,dbaasp_hemolysis,371.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-160,VRRFPWWWPFLRR,dbaasp_hemolysis,380.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3146,GLLKRIKNLL,dbaasp_hemolysis,381.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2196,GLFTLIKGAAKLIGKTVAKEAGKTGLELMACKITNQC,dbaasp_hemolysis,382.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-1888,FLSTALKVAANVVPTLFCKITKKC,dbaasp_hemolysis,389.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-2293,GFLGSLLKTGLKVGSNLL,dbaasp_hemolysis,399.7,SELECTIVE,DBAASP_v3_1meas +DBAASP-874,AGLQFPVGRVHRLLRK,dbaasp_hemolysis,400.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-878,FPVGRVHRLLRK,dbaasp_hemolysis,400.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-1748,ILKWKWPWWPWRR,dbaasp_hemolysis,400.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-933,GLSALISWIKRKRQQ,dbaasp_hemolysis,403.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2207,AVDLAKIANKVLSSLF,dbaasp_hemolysis,422.2,SELECTIVE,DBAASP_v3_1meas +DBAASP-910,KKLFKKILKVL,dbaasp_hemolysis,453.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-2194,GLLDTFKNLALNAAKSAGVSVLNSLSCKLSKTC,dbaasp_hemolysis,471.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-2403,GLKEIFKAGLGSLVKGIAAHVAN,dbaasp_hemolysis,481.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-2510,GFMKYIKPLIPHAVKAIKKLI,dbaasp_hemolysis,481.8,SELECTIVE,DBAASP_v3_2meas +DBAASP-3068,ILPWKWPWWKWRR,dbaasp_hemolysis,490.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2282,GVIKSVLKGVAKTVALGML,dbaasp_hemolysis,527.6,SELECTIVE,DBAASP_v3_1meas +DBAASP-1998,KWASLWNWFNITNWLWYIK,dbaasp_hemolysis,550.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-3069,ILKWKWKWWPWRR,dbaasp_hemolysis,560.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-2197,GLLSGILGAGKNIVCGLSGLC,dbaasp_hemolysis,573.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-2199,GVLGTVKNLLIGAGKSAAQSVLKTLSCKLSNDC,dbaasp_hemolysis,592.1,SELECTIVE,DBAASP_v3_1meas +DBAASP-937,GLPALILWIKRKRQQ,dbaasp_hemolysis,620.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-120,GIGKFIHSAKKFGKLFVGEIMNS,dbaasp_hemolysis,652.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-122,GIGKFIHSAKRFGRAWVGEIMNS,dbaasp_hemolysis,776.2,SELECTIVE,DBAASP_v3_1meas +DBAASP-1977,SLWENFKNAGKQFILNILDKIRCRVAGGCRT,dbaasp_hemolysis,781.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-2018,KILRGVCKKIMRTFLRRISKDILTGKK,dbaasp_hemolysis,900.0,SELECTIVE,DBAASP_v3_2meas +DBAASP-3355,GLNALKKVFQGIHEAIKLINNHVQ,dbaasp_hemolysis,966.6,SELECTIVE,DBAASP_v3_2meas +DBAASP-894,AKKVFKRLEKLFSKIWNWK,dbaasp_hemolysis,974.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-2193,GLFSKFAGKGIKNLIFKGVKHIGKEVGMDVIRTGIDVAGCKIKGEC,dbaasp_hemolysis,1021.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-1639,GLLRASSVWGRKYYVDLAGCAKA,dbaasp_hemolysis,1242.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-118,GIGKFLHSAKKFGKAWVGEIMNS,dbaasp_hemolysis,1275.4,SELECTIVE,DBAASP_v3_1meas +DBAASP-930,LLPALISWIKRKRQQ,dbaasp_hemolysis,1450.0,SELECTIVE,DBAASP_v3_1meas +DBAASP-1970,GAFGDLLKGVAKEAGMKLLNMAQCKLSGKC,dbaasp_hemolysis,1602.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-1971,SILDKIKNVALGVARGAGTGILKALLCKLDKSC,dbaasp_hemolysis,1617.5,SELECTIVE,DBAASP_v3_1meas +DBAASP-3141,GLLKKIKTLL,dbaasp_hemolysis,1636.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-124,GIAKFGKAAAHFGKKWVGELMNS,dbaasp_hemolysis,1713.3,SELECTIVE,DBAASP_v3_1meas +DBAASP-2281,GVIIDTLKGAAKTVAAELLRKAHCKLTNSC,dbaasp_hemolysis,2187.8,SELECTIVE,DBAASP_v3_1meas +DBAASP-3116,GLLKRIKTLL,dbaasp_hemolysis,2779.5,SELECTIVE,DBAASP_v3_1meas diff --git a/scripts/expand_hemolysis_benchmark.py b/scripts/expand_hemolysis_benchmark.py new file mode 100644 index 00000000..a7036550 --- /dev/null +++ b/scripts/expand_hemolysis_benchmark.py @@ -0,0 +1,210 @@ +"""Expand the hemolysis reference set using DBAASP hemolytic activity data. + +The within-AMP selectivity benchmark (PR #113/#114) runs on 42 peptides: +14 hemolytic (HC50 < 25 ug/ml) vs 21 selective (HC50 >= 100 ug/ml) = 35 +for the binary AUROC task. The dedicated hemolysis risk scorer's detection +AUROC=0.9218 (CI 0.82-0.99) is based on this small set. + +DBAASP (data/novelty_db/hemolytic-and-cytotoxic-activities.csv) contains +hemolytic activity measurements for ~880 unique peptides against human +erythrocytes. This script extracts 50%-hemolysis (HC50) values, converts +all measurements to ug/ml using peptide molecular weight, deduplicates by +sequence (median of multiple measurements), and appends new peptides to +examples/validation/hemolysis_reference.csv. + +Unit conversion: ug/ml = uM * MW(kDa) = uM * MW(Da) / 1000 + MW computed from amino acid residue masses (average isotopic masses, + water subtracted for peptide bonds, water added for terminal groups). + +Classification thresholds (unchanged from existing reference): + HEMOLYTIC: HC50 < 25 ug/ml + MODERATE: 25 <= HC50 < 100 ug/ml + SELECTIVE: HC50 >= 100 ug/ml + +Usage: python scripts/expand_hemolysis_benchmark.py +""" + +import csv +import re +from collections import defaultdict +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +DBAASP_CSV = REPO_ROOT / "data" / "novelty_db" / "hemolytic-and-cytotoxic-activities.csv" +OUTPUT_CSV = REPO_ROOT / "examples" / "validation" / "hemolysis_reference.csv" + +# Amino acid residue masses (average, monoisotopic would also work for relative ranking). +# Values from standard biochemical tables (e.g., Sigma-Aldrich residue mass chart). +# Includes water (18.02) in each entry; peptide bond subtracts one water. +_AA_MW: dict[str, float] = { + "A": 89.09, "R": 174.20, "N": 132.12, "D": 133.10, "C": 121.16, + "Q": 146.15, "E": 147.13, "G": 75.03, "H": 155.16, "I": 131.17, + "L": 131.17, "K": 146.19, "M": 149.21, "F": 165.19, "P": 115.13, + "S": 105.09, "T": 119.12, "W": 204.23, "Y": 181.19, "V": 117.15, +} +_H2O = 18.02 + + +def peptide_mw(seq: str) -> float: + """Molecular weight of a peptide in Daltons. + + Sum of amino acid masses minus water for each peptide bond, + plus one water for the intact termini (already included in residue masses). + """ + return sum(_AA_MW[aa] for aa in seq) - (len(seq) - 1) * _H2O + + +def parse_concentration(raw: str) -> tuple[float, bool] | None: + """Extract numeric value from a concentration string. + + Returns (value, is_greater_than) or None if unparseable. + Handles formats like "6", "0.5±0.2", ">200", "39±1.5". + """ + s = raw.strip() + is_gt = ">" in s + s = s.replace("±", " ").replace(">", "").replace("<", "").strip() + m = re.match(r"([\d.]+)", s) + if m: + return float(m.group(1)), is_gt + return None + + +def classify_hemolysis(hc50_ugml: float) -> str: + """Classify a peptide by HC50 in ug/ml.""" + if hc50_ugml < 25: + return "HEMOLYTIC" + elif hc50_ugml < 100: + return "MODERATE" + else: + return "SELECTIVE" + + +def load_existing() -> tuple[set[str], list[dict]]: + """Load existing hemolysis reference entries.""" + existing_seqs: set[str] = set() + existing_rows: list[dict] = [] + with open(OUTPUT_CSV, newline="", encoding="utf-8") as f: + for row in csv.DictReader(f): + seq = row["sequence"].strip().upper() + existing_seqs.add(seq) + existing_rows.append(row) + return existing_seqs, existing_rows + + +def extract_dbaasp() -> dict[str, tuple[str, float, int]]: + """Extract HC50 values from DBAASP hemolysis data. + + Returns: {sequence: (peptide_id, median_hc50_ugml, n_measurements)} + Only includes human erythrocyte 50% hemolysis measurements. + """ + raw: dict[str, list[float]] = defaultdict(list) + pid_map: dict[str, str] = {} + + with open(DBAASP_CSV, newline="", encoding="utf-8") as f: + reader = csv.DictReader(f) + for row in reader: + seq = row.get("Peptide Sequence", "").strip().upper() + pid = row.get("Peptide ID", "").strip() + target = row.get("Target Cell", "").strip() + measure = row.get("Activity Measure for Lysis", "").strip() + conc_raw = row.get("Peptide Concentration", "").strip() + unit = row.get("Unit", "").strip() + + if not seq: + continue + if "human erythrocyte" not in target.lower(): + continue + if "50%" not in measure.lower() and "hc50" not in measure.lower(): + continue + if not all(aa in "ACDEFGHIKLMNPQRSTVWY" for aa in seq): + continue + if len(seq) < 5 or len(seq) > 60: + continue + + parsed = parse_concentration(conc_raw) + if parsed is None: + continue + val, is_gt = parsed + # Skip "greater than" values — they are lower bounds on HC50, not exact + if is_gt: + continue + + # Convert to ug/ml + mw = peptide_mw(seq) + if unit == "µM": + hc50_ugml = val * mw / 1000.0 + elif unit in ("µg/ml", "ug/ml", "ug/mL"): + hc50_ugml = val + else: + continue + + raw[seq].append(hc50_ugml) + if seq not in pid_map: + pid_map[seq] = pid + + # Deduplicate: take median of multiple measurements + result: dict[str, tuple[str, float, int]] = {} + for seq, vals in raw.items(): + median = sorted(vals)[len(vals) // 2] + result[seq] = (pid_map[seq], round(median, 1), len(vals)) + + return result + + +def main() -> None: + existing_seqs, existing_rows = load_existing() + print(f"Existing hemolysis reference: {len(existing_rows)} peptides") + + dbaasp = extract_dbaasp() + print(f"DBAASP extracted (human erythrocyte HC50, standard AAs, 5-60aa): {len(dbaasp)}") + + # Remove sequences already in the reference + new_seqs = {s: v for s, v in dbaasp.items() if s not in existing_seqs} + print(f"New unique peptides (after dedup vs existing): {len(new_seqs)}") + + # Classify and build rows + new_rows: list[dict] = [] + for seq, (pid, hc50, n_meas) in sorted(new_seqs.items(), key=lambda x: x[1][1]): + cls = classify_hemolysis(hc50) + new_rows.append({ + "id": f"DBAASP-{pid}", + "sequence": seq, + "family": "dbaasp_hemolysis", + "hc50_ugml": str(hc50), + "hemolysis_class": cls, + "reference": f"DBAASP_v3_{n_meas}meas", + }) + + # Count by class + from collections import Counter + new_classes = Counter(r["hemolysis_class"] for r in new_rows) + print(f"New peptides by class: {dict(new_classes)}") + + # Append to CSV + with open(OUTPUT_CSV, "a", newline="", encoding="utf-8") as f: + writer = csv.DictWriter( + f, + fieldnames=["id", "sequence", "family", "hc50_ugml", "hemolysis_class", "reference"], + ) + for row in new_rows: + writer.writerow(row) + + print(f"Appended {len(new_rows)} peptides to {OUTPUT_CSV}") + + # Summary after merge + all_seqs, all_rows = load_existing() + all_classes = Counter(r["hemolysis_class"] for r in all_rows) + hemo = sum(1 for r in all_rows if float(r["hc50_ugml"]) < 25) + sel = sum(1 for r in all_rows if float(r["hc50_ugml"]) >= 100) + border = sum(1 for r in all_rows if 25 <= float(r["hc50_ugml"]) < 100) + print(f"\nMerged reference: {len(all_rows)} peptides") + print(f" HEMOLYTIC (HC50<25): {hemo}") + print(f" MODERATE (25-100): {border}") + print(f" SELECTIVE (HC50>=100): {sel}") + print(f" Binary task: {hemo} hemolytic vs {sel} selective = {hemo + sel} total") + print(f" Previous binary task: 14 vs 21 = 35") + print(f" Improvement: {hemo + sel} / 35 = {(hemo + sel) / 35:.1f}x") + + +if __name__ == "__main__": + main() diff --git a/src/openamp_foundry/scoring/expert.py b/src/openamp_foundry/scoring/expert.py index 3cd20f41..65d62335 100644 --- a/src/openamp_foundry/scoring/expert.py +++ b/src/openamp_foundry/scoring/expert.py @@ -203,7 +203,7 @@ def kmer_prior_art(sequence: str, kmer_index: set[str], k: int = 5) -> dict: "serum_stability": 0.05, # proteolytic longevity (informational, low weight) "hinge_selectivity": 0.08, # central helix-hinge bonus (expert motif) "motif_novelty": 0.10, # k-mer prior-art (local novelty beyond global identity) - "hemolysis_safety": 0.10, # dedicated hemolysis risk triage (detection AUROC=0.92) + "hemolysis_safety": 0.10, # dedicated hemolysis risk triage (expanded n=179: AUROC=0.565, CI 0.47-0.66, not significant) } diff --git a/src/openamp_foundry/scoring/hemolysis.py b/src/openamp_foundry/scoring/hemolysis.py index 8c9c35f5..89746159 100644 --- a/src/openamp_foundry/scoring/hemolysis.py +++ b/src/openamp_foundry/scoring/hemolysis.py @@ -40,14 +40,19 @@ aromatic fraction by construction (chymotrypsin cleaves after F/W/Y) and is excluded to avoid double-counting. -Combined 4-component weighted score achieves detection AUROC=0.8912 -(bootstrap CI₉₅: 0.77-0.97), a substantial improvement over the safety -scorer's 0.3844. The CI lower bound (0.77) clears the 0.5 significance -threshold with margin — the first pipeline score to do so on hemolysis. +Combined 4-component weighted score achieved detection AUROC=0.8912 +(bootstrap CI₉₅: 0.77-0.97) on the original 42-peptide reference set. +However, expansion to 238 peptides (v0.5.11, DBAASP data) revealed this +was small-sample inflation: on n=179 (54 hemolytic vs 125 selective), +detection AUROC drops to 0.5650 (CI 0.47-0.66) — direction correct but +NOT statistically significant. The scorer retains weak directional signal +but should not be trusted as a standalone hemolysis detector. IMPORTANT LIMITATIONS: - - The reference set is small (14 hemolytic vs 21 selective, n=35). - The CI is wide. A larger reference set would tighten confidence. + - The original reference set was small (14 hemolytic vs 21 selective, n=35). + The expanded set (54 hemolytic vs 125 selective, n=179) revealed that the + original AUROC=0.92 was small-sample inflation. On the expanded set, + detection AUROC=0.565 (CI 0.47-0.66) — direction correct but not significant. - HC50 values are approximate literature values with high inter-assay variability (RBC source, buffer, incubation time, concentration range). - This is a triage signal, NOT a hemolysis predictor. Every candidate diff --git a/tests/test_hemolysis_risk.py b/tests/test_hemolysis_risk.py index 6c8133ac..e4b947ec 100644 --- a/tests/test_hemolysis_risk.py +++ b/tests/test_hemolysis_risk.py @@ -6,7 +6,13 @@ Key finding being tested: the safety scorer fails hemolysis detection (AUROC=0.3844), but the dedicated hemolysis risk scorer achieves -detection AUROC=0.9218 (CI=[0.82, 0.99]) on the same reference set. +detection AUROC=0.9218 (CI=[0.82, 0.99]) on the original 42-peptide reference set. + +IMPORTANT: The expanded 238-peptide reference set (v0.5.11, DBAASP data) revealed +this was small-sample inflation. On n=179 (54 hemolytic vs 125 selective), the +detection AUROC drops to 0.5650 (CI 0.47-0.66) — direction correct but no longer +statistically significant. The scorer retains weak directional signal but should +NOT be trusted as a hemolysis detector without further improvement. """ from __future__ import annotations diff --git a/tests/test_selectivity_benchmark.py b/tests/test_selectivity_benchmark.py index 7e1b2475..072eb555 100644 --- a/tests/test_selectivity_benchmark.py +++ b/tests/test_selectivity_benchmark.py @@ -7,6 +7,13 @@ AMP-vs-decoy discrimination (where safety/synthesis were anti-signal); this tests within-AMP selectivity, the task those scorers were designed for. +Reference set expanded from 42 to 238 peptides (54 hemolytic vs 125 selective) +in v0.5.11 using DBAASP human erythrocyte hemolysis data. The expanded set +revealed that the hemolysis risk scorer's original detection AUROC=0.9218 +(CI 0.82-0.99) on n=35 was small-sample inflation; on n=179 the detection +AUROC drops to 0.5650 (CI 0.47-0.66) — no longer statistically significant. +This is exactly the kind of self-deception the project is designed to catch. + All results are computational. No biological activity is implied. """ from __future__ import annotations @@ -85,10 +92,26 @@ def test_expert_composite_verdict_present(self, result): def test_hemolysis_risk_verdict_present(self, result): assert "hemolysis risk" in result["hemolysis_risk_verdict"].lower() - def test_hemolysis_risk_is_significant_detector(self, result): - """The dedicated hemolysis risk scorer should be a significant risk detector.""" - assert "hemolysis_risk" in result["risk_detectors"] - assert result["per_score_auroc"]["hemolysis_risk"]["detection_ci95_lo"] > 0.5 + def test_hemolysis_risk_direction_correct(self, result): + """The hemolysis risk scorer should rank hemolytic AMPs higher than selective on average. + + On the expanded reference set (n=179 binary), the detection AUROC is 0.5650 + (CI 0.47-0.66) — direction is correct but not statistically significant. + The original n=35 AUROC=0.9218 was small-sample inflation, corrected by + the DBAASP expansion. This test verifies the direction survives expansion. + """ + info = result["per_score_auroc"]["hemolysis_risk"] + # Direction: hemolytic should score higher (more risk) than selective + assert info["mean_hemolytic"] is not None + assert info["mean_selective"] is not None + assert info["mean_hemolytic"] > info["mean_selective"], ( + f'Hemolysis risk mean_hemolytic={info["mean_hemolytic"]} should exceed ' + f'mean_selective={info["mean_selective"]} - direction must be correct' + ) + # Detection AUROC should be above 0.5 (direction correct) + assert info["hemolysis_detection_auroc"] > 0.50, ( + f'Detection AUROC={info["hemolysis_detection_auroc"]} should be > 0.50' + ) def test_hemolysis_risk_scores_present(self, result): """Per-peptide hemolysis risk scores should be available for inspection."""