Repository navigation
feat: expanded benchmark (95 AMPs + 96 decoys, AUROC 0.7832) + DKP risk + stability calibration - #102
Merged
Merged
Conversation
…72–0.84) Adds 52 well-characterised public-domain AMPs from 12 taxonomic classes (defensins, histatins, dermcidin, cathelicidins, proline-rich Bac/PR-family, plectasin/eurocin, piscidins, tachyplesins/polyphemusins, brevinins, etc.) with clear literature citations. Expands scrambled decoys to 95 (one per AMP). Updates regression sentinels and test assertions. Docs updated: ROADMAP, DISCOVERY_PREDICTION, WET_LAB_PROBABILITY, EXPERT_REVIEW_PACK. n=191 benchmark closes the "small 43+44 set" limitation. AUROC 0.7832 reflects honest generalisation to diverse AMP classes the helic-centric scorer was not calibrated for. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…lization All four SEED-008 pilot variants (FPVTWRFWRWWKG, FPITWRWFKWWKG, FPVSWRWWKFWKG, FPVTWRWWKWYRG) start with F-Pro — the canonical N-terminal motif susceptible to diketopiperazine (DKP) cyclization at physiological pH (t½ hours–days at 37°C). DKP truncates the peptide at position 3, typically causing 5–50× activity loss if undetected at receipt. Changes: - Extract _AA_AVG_MASS to module level (was local dict in _molecular_weight) - Add SynthQC.diketopiperazine_risk field and to_dict() key - Detect X-Pro at N-terminus (excluding P-Pro: secondary amine, no cyclization) - Set n_acetylation_recommended=True for DKP-only cases (no trypsin sites) - Flag text includes: dynamic DKP product mass (e.g. ~244 Da for F-Pro), truncated sequence preview, MS receipt check guidance, Nα-Ac prevention, and Smyth et al. 2000 citation - 15 new tests: detection, PP exclusion, mass value, all 4 SEED-008 pilots, to_dict(), n_acetylation guard for DKP-only sequences, difficulty escalation
…72–0.84) Benchmark expansion: - Adds 52 well-characterised public-domain AMPs from 12 taxonomic classes (defensins, histatins, dermcidin, cathelicidins, proline-rich Bac/PR-family, plectasin/eurocin, piscidins, tachyplesins/polyphemusins, brevinins, etc.) with clear literature citations to known_amps.csv - Expands random_background.csv to 96 decoys and scrambled_decoys.csv to 95 - n=191 benchmark (95 AMPs + 96 decoys) — closes the "small 43+44 set" limitation Pipeline QC (presynth_check.py): - DKP_RISK: N-terminal X-Pro diketopiperazine cyclization check for all 4 SEED-008 pilot variants (FP-starting); excludes Pro-Pro; couples DKP prevention into Nα-Ac recommendation; MW-based MS confirmation guidance - SHORT_PEPTIDE flag for <14 AA sequences: serum stability model calibrated for 18–30 AA — warns wet-lab that model may underestimate actual t½ Doc consistency: - ASSAY_PREREGISTRATION.md: adds SEED-001 (positive control) and SEED-006 (mastoparan-X, Gram-negative cover) to priority organism rationale - EXPERT_REVIEW_PACK.md §8: SEED-007 added to serum stability calibration gap row; AUROC updated throughout - ROADMAP, DISCOVERY_PREDICTION, WET_LAB_PROBABILITY: all benchmark numbers updated to expanded 95+96 values - .commandcode/ added to .gitignore Tests: 1336 passing (up from 1321); regression sentinels updated for new expanded-benchmark baselines (phase3 AUROC ≥0.73, pipeline AUROC <0.78) Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
cschanhniem
changed the title
Raise minimum assertion bounds from 43/44 (old demo set) to 90+ to match the expanded 95+96 benchmark (PR #110). Prevents silent reversion to the smaller 43+44 dataset. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
… docs HIGH fixes: - n_acetylation_reason no longer overwritten by trypsin block when DKP risk also present; both reasons are now concatenated (affected all 4 SEED-008 pilots which have both F-Pro and interior K/R residues) - EXPERT_REVIEW_PACK.md: remove truncated fragment + duplicate boilerplate block from end of file (doc corruption from prior patch) MEDIUM fixes: - EXPERT_REVIEW_PACK.md lines 33 and 63: update stale AUROC 0.8420/43-AMP → 0.7832/95-AMP expanded benchmark numbers - DISCOVERY_PREDICTION.md: fix "~84% of the time" → "~78% of the time" to match new AUROC=0.7832; remove lone '.' + duplicate paragraph from tracking section - ASSAY_PREREGISTRATION.md: add Nα-acetylation REQUIRED note for SEED-008 (F-Pro DKP risk, MW-244 Da MS check, 5% satellite rejection criterion) LOW fix: - test_phase3_benchmark_passes_gate docstring: update sentinel comment to match new bounds (< 0.78 not < 0.84; baseline 0.7448 not 0.8266) New test: test_dkp_reason_preserved_when_trypsin_sites_also_present — regression guard ensuring DKP prevention appears in n_acetylation_reason even when trypsin sites are also detected
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR expands the benchmark from 43+44 (n=87) to 95+96 (n=191), adds DKP_RISK detection in presynth QC, and flags serum stability calibration gaps for short peptides.
Benchmark Expansion
DKP Risk Detection (presynth_check.py)
Stability Calibration
Doc Consistency
Tests
1336 passing (up from 1321). Regression sentinels updated for new expanded-benchmark baselines.