diff --git a/AGENTS.md b/AGENTS.md index d6ded055..886218da 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -423,6 +423,15 @@ Human review is required before: * making external scientific claims; * submitting a paper or press release. +## 9. Keep panel-specific evidence separate + +Do not collapse external-predictor evidence across panels. The generic Gate 6 +workflow remains pending for any future pilot panel until that panel has filled +results and a generated report. Wave 0.5 is separate: its completed external +screen used three submitted activity predictors with CAMPR4 excluded, plus +separate HemoFinder and AntiCP annotations. When docs disagree, treat +`docs/METRICS_CURRENT.md` as the current-state source of truth. + --- # Agent Task Priorities diff --git a/docs/DECISION_RULES.md b/docs/DECISION_RULES.md index 15807c7a..953f8043 100644 --- a/docs/DECISION_RULES.md +++ b/docs/DECISION_RULES.md @@ -57,9 +57,14 @@ |-------|-------| | Threshold | ≥3/5 external tools positive | | Measure | Manual web submission results | -| Status | PENDING (see `outputs/external_predict_checklist.md`) | +| Status | PENDING for any new generic pilot panel until a filled `outputs/external_predict_results.csv` and generated report exist | | Action | If FAIL: Expert reviewer override required | +**Scope note:** This generic Gate 6 is not the same as the completed Wave 0.5 external +screen. Wave 0.5 used three submitted activity predictors with CAMPR4 excluded and +separate HemoFinder/AntiCP safety annotations; see W0.5-3 and W0.5-4 below plus +`docs/METRICS_CURRENT.md`. + ## Gate 7 — Human Expert Review | Field | Value | diff --git a/docs/EXPERT_REVIEW_PACK.md b/docs/EXPERT_REVIEW_PACK.md index e08d7f91..d997fe2c 100644 --- a/docs/EXPERT_REVIEW_PACK.md +++ b/docs/EXPERT_REVIEW_PACK.md @@ -269,9 +269,13 @@ Before any candidate proceeds to synthesis or assay: 2. **Pre-registration lock**: PI signs `docs/ASSAY_PREREGISTRATION.md` before ordering synthesis. 3. **Broad novelty acknowledgement**: File `docs/NOVELTY_BROAD_CHECK.md` with the pre-registration. SEED-003 and SEED-001 candidates are near-known AMPs; novelty claims must be appropriately limited. -4. **External predictor check** (prior to synthesis): Submit `outputs/pilot_panel.fasta` to CAMPR4, - AMPScanner v2, dbAMP, AntiCP2, Macrel web server. Fill checklist `outputs/external_predict_checklist.md`. - Require ≥3/5 agreement before synthesis order. +4. **External predictor check** (prior to synthesis): For a new generic pilot panel, + submit `outputs/pilot_panel.fasta` to CAMPR4, AMPScanner v2, dbAMP, AntiCP2, and the + Macrel web server, then fill `outputs/external_predict_results.csv`. Require ≥3/5 + agreement before synthesis order. For the Wave 0.5 panel, use the completed + current-state summary in `docs/METRICS_CURRENT.md`: CAMPR4 was excluded, so activity + consensus is based on three submitted activity predictors plus separate HemoFinder + and AntiCP annotations. 5. **Serum stability first**: Triage all 20 candidates in 50% human serum (0/30/60/120 min) before committing to full MIC panel. Pass criterion: >50% intact at 60 min (HPLC/MS primary readout). 6. **Synthesis order**: After serum stability triage, `make synthesis-order` generates vendor-ready CSV. diff --git a/docs/EXTERNAL_PREDICTOR_CONSENSUS.md b/docs/EXTERNAL_PREDICTOR_CONSENSUS.md index d1bd900a..e6af97ad 100644 --- a/docs/EXTERNAL_PREDICTOR_CONSENSUS.md +++ b/docs/EXTERNAL_PREDICTOR_CONSENSUS.md @@ -26,7 +26,16 @@ classified as **CONFIDENT** and recommended for synthesis. ## Current Status -**PENDING** — Results require manual web submission for all 5 tools. +This document describes the **generic pilot-panel consensus workflow**. It remains +unresolved for any future panel until that panel has a filled +`outputs/external_predict_results.csv` and a generated consensus report. + +Current Wave 0.5 status is different: the completed Wave 0.5 screen is summarized in +`docs/METRICS_CURRENT.md`. CAMPR4 was not submitted, so Wave 0.5 activity consensus is +based on three activity predictors (AMPScanner v2, AMPActiPred, Macrel web server) plus +separate HemoFinder and AntiCP 2.0 safety annotations. Do not treat this generic 5-tool +template as evidence that Wave 0.5 remains wholly pending. + Macrel v1.6.0 CLI has a known ONNX bug (all candidates classified NAMP — see PR #77); use the web server at `big-data-biology.org/software/macrel` instead. diff --git a/docs/REVIEWER_SUMMARY.md b/docs/REVIEWER_SUMMARY.md index 2f277912..64df997a 100644 --- a/docs/REVIEWER_SUMMARY.md +++ b/docs/REVIEWER_SUMMARY.md @@ -78,7 +78,7 @@ assay is mandatory for all candidates. | No 3D structure or docking | Helical assumption may misclassify non-helical mechanisms | | Near-seed generation only | Novel sequence space not explored by de novo generation | | Melittin safety blind spot | Hemolysis must be assayed — confirmed quantitatively: safety scorer detection AUROC=0.3844 on 14 hemolytic vs 21 selective AMPs (v0.5.9 selectivity benchmark) | -| No external predictor ensemble yet | CAMPR4/AMPScanner/dbAMP/AntiCP2/Macrel web submission checklist exists at `outputs/external_predict_checklist.md` but results not yet obtained | +| External predictor evidence is panel-specific | Wave 0.5 external screen is complete for AMPScanner v2, AMPActiPred, Macrel web, HemoFinder, and AntiCP 2.0 with CAMPR4 excluded; a future generic pilot panel still needs its own filled `outputs/external_predict_results.csv` before Gate 6 can pass | | Probability table uncalibrated | All "P(≥1 active) ≈ 92–97%" statements are upper bounds; honest estimate 55–80% for any hit, 35–60% for novel family | --- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b0f49acd..8b191f34 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -266,7 +266,7 @@ review (2026-06-28). Progress on these would materially raise breakthrough proba | Gap | Why it matters | Effort estimate | |-----|----------------|-----------------| | ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. Cluster-split benchmark added: cluster-aware CI 0.7061–0.8526, representative AUROC 0.7607. 500+ target still deferred to v1.0+) | -| External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Manual web-submission checklist at `outputs/external_predict_checklist.md`. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Medium (checklist generated; web submissions pending) | +| External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Wave 0.5 has completed external evidence for AMPScanner v2, AMPActiPred, Macrel web, HemoFinder, and AntiCP 2.0 with CAMPR4 excluded; see `docs/METRICS_CURRENT.md`. The reusable generic 5-tool pilot-panel workflow still requires a filled `outputs/external_predict_results.csv` per future panel. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Partial (Wave 0.5 complete; generic future-panel Gate 6 remains panel-specific) | | ~~True novelty check against APD3, DRAMP v3.0, dbAMP~~ | ~~Current novelty scored against 45-sequence seed set only; may overestimate novelty~~ | **Done** (v0.5.7: BioPython BLOSUM62 local alignment vs 27,234 AMPs from APD6+DRAMP+UniProt; Wave 0.5 novelty corrected from 53/60 RELATED_NOVEL → 39 CLOSE_RELATIVE + 19 KNOWN_VARIANT + 1 RELATED_NOVEL) | | ~~AUPRC alongside AUROC~~ | ~~Better metric for class-imbalanced AMP datasets~~ | **Done** (PR #58; updated PR #72) — pipeline AUPRC = 0.8627 | | Wet-lab result integration (active-learning round 2) | Required to move from 15–30% to 50%+ credible probability | Requires wet-lab | diff --git a/tests/test_docs_consistent.py b/tests/test_docs_consistent.py index 09e188e7..abef9b06 100644 --- a/tests/test_docs_consistent.py +++ b/tests/test_docs_consistent.py @@ -181,3 +181,27 @@ def test_doc_wave05_historical_docs_do_not_pose_as_live_pending_state(self): assert "Historical placeholder; external predictor review later completed" in prereg_text assert "The external predictor portion has since been completed" in checklist_text assert "External predictor review for Wave 0.5 was completed after this panel recommendation was first drafted." in wave1_text + + def test_doc_external_predictor_gate_distinguishes_generic_from_wave05(self): + """Live docs must not imply the completed Wave 0.5 screen is still wholly pending.""" + consensus_text = (DOCS_DIR / "EXTERNAL_PREDICTOR_CONSENSUS.md").read_text( + encoding="utf-8" + ) + decision_text = (DOCS_DIR / "DECISION_RULES.md").read_text(encoding="utf-8") + reviewer_text = (DOCS_DIR / "REVIEWER_SUMMARY.md").read_text(encoding="utf-8") + roadmap_text = (DOCS_DIR / "ROADMAP.md").read_text(encoding="utf-8") + expert_text = (DOCS_DIR / "EXPERT_REVIEW_PACK.md").read_text(encoding="utf-8") + + assert "generic pilot-panel consensus workflow" in consensus_text + assert "Wave 0.5 activity consensus" in consensus_text + assert "three activity predictors" in consensus_text + assert "generic 5-tool" in consensus_text + assert "Wave 0.5 remains wholly pending" in consensus_text + assert "PENDING (see `outputs/external_predict_checklist.md`)" not in decision_text + assert "generic Gate 6" in decision_text + assert "completed Wave 0.5 external" in decision_text + assert "No external predictor ensemble yet" not in reviewer_text + assert "Wave 0.5 external screen is complete" in reviewer_text + assert "web submissions pending" not in roadmap_text + assert "Wave 0.5 complete; generic future-panel Gate 6 remains panel-specific" in roadmap_text + assert "CAMPR4 was excluded" in expert_text