Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,15 @@ Human review is required before:
* making external scientific claims;
* submitting a paper or press release.

## 9. Keep panel-specific evidence separate

Do not collapse external-predictor evidence across panels. The generic Gate 6
workflow remains pending for any future pilot panel until that panel has filled
results and a generated report. Wave 0.5 is separate: its completed external
screen used three submitted activity predictors with CAMPR4 excluded, plus
separate HemoFinder and AntiCP annotations. When docs disagree, treat
`docs/METRICS_CURRENT.md` as the current-state source of truth.

---

# Agent Task Priorities
Expand Down
7 changes: 6 additions & 1 deletion docs/DECISION_RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,14 @@
|-------|-------|
| Threshold | ≥3/5 external tools positive |
| Measure | Manual web submission results |
| Status | PENDING (see `outputs/external_predict_checklist.md`) |
| Status | PENDING for any new generic pilot panel until a filled `outputs/external_predict_results.csv` and generated report exist |
| Action | If FAIL: Expert reviewer override required |

**Scope note:** This generic Gate 6 is not the same as the completed Wave 0.5 external
screen. Wave 0.5 used three submitted activity predictors with CAMPR4 excluded and
separate HemoFinder/AntiCP safety annotations; see W0.5-3 and W0.5-4 below plus
`docs/METRICS_CURRENT.md`.

## Gate 7 — Human Expert Review

| Field | Value |
Expand Down
10 changes: 7 additions & 3 deletions docs/EXPERT_REVIEW_PACK.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,9 +269,13 @@ Before any candidate proceeds to synthesis or assay:
2. **Pre-registration lock**: PI signs `docs/ASSAY_PREREGISTRATION.md` before ordering synthesis.
3. **Broad novelty acknowledgement**: File `docs/NOVELTY_BROAD_CHECK.md` with the pre-registration.
SEED-003 and SEED-001 candidates are near-known AMPs; novelty claims must be appropriately limited.
4. **External predictor check** (prior to synthesis): Submit `outputs/pilot_panel.fasta` to CAMPR4,
AMPScanner v2, dbAMP, AntiCP2, Macrel web server. Fill checklist `outputs/external_predict_checklist.md`.
Require ≥3/5 agreement before synthesis order.
4. **External predictor check** (prior to synthesis): For a new generic pilot panel,
submit `outputs/pilot_panel.fasta` to CAMPR4, AMPScanner v2, dbAMP, AntiCP2, and the
Macrel web server, then fill `outputs/external_predict_results.csv`. Require ≥3/5
agreement before synthesis order. For the Wave 0.5 panel, use the completed
current-state summary in `docs/METRICS_CURRENT.md`: CAMPR4 was excluded, so activity
consensus is based on three submitted activity predictors plus separate HemoFinder
and AntiCP annotations.
5. **Serum stability first**: Triage all 20 candidates in 50% human serum (0/30/60/120 min) before
committing to full MIC panel. Pass criterion: >50% intact at 60 min (HPLC/MS primary readout).
6. **Synthesis order**: After serum stability triage, `make synthesis-order` generates vendor-ready CSV.
Expand Down
11 changes: 10 additions & 1 deletion docs/EXTERNAL_PREDICTOR_CONSENSUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,16 @@ classified as **CONFIDENT** and recommended for synthesis.

## Current Status

**PENDING** — Results require manual web submission for all 5 tools.
This document describes the **generic pilot-panel consensus workflow**. It remains
unresolved for any future panel until that panel has a filled
`outputs/external_predict_results.csv` and a generated consensus report.

Current Wave 0.5 status is different: the completed Wave 0.5 screen is summarized in
`docs/METRICS_CURRENT.md`. CAMPR4 was not submitted, so Wave 0.5 activity consensus is
based on three activity predictors (AMPScanner v2, AMPActiPred, Macrel web server) plus
separate HemoFinder and AntiCP 2.0 safety annotations. Do not treat this generic 5-tool
template as evidence that Wave 0.5 remains wholly pending.

Macrel v1.6.0 CLI has a known ONNX bug (all candidates classified NAMP — see PR #77);
use the web server at `big-data-biology.org/software/macrel` instead.

Expand Down
2 changes: 1 addition & 1 deletion docs/REVIEWER_SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ assay is mandatory for all candidates.
| No 3D structure or docking | Helical assumption may misclassify non-helical mechanisms |
| Near-seed generation only | Novel sequence space not explored by de novo generation |
| Melittin safety blind spot | Hemolysis must be assayed — confirmed quantitatively: safety scorer detection AUROC=0.3844 on 14 hemolytic vs 21 selective AMPs (v0.5.9 selectivity benchmark) |
| No external predictor ensemble yet | CAMPR4/AMPScanner/dbAMP/AntiCP2/Macrel web submission checklist exists at `outputs/external_predict_checklist.md` but results not yet obtained |
| External predictor evidence is panel-specific | Wave 0.5 external screen is complete for AMPScanner v2, AMPActiPred, Macrel web, HemoFinder, and AntiCP 2.0 with CAMPR4 excluded; a future generic pilot panel still needs its own filled `outputs/external_predict_results.csv` before Gate 6 can pass |
| Probability table uncalibrated | All "P(≥1 active) ≈ 92–97%" statements are upper bounds; honest estimate 55–80% for any hit, 35–60% for novel family |

---
Expand Down
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@ review (2026-06-28). Progress on these would materially raise breakthrough proba
| Gap | Why it matters | Effort estimate |
|-----|----------------|-----------------|
| ~~Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split)~~ | ~~Current AUROC 0.8420 measured on 43+44 demo set (n=87, CI₉₅: 0.76–0.91); may not generalise~~ | **Partial** (PR #110: expanded to 95 AMPs + 96 decoys, n=191, AUROC=0.7832, CI₉₅: 0.72–0.84; 52 new public-domain AMPs from 12 taxonomic classes; covers defensins, proline-rich, lantibiotics — a more honest estimate. Cluster-split benchmark added: cluster-aware CI 0.7061–0.8526, representative AUROC 0.7607. 500+ target still deferred to v1.0+) |
| External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Manual web-submission checklist at `outputs/external_predict_checklist.md`. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Medium (checklist generated; web submissions pending) |
| External predictor ensemble adapters (CAMPR4, AMPScanner, dbAMP, AntiCP2, Macrel) | Independent second opinions on activity; required for scientific credibility. Wave 0.5 has completed external evidence for AMPScanner v2, AMPActiPred, Macrel web, HemoFinder, and AntiCP 2.0 with CAMPR4 excluded; see `docs/METRICS_CURRENT.md`. The reusable generic 5-tool pilot-panel workflow still requires a filled `outputs/external_predict_results.csv` per future panel. Macrel v1.6.0 CLI ONNX bug documented PR #77 — all sequences (incl. canonical AMPs magainin-2, LL-37) misclassified as NAMP; use web server at big-data-biology.org/software/macrel. AMPlify omitted: GPU/ONNX env incompatible with current deps | Partial (Wave 0.5 complete; generic future-panel Gate 6 remains panel-specific) |
| ~~True novelty check against APD3, DRAMP v3.0, dbAMP~~ | ~~Current novelty scored against 45-sequence seed set only; may overestimate novelty~~ | **Done** (v0.5.7: BioPython BLOSUM62 local alignment vs 27,234 AMPs from APD6+DRAMP+UniProt; Wave 0.5 novelty corrected from 53/60 RELATED_NOVEL → 39 CLOSE_RELATIVE + 19 KNOWN_VARIANT + 1 RELATED_NOVEL) |
| ~~AUPRC alongside AUROC~~ | ~~Better metric for class-imbalanced AMP datasets~~ | **Done** (PR #58; updated PR #72) — pipeline AUPRC = 0.8627 |
| Wet-lab result integration (active-learning round 2) | Required to move from 15–30% to 50%+ credible probability | Requires wet-lab |
Expand Down
24 changes: 24 additions & 0 deletions tests/test_docs_consistent.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,3 +181,27 @@ def test_doc_wave05_historical_docs_do_not_pose_as_live_pending_state(self):
assert "Historical placeholder; external predictor review later completed" in prereg_text
assert "The external predictor portion has since been completed" in checklist_text
assert "External predictor review for Wave 0.5 was completed after this panel recommendation was first drafted." in wave1_text

def test_doc_external_predictor_gate_distinguishes_generic_from_wave05(self):
"""Live docs must not imply the completed Wave 0.5 screen is still wholly pending."""
consensus_text = (DOCS_DIR / "EXTERNAL_PREDICTOR_CONSENSUS.md").read_text(
encoding="utf-8"
)
decision_text = (DOCS_DIR / "DECISION_RULES.md").read_text(encoding="utf-8")
reviewer_text = (DOCS_DIR / "REVIEWER_SUMMARY.md").read_text(encoding="utf-8")
roadmap_text = (DOCS_DIR / "ROADMAP.md").read_text(encoding="utf-8")
expert_text = (DOCS_DIR / "EXPERT_REVIEW_PACK.md").read_text(encoding="utf-8")

assert "generic pilot-panel consensus workflow" in consensus_text
assert "Wave 0.5 activity consensus" in consensus_text
assert "three activity predictors" in consensus_text
assert "generic 5-tool" in consensus_text
assert "Wave 0.5 remains wholly pending" in consensus_text
assert "PENDING (see `outputs/external_predict_checklist.md`)" not in decision_text
assert "generic Gate 6" in decision_text
assert "completed Wave 0.5 external" in decision_text
assert "No external predictor ensemble yet" not in reviewer_text
assert "Wave 0.5 external screen is complete" in reviewer_text
assert "web submissions pending" not in roadmap_text
assert "Wave 0.5 complete; generic future-panel Gate 6 remains panel-specific" in roadmap_text
assert "CAMPR4 was excluded" in expert_text
Loading