cschanhniem · Jul 6, 2026 · Jul 6, 2026
diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md
@@ -98,7 +98,7 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte
 
 | Loop | Bottleneck | Deliverable | Verification |
 |------|-----------|-------------|-------------|
-| 30 | No explicit virtual-assay scope document that says what IS and IS NOT being modeled | `docs/VIRTUAL_ASSAY_SCOPE.md` updated with: modeled effects (membrane binding, insertion depth), NOT-modeled effects (full-cell response, immune signaling, metabolism), uncertainty policy | Document is reviewable by a computational biophysicist |
+| 30 ✅ | No explicit virtual-assay scope document; stale BENCHMARKING.md; stale test count in METRICS_CURRENT.md; broken script ref in NOVELTY_AUDIT_GUIDE.md | `docs/VIRTUAL_ASSAY_SCOPE.md` — scope doc with modeled/NOT-modeled effects, uncertainty policy, ablation requirement, integration modes. Rewrote `docs/BENCHMARKING.md` with all 19 benchmarks. Fixed test count, script ref, scripts/__init__.py. | Scope doc reviewable; BENCHMARKING.md matches Makefile; 1843 tests |
 | 31 | No membrane interaction proxy. The simulation module exists but has only a dummy placeholder | `simulation/membrane.py`: implements coarse-grained membrane binding energy proxy using hydrophobicity scale + helical hydrophobic moment. Returns `SimulationResult` with uncertainty | Benchmark: separates melittin (hemolytic) from magainin (selective) better than random (AUROC > 0.6 on strict set) |
 | 32 | No structure ensemble proxy. Current pipeline assumes all AMPs are helical | `simulation/structure.py`: secondary structure prediction (helix/coil/sheet propensity from sequence). Generates 3-state ensemble weights. Flags non-helical candidates | For known beta-sheet AMPs (protegrin, tachyplesin), reports low helix confidence |
 | 33 | No selectivity ratio module that combines bacterial and mammalian membrane scores | `simulation/selectivity_ratio.py`: ratio of bacterial membrane binding to RBC membrane binding proxy. Reports as `selectivity_ratio` with uncertainty | Distinguishes known selective (magainin) from known hemolytic (melittin) at ratio > 2x |
@@ -180,7 +180,7 @@ If no data arrives, virtual assay scaffolding continues independently.
 Phase 0: ✅ Complete (Loops 1–8)
 Phase 1: ✅ Complete (Loops 9–17)
 Phase 2: ✅ Complete (Loops 18–29)
-Phase 3: Not started (Loops 30–39)
+Phase 3: In progress — Loop 30 ✅ (Loops 30–39)
 Phase 4: Not started (Loops 40–49)
 ```
 
@@ -241,7 +241,7 @@ Phase 4: Not started (Loops 40–49)
 | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available |
 | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers |
 
-**Next loop:** Loop 30 — Phase 3 (virtual assay scope document).
+**Next loop:** Loop 31 — Phase 3 (membrane interaction proxy).
 
 **Phase 2 exit criteria (all 5 met ✅):**
 - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest

diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md
@@ -11,40 +11,83 @@ Common artifacts:
 - sequence length differences;
 - charge differences;
 - duplicate or near-duplicate leakage;
-- test peptides too similar to training peptides.
-
-## Required benchmarks
-
-| Benchmark | Purpose | Status |
-|---|---|---|
-| Random baseline | Prove the pipeline is not noise | Implemented |
-| Simple physicochemical baseline | Prove models beat charge/hydrophobicity alone | Implemented |
-| Cluster split | Prevent near-duplicate leakage | Implemented (`bench cluster-split`) |
-| Cluster-aware CI | Honest bootstrap when positives contain near-duplicate families | Implemented (cluster-aware bootstrap in `bench cluster-split`) |
-| Time split | Test future generalization where metadata allows | Deferred (metadata not available) |
-| Hidden-positive recovery | Check whether known actives are ranked high | Implemented |
-| Toxicity down-ranking | Ensure predicted risky candidates are penalized | Implemented |
-| Novelty stress test | Avoid near-clones of references | Implemented |
-| Within-AMP selectivity | Test whether scorers distinguish hemolytic from selective AMPs | Implemented (`bench selectivity`) |
-| Expert ablation | Test whether expert composite adds value over ensemble | Implemented (`bench expert-ablation`) |
-| Multi-class triage | Test selective > hemolytic > decoy ranking in one panel | Implemented (`bench triage`) |
+- test peptides too similar to training peptides;
+- amphipathic-helix bias inflating discrimination on canonical AMPs;
+- small-n benchmark confidence intervals masking uncertainty.
+
+## Benchmarks
+
+Each benchmark has a `make` target, a script in `scripts/`, and a
+documented result in `docs/METRICS_CURRENT.md` and
+`outputs/metrics_snapshot.json`.
+
+### Core discrimination
+
+| Benchmark | Purpose | Target | Verification |
+|-----------|---------|--------|-------------|
+| `bench-baseline` | Random baseline — pipeline is not noise | AUROC > 0.5 | `make bench-baseline` |
+| `bench-hidden-active` | Known active AMPs hidden from training appear in top ranks | Better-than-random recovery | `make bench-hidden-active` |
+| `validate-scoring` | Standard benchmark (n=191, 95 AMPs + 96 decoys) | AUROC > 0.70 | `make validate-scoring` |
+| `bench-500` | Expanded benchmark (500 AMPs + 500 decoys, n=1000) | AUROC > 0.70, ~2.3× tighter CI | `make bench-500` |
+| `bench-cluster-split` | Cluster-aware bootstrap on n=191 — near-duplicate de-inflation | Cluster-aware CI lo > 0.65 | `make bench-cluster-split` |
+| `bench-cluster-split-500` | Cluster-split on expanded 500-AMP set | Cluster-aware CI lo > 0.65 | `make bench-cluster-split-500` |
+
+### Honesty benchmarks
+
+| Benchmark | Purpose | Target | Verification |
+|-----------|---------|--------|-------------|
+| `bench-easy-baseline` | Compare pipeline against single-feature trivial predictors (charge, length, charge density) | Document delta; pipeline expected to trade raw discrimination for safety | `make bench-easy-baseline` |
+| `bench-charge-matched` | Greedy charge-density matching — test signal beyond cationic prior | Informational; documents whether pipeline retains signal after charge matching | `make bench-charge-matched` |
+| `bench-order-dependent` | Test which features survive sequence scrambling (composition-control) | 7/31 features are order-dependent; dipeptide_order_score AUROC 0.786 | `make bench-order-dependent` |
+| `bench-precision-at-k` | Operating characteristic for candidate selection — small-k triage precision/recall | top-20 precision 1.000, best F1 threshold documented | `make bench-precision-at-k` |
+| `bench-multi-negatives` | Test robustness across 4 decoy distributions (random, reverse, shuffled proteome, inactive variants) | Pipeline rank order stable across decoy sources | `make bench-multi-negatives` |
+| `bench-cross-dataset` | Generalization to DRAMP-only AMPs — test source-independence of heuristic features | AUROC within 0.03 of APD6/UniProt baseline | `make bench-cross-dataset` |
+
+### Selectivity / safety
+
+| Benchmark | Purpose | Target | Verification |
+|-----------|---------|--------|-------------|
+| `bench-selectivity` | Within-AMP hemolysis detection — can scorers distinguish hemolytic from selective AMPs? | rich_selectivity AUROC > 0.65 (CI lo > 0.5) | `make bench-selectivity` |
+| `bench-per-family` | Stratify 500 AMPs by 6 heuristic structural classes — reveal charge-dominated pipeline bias | Document per-class AUROC; compensate via diversity selection | `make bench-per-family` |
+| `bench-feature-decomp` | Per-feature selective-vs-hemolytic AUROC — which individual features have signal? | 8/30 features significant; 6/8 unused by current selectivity model | `make bench-feature-decomp` |
+
+### Composite / ablation
+
+| Benchmark | Purpose | Target | Verification |
+|-----------|---------|--------|-------------|
+| `bench-triage` | Multi-class: selective > hemolytic > decoy ranking in a single panel | gate_triage passes all 3 pairwise AUROCs; strict triage (composition-matched) fails honestly | `make bench-triage` |
+| `bench-expert-ablation` | Ablate expert composite components on n=191 | Document per-component AUROC; expert lower than ensemble on AMP-vs-decoy (by design) | `make bench-expert-ablation` |
+| `bench-expert-ablation-500` | Ablate expert composite on expanded n=1000 | Re-run on diverse set; 2 components reclassified from n=191 | `make bench-expert-ablation-500` |
+
+### Regression / CI
+
+| Benchmark | Purpose | Target | Verification |
+|-----------|---------|--------|-------------|
+| `bench-gate` | Multi-metric regression gate: cluster-split, selectivity, triage — enforced in CI | All metrics within tolerance; fails CI if AUROC drops | `make bench-gate` |
+| `bench-leakage` | Train/test leakage detection | Zero sequence overlap between train and test sets | `make bench-leakage` |
+
+## Deferred benchmarks
+
+| Benchmark | Purpose | Reason deferred |
+|-----------|---------|----------------|
+| Time split | Test future generalization | Metadata not available for most sequences |
+| Virtual-assay ablation | Test whether simulation modules improve triage | Phase 3 — no modules exist yet |
+| Biologically plausible charge-balanced negative set | Honest non-charge signal test | Requires curated negative sequences — see v0.5.41 recommendation |
 
 ## Minimum report fields
 
-Every benchmark report should include:
+Every benchmark result stored in `outputs/metrics_snapshot.json` includes:
 
-- dataset source;
-- dataset license;
-- number of positives;
-- number of negatives;
-- deduplication method;
-- split method;
-- similarity threshold;
+- dataset source and license;
+- number of positives and negatives;
+- split method and similarity threshold;
 - baseline performance;
-- model performance;
-- confidence intervals where possible;
+- model performance with confidence intervals;
 - known limitations.
 
 ## Anti-cheating rule
 
-Do not update feature weights, filters, or thresholds after inspecting held-out performance unless you create a new held-out set.
+Do not update feature weights, filters, or thresholds after inspecting
+held-out performance unless you create a new held-out set. Pre-registered
+analysis plans are recommended for any benchmark that gates a release
+decision.
diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md
@@ -1101,7 +1101,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 1832+ |
+| Total tests | 1843 |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 

diff --git a/docs/NOVELTY_AUDIT_GUIDE.md b/docs/NOVELTY_AUDIT_GUIDE.md
@@ -72,8 +72,8 @@ curl -sL "${BASE}/general_amps.fasta" -o data/novelty_db/dramp_general.fasta
 curl -sL "${BASE}/patent_amps.fasta"  -o data/novelty_db/dramp_patent.fasta
 curl -sL "${BASE}/specific_amps.fasta" -o data/novelty_db/dramp_specific.fasta
 
-# Re-download UniProt (run annually) — use the Python paginator in scripts/
-python3 scripts/download_uniprot_amps.py
+# Re-download UniProt (run annually) — use the download-novelty-dbs paginator
+python3 scripts/download_novelty_dbs.py --uniprot
 
 # Re-download ESCAPE (run when Harvard Dataverse dataset is updated)
 python3 scripts/download_novelty_dbs.py --escape

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -1,5 +1,32 @@
 # Roadmap
 
+## v0.5.51 — Virtual Assay Scope + Doc Drift Fix (Loop 30) ✓ (2026-07-06)
+
+Phase 3 begins with a scope document defining what the virtual assay layer
+IS and IS NOT, with explicit uncertainty policy, ablation requirements, and
+integration modes. Accumulated doc drift fixed alongside: BENCHMARKING.md
+was missing ~11 of 19 benchmarks; METRICS_CURRENT.md test count was stale
+(1832+ → 1843); NOVELTY_AUDIT_GUIDE.md referenced a nonexistent script.
+
+Changes:
+- ``docs/VIRTUAL_ASSAY_SCOPE.md`` — Phase 3 scope document covering 4 planned
+  modules (membrane binding, structure ensemble, selectivity ratio, stability),
+  12 explicitly NOT-modeled effects, uncertainty policy (modules with
+  uncertainty > 0.5 cannot affect selection), ablation requirement (each
+  module must beat cheap heuristic baseline), 3 integration modes
+  (off/info/weighted), calibration gate for weighted mode, and 5 exit criteria.
+- ``docs/BENCHMARKING.md`` — Complete rewrite: all 19 benchmarks organized
+  into 6 categories (core discrimination, honesty, selectivity/safety,
+  composite/ablation, regression/CI, deferred). Each benchmark has make target,
+  purpose, target metric, and verification command.
+- ``docs/METRICS_CURRENT.md`` — Test count corrected: 1832+ → 1843.
+- ``docs/NOVELTY_AUDIT_GUIDE.md`` — Fixed stale script reference:
+  ``scripts/download_uniprot_amps.py`` → ``scripts/download_novelty_dbs.py --uniprot``.
+- ``docs/50_LOOP_PLAN.md`` — Loop 30 marked ✅. Phase 3 status updated to
+  "In progress — Loop 30 ✅".
+- ``scripts/__init__.py`` — Populated with module docstring.
+- 1843 tests passing. No code changes to pipeline.
+
 ## v0.5.50 — Negative-Result Archive Template + Phase 2 Closeout ✓ (2026-07-06)
 
 Phase 2 exit criteria required a public negative-result archive format but none