Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/50_LOOP_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Prepare for real wet-lab partners. Make the pack reviewable, the evidence tracea

| Loop | Bottleneck | Deliverable | Verification |
|------|-----------|-------------|-------------|
| 40 | No assay-partner onboarding pack. A new CRO gets no README for how to run the panel | `docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences | A new CRO can read this and know exactly what to do |
| 40 ✅ | No assay-partner onboarding pack. A new CRO gets no README for how to run the panel | `docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences, safety notes, timeline, controls. v0.5.61 | A new CRO can read this and know exactly what to do |
| 41 | No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable | `configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates for Wave 1 (MIC ≤ 32 for ≥ N candidates, hemolysis ≤ 10% for ≥ M candidates, at least 1 control passes) | CI validates criteria against lab result schema |
| 42 | No expert-review GitHub template. Review process is ad-hoc | `.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields |
| 43 | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples |
Expand Down Expand Up @@ -180,8 +180,8 @@ If no data arrives, virtual assay scaffolding continues independently.
Phase 0: ✅ Complete (Loops 1–8)
Phase 1: ✅ Complete (Loops 9–17)
Phase 2: ✅ Complete (Loops 18–29)
Phase 3: In progress — Loop 38 ✅ (Loops 30–39)
Phase 4: Not started (Loops 40–49)
Phase 3: ✅ Complete (Loops 30–39)
Phase 4: In progress — Loop 40 ✅ (Loops 40–49)
```

### Completed
Expand Down Expand Up @@ -241,7 +241,7 @@ Phase 4: Not started (Loops 40–49)
| 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available |
| 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers |

**Next loop:** Loop 40 — lab partner onboarding pack.
**Next loop:** Loop 41 — pre-registered pass/fail criteria (configs/wave1_pass_fail.yaml).

**Phase 2 exit criteria (all 5 met ✅):**
- ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest
Expand Down
211 changes: 211 additions & 0 deletions docs/LAB_PARTNER_ONBOARDING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# Lab Partner Onboarding Guide

## Purpose

This document tells a qualified CRO or academic lab partner everything they
need to run the nominated antimicrobial peptide panel. It covers the candidate
list, synthesis instructions, assay protocols, data return format, and controls.

**This is a dry-lab nomination.** The pipeline scores are computational
predictions, not biological proof. No antimicrobial activity has been
demonstrated in vitro or in vivo.

---

## Candidate Panel

The nominated panel is **Wave 0.5**: 24 candidates across 15 scaffold families.
See `outputs/wave1_final_panel.csv` for the latest list.

### Panel composition

| Role | Count | Description |
|------|:-----:|-------------|
| BALANCED_LEAD | 15 | Predicted active + safe + novel |
| HIGH_UPSIDE_RISKY | 4 | Higher predicted activity, higher safety risk |
| POSITIVE_CONTROL | 1 | Known active AMP (magainin-1 derivative, SEED-001_VAR_064) |
| SAR_CONTROL | 4 | Known-variant sequences for SAR comparison |

### Families represented

| Family | Mechanism | Key risk |
|--------|-----------|----------|
| SEED-006 | Mastoparan-X, wasp-venom helix insertion | Mast-cell degranulation |
| SEED-007 | Bombolitin-II, bumblebee venom | Met oxidation at pos 6 |
| SEED-008 | Puroindoline-a, Trp-rich interfacial | DKP risk (F-Pro), HemoFinder HIGH |
| SEED-009 | Bac2A, proline-rich intracellular | AntiCP risk, RPMI-1640 arm recommended |
| SEED-010–019 | Wave 0.5 diversified scaffolds | See evidence certificates |

---

## Synthesis Instructions

### Peptide supplier

Any qualified SPPS provider (e.g., GenScript, LifeTein, BioMatik).

### Format

- Lyophilized powder, > 95% purity by HPLC.
- 1–2 mg per peptide (sufficient for MIC + hemolysis screening).
- Trifluoroacetate (TFA) counterion is acceptable.
- Store at −20°C, desiccated, upon receipt.

### Modifications

| Modification | Recommendation |
|-------------|---------------|
| C-terminal amidation | Recommended for all candidates (mimics natural processing) |
| N-terminal acetylation | Recommended (prevents pyroglutamate formation if N-term Q) |
| Met-containing peptides | Request norleucine (Nle) substitution to prevent oxidation |
| Cys-containing peptides | Require linear synthesis (no disulfide formation during synthesis) |

### QC requirements

- HPLC trace and MS confirmation required for each peptide.
- Purity ≥ 90% minimum; ≥ 95% preferred.
- Report observed mass vs expected mass (Da).

---

## Assay Protocols

### Primary screen: MIC (minimum inhibitory concentration)

| Parameter | Value |
|-----------|-------|
| Strains | *S. aureus* ATCC 29213, *E. coli* ATCC 25922 (minimum) |
| Method | Broth microdilution (CLSI M07) |
| Medium | Mueller-Hinton broth (MHB) |
| Concentration range | 0.5–256 µg/mL (2-fold serial dilution) |
| Incubation | 18–24 h at 37°C |
| Readout | MIC = lowest concentration with no visible growth |

Reference: CLSI M07, CLSI M100.

### Secondary screen: Hemolysis (RBC)

| Parameter | Value |
|-----------|-------|
| RBC source | Human erythrocytes (fresh, ≤ 24 h old) |
| Method | Hemoglobin release at 540 nm |
| Concentration range | Same as MIC range |
| Positive control | 1% Triton X-100 (100% hemolysis) |
| Negative control | PBS buffer (0% hemolysis) |
| Readout | HC50 = concentration causing 50% hemolysis |

Reference: Standard RBC hemolysis assay (Evans et al. 2013).

### Optional: Serum stability

| Parameter | Value |
|-----------|-------|
| Serum | 25% human serum in PBS |
| Time points | 0, 30, 60, 120, 240 min |
| Readout | RP-HPLC peak area vs t=0 |

Reference: Nguyen et al. (2010) J Biol Chem.

---

## Data Return Format

Return results as **JSON files** matching `schemas/lab_result.schema.json`.
One file per candidate per assay type.

### Required fields per file

```json
{
"candidate_id": "SEED-007_VAR_001",
"assay_type": "MIC",
"organism": "Staphylococcus aureus ATCC 29213",
"result_value": 8.0,
"result_unit": "ug/mL",
"positive_control_value": 2.0,
"positive_control_unit": "ug/mL",
"negative_control_value": null,
"incubation_hours": 20,
"temperature_celsius": 37,
"medium": "MHB",
"lab": "Partner Lab Name",
"assay_date": "2026-08-15",
"notes": "",
"disclaimer": "SYNTHETIC — replace with real data"
}
```

### Files

- One JSON per candidate per assay → `SEED-007_VAR_001_MIC.json`
- Place all files in a single directory.
- See `schemas/lab_result.schema.json` for the full schema specification.

---

## Controls

### Positive control

| ID | Sequence | Expected MIC | Purpose |
|----|----------|:-----------:|---------|
| SEED-001_VAR_064 | (magainin-1 derivative) | 4–32 µg/mL | Validate assay is working |

### Negative control (if applicable)

A scrambled / inactive variant can be provided upon request to validate
specificity.

### Assay acceptance criteria

The batch is valid only if:

- Positive control MIC falls within 2-fold of expected value.
- No contamination in negative controls (sterility check).
- Hemolysis positive control (Triton X-100) produces ≥ 90% hemolysis.
- Hemolysis negative control (PBS) produces ≤ 5% hemolysis.

---

## Safety and Biosecurity

All candidates are short antimicrobial peptides (10–30 AA). They are not:

- Known toxins or virulence factors.
- Genetically modified organisms.
- Select agents (CDC/HHS list).

Standard BSL-2 practices are sufficient for MIC and hemolysis assays.
No special biocontainment is required.

---

## Timeline

| Milestone | Typical duration |
|-----------|:----------------:|
| Synthesis and QC | 2–4 weeks |
| MIC screening (2 strains × 24 candidates) | 1–2 weeks |
| Hemolysis assay (24 candidates) | 1 week |
| Data return and analysis | 1 week |

Total estimated: **5–8 weeks** from synthesis order to data return.

---

## Contact

For questions about candidate selection, protocol modifications, or data return:
open an issue at `github.com/Open-Problem-Lab/OpenAMP-Foundry` with the
`lab-partner` label.

---

## Related Documents

- `docs/WET_LAB_HANDOFF.md` — Detailed wet-lab procedure and score interpretation.
- `docs/ASSAY_PREREGISTRATION.md` — Pre-registered assay parameters.
- `docs/EXPERT_REVIEW_PACK.md` — Expert review documentation.
- `schemas/lab_result.schema.json` — Machine-readable data return schema.
- `outputs/wave1_final_panel.csv` — Current candidate list.
- `outputs/evidence_wave0_5/` — Evidence certificates for each candidate.
3 changes: 2 additions & 1 deletion docs/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-06 (simulation benchmark report — v0.5.60)
> **Last updated:** 2026-07-06 (lab partner onboarding — v0.5.61)
> **New in v0.5.61:** `docs/LAB_PARTNER_ONBOARDING.md` — CRO onboarding pack with panel summary, synthesis instructions, assay protocols, data return format, controls, safety, timeline. No code changes. 1966 tests.
> **New in v0.5.60:** `docs/SIMULATION_BENCHMARK.md` consolidates simulation ablation, cheap-baseline comparison, and weighted-mode gate results. Current conclusion: simulation does not improve ranking; `weighted` remains blocked.
> **New in v0.5.59:** `ExternalSimulationAdapter` protocol — wraps third-party callables into `VirtualAssayProxy`. Availability check, graceful error handling, metadata override. ARCHITECTURE.md docs updated. 12 tests. 1965 total.
> **New in v0.5.58:** Per-signal cheap-baseline comparison: 0/4 simulation signals beat their cheapest heuristic. All simulation modules remain permanently experimental. `make bench-simulation-baselines`. 13 tests. 1953 total.
Expand Down
16 changes: 16 additions & 0 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# Roadmap

## v0.5.61 — Lab Partner Onboarding Pack (Loop 40) ✓ (2026-07-06)

Phase 4 begins with a comprehensive onboarding document for CRO or academic
lab partners. Covers everything needed to run the Wave 0.5 panel.

Changes:
- ``docs/LAB_PARTNER_ONBOARDING.md`` — Full onboarding pack: candidate panel
summary (24 candidates, 15 families), synthesis instructions (SPPS, purity,
modifications), assay protocols (MIC, hemolysis, serum stability), data
return JSON format with required fields, positive/negative controls, safety
and biosecurity (BSL-2), timeline (5–8 weeks), and related document links.
- ``docs/50_LOOP_PLAN.md`` — Loop 40 ✅. Phase 4 status: "In progress — Loop 40 ✅".
Next: Loop 41.
- ``docs/METRICS_CURRENT.md`` — Changelog entry updated.
- No code changes. 1966 tests still pass.

## v0.5.60 — Simulation Benchmark Report (Loop 39) ✓ (2026-07-06)

Closed the benchmark-report gap with `docs/SIMULATION_BENCHMARK.md`.
Expand Down
89 changes: 79 additions & 10 deletions docs/SIMULATION_BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,50 @@
# Simulation Benchmark Report

## Status
## Phase 3 Closeout

Phase 3 (virtual assay scaffolding, Loops 30–39) built two simulation modules,
three benchmarks, a weighted-mode gate, a CLI flag, and an external adapter
protocol. Simulation does not improve ranking. The honest finding across all
benchmarks: **simulation does not improve ranking**.

Current verdict: simulation remains informational only.
Weighted mode is blocked by the gate and will remain blocked until a
future module demonstrably beats its cheapest heuristic baseline.

`weighted` simulation remains blocked.

---

## What Was Built

| Loop | Module | Status |
|:----:|--------|:------:|
| 30 | Virtual assay scope document | ✅ Scope defined |
| 31 | MembraneProxy (Wimley-White scales) | ✅ Ships in info mode |
| 32 | StructureProxy (Chou-Fasman 3-state) | ✅ Ships in info mode |
| 33 | AMP-vs-decoy ablation benchmark | ✅ No improvement |
| 34 | Within-AMP hemolysis ablation | ✅ No improvement |
| 35 | Weighted-mode simulation gate | ✅ Blocks weighted mode |
| 36 | Cheap-baseline comparison benchmark | ✅ 0/4 signals beat baselines |
| 37 | `rank --simulation-mode` CLI flag | ✅ Users can inspect scores |
| 38 | `ExternalSimulationAdapter` protocol | ✅ Third-party integration ready |
| 39 | This report | ✅ Phase 3 closed |

The shipped membrane and structure proxies do not improve candidate ranking
against the repo's current benchmark bar. They may still help humans inspect a
candidate, but they must not affect ranking weights.
---

Simulation does not improve ranking on the current benchmark set.
## Phase 3 Exit Criteria

| Criterion | Required | Status | Evidence |
|-----------|----------|:------:|----------|
| ≥2 simulation modules exist and are benchmarked | ✅ | ✅ Met | MembraneProxy (Loop 31), StructureProxy (Loop 32). Both benchmarked (Loops 33–36) |
| Simulation improves strict triage by >0.03 AUROC on any pairwise metric | Delta > 0.03 | ❌ Not met | Best simulation helix_weight AUROC 0.6458; best existing rich_selectivity AUROC 0.7453 (delta −0.0995) |
| Modules that fail ablation are removed or permanently experimental | All failed | ✅ Met | All modules flagged permanent experimental (Loop 36 verdict) |
| Uncertainty propagated through to evidence certificate | Via `SimulationResult` | ✅ Met | Each module returns `uncertainty` in range [0, 1]; information added to scores dict in info mode |
| External adapter protocol documented | ARCHITECTURE.md | ✅ Met | `ExternalSimulationAdapter` documented in ARCHITECTURE.md extension points (Loop 38) |

**Verdict: Phase 3 exit criteria partially met. Simulation does NOT earn ranking impact, but the infrastructure and honesty guardrails are complete.**

---

## Reproducible Commands

Expand All @@ -31,6 +67,8 @@ PYTHONPATH=src python3 -m openamp_foundry.cli bench simulation-gate \
Exit code `3` is expected for benchmarks or gates that correctly reject
weighted simulation.

---

## Current Results

| Benchmark | Baseline / incumbent | Simulation result | Delta | Verdict |
Expand All @@ -42,6 +80,8 @@ weighted simulation.
| Cheap baseline: helix weight | Helix propensity AUROC `0.6489` | Helix weight AUROC `0.6458` | `-0.0031` | No improvement |
| Cheap baseline: non-helical flag | Proline fraction AUROC `0.4929` | Non-helical flag AUROC `0.4124` | `-0.0805` | No improvement |

---

## Gate Decision

`openamp-foundry bench simulation-gate` blocks `weighted` mode.
Expand All @@ -53,14 +93,29 @@ Reasons:
- Cheap-baseline comparison shows zero of four simulation signals beat their
simplest meaningful heuristic.

Weighted mode will remain blocked until a future module passes all three checks.

---

## Interpretation

Known fact from current benchmark artifacts: the current simulation layer is
not ready to influence ranking.
The current simulation layer (1D Chou-Fasman propensities, averaged Wimley-White
scales) is not rich enough to beat existing heuristics. Two reasons dominate:

1. **Composition dominance**: The existing pipeline already extracts charge,
hydrophobicity, aromatic fraction, and GRAVY — features that correlate
strongly with membrane interaction. Averaged hydrophobicity scales add
little beyond what these composition features already capture.

2. **No structural context**: 1D per-residue averages lose all sequence-order
and spatial information. A proper membrane interaction model requires
coarse-grained dynamics (Martini), structural ensembles (AlphaFold), or
at minimum a windowed context that captures local sequence patterns.

Future simulation work must use fundamentally richer inputs than 1D propensities.
Any new module must still beat its cheap baseline before earning ranking impact.

Logical inference: future simulation work needs richer inputs than the current
1D propensities. A more complex module still has to beat cheap baselines before
it earns ranking impact.
---

## Policy

Expand All @@ -71,3 +126,17 @@ Until this report changes with stronger evidence:
- Evidence certificates may include simulation values only as experimental
context, not as activity, safety, or selectivity proof.
- No public claim may imply simulation-validated antimicrobial activity.
- Any future simulation module must pass the gate (Loop 35) before affecting
ranking. The gate will not be bypassed even if the module appears plausible.

---

## Transition to Phase 4

Phase 4 (wet-lab readiness, Loops 40–49) shifts focus from virtual assay
scaffolding to preparing for real lab partners. The simulation modules ship
in info mode — usable for human inspection but not trusted for ranking.

The honest signal from Phase 3 is: **simulation is not ready to compress
wet-lab experiments.** Phase 4 work should prioritize getting the existing
pipeline into a lab partnership rather than deepening the simulation layer.
Loading