From fe21cc294d4bef92dc8e1a00a00062ee96b968d6 Mon Sep 17 00:00:00 2001 From: OpenCode Date: Mon, 6 Jul 2026 13:07:54 +0700 Subject: [PATCH] =?UTF-8?q?v0.5.51=20=E2=80=94=20Virtual=20Assay=20Scope?= =?UTF-8?q?=20Document=20+=20Doc=20Drift=20Fix=20(Loop=2030)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 begins with a scope document defining what the virtual assay layer is and is not, with explicit uncertainty policy, ablation requirements, and integration modes. Accumulated doc drift fixed alongside. Changes: - docs/VIRTUAL_ASSAY_SCOPE.md — Phase 3 scope: 4 planned modules, 12 not-modeled effects, uncertainty gate (uncertainty>0.5 blocks selection), ablation requirement versus cheap baselines, 3 integration modes (off/info/weighted), calibration gate for weighted mode, 5 exit criteria. - docs/BENCHMARKING.md — Complete rewrite: all 19 benchmarks in 6 categories (core, honesty, selectivity, ablation, regression, deferred). Each has make target, purpose, target metric, verification command. - docs/METRICS_CURRENT.md — Test count corrected 1832+ → 1843. - docs/NOVELTY_AUDIT_GUIDE.md — Stale script reference fixed: download_uniprot_amps.py → download_novelty_dbs.py --uniprot. - docs/50_LOOP_PLAN.md — Loop 30 marked complete, Phase 3 now 'In progress', next is Loop 31 (membrane proxy). - docs/ROADMAP.md — v0.5.51 entry with all change details. - scripts/__init__.py — Populated with module docstring. 1843 tests passing. No pipeline code changes. --- docs/50_LOOP_PLAN.md | 6 +- docs/BENCHMARKING.md | 99 ++++++++++++++++------- docs/METRICS_CURRENT.md | 2 +- docs/NOVELTY_AUDIT_GUIDE.md | 4 +- docs/ROADMAP.md | 27 +++++++ docs/VIRTUAL_ASSAY_SCOPE.md | 154 ++++++++++++++++++++++++++++++------ scripts/__init__.py | 4 + 7 files changed, 240 insertions(+), 56 deletions(-) diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md index 63dc9271..a6167ee2 100644 --- a/docs/50_LOOP_PLAN.md +++ b/docs/50_LOOP_PLAN.md @@ -98,7 +98,7 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte | Loop | Bottleneck | Deliverable | Verification | |------|-----------|-------------|-------------| -| 30 | No explicit virtual-assay scope document that says what IS and IS NOT being modeled | `docs/VIRTUAL_ASSAY_SCOPE.md` updated with: modeled effects (membrane binding, insertion depth), NOT-modeled effects (full-cell response, immune signaling, metabolism), uncertainty policy | Document is reviewable by a computational biophysicist | +| 30 ✅ | No explicit virtual-assay scope document; stale BENCHMARKING.md; stale test count in METRICS_CURRENT.md; broken script ref in NOVELTY_AUDIT_GUIDE.md | `docs/VIRTUAL_ASSAY_SCOPE.md` — scope doc with modeled/NOT-modeled effects, uncertainty policy, ablation requirement, integration modes. Rewrote `docs/BENCHMARKING.md` with all 19 benchmarks. Fixed test count, script ref, scripts/__init__.py. | Scope doc reviewable; BENCHMARKING.md matches Makefile; 1843 tests | | 31 | No membrane interaction proxy. The simulation module exists but has only a dummy placeholder | `simulation/membrane.py`: implements coarse-grained membrane binding energy proxy using hydrophobicity scale + helical hydrophobic moment. Returns `SimulationResult` with uncertainty | Benchmark: separates melittin (hemolytic) from magainin (selective) better than random (AUROC > 0.6 on strict set) | | 32 | No structure ensemble proxy. Current pipeline assumes all AMPs are helical | `simulation/structure.py`: secondary structure prediction (helix/coil/sheet propensity from sequence). Generates 3-state ensemble weights. Flags non-helical candidates | For known beta-sheet AMPs (protegrin, tachyplesin), reports low helix confidence | | 33 | No selectivity ratio module that combines bacterial and mammalian membrane scores | `simulation/selectivity_ratio.py`: ratio of bacterial membrane binding to RBC membrane binding proxy. Reports as `selectivity_ratio` with uncertainty | Distinguishes known selective (magainin) from known hemolytic (melittin) at ratio > 2x | @@ -180,7 +180,7 @@ If no data arrives, virtual assay scaffolding continues independently. Phase 0: ✅ Complete (Loops 1–8) Phase 1: ✅ Complete (Loops 9–17) Phase 2: ✅ Complete (Loops 18–29) -Phase 3: Not started (Loops 30–39) +Phase 3: In progress — Loop 30 ✅ (Loops 30–39) Phase 4: Not started (Loops 40–49) ``` @@ -241,7 +241,7 @@ Phase 4: Not started (Loops 40–49) | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available | | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers | -**Next loop:** Loop 30 — Phase 3 (virtual assay scope document). +**Next loop:** Loop 31 — Phase 3 (membrane interaction proxy). **Phase 2 exit criteria (all 5 met ✅):** - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md index ce151f2d..31430a6d 100644 --- a/docs/BENCHMARKING.md +++ b/docs/BENCHMARKING.md @@ -11,40 +11,83 @@ Common artifacts: - sequence length differences; - charge differences; - duplicate or near-duplicate leakage; -- test peptides too similar to training peptides. - -## Required benchmarks - -| Benchmark | Purpose | Status | -|---|---|---| -| Random baseline | Prove the pipeline is not noise | Implemented | -| Simple physicochemical baseline | Prove models beat charge/hydrophobicity alone | Implemented | -| Cluster split | Prevent near-duplicate leakage | Implemented (`bench cluster-split`) | -| Cluster-aware CI | Honest bootstrap when positives contain near-duplicate families | Implemented (cluster-aware bootstrap in `bench cluster-split`) | -| Time split | Test future generalization where metadata allows | Deferred (metadata not available) | -| Hidden-positive recovery | Check whether known actives are ranked high | Implemented | -| Toxicity down-ranking | Ensure predicted risky candidates are penalized | Implemented | -| Novelty stress test | Avoid near-clones of references | Implemented | -| Within-AMP selectivity | Test whether scorers distinguish hemolytic from selective AMPs | Implemented (`bench selectivity`) | -| Expert ablation | Test whether expert composite adds value over ensemble | Implemented (`bench expert-ablation`) | -| Multi-class triage | Test selective > hemolytic > decoy ranking in one panel | Implemented (`bench triage`) | +- test peptides too similar to training peptides; +- amphipathic-helix bias inflating discrimination on canonical AMPs; +- small-n benchmark confidence intervals masking uncertainty. + +## Benchmarks + +Each benchmark has a `make` target, a script in `scripts/`, and a +documented result in `docs/METRICS_CURRENT.md` and +`outputs/metrics_snapshot.json`. + +### Core discrimination + +| Benchmark | Purpose | Target | Verification | +|-----------|---------|--------|-------------| +| `bench-baseline` | Random baseline — pipeline is not noise | AUROC > 0.5 | `make bench-baseline` | +| `bench-hidden-active` | Known active AMPs hidden from training appear in top ranks | Better-than-random recovery | `make bench-hidden-active` | +| `validate-scoring` | Standard benchmark (n=191, 95 AMPs + 96 decoys) | AUROC > 0.70 | `make validate-scoring` | +| `bench-500` | Expanded benchmark (500 AMPs + 500 decoys, n=1000) | AUROC > 0.70, ~2.3× tighter CI | `make bench-500` | +| `bench-cluster-split` | Cluster-aware bootstrap on n=191 — near-duplicate de-inflation | Cluster-aware CI lo > 0.65 | `make bench-cluster-split` | +| `bench-cluster-split-500` | Cluster-split on expanded 500-AMP set | Cluster-aware CI lo > 0.65 | `make bench-cluster-split-500` | + +### Honesty benchmarks + +| Benchmark | Purpose | Target | Verification | +|-----------|---------|--------|-------------| +| `bench-easy-baseline` | Compare pipeline against single-feature trivial predictors (charge, length, charge density) | Document delta; pipeline expected to trade raw discrimination for safety | `make bench-easy-baseline` | +| `bench-charge-matched` | Greedy charge-density matching — test signal beyond cationic prior | Informational; documents whether pipeline retains signal after charge matching | `make bench-charge-matched` | +| `bench-order-dependent` | Test which features survive sequence scrambling (composition-control) | 7/31 features are order-dependent; dipeptide_order_score AUROC 0.786 | `make bench-order-dependent` | +| `bench-precision-at-k` | Operating characteristic for candidate selection — small-k triage precision/recall | top-20 precision 1.000, best F1 threshold documented | `make bench-precision-at-k` | +| `bench-multi-negatives` | Test robustness across 4 decoy distributions (random, reverse, shuffled proteome, inactive variants) | Pipeline rank order stable across decoy sources | `make bench-multi-negatives` | +| `bench-cross-dataset` | Generalization to DRAMP-only AMPs — test source-independence of heuristic features | AUROC within 0.03 of APD6/UniProt baseline | `make bench-cross-dataset` | + +### Selectivity / safety + +| Benchmark | Purpose | Target | Verification | +|-----------|---------|--------|-------------| +| `bench-selectivity` | Within-AMP hemolysis detection — can scorers distinguish hemolytic from selective AMPs? | rich_selectivity AUROC > 0.65 (CI lo > 0.5) | `make bench-selectivity` | +| `bench-per-family` | Stratify 500 AMPs by 6 heuristic structural classes — reveal charge-dominated pipeline bias | Document per-class AUROC; compensate via diversity selection | `make bench-per-family` | +| `bench-feature-decomp` | Per-feature selective-vs-hemolytic AUROC — which individual features have signal? | 8/30 features significant; 6/8 unused by current selectivity model | `make bench-feature-decomp` | + +### Composite / ablation + +| Benchmark | Purpose | Target | Verification | +|-----------|---------|--------|-------------| +| `bench-triage` | Multi-class: selective > hemolytic > decoy ranking in a single panel | gate_triage passes all 3 pairwise AUROCs; strict triage (composition-matched) fails honestly | `make bench-triage` | +| `bench-expert-ablation` | Ablate expert composite components on n=191 | Document per-component AUROC; expert lower than ensemble on AMP-vs-decoy (by design) | `make bench-expert-ablation` | +| `bench-expert-ablation-500` | Ablate expert composite on expanded n=1000 | Re-run on diverse set; 2 components reclassified from n=191 | `make bench-expert-ablation-500` | + +### Regression / CI + +| Benchmark | Purpose | Target | Verification | +|-----------|---------|--------|-------------| +| `bench-gate` | Multi-metric regression gate: cluster-split, selectivity, triage — enforced in CI | All metrics within tolerance; fails CI if AUROC drops | `make bench-gate` | +| `bench-leakage` | Train/test leakage detection | Zero sequence overlap between train and test sets | `make bench-leakage` | + +## Deferred benchmarks + +| Benchmark | Purpose | Reason deferred | +|-----------|---------|----------------| +| Time split | Test future generalization | Metadata not available for most sequences | +| Virtual-assay ablation | Test whether simulation modules improve triage | Phase 3 — no modules exist yet | +| Biologically plausible charge-balanced negative set | Honest non-charge signal test | Requires curated negative sequences — see v0.5.41 recommendation | ## Minimum report fields -Every benchmark report should include: +Every benchmark result stored in `outputs/metrics_snapshot.json` includes: -- dataset source; -- dataset license; -- number of positives; -- number of negatives; -- deduplication method; -- split method; -- similarity threshold; +- dataset source and license; +- number of positives and negatives; +- split method and similarity threshold; - baseline performance; -- model performance; -- confidence intervals where possible; +- model performance with confidence intervals; - known limitations. ## Anti-cheating rule -Do not update feature weights, filters, or thresholds after inspecting held-out performance unless you create a new held-out set. +Do not update feature weights, filters, or thresholds after inspecting +held-out performance unless you create a new held-out set. Pre-registered +analysis plans are recommended for any benchmark that gates a release +decision. diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index d61d0649..f0ee43f4 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -1101,7 +1101,7 @@ Decoys score low on activity. Selective AMPs score moderately on both. | Metric | Value | |--------|-------| -| Total tests | 1832+ | +| Total tests | 1843 | | Coverage (branch) | 99% (6 CLI guard lines only) | | Source modules at 100% | All pipeline, QC, scoring modules | diff --git a/docs/NOVELTY_AUDIT_GUIDE.md b/docs/NOVELTY_AUDIT_GUIDE.md index 0329dc2d..e769d182 100644 --- a/docs/NOVELTY_AUDIT_GUIDE.md +++ b/docs/NOVELTY_AUDIT_GUIDE.md @@ -72,8 +72,8 @@ curl -sL "${BASE}/general_amps.fasta" -o data/novelty_db/dramp_general.fasta curl -sL "${BASE}/patent_amps.fasta" -o data/novelty_db/dramp_patent.fasta curl -sL "${BASE}/specific_amps.fasta" -o data/novelty_db/dramp_specific.fasta -# Re-download UniProt (run annually) — use the Python paginator in scripts/ -python3 scripts/download_uniprot_amps.py +# Re-download UniProt (run annually) — use the download-novelty-dbs paginator +python3 scripts/download_novelty_dbs.py --uniprot # Re-download ESCAPE (run when Harvard Dataverse dataset is updated) python3 scripts/download_novelty_dbs.py --escape diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 5015b46e..9f0f91f6 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,5 +1,32 @@ # Roadmap +## v0.5.51 — Virtual Assay Scope + Doc Drift Fix (Loop 30) ✓ (2026-07-06) + +Phase 3 begins with a scope document defining what the virtual assay layer +IS and IS NOT, with explicit uncertainty policy, ablation requirements, and +integration modes. Accumulated doc drift fixed alongside: BENCHMARKING.md +was missing ~11 of 19 benchmarks; METRICS_CURRENT.md test count was stale +(1832+ → 1843); NOVELTY_AUDIT_GUIDE.md referenced a nonexistent script. + +Changes: +- ``docs/VIRTUAL_ASSAY_SCOPE.md`` — Phase 3 scope document covering 4 planned + modules (membrane binding, structure ensemble, selectivity ratio, stability), + 12 explicitly NOT-modeled effects, uncertainty policy (modules with + uncertainty > 0.5 cannot affect selection), ablation requirement (each + module must beat cheap heuristic baseline), 3 integration modes + (off/info/weighted), calibration gate for weighted mode, and 5 exit criteria. +- ``docs/BENCHMARKING.md`` — Complete rewrite: all 19 benchmarks organized + into 6 categories (core discrimination, honesty, selectivity/safety, + composite/ablation, regression/CI, deferred). Each benchmark has make target, + purpose, target metric, and verification command. +- ``docs/METRICS_CURRENT.md`` — Test count corrected: 1832+ → 1843. +- ``docs/NOVELTY_AUDIT_GUIDE.md`` — Fixed stale script reference: + ``scripts/download_uniprot_amps.py`` → ``scripts/download_novelty_dbs.py --uniprot``. +- ``docs/50_LOOP_PLAN.md`` — Loop 30 marked ✅. Phase 3 status updated to + "In progress — Loop 30 ✅". +- ``scripts/__init__.py`` — Populated with module docstring. +- 1843 tests passing. No code changes to pipeline. + ## v0.5.50 — Negative-Result Archive Template + Phase 2 Closeout ✓ (2026-07-06) Phase 2 exit criteria required a public negative-result archive format but none diff --git a/docs/VIRTUAL_ASSAY_SCOPE.md b/docs/VIRTUAL_ASSAY_SCOPE.md index 5bfff843..1b327cec 100644 --- a/docs/VIRTUAL_ASSAY_SCOPE.md +++ b/docs/VIRTUAL_ASSAY_SCOPE.md @@ -2,34 +2,144 @@ ## Purpose -The OpenAMP virtual assay layer exists to perform one job: **reduce the number of wet-lab experiments required to find a validated AMP.** +This document defines what the Phase 3 virtual assay layer IS and IS NOT, +which modules are planned, and the bar each module must clear before +affecting candidate selection. It exists to prevent simulation theater: +an impressive-looking model that does not improve decision quality is noise. -It does this by serving as a multi-resolution triage filter between cheap dry-lab screening (Phase 1) and actual wet-lab testing. It is *not* intended to be a perfect biological simulation, and it does not replace the wet lab. +## Status -The core operating principle is **"Wet lab is the expensive oracle."** The virtual assay layer asks the oracle fewer, smarter questions. +**Phase 3 — Planning.** No virtual-assay modules are implemented in the +production pipeline. This document describes the intended scope and +constraints. Implementation starts with Loop 31. -## What OpenAMP WILL Model +## Core Principle -1. **Multi-Resolution Proxies**: Moving beyond basic sequence heuristics (like charge and hydrophobicity) to include: - * **Structure Ensembles**: Predicted secondary/tertiary structures. - * **Coarse-Grained Membrane Interactions**: Simplified interaction simulations with bacterial (e.g., E. coli, S. aureus) and mammalian (RBC-like) membrane proxies (e.g., binding energy, insertion depth). - * **Serum Stability / Protease Resistance**: Proxies for half-life in physiological conditions. -2. **Learned Surrogate Emulators**: Machine learning models trained to predict MIC, HC50, and cytotoxicity from the structural and membrane interaction features. -3. **Uncertainty**: Every virtual assay must output an explicit uncertainty score. If the model is extrapolating outside its calibration data, it must abstain or flag low confidence. -4. **Calibration via Active Learning**: The models will be updated using true wet-lab assay outcomes (both hits and failures) from small, targeted batches (8-12 peptides). +> An impressive-looking simulator that does not improve real decision +> quality is noise, not progress. — AGENTS.md §7.5 -## What OpenAMP WILL NOT Model (No Simulation Theater) +Every module must demonstrate that it improves triage or selection quality +relative to a cheap heuristic baseline before it may affect candidate +ranking. Modules that fail the ablation must remain flagged as experimental. -1. **Whole-Organism Simulation**: We are not simulating entire E. coli bacteria from atoms upward. -2. **Uncalibrated "Truth"**: We will not treat molecular dynamics (MD) or surrogate model outputs as biological proof. -3. **Simulation for Simulation's Sake**: A virtual assay module will only be integrated into the selection pipeline if it empirically improves candidate triage relative to a cheap heuristic baseline. If a complex 24-hour MD simulation doesn't select better candidates than a 2-second Boman index calculation, it will be discarded. -4. **Cherry-Picked Validation**: We will not adjust success criteria after running simulations to make the pipeline look better. +## What the Virtual Assay Layer Models -## Integration Rules +| Module | Planned loop | Models | Output | +|--------|-------------|--------|--------| +| Membrane binding proxy | 31 | Coarse-grained membrane binding energy using hydrophobicity scale + helical hydrophobic moment. Bacterial membrane proxy (anionic lipid headgroups) and mammalian membrane proxy (zwitterionic/cholesterol). | `SimulationResult` with scores per membrane type, uncertainty | +| Structure ensemble proxy | 32 | Secondary structure propensity (helix/coil/sheet) from sequence. Flags non-helical candidates for which the helic-centric activity scorer is unreliable. | 3-state ensemble weights (helix/coil/sheet), confidence | +| Selectivity ratio | 33 | Ratio of bacterial membrane binding score to RBC/mammalian membrane binding score. Flags candidates where selectivity index is low. | `selectivity_ratio` with uncertainty | +| Stability proxy | Deferred | Protease susceptibility, serum half-life proxy. Deferred because no calibration data exists. | Not scoped until calibration data arrives | -Any new virtual assay module added to the `openamp_foundry.simulation` package must satisfy the following contract: +## What the Virtual Assay Layer Does NOT Model -1. **Implements `VirtualAssayProxy`**: Must conform to the standard interface returning a `SimulationResult`. -2. **Explicit Baseline Comparison**: Must demonstrate performance against the `EmulatorBaseline` contract on a benchmark set before affecting actual candidate selection. -3. **Strict Scope Definition**: Must explicitly list the scopes it targets (e.g., `["bacterial_membrane_binding", "rbc_membrane_disruption"]`). -4. **Uncertainty Reporting**: Must provide an explicit uncertainty metric (0.0 to 1.0) and identify its calibration dataset. +The virtual assay layer explicitly DOES NOT model: + +- Full-cell biological response (signaling, metabolism, stress response) +- Immune system interactions (cytokine release, immunogenicity) +- Pharmacokinetics or biodistribution +- Synergistic effects with other compounds +- Biofilm penetration or extracellular matrix interaction +- Intracellular target engagement (DNA, ribosome, protein binding) +- Resistance development or evolutionary dynamics +- In vivo efficacy in any animal model +- Human clinical outcomes + +Any claim that the virtual assay layer predicts these is a category error. + +## Uncertainty Policy + +Every `SimulationResult` MUST include an `uncertainty` field. The +uncertainty reflects: + +- **Parameter uncertainty**: how well the module's parameters are + constrained by available calibration data +- **Model-form uncertainty**: whether the module's functional form + captures the relevant physics +- **Calibration uncertainty**: whether calibration data exists and how + well it matches the target domain + +If calibration data is absent or weak, the `uncertainty` field must +surface that directly. A module with `uncertainty > 0.5` (on a 0–1 scale +where 0 = fully constrained, 1 = unconstrained) is experimental and must +not affect selection. + +## Ablation Requirement + +Each module must pass an ablation benchmark before it can affect +selection. The ablation test compares the module's discrimination against +a trivial baseline (single-feature proxy): + +| Metric | Required bar | +|--------|-------------| +| Selective-vs-hemolytic AUROC | > 0.65 with CI lower bound > 0.50 | +| Improvement over best single feature | Delta AUROC > 0.05 | +| Uncertainty | < 0.5 on relevant domain | + +Modules that do NOT pass the ablation are permanently marked as +experimental and flagged with `SIMULATION_THEATER_RISK` in their +`SimulationResult`. + +## Integration Modes + +The virtual assay layer supports three modes: + +| Mode | Effect | When to use | +|------|--------|-------------| +| `off` (default) | No simulation scores computed. Current pipeline. | Production candidate ranking | +| `info` | Simulation scores computed and included in reports only. No ranking impact. | Benchmarking, exploration | +| `weighted` | Simulation scores affect ranking if each module clears its ablation bar. | Only after ablation validation | + +The `rank` CLI accepts `--simulation-mode` to select the mode. + +## Calibration Requirement + +No virtual-assay module may be used in `weighted` mode unless it has +calibration data from at least one wet-lab batch showing that its +predictions correlate with measured outcomes (MIC, hemolysis, or +selectivity index). The calibration data must be documented in the +module's `calibration_set` field. + +## Known Risks + +| Risk | Mitigation | +|------|-----------| +| Overinterpretation of coarse-grained proxies | Ablation requirement, uncertainty propagation, explicit scope document | +| False confidence from simulation | Uncertainty > 0.5 gates off selection impact | +| Helical bias propagated to simulation | Structure ensemble proxy flags non-helical candidates | +| Simulation results mistaken for biology | All output labeled with `simulation: true`, no biological claims allowed | +| Ablation benchmarks overfit to reference set | Cross-validate on held-out membrane data, update reference set annually | + +## Relationship to Calibration Pipeline + +The virtual assay layer feeds INTO the calibration pipeline but does not +replace it. The order is: + +```text +candidate source + → dry-lab foundry (current pipeline) + → virtual assay proxies (Phase 3) + → uncertainty estimation + → informative batch selection + → lab assay + → calibration intake → gate → engine (Phase 2) + → next-round candidate selection +``` + +The virtual assay layer proposes which candidates are informative. The +calibration layer decides whether and how to update weights based on +actual outcomes. Neither layer operates without human review. + +## Exit Criteria for Phase 3 + +Phase 3 is complete when: + +1. At least 2 simulation modules exist and are benchmarked. +2. A module that improves strict-triage selective-vs-hemolytic AUROC + by > 0.03 is flagged as "improves selection" and available in + `weighted` mode. +3. Modules that fail the ablation are either removed or permanently + marked experimental. +4. Uncertainty is propagated through to the evidence certificate. +5. The external adapter protocol (`ExternalSimulationAdapter`) is + documented in ARCHITECTURE.md. diff --git a/scripts/__init__.py b/scripts/__init__.py index e69de29b..00469a1c 100644 --- a/scripts/__init__.py +++ b/scripts/__init__.py @@ -0,0 +1,4 @@ +# OpenAMP Foundry — reusable entrypoint scripts. + +# Scripts in this directory are standalone entrypoints. +# They are not imported as a Python package.