diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md index aac414d5..eb4046d8 100644 --- a/docs/50_LOOP_PLAN.md +++ b/docs/50_LOOP_PLAN.md @@ -106,7 +106,7 @@ Build the multi-resolution virtual assay layer: structure proxies, membrane inte | 35 ✅ | No executable brake prevented future agents from enabling weighted simulation despite negative ablation results | `simulation/gate.py` + `openamp-foundry bench simulation-gate`: fail-closed permission gate consuming both current ablation artifacts. `make simulation-gate` writes a machine-readable verdict. | 6 tests added; current suite 1927 passing, 7 skipped. Current verdict blocks weighted mode and downgrades to info-only | | 36 ✅ | No per-module cheap-baseline benchmark for within-domain simulation value | `scripts/benchmark_simulation_baselines.py`: compares each simulation signal (bacterial_binding, selectivity_ratio, helix_weight, non_helical) against its cheapest heuristic baseline (mean_eisenberg, selectivity_proxy, helix_propensity, proline_fraction). v0.5.58 | 13 tests, 1953 total. Honest finding: 0/4 signals beat baselines. All simulation modules remain permanently experimental. Weighted mode remains blocked | | 37 ✅ | No `--simulation-mode` CLI flag. Simulation modules exist but users cannot access them | Added `--simulation-mode` to `rank` CLI: `off` (default) and `info` (runs MembraneProxy + StructureProxy, adds `sim_*` keys to scores, includes in JSONL/report). Weighted blocked by gate (Loop 35). v0.5.57 | 6 tests, 1940 total. Demo pipeline with `--simulation-mode info` produces sim scores for all 10 candidates. Report includes sim columns | -| 38 | No API contract for third-party simulation modules (e.g. Martini MD, AlphaFold) | `simulation/interfaces.py` extended with `ExternalSimulationAdapter` protocol. Documented in ARCHITECTURE.md extension points | Third-party can implement the protocol without reading pipeline internals | +| 38 ✅ | No API contract for third-party simulation modules (e.g. Martini MD, AlphaFold) | `simulation/interfaces.py` extended with `ExternalSimulationAdapter` protocol and `validate_external_adapter_result()`. Documented in ARCHITECTURE.md extension points. v0.5.59. | Third-party adapters can implement the protocol without reading pipeline internals; validation rejects length, module, version, and uncertainty mismatches. | | 39 | Virtual-assay benchmark report comparing simulation-augmented vs simulation-free ranking on the strict triage set | `docs/SIMULATION_BENCHMARK.md`: head-to-head comparison of sel_vs_dec, hem_vs_dec, sel_vs_hem AUROCs with and without simulation. If simulation doesn't improve strict triage, say so explicitly | Honest: if delta ≤ 0, report "simulation did not improve triage" | **Phase 3 exit criteria:** @@ -241,7 +241,7 @@ Phase 4: Not started (Loops 40–49) | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available | | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers | -**Next loop:** Loop 38 — Phase 3 (external adapter protocol). +**Next loop:** Loop 39 — simulation benchmark report against strict triage. **Phase 2 exit criteria (all 5 met ✅):** - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index df0f0890..c703b37b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -42,7 +42,7 @@ The purpose of the added layers is not simulation theater. It is to improve whic | `openamp_foundry.reports` | human-readable and machine-readable batch, calibration, and wet-lab review reports | | `openamp_foundry.benchmark` | leakage checks, cluster-split benchmark, expert ablation benchmark, within-AMP selectivity benchmark, triage benchmark, per-feature selectivity decomposition, and evaluation scaffolding | | `openamp_foundry.generators` | safe, bounded toy candidate generation | -| `openamp_foundry.simulation` | membrane proxy (Loop 31), structure proxy (Loop 32), simulation ablation benchmark (Loop 33), dummy stub, interfaces | +| `openamp_foundry.simulation` | membrane proxy (Loop 31), structure proxy (Loop 32), simulation ablation benchmark (Loop 33), fail-closed weighted-mode gate, external adapter protocol, dummy stub, interfaces | | `openamp_foundry.calibration` | lab-result intake (v0.5.19), pre-registered recalibration policy + gate (v0.5.20), engine + report (v0.5.36/v0.5.44), policy version tracking (v0.5.40) | | `openamp_foundry.active_learning` | batch-2 selector (v0.5.45) and recovery benchmark (v0.5.46) for choosing informative next experiments under uncertainty | | `openamp_foundry.analysis` | diversity clustering, panel similarity, family structural warnings, audit helpers | @@ -115,7 +115,7 @@ Later external predictors should be added as adapters. Each adapter must return: Adapters must not silently download model weights or send sequences to third-party services without explicit user consent. -Any future simulation or emulator module must implement the `VirtualAssayProxy` interface (in `openamp_foundry.simulation`) and return a `SimulationResult` object matching this schema: +Any future in-repo simulation or emulator module must implement the `VirtualAssayProxy` interface (in `openamp_foundry.simulation`) and return a `SimulationResult` object matching this schema: ```python @dataclass @@ -131,3 +131,12 @@ class SimulationResult: ``` If calibration data is absent or weak, the `uncertainty` field must surface that directly. + +Third-party or heavyweight simulation integrations must implement +`ExternalSimulationAdapter` instead. That protocol requires adapter metadata +(`name`, `version`), explicit consent flags (`requires_network`, +`requires_model_download`, `requires_explicit_consent`), a `describe()` method, +and `simulate_batch(sequences) -> list[SimulationResult]`. Adapter outputs +should be checked with `validate_external_adapter_result()` before they enter +reports or benchmarks. External adapters must not silently download model +weights, transmit sequences, or run long jobs without explicit user consent. diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index 204792a3..aaaa21d2 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-06 (simulation cheap-baseline benchmark — v0.5.58) +> **Last updated:** 2026-07-06 (external simulation adapter contract — v0.5.59) +> **New in v0.5.59:** `ExternalSimulationAdapter` protocol and `validate_external_adapter_result()` define a safe contract for third-party or heavyweight simulation integrations. No external simulator was added; this is an interface and guardrail change only. > **New in v0.5.58:** Per-signal cheap-baseline comparison: 0/4 simulation signals beat their cheapest heuristic. All simulation modules remain permanently experimental. `make bench-simulation-baselines`. 13 tests. 1953 total. > **New in v0.5.57:** `rank --simulation-mode info` runs MembraneProxy + StructureProxy on every candidate, adds `sim_*` scores to JSONL output and Markdown report. 6 new CLI tests. 1940 total. > **New in v0.5.56:** `openamp-foundry bench simulation-gate` converts both @@ -43,7 +44,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test). > **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination. > **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison. -> **Pipeline version:** v0.5.58 +> **Pipeline version:** v0.5.59 > **Branch:** main --- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b80d06a9..1f8ed003 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,5 +1,26 @@ # Roadmap +## v0.5.59 — External Simulation Adapter Contract (Loop 38) ✓ (2026-07-06) + +Added the missing contract for third-party or heavyweight simulation modules. +This does not add any external simulator. It makes future integrations safer +and easier to review before they can enter reports or benchmarks. + +Changes: +- ``ExternalSimulationAdapter`` protocol in ``simulation/interfaces.py`` with + required metadata, consent flags, ``describe()``, and ``simulate_batch()``. +- ``validate_external_adapter_result()`` fail-loud helper for batch length, + module name, version, and uncertainty-range mismatches. +- Public exports from ``openamp_foundry.simulation``. +- Interface tests using a local fake adapter. +- Architecture and virtual-assay docs updated with consent requirements. + +**Honest scope:** +- No network, model download, molecular dynamics, AlphaFold, or remote service + integration was added. +- The contract only prevents hidden behavior and output-shape drift. It does + not validate biological usefulness. + ## v0.5.58 — Simulation Cheap-Baseline Benchmark (Loop 36) ✓ (2026-07-06) Per-signal comparison: does each simulation output beat its cheapest diff --git a/docs/VIRTUAL_ASSAY_SCOPE.md b/docs/VIRTUAL_ASSAY_SCOPE.md index 68f4fbdd..4677ac14 100644 --- a/docs/VIRTUAL_ASSAY_SCOPE.md +++ b/docs/VIRTUAL_ASSAY_SCOPE.md @@ -96,11 +96,16 @@ The virtual assay layer supports three modes: | `info` | Simulation scores computed and included in reports only. No ranking impact. | Benchmarking, exploration | | `weighted` | Simulation scores affect ranking only if the simulation gate passes on current ablation artifacts. | Only after ablation validation | -The `rank` CLI does **not** yet expose `--simulation-mode`. That omission is -intentional until a module clears the gate honestly. Current executable policy: +The `rank` CLI exposes `--simulation-mode off|info`. `weighted` mode is still +not exposed because the gate blocks it. Current executable policy: `openamp-foundry bench simulation-gate` may return `weighted` permission in the future, but production ranking remains simulation-free today. +Third-party or heavyweight simulators must implement `ExternalSimulationAdapter` +and pass `validate_external_adapter_result()` before their outputs are used. +Adapters must declare whether they require network access, model downloads, or +explicit user consent. + ## Calibration Requirement No virtual-assay module may be used in `weighted` mode unless it has diff --git a/src/openamp_foundry/simulation/__init__.py b/src/openamp_foundry/simulation/__init__.py index fe391943..8a836b6e 100644 --- a/src/openamp_foundry/simulation/__init__.py +++ b/src/openamp_foundry/simulation/__init__.py @@ -3,8 +3,10 @@ from .gate import SimulationGateVerdict, evaluate_simulation_gate from .interfaces import ( EmulatorBaseline, + ExternalSimulationAdapter, SimulationResult, VirtualAssayProxy, + validate_external_adapter_result, ) from .membrane import MembraneProxy from .structure import StructureProxy @@ -12,7 +14,9 @@ __all__ = [ "SimulationResult", "VirtualAssayProxy", + "ExternalSimulationAdapter", "EmulatorBaseline", + "validate_external_adapter_result", "MembraneProxy", "StructureProxy", "SimulationGateVerdict", diff --git a/src/openamp_foundry/simulation/interfaces.py b/src/openamp_foundry/simulation/interfaces.py index 25b5073c..e626cdf7 100644 --- a/src/openamp_foundry/simulation/interfaces.py +++ b/src/openamp_foundry/simulation/interfaces.py @@ -2,7 +2,7 @@ import abc from dataclasses import dataclass, field -from typing import Dict, List, Optional +from typing import Dict, List, Optional, Protocol, runtime_checkable @dataclass @@ -54,3 +54,60 @@ def simulate(self, sequence: str) -> SimulationResult: def get_baseline(self) -> EmulatorBaseline: """Return the baseline heuristic this simulation must beat.""" pass + + +@runtime_checkable +class ExternalSimulationAdapter(Protocol): + """Protocol for third-party or high-cost simulation adapters. + + External adapters may wrap molecular dynamics packages, structure predictors, + or remote inference services. They must expose consent-sensitive behavior so + OpenAMP never silently downloads model weights, transmits sequences, or runs + heavyweight jobs. + """ + + name: str + version: str + requires_network: bool + requires_model_download: bool + requires_explicit_consent: bool + + def describe(self) -> dict[str, object]: + """Return metadata, scope, limits, and consent requirements.""" + ... + + def simulate_batch(self, sequences: list[str]) -> list[SimulationResult]: + """Run the adapter and return one result per input sequence.""" + ... + + +def validate_external_adapter_result( + adapter: ExternalSimulationAdapter, + sequences: list[str], + results: list[SimulationResult], +) -> None: + """Validate the minimum output contract for external adapters.""" + if adapter.requires_explicit_consent and not ( + adapter.requires_network or adapter.requires_model_download + ): + raise ValueError( + "requires_explicit_consent may only be true when network or model " + "download consent is actually needed." + ) + if len(results) != len(sequences): + raise ValueError( + f"adapter returned {len(results)} result(s) for {len(sequences)} sequence(s)" + ) + for result in results: + if result.module != adapter.name: + raise ValueError( + f"adapter result module {result.module!r} does not match adapter " + f"name {adapter.name!r}" + ) + if result.version != adapter.version: + raise ValueError( + f"adapter result version {result.version!r} does not match adapter " + f"version {adapter.version!r}" + ) + if not 0.0 <= result.uncertainty <= 1.0: + raise ValueError("adapter uncertainty must be in [0, 1]") diff --git a/tests/test_benchmark_simulation_baselines.py b/tests/test_benchmark_simulation_baselines.py index 20e083dc..310efc00 100644 --- a/tests/test_benchmark_simulation_baselines.py +++ b/tests/test_benchmark_simulation_baselines.py @@ -92,7 +92,7 @@ def test_cli_output_contains_verdict(): def test_cli_writes_output_json(tmp_path): out = tmp_path / "baselines.json" - result = subprocess.run( + subprocess.run( [sys.executable, "scripts/benchmark_simulation_baselines.py", "--out", str(out)], capture_output=True, text=True, env={"PYTHONPATH": "src"}, diff --git a/tests/test_simulation_interfaces.py b/tests/test_simulation_interfaces.py index f85cad65..4792d350 100644 --- a/tests/test_simulation_interfaces.py +++ b/tests/test_simulation_interfaces.py @@ -1,8 +1,45 @@ """Tests for the virtual assay layer interfaces and contracts.""" +import pytest + from openamp_foundry.simulation.dummy import DummyMembraneProxy -from openamp_foundry.simulation.interfaces import SimulationResult +from openamp_foundry.simulation.interfaces import ( + ExternalSimulationAdapter, + SimulationResult, + validate_external_adapter_result, +) + + +class ExampleExternalAdapter: + name = "example_external_adapter" + version = "0.1.0" + requires_network = False + requires_model_download = False + requires_explicit_consent = False + + def describe(self): + return { + "name": self.name, + "version": self.version, + "scope": ["contract_test_only"], + "limits": ["does not model biology"], + } + + def simulate_batch(self, sequences): + return [ + SimulationResult( + module=self.name, + version=self.version, + scope=["contract_test_only"], + scores={"example_score": 0.5}, + uncertainty=1.0, + calibration_set=None, + validated_against=[], + notes=["Contract fixture only."], + ) + for _seq in sequences + ] def test_simulation_result_schema(): @@ -39,3 +76,56 @@ def test_dummy_proxy_contract(): baseline = proxy.get_baseline() baseline_score = baseline.evaluate("KKLFKKILKYL") assert isinstance(baseline_score, float) + + +def test_external_adapter_protocol_runtime_checkable(): + adapter = ExampleExternalAdapter() + + assert isinstance(adapter, ExternalSimulationAdapter) + assert adapter.describe()["name"] == "example_external_adapter" + + +def test_external_adapter_result_validation_accepts_matching_batch(): + adapter = ExampleExternalAdapter() + sequences = ["KKLFKKILKYL", "GIGKFLHSAKK"] + results = adapter.simulate_batch(sequences) + + validate_external_adapter_result(adapter, sequences, results) + + +def test_external_adapter_result_validation_rejects_length_mismatch(): + adapter = ExampleExternalAdapter() + + with pytest.raises(ValueError, match="returned 1 result"): + validate_external_adapter_result( + adapter, + ["AAA", "KKK"], + adapter.simulate_batch(["AAA"]), + ) + + +def test_external_adapter_result_validation_rejects_module_mismatch(): + adapter = ExampleExternalAdapter() + result = adapter.simulate_batch(["AAA"])[0] + result.module = "wrong" + + with pytest.raises(ValueError, match="does not match adapter name"): + validate_external_adapter_result(adapter, ["AAA"], [result]) + + +def test_external_adapter_result_validation_rejects_version_mismatch(): + adapter = ExampleExternalAdapter() + result = adapter.simulate_batch(["AAA"])[0] + result.version = "wrong" + + with pytest.raises(ValueError, match="does not match adapter version"): + validate_external_adapter_result(adapter, ["AAA"], [result]) + + +def test_external_adapter_result_validation_rejects_bad_uncertainty(): + adapter = ExampleExternalAdapter() + result = adapter.simulate_batch(["AAA"])[0] + result.uncertainty = 1.5 + + with pytest.raises(ValueError, match="uncertainty"): + validate_external_adapter_result(adapter, ["AAA"], [result])