Add 10 adapter and benchmark infrastructure files

docs/evidence/CHECKLIST_ADAPTER_PRIVACY.md

@@ -0,0 +1,17 @@
+# Adapter Privacy Checklist
+
+Privacy checklist for external adapters.
+
+## Checklist
+- [ ] Adapter declares whether network access is required.
+- [ ] No data is sent without user consent.
+- [ ] Adapter does not collect telemetry.
+- [ ] Credentials are not stored in code.
+- [ ] Authentication uses environment variables or secure storage.
+- [ ] Data sent to external services is documented.
+- [ ] Users are informed before any network request is made.
+
+## Rules
+- All adapters must pass this checklist before being used with real data.
+- Adapters that fail the checklist may only be used with synthetic data.
+- This checklist should be reviewed when adding new adapters.

docs/evidence/POLICY_ADAPTER_TIMEOUT.md

@@ -0,0 +1,17 @@
+# Adapter Timeout Policy
+
+Timeout rules for external adapter calls.
+
+## Default Timeouts
+| Adapter Type | Default Timeout | Max Timeout |
+|-------------|:---------------:|:-----------:|
+| External API | 30 seconds | 120 seconds |
+| Local process | 60 seconds | 300 seconds |
+| File I/O | 10 seconds | 60 seconds |
+
+## Rules
+- Adapters should have configurable timeout values.
+- Timeout errors should be retried at least once.
+- If a timeout persists, return uncertainty=1.0.
+- Document timeout behavior in each adapter's docstring.
+- Timeouts should not block the pipeline — use async or thread-based timeout.

docs/evidence/REGISTRY_BENCHMARK_LIMITATIONS.md

@@ -0,0 +1,17 @@
+# Benchmark Limitation Registry
+
+Known limitations of the project's benchmarks.
+
+| Benchmark | Limitation |
+|-----------|------------|
+| 500-AMP benchmark | AUROC is charge-inflated (0.7792; collapses to 0.5103 under exact charge control) |
+| Cross-dataset | DRAMP + Swiss-Prot decoys may share database artifacts |
+| Simulation ablation | Modules are tested on AMP-vs-decoy, not their designed task |
+| Within-AMP hemolysis | Small reference set (n=179), HC50 values from literature with high variability |
+| Easy baseline | Charge density beats ensemble — expected, pipeline optimizes for safety |
+
+## Rules
+- Every benchmark result must reference its limitations.
+- Limitations must be stated alongside the result, not hidden.
+- When a limitation is resolved, update this registry.
+- If a benchmark has no documented limitations, it's incomplete.

docs/evidence/SCHEMA_ADAPTER_METADATA.md

@@ -0,0 +1,31 @@
+# Adapter Metadata Schema
+
+Standard metadata fields for external adapters.
+
+## Required Fields
+| Field | Type | Description |
+|-------|------|-------------|
+| name | string | Adapter name |
+| version | string | Adapter version |
+| type | string | simulation, external_api, local_process |
+| requires_network | boolean | Whether network access is needed |
+
+## Optional Fields
+| Field | Type | Description |
+|-------|------|-------------|
+| timeout | integer | Default timeout in seconds |
+| retry_count | integer | Number of retries on failure |
+| rate_limit | integer | Maximum requests per minute |
+| dependencies | list | Required Python packages |
+
+## Example
+```json
+{
+  "name": "example_adapter",
+  "version": "0.1.0",
+  "type": "external_api",
+  "requires_network": true,
+  "timeout": 30,
+  "retry_count": 2
+}
+```

docs/evidence/SCHEMA_ADAPTER_OUTPUT.md

@@ -0,0 +1,21 @@
+# Adapter Output Schema
+
+Standard output schema for external adapters.
+
+## SimulationResult Fields
+| Field | Type | Description |
+|-------|------|-------------|
+| module | string | Adapter name |
+| version | string | Adapter version |
+| scope | list | What was modeled |
+| scores | dict | Score key-value pairs |
+| uncertainty | float | 0-1 scale, higher = less certain |
+| calibration_set | string or null | Reference to calibration data |
+| validated_against | list | Validation datasets |
+| notes | list | Human-readable notes |
+
+## Rules
+- All adapters must return a SimulationResult.
+- uncertainty=1.0 means completely uncertain (do not use).
+- calibration_set=null means no calibration data exists.
+- notes should include warnings, limitations, and context.

docs/evidence/SCHEMA_EXTERNAL_CACHE.md

@@ -0,0 +1,27 @@
+# External Result Cache Manifest
+
+Records cached results from external adapter calls.
+
+## Format
+```json
+{
+  "cache_version": "1.0",
+  "created_at": "ISO 8601",
+  "entries": [
+    {
+      "adapter": "adapter_name",
+      "sequence": "peptide_sequence",
+      "result_hash": "sha256_of_result",
+      "cached_at": "ISO 8601",
+      "ttl_seconds": 86400
+    }
+  ]
+}
+```
+
+## Rules
+- Cache entries should have a TTL (time-to-live).
+- Cached results should be validated against the adapter schema.
+- Cache should be invalidated when the adapter version changes.
+- Cached results must be clearly labeled as cached.
+- Cache location: `outputs/adapter_cache/`.

docs/evidence/TAXONOMY_ADAPTER_ERRORS.md

@@ -0,0 +1,19 @@
+# Adapter Error Taxonomy
+
+Types of errors that external adapters can produce.
+
+| Error Type | Meaning | Recovery |
+|-----------|---------|----------|
+| connection | Network or DNS failure | Retry with backoff |
+| timeout | Request exceeded time limit | Increase timeout or retry |
+| rate_limit | Too many requests | Reduce frequency, wait |
+| auth | Authentication failure | Check credentials |
+| validation | Response failed schema check | Check adapter version |
+| internal | Adapter raised an exception | Check adapter code |
+| not_found | Resource not available | Check resource path |
+
+## Rules
+- All adapters should handle all error types gracefully.
+- Errors should return a SimulationResult with uncertainty=1.0.
+- Errors should include a human-readable message.
+- Connection and timeout errors should be retryable.

tests/test_adapter_replay_fixtures.py

@@ -0,0 +1,40 @@
+"""Provide replay fixtures for testing adapters offline."""
+from openamp_foundry.simulation.interfaces import SimulationResult
+
+
+def sample_membrane_result() -> SimulationResult:
+    return SimulationResult(
+        module="membrane_proxy",
+        version="0.1.0",
+        scope=["bacterial_membrane_binding", "mammalian_membrane_binding"],
+        scores={"bacterial_binding": 0.75, "mammalian_binding": 0.3, "selectivity_ratio": 1.5},
+        uncertainty=0.2,
+        calibration_set=None,
+        validated_against=[],
+        notes=["Sample result for offline testing"],
+    )
+
+
+def sample_structure_result() -> SimulationResult:
+    return SimulationResult(
+        module="structure_proxy",
+        version="0.1.0",
+        scope=["secondary_structure_prediction"],
+        scores={"helix_weight": 0.6, "sheet_weight": 0.2, "coil_weight": 0.2, "non_helical": 0.0},
+        uncertainty=0.15,
+        calibration_set=None,
+        validated_against=[],
+        notes=["Sample result for offline testing"],
+    )
+
+
+def test_sample_membrane():
+    r = sample_membrane_result()
+    assert r.module == "membrane_proxy"
+    assert 0 <= r.scores["bacterial_binding"] <= 1
+
+
+def test_sample_structure():
+    r = sample_structure_result()
+    assert r.module == "structure_proxy"
+    assert abs(sum([r.scores["helix_weight"], r.scores["sheet_weight"], r.scores["coil_weight"]]) - 1.0) < 0.01

tests/test_benchmark_delta_sign.py

@@ -0,0 +1,19 @@
+"""Test that benchmark delta signs are correct (improvement vs regression)."""
+
+def test_benchmark_gate_delta_sign():
+    """Verify the benchmark gate correctly identifies improvements vs regressions."""
+    import json
+    from pathlib import Path
+    gate_output = Path("outputs/benchmark_gate_output.json")
+    if not gate_output.exists():
+        return  # Skip if no gate output exists
+    data = json.loads(gate_output.read_text())
+    for metric, info in data.items():
+        if "delta" in info:
+            assert isinstance(info["delta"], (int, float)), f"Delta for {metric} is not numeric"
+
+
+def test_auroc_delta_interpretation():
+    """A positive delta in AUROC means improvement."""
+    from openamp_foundry.scoring.ensemble import ensemble_score
+    assert ensemble_score({"a": 1.0}, {"a": 1.0}) > ensemble_score({"a": 0.0}, {"a": 1.0})

tests/test_benchmark_footnotes.py

@@ -0,0 +1,21 @@
+"""Test that benchmark report footnotes are consistent."""
+from pathlib import Path
+
+
+def test_metrics_current_has_caveats():
+    """METRICS_CURRENT.md should include caveats with benchmark results."""
+    text = Path("docs/evidence/METRICS_CURRENT.md").read_text()
+    assert "limitation" in text.lower() or "caveat" in text.lower() or "note" in text.lower()
+
+
+def test_simulation_benchmark_has_verdict():
+    """Simulation benchmark should clearly state the verdict."""
+    text = Path("docs/evidence/SIMULATION_BENCHMARK.md").read_text()
+    assert "No improvement" in text or "blocked" in text
+
+
+def test_each_benchmark_has_interpretation():
+    """Each benchmark section should include an interpretation."""
+    text = Path("docs/evidence/BENCHMARKING.md").read_text()
+    assert "Benchmark" in text
+    assert len(text) > 500