Add 10 benchmark infrastructure files

docs/evidence/GALLERY_BASELINE_FAILURES.md

@@ -0,0 +1,18 @@
+# Baseline Failure Gallery
+
+Cases where simple baselines outperform or match the pipeline.
+
+| Baseline | Task | Performance | Pipeline Performance | Delta |
+|----------|:----:|:-----------:|:-------------------:|:-----:|
+| Charge density | AMP-vs-decoy | AUROC 0.8166 | AUROC 0.7792 | −0.0374 |
+| Helix propensity | Hemolysis detection | AUROC 0.6489 | Best sim 0.6458 | −0.0031 |
+| Selectivity proxy | Hemolysis detection | AUROC 0.3905 | Sim ratio 0.3615 | −0.0290 |
+
+## What This Means
+- The pipeline's primary discriminative signal is charge, not sophisticated scoring.
+- Simulation modules do not improve over cheap heuristics (0/4 signals).
+- Multi-objective optimization (safety, novelty) trades off raw discrimination.
+
+## Related
+- `scripts/baseline_trivial.py` — easy baseline benchmark
+- `scripts/benchmark_simulation_baselines.py` — simulation cheap-baseline comparison

docs/evidence/GUIDE_BENCHMARK_DRY_RUN.md

@@ -0,0 +1,26 @@
+# Benchmark Dry-Run Mode
+
+A dry-run mode for benchmarks that checks configuration without running.
+
+## Usage
+```bash
+make bench-500  # Runs the benchmark
+# For dry-run, check config validity first:
+python -c "
+import yaml
+from pathlib import Path
+config = yaml.safe_load(Path('configs/pipeline.yaml').read_text())
+assert 'weights' in config, 'Missing weights in config'
+assert 'filters' in config, 'Missing filters in config'
+print('Config is valid')
+"
+```
+
+## What a Dry-Run Checks
+- Configuration file is valid YAML.
+- Required weights are present.
+- Filter thresholds are within expected ranges.
+- Input files exist.
+
+## Status
+Dry-run mode is not yet implemented as a CLI flag. Use config validation as a substitute.

docs/evidence/GUIDE_BENCHMARK_OUTLIERS.md

@@ -0,0 +1,20 @@
+# Benchmark Outlier Analysis Report
+
+How to identify and handle outlier results in benchmarks.
+
+## Identifying Outliers
+- Per-family AUROC that differs from the overall AUROC by > 0.20.
+- Individual candidates that score far outside their family's range.
+- Benchmarks where the CI width is > 2x the average.
+
+## Examples
+| Class | Overall AUROC | Per-family AUROC | Delta | Verdict |
+|-------|:-------------:|:----------------:|:-----:|:--------|
+| highly_cationic | 0.779 | 0.958 | +0.179 | Overperforms |
+| proline_rich | 0.779 | 0.586 | -0.193 | Underperforms |
+
+## Response
+- Outliers should be investigated and documented.
+- If an outlier has a known cause, document it in the benchmark card.
+- If an outlier cannot be explained, flag it for investigation.
+- Outliers may indicate dataset bias or model blind spots.

docs/evidence/GUIDE_BENCHMARK_PLOT_DATA.md

@@ -0,0 +1,25 @@
+# Benchmark Plot Data Export
+
+How to export benchmark data for plotting.
+
+## Available Data
+Benchmark results are available in JSON format in `outputs/`:
+- `outputs/validate_scoring_500.json` — 500-AMP benchmark
+- `outputs/cross_dataset_benchmark.json` — cross-dataset
+- `outputs/benchmark_per_family.json` — per-family breakdown
+
+## Format
+Each JSON file contains:
+```json
+{
+  "auroc": 0.7792,
+  "auprc": 0.7705,
+  "ci_95": [0.7505, 0.8065],
+  "recall_at_10": 0.02,
+  "recall_at_20": 0.04
+}
+```
+
+## Plotting
+Use any JSON-compatible plotting tool (matplotlib, seaborn, etc.).
+For per-family data, the JSON includes per-class metrics for grouped bar charts.

docs/evidence/GUIDE_BENCHMARK_VOLATILITY.md

@@ -0,0 +1,18 @@
+# Benchmark Volatility Report
+
+Measures how much benchmark results vary across runs.
+
+## Sources of Volatility
+| Source | Impact | Mitigation |
+|--------|:------:|------------|
+| Random seed | Low | Deterministic seeds in all scripts |
+| Dataset ordering | Low | Stable sorting in all loaders |
+| System load | Negligible | No parallel processing dependencies |
+
+## Measurement
+Benchmark volatility is measured by running the same benchmark 3 times
+and recording the range of AUROC values.
+
+## Current State
+All benchmarks use fixed random seeds. Volatility is expected to be < 0.005 AUROC.
+If volatility exceeds 0.01, investigate the cause.

docs/evidence/POLICY_BENCHMARK_MIN_SAMPLES.md

@@ -0,0 +1,16 @@
+# Benchmark Minimum Sample Policy
+
+Minimum sample sizes for reliable benchmark results.
+
+| Metric | Minimum Samples | Reason |
+|:------:|:---------------:|--------|
+| AUROC | 50 positives + 50 negatives | Bootstrap CI is reliable |
+| Per-family AUROC | 20 per class | Smaller = wide CI |
+| Hemolysis detection | 30 hemolytic + 30 selective | Moderate statistical power |
+| Precision@k | k <= 10% of set | Small-k precision is noisy |
+
+## Rules
+- Benchmarks with fewer than minimum samples should be labeled `informational`.
+- Informational benchmarks must not be used for regression gates.
+- The benchmark gate should skip metrics that don't meet minimums.
+- Document sample sizes alongside all benchmark results.

docs/evidence/SCHEMA_BENCHMARK_REPLAY.md

@@ -0,0 +1,26 @@
+# Benchmark Replay Manifest
+
+Records the inputs and configuration used for a benchmark run.
+
+## Format
+```json
+{
+  "benchmark": "bench-500",
+  "timestamp": "ISO 8601",
+  "amp_csv": "path/to/amps.csv",
+  "decoy_csv": "path/to/decoys.csv",
+  "config": "configs/pipeline.yaml",
+  "git_sha": "abc123",
+  "seed": 20260705,
+  "results": {
+    "auroc": 0.7792,
+    "auprc": 0.7705
+  }
+}
+```
+
+## Rules
+- Every benchmark run should produce a replay manifest.
+- Manifests are stored in `outputs/replay_manifests/`.
+- Manifests enable exact reproduction of benchmark results.
+- Include the git SHA and seed for reproducibility.

tests/test_benchmark_checksums.py

@@ -0,0 +1,20 @@
+"""Test that benchmark inputs have consistent checksums."""
+from pathlib import Path
+import hashlib
+
+
+def test_amp_csv_exists():
+    assert Path("examples/validation/known_amps_500.csv").exists()
+
+
+def test_decoy_csv_exists():
+    assert Path("examples/validation/random_background_500.csv").exists()
+
+
+def test_benchmark_inputs_deterministic():
+    """Reading benchmark CSVs should produce consistent results."""
+    amp = Path("examples/validation/known_amps_500.csv").read_bytes()
+    decoy = Path("examples/validation/random_background_500.csv").read_bytes()
+    h = hashlib.sha256(amp + decoy).hexdigest()
+    assert len(h) == 64
+    assert h.isalnum()

tests/test_benchmark_env_metadata.py

@@ -0,0 +1,22 @@
+"""Capture benchmark environment metadata for reproducibility."""
+import sys
+import platform
+
+
+def test_python_version():
+    assert sys.version_info >= (3, 11), f"Python {sys.version} is too old"
+
+
+def test_platform_info():
+    info = {
+        "python": sys.version,
+        "platform": platform.platform(),
+        "architecture": platform.architecture()[0],
+    }
+    assert len(info["python"]) > 0
+    assert len(info["platform"]) > 0
+
+
+def test_main_package_importable():
+    import openamp_foundry  # noqa: F401
+    assert hasattr(openamp_foundry, "__version__")

tests/test_threshold_decision_validator.py

@@ -0,0 +1,34 @@
+"""Test that threshold decisions are recorded and valid."""
+import json
+from pathlib import Path
+
+
+def test_decision_log_schema_exists():
+    assert Path("schemas/decision_log.schema.json").exists()
+
+
+def test_decision_log_is_valid_json():
+    schema = json.loads(Path("schemas/decision_log.schema.json").read_text())
+    assert "required" in schema
+    assert "properties" in schema
+
+
+def test_decision_log_has_threshold_types():
+    schema = json.loads(Path("schemas/decision_log.schema.json").read_text())
+    decision_types = schema.get("properties", {}).get("decision_type", {}).get("enum", [])
+    assert "recalibration_approval" in decision_types
+
+
+def test_example_decision_passes_schema():
+    from openamp_foundry.evidence.schemas import validate_json_schema
+    example = {
+        "entry_id": "DR-THRESHOLD-001",
+        "date": "2026-07-08",
+        "reviewer_name": "Test Reviewer",
+        "decision_type": "recalibration_approval",
+        "decision": "approved",
+        "evidence_refs": ["test-evidence"],
+        "reasoning_notes": "Threshold change approved for testing",
+        "dissent_flag": False,
+    }
+    validate_json_schema(example, Path("schemas/decision_log.schema.json"))