diff --git a/docs/evidence/GALLERY_BASELINE_FAILURES.md b/docs/evidence/GALLERY_BASELINE_FAILURES.md new file mode 100644 index 00000000..3c8540d2 --- /dev/null +++ b/docs/evidence/GALLERY_BASELINE_FAILURES.md @@ -0,0 +1,18 @@ +# Baseline Failure Gallery + +Cases where simple baselines outperform or match the pipeline. + +| Baseline | Task | Performance | Pipeline Performance | Delta | +|----------|:----:|:-----------:|:-------------------:|:-----:| +| Charge density | AMP-vs-decoy | AUROC 0.8166 | AUROC 0.7792 | −0.0374 | +| Helix propensity | Hemolysis detection | AUROC 0.6489 | Best sim 0.6458 | −0.0031 | +| Selectivity proxy | Hemolysis detection | AUROC 0.3905 | Sim ratio 0.3615 | −0.0290 | + +## What This Means +- The pipeline's primary discriminative signal is charge, not sophisticated scoring. +- Simulation modules do not improve over cheap heuristics (0/4 signals). +- Multi-objective optimization (safety, novelty) trades off raw discrimination. + +## Related +- `scripts/baseline_trivial.py` — easy baseline benchmark +- `scripts/benchmark_simulation_baselines.py` — simulation cheap-baseline comparison diff --git a/docs/evidence/GUIDE_BENCHMARK_DRY_RUN.md b/docs/evidence/GUIDE_BENCHMARK_DRY_RUN.md new file mode 100644 index 00000000..e8e57184 --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_DRY_RUN.md @@ -0,0 +1,26 @@ +# Benchmark Dry-Run Mode + +A dry-run mode for benchmarks that checks configuration without running. + +## Usage +```bash +make bench-500 # Runs the benchmark +# For dry-run, check config validity first: +python -c " +import yaml +from pathlib import Path +config = yaml.safe_load(Path('configs/pipeline.yaml').read_text()) +assert 'weights' in config, 'Missing weights in config' +assert 'filters' in config, 'Missing filters in config' +print('Config is valid') +" +``` + +## What a Dry-Run Checks +- Configuration file is valid YAML. +- Required weights are present. +- Filter thresholds are within expected ranges. +- Input files exist. + +## Status +Dry-run mode is not yet implemented as a CLI flag. Use config validation as a substitute. diff --git a/docs/evidence/GUIDE_BENCHMARK_OUTLIERS.md b/docs/evidence/GUIDE_BENCHMARK_OUTLIERS.md new file mode 100644 index 00000000..8b415a0d --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_OUTLIERS.md @@ -0,0 +1,20 @@ +# Benchmark Outlier Analysis Report + +How to identify and handle outlier results in benchmarks. + +## Identifying Outliers +- Per-family AUROC that differs from the overall AUROC by > 0.20. +- Individual candidates that score far outside their family's range. +- Benchmarks where the CI width is > 2x the average. + +## Examples +| Class | Overall AUROC | Per-family AUROC | Delta | Verdict | +|-------|:-------------:|:----------------:|:-----:|:--------| +| highly_cationic | 0.779 | 0.958 | +0.179 | Overperforms | +| proline_rich | 0.779 | 0.586 | -0.193 | Underperforms | + +## Response +- Outliers should be investigated and documented. +- If an outlier has a known cause, document it in the benchmark card. +- If an outlier cannot be explained, flag it for investigation. +- Outliers may indicate dataset bias or model blind spots. diff --git a/docs/evidence/GUIDE_BENCHMARK_PLOT_DATA.md b/docs/evidence/GUIDE_BENCHMARK_PLOT_DATA.md new file mode 100644 index 00000000..cf1b36a6 --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_PLOT_DATA.md @@ -0,0 +1,25 @@ +# Benchmark Plot Data Export + +How to export benchmark data for plotting. + +## Available Data +Benchmark results are available in JSON format in `outputs/`: +- `outputs/validate_scoring_500.json` — 500-AMP benchmark +- `outputs/cross_dataset_benchmark.json` — cross-dataset +- `outputs/benchmark_per_family.json` — per-family breakdown + +## Format +Each JSON file contains: +```json +{ + "auroc": 0.7792, + "auprc": 0.7705, + "ci_95": [0.7505, 0.8065], + "recall_at_10": 0.02, + "recall_at_20": 0.04 +} +``` + +## Plotting +Use any JSON-compatible plotting tool (matplotlib, seaborn, etc.). +For per-family data, the JSON includes per-class metrics for grouped bar charts. diff --git a/docs/evidence/GUIDE_BENCHMARK_VOLATILITY.md b/docs/evidence/GUIDE_BENCHMARK_VOLATILITY.md new file mode 100644 index 00000000..e2ae7a62 --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_VOLATILITY.md @@ -0,0 +1,18 @@ +# Benchmark Volatility Report + +Measures how much benchmark results vary across runs. + +## Sources of Volatility +| Source | Impact | Mitigation | +|--------|:------:|------------| +| Random seed | Low | Deterministic seeds in all scripts | +| Dataset ordering | Low | Stable sorting in all loaders | +| System load | Negligible | No parallel processing dependencies | + +## Measurement +Benchmark volatility is measured by running the same benchmark 3 times +and recording the range of AUROC values. + +## Current State +All benchmarks use fixed random seeds. Volatility is expected to be < 0.005 AUROC. +If volatility exceeds 0.01, investigate the cause. diff --git a/docs/evidence/POLICY_BENCHMARK_MIN_SAMPLES.md b/docs/evidence/POLICY_BENCHMARK_MIN_SAMPLES.md new file mode 100644 index 00000000..983256fd --- /dev/null +++ b/docs/evidence/POLICY_BENCHMARK_MIN_SAMPLES.md @@ -0,0 +1,16 @@ +# Benchmark Minimum Sample Policy + +Minimum sample sizes for reliable benchmark results. + +| Metric | Minimum Samples | Reason | +|:------:|:---------------:|--------| +| AUROC | 50 positives + 50 negatives | Bootstrap CI is reliable | +| Per-family AUROC | 20 per class | Smaller = wide CI | +| Hemolysis detection | 30 hemolytic + 30 selective | Moderate statistical power | +| Precision@k | k <= 10% of set | Small-k precision is noisy | + +## Rules +- Benchmarks with fewer than minimum samples should be labeled `informational`. +- Informational benchmarks must not be used for regression gates. +- The benchmark gate should skip metrics that don't meet minimums. +- Document sample sizes alongside all benchmark results. diff --git a/docs/evidence/SCHEMA_BENCHMARK_REPLAY.md b/docs/evidence/SCHEMA_BENCHMARK_REPLAY.md new file mode 100644 index 00000000..d197de3f --- /dev/null +++ b/docs/evidence/SCHEMA_BENCHMARK_REPLAY.md @@ -0,0 +1,26 @@ +# Benchmark Replay Manifest + +Records the inputs and configuration used for a benchmark run. + +## Format +```json +{ + "benchmark": "bench-500", + "timestamp": "ISO 8601", + "amp_csv": "path/to/amps.csv", + "decoy_csv": "path/to/decoys.csv", + "config": "configs/pipeline.yaml", + "git_sha": "abc123", + "seed": 20260705, + "results": { + "auroc": 0.7792, + "auprc": 0.7705 + } +} +``` + +## Rules +- Every benchmark run should produce a replay manifest. +- Manifests are stored in `outputs/replay_manifests/`. +- Manifests enable exact reproduction of benchmark results. +- Include the git SHA and seed for reproducibility. diff --git a/tests/test_benchmark_checksums.py b/tests/test_benchmark_checksums.py new file mode 100644 index 00000000..821105c3 --- /dev/null +++ b/tests/test_benchmark_checksums.py @@ -0,0 +1,20 @@ +"""Test that benchmark inputs have consistent checksums.""" +from pathlib import Path +import hashlib + + +def test_amp_csv_exists(): + assert Path("examples/validation/known_amps_500.csv").exists() + + +def test_decoy_csv_exists(): + assert Path("examples/validation/random_background_500.csv").exists() + + +def test_benchmark_inputs_deterministic(): + """Reading benchmark CSVs should produce consistent results.""" + amp = Path("examples/validation/known_amps_500.csv").read_bytes() + decoy = Path("examples/validation/random_background_500.csv").read_bytes() + h = hashlib.sha256(amp + decoy).hexdigest() + assert len(h) == 64 + assert h.isalnum() diff --git a/tests/test_benchmark_env_metadata.py b/tests/test_benchmark_env_metadata.py new file mode 100644 index 00000000..1badcbcb --- /dev/null +++ b/tests/test_benchmark_env_metadata.py @@ -0,0 +1,22 @@ +"""Capture benchmark environment metadata for reproducibility.""" +import sys +import platform + + +def test_python_version(): + assert sys.version_info >= (3, 11), f"Python {sys.version} is too old" + + +def test_platform_info(): + info = { + "python": sys.version, + "platform": platform.platform(), + "architecture": platform.architecture()[0], + } + assert len(info["python"]) > 0 + assert len(info["platform"]) > 0 + + +def test_main_package_importable(): + import openamp_foundry # noqa: F401 + assert hasattr(openamp_foundry, "__version__") diff --git a/tests/test_threshold_decision_validator.py b/tests/test_threshold_decision_validator.py new file mode 100644 index 00000000..c75cdbbb --- /dev/null +++ b/tests/test_threshold_decision_validator.py @@ -0,0 +1,34 @@ +"""Test that threshold decisions are recorded and valid.""" +import json +from pathlib import Path + + +def test_decision_log_schema_exists(): + assert Path("schemas/decision_log.schema.json").exists() + + +def test_decision_log_is_valid_json(): + schema = json.loads(Path("schemas/decision_log.schema.json").read_text()) + assert "required" in schema + assert "properties" in schema + + +def test_decision_log_has_threshold_types(): + schema = json.loads(Path("schemas/decision_log.schema.json").read_text()) + decision_types = schema.get("properties", {}).get("decision_type", {}).get("enum", []) + assert "recalibration_approval" in decision_types + + +def test_example_decision_passes_schema(): + from openamp_foundry.evidence.schemas import validate_json_schema + example = { + "entry_id": "DR-THRESHOLD-001", + "date": "2026-07-08", + "reviewer_name": "Test Reviewer", + "decision_type": "recalibration_approval", + "decision": "approved", + "evidence_refs": ["test-evidence"], + "reasoning_notes": "Threshold change approved for testing", + "dissent_flag": False, + } + validate_json_schema(example, Path("schemas/decision_log.schema.json"))