Add 10 benchmark and concept guides

docs/evidence/CONCEPT_LEAKAGE.md

@@ -0,0 +1,19 @@
+# Leakage Concept Guide
+
+Leakage occurs when information from outside the training set influences the model.
+
+## Types of Leakage
+| Type | Description | Example |
+|------|-------------|---------|
+| Near-duplicate leakage | Test sequence is similar to a training sequence | Cluster split prevents this |
+| Composition leakage | Test and train share composition features | Hard to avoid; tracked by charge-matched benchmark |
+| Temporal leakage | Future data influences past predictions | Time splits where metadata permits |
+| Feature leakage | Features indirectly encode the target | Avoided by using transparent heuristics |
+
+## Detection
+- Cluster split benchmarks detect near-duplicate leakage.
+- Charge-matched benchmarks detect composition leakage.
+- Cross-dataset benchmarks detect overfitting to database artifacts.
+
+## Related
+- `docs/evidence/LEAKAGE_AUDIT_CHECKLIST.md`

docs/evidence/CONCEPT_STALE_ARTIFACT.md

@@ -0,0 +1,19 @@
+# Stale Artifact Concept Guide
+
+An artifact is stale when it no longer reflects the current pipeline or data.
+
+## Signs of Staleness
+- Schema version doesn't match current code
+- Benchmark results don't match current pipeline output
+- Config values differ from the values used to generate the artifact
+- The artifact references features or commands that no longer exist
+
+## Causes
+- Pipeline code updated without regenerating artifacts
+- Schema changed without migrating existing artifacts
+- Config changed without re-running dependent processes
+
+## Prevention
+- Regenerate artifacts after pipeline changes.
+- Version schemas and check version alignment in CI.
+- Run `make regenerate-all` before releases.

docs/evidence/GUIDE_BENCHMARK_CARD_WRITING.md

@@ -0,0 +1,36 @@
+# Benchmark Card Writing Guide
+
+How to write a benchmark card.
+
+## Required Sections
+- **Purpose**: What the benchmark measures
+- **Dataset**: What data is used and its source
+- **Metrics**: What metrics are reported
+- **Caveats**: What the benchmark does NOT prove
+- **Reproduction**: How to run the benchmark
+
+## Format
+```markdown
+# Benchmark: [Name]
+
+## Purpose
+One paragraph describing what this benchmark measures.
+
+## Dataset
+- Positives: N sequences from [source]
+- Negatives: N sequences from [source]
+- Split method: [method]
+
+## Metrics
+| Metric | Value | Interpretation |
+|--------|-------|----------------|
+
+## Caveats
+- Known limitations of this benchmark.
+- What this benchmark does not test.
+
+## Reproduction
+```bash
+make bench-500
+```
+```

docs/evidence/GUIDE_BENCHMARK_PHILOSOPHY.md

@@ -0,0 +1,21 @@
+# Benchmark Philosophy Page
+
+The project's approach to benchmarking.
+
+## Principles
+1. Benchmarks measure discrimination, not biological truth.
+2. Every benchmark has caveats that must be documented.
+3. No single benchmark determines pipeline quality — use the full suite.
+4. Benchmarks must be reproducible (deterministic, versioned inputs).
+5. Benchmarks must be adversarial — actively look for ways the pipeline is wrong.
+
+## What Benchmarks Do NOT Measure
+- Real-world antimicrobial activity
+- Safety in humans
+- Clinical utility
+- Synthesis success rate
+- Actual hit rate in a screening campaign
+
+## Related
+- `docs/evidence/BENCHMARK_GOVERNANCE.md` — benchmark policies
+- `docs/evidence/METRICS_CURRENT.md` — current results

docs/evidence/GUIDE_BENCHMARK_REGRESSION_INTERP.md

@@ -0,0 +1,22 @@
+# Benchmark Regression Interpretation Guide
+
+How to interpret a benchmark regression.
+
+## What to Check
+1. Is the regression > 0.02 AUROC? (gate threshold)
+2. Is the regression in the primary benchmark or a slice?
+3. Does the regression affect all classes or just one?
+4. Is the change intentional (tradeoff) or accidental (bug)?
+
+## Common Causes
+| Cause | Pattern | Action |
+|-------|---------|--------|
+| Safety penalty increase | AUROC drops, safety improves | Accept tradeoff |
+| Feature change | One class regresses | Isolate the class |
+| Data change | Cross-dataset validation | Update benchmark |
+| Bug | Unexpected drop everywhere | Fix and revert |
+
+## Response
+- Acceptable regression (< 0.02): Document and proceed.
+- Unacceptable regression (> 0.02): Investigate and fix.
+- Deliberate tradeoff: Document the rationale in the decision log.

docs/evidence/GUIDE_BENCHMARK_SLICE.md

@@ -0,0 +1,19 @@
+# Benchmark Slice Explanation
+
+A benchmark slice measures performance on a specific subset of data.
+
+## Why Slice
+- Overall AUROC can hide poor performance on important subgroups.
+- Slicing reveals which candidate classes the pipeline handles well or poorly.
+
+## Current Slices
+| Slice | Description | AUROC | Finding |
+|-------|-------------|:-----:|---------|
+| Highly cationic | Charge >= 4.0 | 0.958 | Well handled |
+| Proline-rich | Pro fraction >= 15% | 0.586 | Poorly handled |
+| Cysteine-rich | >= 2 Cys | 0.723 | Moderately handled |
+| Short | <= 12 AA | 0.610 | Poorly handled |
+
+## Related
+- `docs/evidence/METRICS_CURRENT.md` — per-family benchmark table
+- `scripts/benchmark_per_family.py` — how slices are computed

docs/evidence/GUIDE_CHEAP_BASELINE.md

@@ -0,0 +1,16 @@
+# Cheap Baseline Explanation
+
+A cheap baseline is a simple heuristic that any new method must beat.
+
+## Examples in This Project
+| Baseline | What It Tests | Current Performance |
+|----------|---------------|:------------------:|
+| Charge density | How much of AUROC is charge-driven | AUROC 0.8166 (beats ensemble) |
+| Length | Sequence length alone | AUROC 0.5000 (no signal) |
+| Selectivity proxy (charge + GRAVY) | Simple selectivity heuristic | Detection AUROC 0.5744 |
+| Helix propensity | Chou-Fasman P alpha alone | Detection AUROC 0.6489 |
+
+## Why They Matter
+- Without cheap baselines, you can't tell if your complex model is adding value.
+- A model that can't beat a cheap baseline is not ready for production.
+- Cheap baselines set the minimum bar for acceptance.

docs/evidence/GUIDE_CONFIDENCE_INTERVALS.md

@@ -0,0 +1,24 @@
+# Confidence Interval Interpretation Guide
+
+How to read and interpret confidence intervals in benchmark results.
+
+## What a CI Is
+A confidence interval estimates the range where the true value likely falls.
+A 95% CI means that if the experiment were repeated 100 times, the true value
+would fall within the interval ~95 times.
+
+## How to Read
+- Narrow CI = more precise estimate (more data or less variability).
+- Wide CI = less precise estimate (less data or more variability).
+- If the CI includes 0.5, the result is not statistically significant.
+
+## Examples
+| Metric | Value | 95% CI | Interpretation |
+|--------|-------|--------|----------------|
+| AUROC | 0.7792 | 0.7505–0.8065 | Strong signal (CI above 0.5) |
+| Charge density AUROC | 0.8166 | — | Strong trivial baseline |
+| Helix_weight detection | 0.6458 | — | Moderate signal |
+
+## Related
+- All CIs are bootstrap-based (n=2000 resamples).
+- Cluster-aware CIs account for near-duplicate inflation.

docs/evidence/GUIDE_FOLDER_OWNERSHIP.md

@@ -0,0 +1,19 @@
+# Folder Ownership Guide
+
+Each top-level directory has an owner responsible for its content.
+
+| Path | Owner | Review Required |
+|------|-------|:---------------:|
+| `src/` | Maintainer | Yes |
+| `tests/` | Maintainer | Yes |
+| `docs/` | Maintainer | For new docs |
+| `configs/` | Maintainer | Yes |
+| `schemas/` | Maintainer | Yes |
+| `scripts/` | Maintainer | For new scripts |
+| `examples/` | Contributor | No |
+| `outputs/` | Generated | N/A (git-ignored) |
+
+## Rules
+- Changes outside the owned path require the path owner's review.
+- If no owner is listed for a path, the default owner is the project lead.
+- Ownership can be delegated but must be documented.

docs/evidence/GUIDE_METRIC_POLARITY.md

@@ -0,0 +1,18 @@
+# Metric Polarity Guide
+
+Metric polarity defines whether higher or lower values are better.
+
+## Common Polarity
+| Metric | Polarity | Meaning |
+|--------|:--------:|---------|
+| AUROC | Higher is better | Better discrimination |
+| AUPRC | Higher is better | Better precision-recall |
+| Recall@k | Higher is better | More positives in top k |
+| Disagreement | Lower is better | Better model consensus |
+| Charge bias | Lower is better | Less charge-dependence |
+| Uncertainty | Lower is better | More confident prediction |
+| Hemolysis risk | Lower is better | Less predicted toxicity |
+
+## Importance
+Knowing polarity prevents misinterpretation. A low AUROC is bad;
+a low disagreement is good. Always check polarity before interpreting a metric.

tests/test_benchmark_concepts_batch.py

@@ -0,0 +1,20 @@
+"""Tests for benchmark concept guides batch."""
+from pathlib import Path
+
+FILES = [
+    "CONCEPT_STALE_ARTIFACT.md",
+    "GUIDE_FOLDER_OWNERSHIP.md",
+    "GUIDE_BENCHMARK_PHILOSOPHY.md",
+    "GUIDE_BENCHMARK_CARD_WRITING.md",
+    "GUIDE_CHEAP_BASELINE.md",
+    "CONCEPT_LEAKAGE.md",
+    "GUIDE_BENCHMARK_SLICE.md",
+    "GUIDE_CONFIDENCE_INTERVALS.md",
+    "GUIDE_METRIC_POLARITY.md",
+    "GUIDE_BENCHMARK_REGRESSION_INTERP.md",
+]
+
+
+def test_all_exist():
+    for f in FILES:
+        assert Path("docs/evidence", f).exists(), f"Missing: {f}"