diff --git a/docs/evidence/CONCEPT_LEAKAGE.md b/docs/evidence/CONCEPT_LEAKAGE.md new file mode 100644 index 00000000..fc574319 --- /dev/null +++ b/docs/evidence/CONCEPT_LEAKAGE.md @@ -0,0 +1,19 @@ +# Leakage Concept Guide + +Leakage occurs when information from outside the training set influences the model. + +## Types of Leakage +| Type | Description | Example | +|------|-------------|---------| +| Near-duplicate leakage | Test sequence is similar to a training sequence | Cluster split prevents this | +| Composition leakage | Test and train share composition features | Hard to avoid; tracked by charge-matched benchmark | +| Temporal leakage | Future data influences past predictions | Time splits where metadata permits | +| Feature leakage | Features indirectly encode the target | Avoided by using transparent heuristics | + +## Detection +- Cluster split benchmarks detect near-duplicate leakage. +- Charge-matched benchmarks detect composition leakage. +- Cross-dataset benchmarks detect overfitting to database artifacts. + +## Related +- `docs/evidence/LEAKAGE_AUDIT_CHECKLIST.md` diff --git a/docs/evidence/CONCEPT_STALE_ARTIFACT.md b/docs/evidence/CONCEPT_STALE_ARTIFACT.md new file mode 100644 index 00000000..9f822dd4 --- /dev/null +++ b/docs/evidence/CONCEPT_STALE_ARTIFACT.md @@ -0,0 +1,19 @@ +# Stale Artifact Concept Guide + +An artifact is stale when it no longer reflects the current pipeline or data. + +## Signs of Staleness +- Schema version doesn't match current code +- Benchmark results don't match current pipeline output +- Config values differ from the values used to generate the artifact +- The artifact references features or commands that no longer exist + +## Causes +- Pipeline code updated without regenerating artifacts +- Schema changed without migrating existing artifacts +- Config changed without re-running dependent processes + +## Prevention +- Regenerate artifacts after pipeline changes. +- Version schemas and check version alignment in CI. +- Run `make regenerate-all` before releases. diff --git a/docs/evidence/GUIDE_BENCHMARK_CARD_WRITING.md b/docs/evidence/GUIDE_BENCHMARK_CARD_WRITING.md new file mode 100644 index 00000000..dfee627b --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_CARD_WRITING.md @@ -0,0 +1,36 @@ +# Benchmark Card Writing Guide + +How to write a benchmark card. + +## Required Sections +- **Purpose**: What the benchmark measures +- **Dataset**: What data is used and its source +- **Metrics**: What metrics are reported +- **Caveats**: What the benchmark does NOT prove +- **Reproduction**: How to run the benchmark + +## Format +```markdown +# Benchmark: [Name] + +## Purpose +One paragraph describing what this benchmark measures. + +## Dataset +- Positives: N sequences from [source] +- Negatives: N sequences from [source] +- Split method: [method] + +## Metrics +| Metric | Value | Interpretation | +|--------|-------|----------------| + +## Caveats +- Known limitations of this benchmark. +- What this benchmark does not test. + +## Reproduction +```bash +make bench-500 +``` +``` diff --git a/docs/evidence/GUIDE_BENCHMARK_PHILOSOPHY.md b/docs/evidence/GUIDE_BENCHMARK_PHILOSOPHY.md new file mode 100644 index 00000000..528f7f3b --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_PHILOSOPHY.md @@ -0,0 +1,21 @@ +# Benchmark Philosophy Page + +The project's approach to benchmarking. + +## Principles +1. Benchmarks measure discrimination, not biological truth. +2. Every benchmark has caveats that must be documented. +3. No single benchmark determines pipeline quality — use the full suite. +4. Benchmarks must be reproducible (deterministic, versioned inputs). +5. Benchmarks must be adversarial — actively look for ways the pipeline is wrong. + +## What Benchmarks Do NOT Measure +- Real-world antimicrobial activity +- Safety in humans +- Clinical utility +- Synthesis success rate +- Actual hit rate in a screening campaign + +## Related +- `docs/evidence/BENCHMARK_GOVERNANCE.md` — benchmark policies +- `docs/evidence/METRICS_CURRENT.md` — current results diff --git a/docs/evidence/GUIDE_BENCHMARK_REGRESSION_INTERP.md b/docs/evidence/GUIDE_BENCHMARK_REGRESSION_INTERP.md new file mode 100644 index 00000000..19a5c998 --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_REGRESSION_INTERP.md @@ -0,0 +1,22 @@ +# Benchmark Regression Interpretation Guide + +How to interpret a benchmark regression. + +## What to Check +1. Is the regression > 0.02 AUROC? (gate threshold) +2. Is the regression in the primary benchmark or a slice? +3. Does the regression affect all classes or just one? +4. Is the change intentional (tradeoff) or accidental (bug)? + +## Common Causes +| Cause | Pattern | Action | +|-------|---------|--------| +| Safety penalty increase | AUROC drops, safety improves | Accept tradeoff | +| Feature change | One class regresses | Isolate the class | +| Data change | Cross-dataset validation | Update benchmark | +| Bug | Unexpected drop everywhere | Fix and revert | + +## Response +- Acceptable regression (< 0.02): Document and proceed. +- Unacceptable regression (> 0.02): Investigate and fix. +- Deliberate tradeoff: Document the rationale in the decision log. diff --git a/docs/evidence/GUIDE_BENCHMARK_SLICE.md b/docs/evidence/GUIDE_BENCHMARK_SLICE.md new file mode 100644 index 00000000..3003cc06 --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_SLICE.md @@ -0,0 +1,19 @@ +# Benchmark Slice Explanation + +A benchmark slice measures performance on a specific subset of data. + +## Why Slice +- Overall AUROC can hide poor performance on important subgroups. +- Slicing reveals which candidate classes the pipeline handles well or poorly. + +## Current Slices +| Slice | Description | AUROC | Finding | +|-------|-------------|:-----:|---------| +| Highly cationic | Charge >= 4.0 | 0.958 | Well handled | +| Proline-rich | Pro fraction >= 15% | 0.586 | Poorly handled | +| Cysteine-rich | >= 2 Cys | 0.723 | Moderately handled | +| Short | <= 12 AA | 0.610 | Poorly handled | + +## Related +- `docs/evidence/METRICS_CURRENT.md` — per-family benchmark table +- `scripts/benchmark_per_family.py` — how slices are computed diff --git a/docs/evidence/GUIDE_CHEAP_BASELINE.md b/docs/evidence/GUIDE_CHEAP_BASELINE.md new file mode 100644 index 00000000..d9c3e907 --- /dev/null +++ b/docs/evidence/GUIDE_CHEAP_BASELINE.md @@ -0,0 +1,16 @@ +# Cheap Baseline Explanation + +A cheap baseline is a simple heuristic that any new method must beat. + +## Examples in This Project +| Baseline | What It Tests | Current Performance | +|----------|---------------|:------------------:| +| Charge density | How much of AUROC is charge-driven | AUROC 0.8166 (beats ensemble) | +| Length | Sequence length alone | AUROC 0.5000 (no signal) | +| Selectivity proxy (charge + GRAVY) | Simple selectivity heuristic | Detection AUROC 0.5744 | +| Helix propensity | Chou-Fasman P alpha alone | Detection AUROC 0.6489 | + +## Why They Matter +- Without cheap baselines, you can't tell if your complex model is adding value. +- A model that can't beat a cheap baseline is not ready for production. +- Cheap baselines set the minimum bar for acceptance. diff --git a/docs/evidence/GUIDE_CONFIDENCE_INTERVALS.md b/docs/evidence/GUIDE_CONFIDENCE_INTERVALS.md new file mode 100644 index 00000000..9850ea4f --- /dev/null +++ b/docs/evidence/GUIDE_CONFIDENCE_INTERVALS.md @@ -0,0 +1,24 @@ +# Confidence Interval Interpretation Guide + +How to read and interpret confidence intervals in benchmark results. + +## What a CI Is +A confidence interval estimates the range where the true value likely falls. +A 95% CI means that if the experiment were repeated 100 times, the true value +would fall within the interval ~95 times. + +## How to Read +- Narrow CI = more precise estimate (more data or less variability). +- Wide CI = less precise estimate (less data or more variability). +- If the CI includes 0.5, the result is not statistically significant. + +## Examples +| Metric | Value | 95% CI | Interpretation | +|--------|-------|--------|----------------| +| AUROC | 0.7792 | 0.7505–0.8065 | Strong signal (CI above 0.5) | +| Charge density AUROC | 0.8166 | — | Strong trivial baseline | +| Helix_weight detection | 0.6458 | — | Moderate signal | + +## Related +- All CIs are bootstrap-based (n=2000 resamples). +- Cluster-aware CIs account for near-duplicate inflation. diff --git a/docs/evidence/GUIDE_FOLDER_OWNERSHIP.md b/docs/evidence/GUIDE_FOLDER_OWNERSHIP.md new file mode 100644 index 00000000..13484af3 --- /dev/null +++ b/docs/evidence/GUIDE_FOLDER_OWNERSHIP.md @@ -0,0 +1,19 @@ +# Folder Ownership Guide + +Each top-level directory has an owner responsible for its content. + +| Path | Owner | Review Required | +|------|-------|:---------------:| +| `src/` | Maintainer | Yes | +| `tests/` | Maintainer | Yes | +| `docs/` | Maintainer | For new docs | +| `configs/` | Maintainer | Yes | +| `schemas/` | Maintainer | Yes | +| `scripts/` | Maintainer | For new scripts | +| `examples/` | Contributor | No | +| `outputs/` | Generated | N/A (git-ignored) | + +## Rules +- Changes outside the owned path require the path owner's review. +- If no owner is listed for a path, the default owner is the project lead. +- Ownership can be delegated but must be documented. diff --git a/docs/evidence/GUIDE_METRIC_POLARITY.md b/docs/evidence/GUIDE_METRIC_POLARITY.md new file mode 100644 index 00000000..09e8bdcb --- /dev/null +++ b/docs/evidence/GUIDE_METRIC_POLARITY.md @@ -0,0 +1,18 @@ +# Metric Polarity Guide + +Metric polarity defines whether higher or lower values are better. + +## Common Polarity +| Metric | Polarity | Meaning | +|--------|:--------:|---------| +| AUROC | Higher is better | Better discrimination | +| AUPRC | Higher is better | Better precision-recall | +| Recall@k | Higher is better | More positives in top k | +| Disagreement | Lower is better | Better model consensus | +| Charge bias | Lower is better | Less charge-dependence | +| Uncertainty | Lower is better | More confident prediction | +| Hemolysis risk | Lower is better | Less predicted toxicity | + +## Importance +Knowing polarity prevents misinterpretation. A low AUROC is bad; +a low disagreement is good. Always check polarity before interpreting a metric. diff --git a/tests/test_benchmark_concepts_batch.py b/tests/test_benchmark_concepts_batch.py new file mode 100644 index 00000000..35d9506d --- /dev/null +++ b/tests/test_benchmark_concepts_batch.py @@ -0,0 +1,20 @@ +"""Tests for benchmark concept guides batch.""" +from pathlib import Path + +FILES = [ + "CONCEPT_STALE_ARTIFACT.md", + "GUIDE_FOLDER_OWNERSHIP.md", + "GUIDE_BENCHMARK_PHILOSOPHY.md", + "GUIDE_BENCHMARK_CARD_WRITING.md", + "GUIDE_CHEAP_BASELINE.md", + "CONCEPT_LEAKAGE.md", + "GUIDE_BENCHMARK_SLICE.md", + "GUIDE_CONFIDENCE_INTERVALS.md", + "GUIDE_METRIC_POLARITY.md", + "GUIDE_BENCHMARK_REGRESSION_INTERP.md", +] + + +def test_all_exist(): + for f in FILES: + assert Path("docs/evidence", f).exists(), f"Missing: {f}"