diff --git a/docs/evidence/GUIDE_BENCHMARK_REPORT_READING.md b/docs/evidence/GUIDE_BENCHMARK_REPORT_READING.md new file mode 100644 index 00000000..ab65793d --- /dev/null +++ b/docs/evidence/GUIDE_BENCHMARK_REPORT_READING.md @@ -0,0 +1,18 @@ +# Benchmark Report Reading Guide + +How to read a benchmark report. + +## Key Sections +| Section | What to Look For | Action if Concerning | +|---------|------------------|---------------------| +| AUROC | Overall discrimination | Check if > 0.70 | +| Per-family AUROC | Class-specific performance | Investigate weak classes | +| CI width | Precision of estimate | Wider CI = less data | +| Simulation verdict | Module status | Should be NO_IMPROVEMENT | +| Gate verdict | Regression check | Should pass | + +## Red Flags +- AUROC drop > 0.02 from baseline. +- Any gate check failing. +- Unexpected simulation IMPROVEMENT. +- CI lower bound below 0.65. diff --git a/docs/evidence/GUIDE_CALIBRATION_ANTI_PATTERNS.md b/docs/evidence/GUIDE_CALIBRATION_ANTI_PATTERNS.md new file mode 100644 index 00000000..3fa17051 --- /dev/null +++ b/docs/evidence/GUIDE_CALIBRATION_ANTI_PATTERNS.md @@ -0,0 +1,15 @@ +# Calibration Anti-Patterns + +Common mistakes in calibration workflow. + +| Anti-Pattern | Problem | Solution | +|-------------|---------|----------| +| Calibrating on synthetic data | No signal from real biology | Only calibrate with real lab results | +| Post-hoc threshold changes | Changes success definition | Lock thresholds before results arrive | +| Ignoring the gate | Skips safety checks | Gate is not bypassable by design | +| Over-fitting to small data | 1-2 results don't justify weight changes | Minimum cohort size enforced | +| Calibrating without human review | Automates decision-making | Human review always required | + +## Related +- `configs/recalibration_policy.yaml` — the gate policy +- `docs/evidence/CALIBRATION_POLICY.md` — full policy diff --git a/docs/evidence/GUIDE_CLI_ARCHITECTURE.md b/docs/evidence/GUIDE_CLI_ARCHITECTURE.md new file mode 100644 index 00000000..622c9d7b --- /dev/null +++ b/docs/evidence/GUIDE_CLI_ARCHITECTURE.md @@ -0,0 +1,22 @@ +# CLI Architecture Explainer + +How the CLI is structured and how commands are organized. + +## Structure +``` +openamp-foundry — main CLI entry point + rank — score and rank candidates + validate — validate evidence certificates + bench — run benchmarks + leakage — check for near-duplicate leakage + baseline — baseline comparison + cluster-split — cluster-aware bootstrap + pilot-panel — select pilot panel candidates + calibration-intake — ingest lab results + recalibration-gate — evaluate recalibration permission +``` + +## Pattern +Each command is a subparser in `src/openamp_foundry/cli/main.py`. +The handler calls a function from the relevant package. +New commands should follow the same pattern. diff --git a/docs/evidence/GUIDE_EVIDENCE_DOWNGRADE.md b/docs/evidence/GUIDE_EVIDENCE_DOWNGRADE.md new file mode 100644 index 00000000..ac5e97c6 --- /dev/null +++ b/docs/evidence/GUIDE_EVIDENCE_DOWNGRADE.md @@ -0,0 +1,18 @@ +# Evidence Downgrade Guide + +When and how to lower the evidence level of a claim. + +## When to Downgrade +- New information contradicts the previous evidence. +- The evidence was found to have methodological flaws. +- The evidence cannot be reproduced. +- Caveats were discovered that were previously unknown. + +## Levels +See PROOF_LADDER.md for level definitions. Downgrade one level at a time. + +## Process +1. Document the reason for the downgrade in the decision log. +2. Update the affected claims and documents. +3. Notify anyone who relied on the previous evidence level. +4. If the downgrade affects a release, block the release until resolved. diff --git a/docs/evidence/GUIDE_LEARNING_FROM_NEGATIVES.md b/docs/evidence/GUIDE_LEARNING_FROM_NEGATIVES.md new file mode 100644 index 00000000..9f8f3669 --- /dev/null +++ b/docs/evidence/GUIDE_LEARNING_FROM_NEGATIVES.md @@ -0,0 +1,20 @@ +# Negative-Result Learning Guide + +How to extract value from negative results. + +## What Negative Results Teach +- Which candidate classes are overpredicted +- Which safety markers are insufficient +- Which benchmark assumptions need revision +- Where the pipeline needs improvement + +## Process +1. Record in the negative result archive. +2. Analyze whether the failure was model error, data error, or genuine negative. +3. Document findings in the decision log. +4. If multiple negatives point to the same issue, prioritize fixing it. + +## Rules +- Negative results are calibration data, not failures. +- Never discard a negative result. +- Never change weights based on a single negative result. diff --git a/docs/evidence/GUIDE_REVIEWER_ROLE.md b/docs/evidence/GUIDE_REVIEWER_ROLE.md new file mode 100644 index 00000000..9efc3f5a --- /dev/null +++ b/docs/evidence/GUIDE_REVIEWER_ROLE.md @@ -0,0 +1,17 @@ +# Reviewer Role Guide + +Responsibilities of different reviewer roles. + +| Role | Can Approve | Required For | +|------|:-----------:|--------------| +| Contributor | No | Can comment, suggest changes | +| Reviewer | Low-risk PRs | Medium-risk PRs | +| Maintainer | High-risk PRs | Policy, safety, calibration changes | +| Safety reviewer | Safety changes | Safety policy updates | +| Project lead | All changes | Final approval on releases | + +## Guidelines +- Reviewers should match their expertise to the change. +- Safety changes always require a safety reviewer. +- High-risk changes require at least 2 reviewers. +- The project lead has final authority on disputed reviews. diff --git a/docs/evidence/GUIDE_REVIEW_PACKET_OVERVIEW.md b/docs/evidence/GUIDE_REVIEW_PACKET_OVERVIEW.md new file mode 100644 index 00000000..fe70d8b2 --- /dev/null +++ b/docs/evidence/GUIDE_REVIEW_PACKET_OVERVIEW.md @@ -0,0 +1,19 @@ +# Review Packet Overview + +A review packet contains everything a reviewer needs to evaluate a claim. + +## Required Contents +- Candidate list with scores +- Evidence certificates for selected candidates +- LIMITATIONS_OVERVIEW.md +- The specific claim or question being reviewed + +## Optional Contents +- METRICS_CURRENT.md for benchmark context +- Simulation scores (if used) +- Expert review template + +## Rules +- Packets must not overclaim. +- Packets must state which evidence is computational vs biological. +- Reviewers must be told this is a dry-lab nomination only. diff --git a/docs/evidence/GUIDE_SCHEMA_ARCHITECTURE.md b/docs/evidence/GUIDE_SCHEMA_ARCHITECTURE.md new file mode 100644 index 00000000..3f6742a5 --- /dev/null +++ b/docs/evidence/GUIDE_SCHEMA_ARCHITECTURE.md @@ -0,0 +1,20 @@ +# Schema Architecture Explainer + +How schemas are structured and maintained. + +## Schema Locations +| Schema | Location | Purpose | +|--------|----------|---------| +| candidate.schema.json | schemas/ | Evidence certificates | +| lab_result.schema.json | schemas/ | Lab assay results | +| batch_report.schema.json | schemas/ | Batch reports | +| run_manifest.schema.json | schemas/ | Run manifests | +| recalibration_report.schema.json | schemas/ | Calibration reports | +| decision_log.schema.json | schemas/ | Human review decisions | +| panel_csv.schema.json | schemas/ | Panel CSVs | + +## Rules +- All schemas use Draft 2020-12. +- Required fields must be documented in the schema. +- Adding fields is backward-compatible; removing fields is not. +- Schema changes require a corresponding version bump. diff --git a/docs/evidence/NARRATIVE_FIRST_PRINCIPLES.md b/docs/evidence/NARRATIVE_FIRST_PRINCIPLES.md new file mode 100644 index 00000000..51c05fc6 --- /dev/null +++ b/docs/evidence/NARRATIVE_FIRST_PRINCIPLES.md @@ -0,0 +1,20 @@ +# First-Principles Project Narrative + +OpenAMP Foundry exists to answer one question: Can an open, reproducible, +safety-constrained computational pipeline nominate antimicrobial peptide +candidates that are worth testing in a lab? + +## Why This Matters +Antimicrobial resistance is a global threat. Current discovery methods are slow +and expensive. Computation can help triage candidates, but only if the pipeline +is honest about its limitations. + +## Our Approach +1. Transparent heuristics, not black-box models. +2. Benchmark honesty over impressive metrics. +3. Safety constraints built into the scoring. +4. Reproducibility as a first-class requirement. +5. Human review as a mandatory gate. + +## Our Non-Goal +We do not claim that computation replaces biology. The lab is the judge. diff --git a/docs/evidence/PLAYBOOK_BENCHMARK_DRIFT.md b/docs/evidence/PLAYBOOK_BENCHMARK_DRIFT.md new file mode 100644 index 00000000..f8a2e4ef --- /dev/null +++ b/docs/evidence/PLAYBOOK_BENCHMARK_DRIFT.md @@ -0,0 +1,20 @@ +# Benchmark Drift Playbook + +How to detect and respond to benchmark drift. + +## Detection +- Run `make bench-gate` to check regression gates. +- Compare current AUROC to METRICS_CURRENT.md values. +- Check per-family benchmarks for class-specific drift. + +## Response +| Drift Magnitude | Action | +|:---------------:|--------| +| < 0.01 | Document, no action needed | +| 0.01 - 0.02 | Investigate, document | +| > 0.02 | Block release, fix required | + +## Prevention +- Run benchmarks after every scoring change. +- Gate benchmark regressions in CI. +- Maintain benchmark history in METRICS_CURRENT.md. diff --git a/tests/test_final_issues_batch.py b/tests/test_final_issues_batch.py new file mode 100644 index 00000000..6262d5c2 --- /dev/null +++ b/tests/test_final_issues_batch.py @@ -0,0 +1,20 @@ +"""Tests for final remaining issues batch.""" +from pathlib import Path + +FILES = [ + "PLAYBOOK_BENCHMARK_DRIFT.md", + "GUIDE_BENCHMARK_REPORT_READING.md", + "GUIDE_LEARNING_FROM_NEGATIVES.md", + "GUIDE_EVIDENCE_DOWNGRADE.md", + "GUIDE_CALIBRATION_ANTI_PATTERNS.md", + "GUIDE_REVIEW_PACKET_OVERVIEW.md", + "GUIDE_REVIEWER_ROLE.md", + "NARRATIVE_FIRST_PRINCIPLES.md", + "GUIDE_CLI_ARCHITECTURE.md", + "GUIDE_SCHEMA_ARCHITECTURE.md", +] + + +def test_all_exist(): + for f in FILES: + assert Path("docs/evidence", f).exists(), f"Missing: {f}"