Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/evidence/GUIDE_BENCHMARK_REPORT_READING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Benchmark Report Reading Guide

How to read a benchmark report.

## Key Sections
| Section | What to Look For | Action if Concerning |
|---------|------------------|---------------------|
| AUROC | Overall discrimination | Check if > 0.70 |
| Per-family AUROC | Class-specific performance | Investigate weak classes |
| CI width | Precision of estimate | Wider CI = less data |
| Simulation verdict | Module status | Should be NO_IMPROVEMENT |
| Gate verdict | Regression check | Should pass |

## Red Flags
- AUROC drop > 0.02 from baseline.
- Any gate check failing.
- Unexpected simulation IMPROVEMENT.
- CI lower bound below 0.65.
15 changes: 15 additions & 0 deletions docs/evidence/GUIDE_CALIBRATION_ANTI_PATTERNS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Calibration Anti-Patterns

Common mistakes in calibration workflow.

| Anti-Pattern | Problem | Solution |
|-------------|---------|----------|
| Calibrating on synthetic data | No signal from real biology | Only calibrate with real lab results |
| Post-hoc threshold changes | Changes success definition | Lock thresholds before results arrive |
| Ignoring the gate | Skips safety checks | Gate is not bypassable by design |
| Over-fitting to small data | 1-2 results don't justify weight changes | Minimum cohort size enforced |
| Calibrating without human review | Automates decision-making | Human review always required |

## Related
- `configs/recalibration_policy.yaml` — the gate policy
- `docs/evidence/CALIBRATION_POLICY.md` — full policy
22 changes: 22 additions & 0 deletions docs/evidence/GUIDE_CLI_ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# CLI Architecture Explainer

How the CLI is structured and how commands are organized.

## Structure
```
openamp-foundry — main CLI entry point
rank — score and rank candidates
validate — validate evidence certificates
bench — run benchmarks
leakage — check for near-duplicate leakage
baseline — baseline comparison
cluster-split — cluster-aware bootstrap
pilot-panel — select pilot panel candidates
calibration-intake — ingest lab results
recalibration-gate — evaluate recalibration permission
```

## Pattern
Each command is a subparser in `src/openamp_foundry/cli/main.py`.
The handler calls a function from the relevant package.
New commands should follow the same pattern.
18 changes: 18 additions & 0 deletions docs/evidence/GUIDE_EVIDENCE_DOWNGRADE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Evidence Downgrade Guide

When and how to lower the evidence level of a claim.

## When to Downgrade
- New information contradicts the previous evidence.
- The evidence was found to have methodological flaws.
- The evidence cannot be reproduced.
- Caveats were discovered that were previously unknown.

## Levels
See PROOF_LADDER.md for level definitions. Downgrade one level at a time.

## Process
1. Document the reason for the downgrade in the decision log.
2. Update the affected claims and documents.
3. Notify anyone who relied on the previous evidence level.
4. If the downgrade affects a release, block the release until resolved.
20 changes: 20 additions & 0 deletions docs/evidence/GUIDE_LEARNING_FROM_NEGATIVES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Negative-Result Learning Guide

How to extract value from negative results.

## What Negative Results Teach
- Which candidate classes are overpredicted
- Which safety markers are insufficient
- Which benchmark assumptions need revision
- Where the pipeline needs improvement

## Process
1. Record in the negative result archive.
2. Analyze whether the failure was model error, data error, or genuine negative.
3. Document findings in the decision log.
4. If multiple negatives point to the same issue, prioritize fixing it.

## Rules
- Negative results are calibration data, not failures.
- Never discard a negative result.
- Never change weights based on a single negative result.
17 changes: 17 additions & 0 deletions docs/evidence/GUIDE_REVIEWER_ROLE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Reviewer Role Guide

Responsibilities of different reviewer roles.

| Role | Can Approve | Required For |
|------|:-----------:|--------------|
| Contributor | No | Can comment, suggest changes |
| Reviewer | Low-risk PRs | Medium-risk PRs |
| Maintainer | High-risk PRs | Policy, safety, calibration changes |
| Safety reviewer | Safety changes | Safety policy updates |
| Project lead | All changes | Final approval on releases |

## Guidelines
- Reviewers should match their expertise to the change.
- Safety changes always require a safety reviewer.
- High-risk changes require at least 2 reviewers.
- The project lead has final authority on disputed reviews.
19 changes: 19 additions & 0 deletions docs/evidence/GUIDE_REVIEW_PACKET_OVERVIEW.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Review Packet Overview

A review packet contains everything a reviewer needs to evaluate a claim.

## Required Contents
- Candidate list with scores
- Evidence certificates for selected candidates
- LIMITATIONS_OVERVIEW.md
- The specific claim or question being reviewed

## Optional Contents
- METRICS_CURRENT.md for benchmark context
- Simulation scores (if used)
- Expert review template

## Rules
- Packets must not overclaim.
- Packets must state which evidence is computational vs biological.
- Reviewers must be told this is a dry-lab nomination only.
20 changes: 20 additions & 0 deletions docs/evidence/GUIDE_SCHEMA_ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Schema Architecture Explainer

How schemas are structured and maintained.

## Schema Locations
| Schema | Location | Purpose |
|--------|----------|---------|
| candidate.schema.json | schemas/ | Evidence certificates |
| lab_result.schema.json | schemas/ | Lab assay results |
| batch_report.schema.json | schemas/ | Batch reports |
| run_manifest.schema.json | schemas/ | Run manifests |
| recalibration_report.schema.json | schemas/ | Calibration reports |
| decision_log.schema.json | schemas/ | Human review decisions |
| panel_csv.schema.json | schemas/ | Panel CSVs |

## Rules
- All schemas use Draft 2020-12.
- Required fields must be documented in the schema.
- Adding fields is backward-compatible; removing fields is not.
- Schema changes require a corresponding version bump.
20 changes: 20 additions & 0 deletions docs/evidence/NARRATIVE_FIRST_PRINCIPLES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# First-Principles Project Narrative

OpenAMP Foundry exists to answer one question: Can an open, reproducible,
safety-constrained computational pipeline nominate antimicrobial peptide
candidates that are worth testing in a lab?

## Why This Matters
Antimicrobial resistance is a global threat. Current discovery methods are slow
and expensive. Computation can help triage candidates, but only if the pipeline
is honest about its limitations.

## Our Approach
1. Transparent heuristics, not black-box models.
2. Benchmark honesty over impressive metrics.
3. Safety constraints built into the scoring.
4. Reproducibility as a first-class requirement.
5. Human review as a mandatory gate.

## Our Non-Goal
We do not claim that computation replaces biology. The lab is the judge.
20 changes: 20 additions & 0 deletions docs/evidence/PLAYBOOK_BENCHMARK_DRIFT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Benchmark Drift Playbook

How to detect and respond to benchmark drift.

## Detection
- Run `make bench-gate` to check regression gates.
- Compare current AUROC to METRICS_CURRENT.md values.
- Check per-family benchmarks for class-specific drift.

## Response
| Drift Magnitude | Action |
|:---------------:|--------|
| < 0.01 | Document, no action needed |
| 0.01 - 0.02 | Investigate, document |
| > 0.02 | Block release, fix required |

## Prevention
- Run benchmarks after every scoring change.
- Gate benchmark regressions in CI.
- Maintain benchmark history in METRICS_CURRENT.md.
20 changes: 20 additions & 0 deletions tests/test_final_issues_batch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
"""Tests for final remaining issues batch."""
from pathlib import Path

FILES = [
"PLAYBOOK_BENCHMARK_DRIFT.md",
"GUIDE_BENCHMARK_REPORT_READING.md",
"GUIDE_LEARNING_FROM_NEGATIVES.md",
"GUIDE_EVIDENCE_DOWNGRADE.md",
"GUIDE_CALIBRATION_ANTI_PATTERNS.md",
"GUIDE_REVIEW_PACKET_OVERVIEW.md",
"GUIDE_REVIEWER_ROLE.md",
"NARRATIVE_FIRST_PRINCIPLES.md",
"GUIDE_CLI_ARCHITECTURE.md",
"GUIDE_SCHEMA_ARCHITECTURE.md",
]


def test_all_exist():
for f in FILES:
assert Path("docs/evidence", f).exists(), f"Missing: {f}"
Loading