From 662d1001cfdf28b4db4fa8bf39aee429f8fc3256 Mon Sep 17 00:00:00 2001 From: OpenCode Date: Tue, 7 Jul 2026 10:45:37 +0700 Subject: [PATCH] Add limitations overview and quickstart guide Closes #420, Closes #424 - docs/evidence/LIMITATIONS_OVERVIEW.md -- honest limitations across 6 categories (model, data, benchmark, calibration, safety, reproducibility) - docs/getting-started/QUICKSTART.md -- 15-minute path from zero to demo - 8 tests total --- docs/evidence/LIMITATIONS_OVERVIEW.md | 74 +++++++++++++++++++++++++++ docs/getting-started/QUICKSTART.md | 68 ++++++++++++++++++++++++ tests/test_limitations_overview.py | 26 ++++++++++ tests/test_quickstart.py | 22 ++++++++ 4 files changed, 190 insertions(+) create mode 100644 docs/evidence/LIMITATIONS_OVERVIEW.md create mode 100644 docs/getting-started/QUICKSTART.md create mode 100644 tests/test_limitations_overview.py create mode 100644 tests/test_quickstart.py diff --git a/docs/evidence/LIMITATIONS_OVERVIEW.md b/docs/evidence/LIMITATIONS_OVERVIEW.md new file mode 100644 index 00000000..e9175644 --- /dev/null +++ b/docs/evidence/LIMITATIONS_OVERVIEW.md @@ -0,0 +1,74 @@ +# Honest Limitations Overview + +OpenAMP Foundry is a serious computational project. It also has real +limitations. This page collects them in one place so readers and reviewers +can assess credibility without hunting through 50 documents. + +--- + +## 1. Model Limitations + +| Limitation | Detail | Status | +|---|---|---| +| Charge-dominated signal | Pipeline AUROC 0.7792 is inflated by charge density. Under exact charge-balanced synthetic controls, AUROC falls to 0.5103 (near chance). | Documented. `charge_bias` per candidate available. | +| Amphipathic-helix bias | Activity scorer assumes helical structure. Proline-rich, cysteine-rich (beta-sheet), and short AMPs are systematically undervalued (proline_rich AUROC 0.586 vs highly_cationic 0.958). | Per-family benchmark exists. Structural class floor in pilot-panel. | +| No sequence-order awareness | 24/31 features are composition-only. Only hydrophobic moment and dipeptide order score capture sequence order. Strict triage (scrambled control) AUROC 0.572. | Dipeptide order score added. Rich modeling deferred. | +| Simulation modules experimental | MembraneProxy and StructureProxy do not beat their cheapest heuristic baselines (0/4 signals). Weighted mode blocked by gate. | Info-only in `--simulation-mode info`. | + +## 2. Data Limitations + +| Limitation | Detail | Status | +|---|---|---| +| No wet-lab data | All scores are computational predictions. No antimicrobial activity has been demonstrated. | All output labeled as computational. | +| Benchmark decoys are synthetic | The 500 Swiss-Prot frequency decoys are not biologically validated negatives. Charge-balanced controls are synthetic. | Adversarial benchmarks track this gap. | +| Hemolysis reference small | 239 peptides (45 hemolytic, 125 selective after filtering) — moderate statistical power. HC50 values are literature-sourced with high inter-assay variability. | All confidence intervals reported. | + +## 3. Benchmark Limitations + +| Limitation | Detail | Status | +|---|---|---| +| AMP-vs-decoy is not the real task | Pipeline optimizes for multi-objective selection (active + safe + novel + synthesizable), not binary AMP/non-AMP discrimination. Charge density alone beats ensemble on the decoy task. | Easy baseline, charge-matched, and within-AMP benchmarks document this gap. | +| Within-AMP detection is weak | Best scorer (rich_selectivity) AUROC 0.7453 for hemolysis detection. No existing scorer passes strict triage (composition-matched decoys) at all three pairwise comparisons. | gate_triage passes standard triage; strict triage is the honest bar. | +| Small-k precision is high but unvalidated | Top-20 precision is 1.000 on the benchmark, but base rate in real screening is unknown (likely 1-10%). | Precision@k calibrated but caveated. | + +## 4. Calibration Limitations + +| Limitation | Detail | Status | +|---|---|---| +| No real calibration data | Recalibration pipeline (intake, gate, engine) is complete but has never been tested on real wet-lab results. | Synthetic data used for all tests. | +| Recalibration policy untested | `configs/recalibration_policy.yaml` defines 7 conditions and 5 prohibitions. None have been exercised against real data. | Gate logic is tested. Real exercise is pending. | + +## 5. Safety and Review Limitations + +| Limitation | Detail | Status | +|---|---|---| +| Safety scorer has blind spots | Melittin scores Safety=1.0 despite known hemolysis. Hemolysis cannot be predicted by current physicochemical scorers. | Mandatory hemolysis assay for all candidates. | +| No external expert review completed | Expert review template exists. No qualified domain expert has reviewed the candidate panel. | Template is ready. Review is pending. | +| Dual-use review not yet performed | Safety review template and dual-use checklist exist. No formal biosecurity review has been conducted. | Governance framework is in place. Review is pending. | + +## 6. Reproducibility Limitations + +| Limitation | Detail | Status | +|---|---|---| +| Some benchmark outputs are git-ignored | Generated files in `outputs/` are not committed. Regeneration requires running the pipeline. | `make regenerate-all` reproduces all outputs. | +| Random seeds not tracked per-run | Benchmark scripts use deterministic seeds, but run-level seed tracking is not automated. | Manual seed documentation in each script. | + +--- + +## What This Does Not Mean + +- **Not a failure.** Documentation of limitations is a sign of scientific maturity, not weakness. +- **Not an excuse.** Each limitation has a documented plan or mitigation. +- **Not a comprehensive risk register.** See `docs/evidence/RISK_REGISTER.md` for operational risks. + +--- + +## Related Documents + +- `docs/evidence/METRICS_CURRENT.md` — current benchmark numbers (single source of truth) +- `docs/evidence/BENCHMARK_GOVERNANCE.md` — benchmark policies +- `docs/evidence/SIMULATION_BENCHMARK.md` — simulation module evaluation +- `docs/evidence/CALIBRATION_POLICY.md` — recalibration gate rules +- `docs/evidence/EVIDENCE_CERTIFICATE.md` — per-candidate evidence schema +- `docs/review/EXPERT_REVIEW_PACK.md` — expert review guidelines +- `docs/trust/RISK_REGISTER.md` — operational risk register diff --git a/docs/getting-started/QUICKSTART.md b/docs/getting-started/QUICKSTART.md new file mode 100644 index 00000000..52804287 --- /dev/null +++ b/docs/getting-started/QUICKSTART.md @@ -0,0 +1,68 @@ +# 15-Minute Quickstart + +**Goal:** Go from zero to running the full demo pipeline in 15 minutes. + +--- + +## Prerequisites (2 min) + +- Python 3.11+ +- git + +## Setup (5 min) + +```bash +git clone https://github.com/Open-Problem-Lab/OpenAMP-Foundry.git +cd OpenAMP-Foundry +python3 -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" +``` + +## Run the Demo (3 min) + +```bash +make demo +``` + +This will rank 10 demo candidates, generate evidence certificates in `outputs/evidence/`, and produce a report at `outputs/demo_report.md`. + +## View Results (2 min) + +```bash +# View the report +cat outputs/demo_report.md + +# View a candidate's evidence certificate +cat outputs/evidence/AMPF-000001.json + +# View the run manifest +cat outputs/run_manifest.json +``` + +## Run with Simulation Scores (2 min) + +```bash +PYTHONPATH=src python3 -m openamp_foundry.cli rank \ + --candidates examples/sequences/demo_candidates.csv \ + --references examples/known_reference/demo_known_amps.csv \ + --out outputs/demo_ranked.jsonl \ + --report outputs/demo_report.md \ + --simulation-mode info +``` + +This adds membrane binding and structure proxy scores to every candidate. + +## Run the Full Test Suite (1 min) + +```bash +python3 -m pytest -q --tb=short +``` + +> **Note:** Full suite takes ~80s. Use `-k` to run specific tests. + +## Next Steps + +- Read [`docs/evidence/LIMITATIONS_OVERVIEW.md`](docs/evidence/LIMITATIONS_OVERVIEW.md) for honest limitations +- Read [`CONTRIBUTING.md`](../CONTRIBUTING.md) for contribution guidelines +- Read [`AGENTS.md`](../AGENTS.md) for the agent operating contract diff --git a/tests/test_limitations_overview.py b/tests/test_limitations_overview.py new file mode 100644 index 00000000..fe04d90b --- /dev/null +++ b/tests/test_limitations_overview.py @@ -0,0 +1,26 @@ +"""Tests for limitations overview doc.""" +from pathlib import Path + + +def test_file_exists(): + assert Path("docs/evidence/LIMITATIONS_OVERVIEW.md").exists() + + +def test_has_required_sections(): + text = Path("docs/evidence/LIMITATIONS_OVERVIEW.md").read_text() + for section in ["Model Limitations", "Data Limitations", "Benchmark Limitations", + "Calibration Limitations", "Safety and Review Limitations", + "Reproducibility Limitations"]: + assert section in text, f"Missing section: {section}" + + +def test_does_not_overclaim(): + text = Path("docs/evidence/LIMITATIONS_OVERVIEW.md").read_text() + assert "proven" not in text.lower() + assert "No antimicrobial activity has been demonstrated" in text + + +def test_links_to_related_docs(): + text = Path("docs/evidence/LIMITATIONS_OVERVIEW.md").read_text() + assert "METRICS_CURRENT.md" in text + assert "RISK_REGISTER.md" in text diff --git a/tests/test_quickstart.py b/tests/test_quickstart.py new file mode 100644 index 00000000..c86d0479 --- /dev/null +++ b/tests/test_quickstart.py @@ -0,0 +1,22 @@ +"""Tests for quickstart guide.""" +from pathlib import Path + + +def test_file_exists(): + assert Path("docs/getting-started/QUICKSTART.md").exists() + + +def test_has_demo_command(): + text = Path("docs/getting-started/QUICKSTART.md").read_text() + assert "make demo" in text + + +def test_has_setup_instructions(): + text = Path("docs/getting-started/QUICKSTART.md").read_text() + assert "pip install" in text + assert "git clone" in text + + +def test_has_next_steps(): + text = Path("docs/getting-started/QUICKSTART.md").read_text() + assert "Next Steps" in text