diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md index e6acade8..83b21cdc 100644 --- a/docs/50_LOOP_PLAN.md +++ b/docs/50_LOOP_PLAN.md @@ -133,7 +133,7 @@ Prepare for real wet-lab partners. Make the pack reviewable, the evidence tracea | 45 ✅ | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | `scripts/build_lab_batch_pack.py` now writes `chain_of_custody.json` + `MANIFEST.json` into the lab batch pack, including SHA-256 hashes for `panel.csv`, the ordered candidate list, each candidate sequence, and each evidence certificate. `--verify-pack` re-checks archive integrity end to end. v0.5.65 | Verification script passes on synthetic data and fails on tampered packs | | 46 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return | | 47 ✅ | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: validates JSON files against `schemas/lab_result.schema.json`, reports per-file errors, detects control failures. v0.5.67 | 8 tests, 2008 total. Detects missing fields, invalid JSON, control failures | -| 48 | No public-data release package. If results are publishable, what goes in the supplement? | `docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings | +| 48 ✅ | No public-data release package. If results are publishable, what goes in the supplement? | `docs/PUBLICATION_PACK.md`: 12-section checklist covering pipeline code, input data, scores, benchmarks, panel, wet-lab results, calibration, reproducibility, negative results, safety, README, preprint prep. v0.5.68 | Pack is complete enough for a reviewer to reproduce all findings | | 49 | **Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty | `make full-reproducibility-report` succeeds; all 49 gates pass; report is publishable as a preprint | **Phase 4 exit criteria:** @@ -242,7 +242,7 @@ Phase 4: In progress — Loop 43 ✅ (Loops 40–49) | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available | | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers | -**Next loop:** Loop 48 — public-data release package. +**Next loop:** Loop 49 — culmination reproducibility report. **Phase 2 exit criteria (all 5 met ✅):** - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index cb8f4a34..89d0e406 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-06 (data-return validation — v0.5.67) +> **Last updated:** 2026-07-06 (publication pack — v0.5.68) > **New in v0.5.65:** `scripts/build_lab_batch_pack.py` now writes `chain_of_custody.json` and `MANIFEST.json` into lab batch packs. The custody file includes SHA-256 hashes for `panel.csv`, the ordered candidate list, each candidate sequence, and each evidence certificate. `--verify-pack` verifies archive integrity and detects tampering. These hashes prove identity/integrity only, not biological activity, safety, or synthesis success. 5 tests. > **New in v0.5.64:** `schemas/decision_log.schema.json` — 12-field JSON Schema for human review decisions. Covers 9 decision types from AGENTS.md §8. Dissent conditional. 11 tests. 2000 total. > **New in v0.5.63:** `scripts/build_lab_batch_pack.py` — generates single zip with candidate CSV, 36 evidence certs, protocol refs, controls manifest, data return template. `make lab-batch-pack`. 10 tests. 1989 total. diff --git a/docs/PUBLICATION_PACK.md b/docs/PUBLICATION_PACK.md new file mode 100644 index 00000000..0fe921c8 --- /dev/null +++ b/docs/PUBLICATION_PACK.md @@ -0,0 +1,156 @@ +# Publication Pack Checklist + +**Purpose:** What to include in a public release of OpenAMP Foundry results. +A reviewer should be able to reproduce all findings from the contents of this pack. + +--- + +## 1. Pipeline Code and Version + +- [ ] Git commit SHA of the pipeline version used. +- [ ] `configs/pipeline.yaml` and `configs/phase3.yaml` used for scoring. +- [ ] List of all Python package dependencies with versions (`pip freeze` or `pyproject.toml`). +- [ ] Instructions to recreate the Python environment (`make install` or `pip install -e .[dev]`). + +--- + +## 2. Input Data + +- [ ] Candidate sequences (FASTA or CSV). +- [ ] Reference AMP database version and download instructions (APD6, DRAMP, UniProt). +- [ ] Decoy sequences used for benchmarking. +- [ ] Dataset cards for each dataset (source, license, date, known biases). +- [ ] Input hashes (SHA-256) for reproducibility verification. + +--- + +## 3. Scoring and Ranking Outputs + +- [ ] Per-candidate scores (JSONL or CSV): ensemble, activity, safety, synthesis, novelty. +- [ ] Per-candidate simulation scores (if `--simulation-mode info` was used). +- [ ] Selection rule used (`configs/pipeline.yaml` weights). +- [ ] Ranked candidate list with selection reasons. +- [ ] Evidence certificates (JSON) for all selected candidates. + +--- + +## 4. Benchmark Results + +- [ ] Expanded benchmark (500 AMP + 500 decoy): AUROC, AUPRC, CI. +- [ ] Cluster-split benchmark with cluster-aware CI. +- [ ] Easy baseline comparison (charge density vs pipeline). +- [ ] Cross-dataset generalization (DRAMP). +- [ ] Per-family breakdown (6 structural classes). +- [ ] Precision@k calibration. +- [ ] Simulation ablation results (AMP-vs-decoy + within-AMP). +- [ ] Simulation cheap-baseline comparison (0/4 signals beat baselines). +- [ ] Benchmark gate verdict (all metrics within tolerance). + +--- + +## 5. Candidate Panel + +- [ ] Final panel CSV with candidate IDs, sequences, scores, and selection reasons. +- [ ] Diversity clustering report. +- [ ] Novelty report against known AMP references (BLOSUM62 local alignment). +- [ ] Broad novelty check results (72-AMP or 27k-sequence reference). +- [ ] Toxicity / hemolysis risk report. +- [ ] Synthesis feasibility report. +- [ ] Pre-registered selection rule document. +- [ ] Evidence certificates for all selected candidates. + +--- + +## 6. Wet-Lab Results (if available) + +- [ ] Raw lab result JSONs (one per candidate per assay). +- [ ] Pass/fail criteria check output (`check_wave1_pass_fail.py` verdict). +- [ ] Primary analysis results (MIC ≤ 32 hit rate). +- [ ] Secondary analysis (selectivity index, TI > 10). +- [ ] Exploratory analysis (per-family, structural class, calibration). +- [ ] Positive control results. +- [ ] Negative results archive entries. +- [ ] Lab batch pack zip used. + +--- + +## 7. Calibration and Active Learning (if available) + +- [ ] Calibration intake report (`calibration-intake` output). +- [ ] Recalibration gate verdict (`recalibration-gate` output). +- [ ] Weight update proposal (if gate passed). +- [ ] Decision log entries for each human review decision. +- [ ] Batch-2 selection manifest (if applicable). + +--- + +## 8. Reproducibility Evidence + +- [ ] Run manifest JSON for each pipeline run (input hashes, config hash, outputs). +- [ ] `make regenerated-all` output (all derived outputs from versioned inputs). +- [ ] Benchmark gate output (`outputs/metrics_snapshot.json`). +- [ ] Pipeline version string (`openamp-foundry --version`). +- [ ] Random seed documentation (all benchmark scripts use deterministic seeds). + +--- + +## 9. Negative Results + +- [ ] Pre-selection rejects (candidates eliminated by gates). +- [ ] Selected-but-untested candidates. +- [ ] Lab-tested inactives (MIC > 32 µg/mL). +- [ ] Lab-tested toxic (TI < 2). +- [ ] Control failures. +- [ ] Simulation ablation negative results (0/4 signals beat baselines). +- [ ] All entries in `outputs/negative_result_archive.csv` format. + +--- + +## 10. Safety and Governance + +- [ ] Safety review documentation (dual-use, biosecurity). +- [ ] Decision log entries for all human review decisions. +- [ ] Pre-registration documents (assay preregistration, analysis plan). +- [ ] Model release policy compliance (no unscreened weights shipped). +- [ ] Responsible use statement. +- [ ] License information (Apache-2.0 for code, CC BY 4.0 for docs). + +--- + +## 11. README for Reproduction + +A top-level README in the release package should include: + +1. One-paragraph summary of what was done. +2. System requirements (Python 3.11+, RAM, OS). +3. Installation instructions. +4. Command to regenerate scoring: `make demo` or `make regenerate-all`. +5. Command to reproduce benchmarks: `make validate-scoring`, `make bench-500`, etc. +6. Command to build the lab batch pack: `make lab-batch-pack`. +7. Where to find evidence certificates, scores, and manifests. +8. How to verify data integrity (SHA-256 hashes). +9. Citation instructions and license. + +--- + +## 12. Preprint / Publication Preparation + +- [ ] Methods section drafted from pipeline documentation. +- [ ] Benchmark table prepared from METRICS_CURRENT.md. +- [ ] Honest limitations section (charge-domination, no wet-lab data, simulation modules experimental). +- [ ] Negative result section (all negative findings from Phase 3). +- [ ] Author contributions and acknowledgments. +- [ ] Data and code availability statement. +- [ ] Competing interests declaration. +- [ ] Pre-registration DOIs or timestamps for all locked plans. + +--- + +## Related Documents + +- `docs/RELEASE_CHECKLIST.md` — General release process. +- `docs/CLAIM_REVIEW_CHECKLIST.md` — Claim verification before publication. +- `docs/DATA_GOVERNANCE.md` — Data licensing and redistribution policy. +- `docs/ARTIFACT_VERSIONING.md` — Versioning scheme for pipeline outputs. +- `docs/MODEL_CARD_TEMPLATE.md` — Model card format for released scorers. +- `docs/TRUST_CENTER.md` — Trust and safety documentation hub. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index eaac5afe..1575dc45 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,5 +1,20 @@ # Roadmap +## v0.5.68 — Publication Pack Checklist (Loop 48) ✓ (2026-07-06) + +Checklist for public release of OpenAMP results. A reviewer can reproduce +all findings from the contents of this pack. + +Changes: +- ``docs/PUBLICATION_PACK.md`` — 12 sections: pipeline code and version, + input data, scoring outputs, benchmark results, candidate panel, wet-lab + results, calibration, reproducibility evidence, negative results, safety + and governance, README for reproduction, preprint preparation. References + 7 related governance docs. +- ``docs/50_LOOP_PLAN.md`` — Loop 48 ✅. Next: Loop 49. +- ``docs/METRICS_CURRENT.md`` — Changelog updated. +- No code changes. 2013 tests still pass. + ## v0.5.67 — Data-Return Validation Tool (Loop 47) ✓ (2026-07-06) CLI to validate lab result JSON files against the schema.