Loop 71: External review packet schemas (Phase E, E1-E3)

CLAUDE.md

@@ -47,6 +47,16 @@ OpenAMP Foundry is a verification-first, safety-constrained antimicrobial peptid
 - [docs/trust/MODEL_CARD_TEMPLATE.md](docs/trust/MODEL_CARD_TEMPLATE.md)
 - [docs/trust/RELEASE_CHECKLIST.md](docs/trust/RELEASE_CHECKLIST.md)
 
+## Verification commands
+
+Before creating a PR, run:
+
+```bash
+make agent-check     # claim scan + doc link check + benchmark deprecation
+make doctor          # environment diagnostic
+make bench-calibration  # verify calibration hasn't degraded (optional)
+```
+
 ## Default work loop
 
 ```text
@@ -57,6 +67,7 @@ read relevant docs
   -> add tests or explain docs-only status
   -> update source-of-truth docs
   -> preserve limitations
+  -> run make agent-check
   -> stop before scope creep
 ```
 

Makefile

@@ -5,7 +5,18 @@ PYTEST  := $(shell [ -f .venv/bin/pytest ] && echo .venv/bin/pytest || echo pyte
 RUFF    := $(shell [ -f .venv/bin/ruff ] && echo .venv/bin/ruff || echo ruff)
 
 help:
-	@echo "OpenAMP Foundry — wet-lab pipeline targets"
+	@echo "OpenAMP Foundry — dry-lab pipeline targets"
+	@echo ""
+	@echo "Quick verification:"
+	@echo "  make pr-ready           Pre-PR check (agent-check + doctor)"
+	@echo "  make agent-check        Claim scan + doc links + benchmark deprecation"
+	@echo "  make doctor             Environment diagnostic"
+	@echo ""
+	@echo "Pipeline:"
+	@echo "  make demo               Rank demo candidates, produce report + evidence certs"
+	@echo ""
+	@echo "Benchmarks (key):"
+	@echo "  make bench-500          Full expanded benchmark (AUROC, AUPRC, recall)"
 	@echo ""
 	@echo "  make demo               Rank demo candidates, produce report + evidence certs"
 	@echo "  make phase3             Generate + rank Phase 3 pool (89 nominees)"
@@ -71,6 +82,9 @@ help:
 	@echo "  make ci                 lint + test (CI gate)"
 	@echo "  make clean              Remove outputs/ (except CSV pilot panel)"
 
+pr-ready: agent-check doctor
+	@echo "Ready for PR."
+
 agent-check: claim-check doc-links-check bench-deprecation-check
 
 docs-index-check:
@@ -141,6 +155,9 @@ coverage:
 typecheck:
 	uv run mypy src/ --ignore-missing-imports --no-error-summary 2>&1 | head -30 || true
 
+bench-all: bench-500 bench-calibration bench-cheap-enemies bench-charge-distribution bench-simulation-calibration
+	@echo "All key benchmarks complete."
+
 bench-leakage:
 	PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli bench leakage \
 		--candidates examples/sequences/demo_candidates.csv \

configs/exit_code_registry.json

@@ -16,6 +16,7 @@
     "validate_lab_data_return.py": {"expected": [0, 2, 3], "notes": "0=all valid, 2=input error, 3=validation fail"},
     "validate_negative_archive.py": {"expected": [0, 2, 3], "notes": "0=all valid, 2=input error, 3=validation fail"},
     "validate_panel_csv.py": {"expected": [0, 2, 3], "notes": "0=valid, 2=input error, 3=invalid"},
+    "validate-cert-quality": {"expected": [0, 3], "notes": "0=pass/warn, 3=fail"},
     "check_claims.py": {"expected": [0, 3], "notes": "0=ok, 3=findings (advisory)"},
     "check_doc_links.py": {"expected": [0, 3], "notes": "0=ok, 3=broken links found"},
     "check_pyproject.py": {"expected": [0, 2, 3], "notes": "0=ok, 2=input error, 3=inconsistency found"}

docs/PROJECT_INDEX.md

@@ -291,13 +291,16 @@ Do not change a success definition after seeing results.
 
 Do not hide negative results merely because they weaken the story.
 
-## Current strategic bottleneck
+## Current status — 60-loop milestone
 
-The project already has unusually strong dry-lab discipline.
+**Dry-lab infrastructure complete** as of Loop 60 (2026-07-08). All internal
+quality gates, benchmarks, evidence chain, agent workflow, and documentation
+are in place and tested. Key metrics: 2427+ test baseline, `make pr-ready`
+passes on clean checkout, all 10 JSON schemas validated, 16 packages documented.
 
-The next bottleneck is not more impressive language.
+## Current strategic bottleneck
 
-The next bottleneck is external truth:
+The next bottleneck is **external truth**:
 
 - independent review of candidate evidence packages;
 - fair comparison against charge/similarity baselines;

docs/evidence/ARTIFACT_CLAIM_BOUNDARIES.md

@@ -0,0 +1,38 @@
+# Artifact Claim Boundaries
+
+## Purpose
+
+Each artifact type produced by the pipeline supports a specific maximum claim level.
+This document prevents overclaiming by mapping artifact types to their supported
+proof-ladder levels.
+
+## Artifact map
+
+| Artifact | Max claim level | What it supports | What it does NOT support |
+|----------|----------------|------------------|-------------------------|
+| Candidate certificate | multi_signal_candidate_evidence | Candidate passed dry-lab filters and multi-signature scoring | Biological activity, safety, clinical usefulness |
+| Batch report | multi_signal_candidate_evidence | Summary of scored candidates in a batch | Any individual candidate claim |
+| Evidence certificate | baseline_triaged to multi_signal_candidate_evidence | Reproducible selection rationale | Biological validation |
+| Simulation result | informational only | Experimental proxy output, always informational | Ranking impact unless gate permits |
+| Lab result report | initial_qualified_assay_result | Machine-readable assay outcome | Conclusions about efficacy |
+| Calibration intake report | informational only | Descriptive join of predictions and outcomes | Recalibration authority |
+| Recalibration gate verdict | informational only | Whether recalibration conditions are met | Permission to change weights |
+| Benchmark report | leakage_aware_benchmark | Retrospective performance estimate | Real-world generalization |
+| Review packet | expert_reviewed_assay_proposal | Candidate evidence for human review | Approval or recommendation |
+
+## Rule
+
+**No artifact may assert a claim higher than its max claim level.**
+
+If an artifact needs to support a higher claim level, it must:
+1. Include evidence at that level (e.g., assay results for Level 6)
+2. Be reviewed by a qualified human
+3. Have its schema updated to reflect the higher level
+
+## Exceptions
+
+The only exception is the final validated discovery claim:
+> "An open AI discovery pipeline produced a newly validated antimicrobial peptide family."
+
+This claim requires all 10 conditions from MISSION.md §Headline-grade result and may
+only be made after independent wet-lab validation with full evidence trail.

docs/evidence/METRICS_CURRENT.md

@@ -5,7 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-08 (calibration benchmark — v0.5.71)
+> **Last updated:** 2026-07-09 (Phase E external review schemas — v0.5.72)
+> **New in v0.5.72:** External review packet schema (E1), example packet (E2), reviewer questionnaire schema (E3). 27 tests. 2666 total.
+>
 > **New in v0.5.71:** Calibration benchmark added — Brier score, reliability diagram, and calibration slope for pipeline ensemble scores. Honest finding: Brier=0.318 (>0.25=uninformative), slope=0.43 (ideal=1.0). Pipeline ranks well but scores are not meaningful probabilities. `make bench-calibration`. 6 tests. ~2100 total.
 >
 > **New in v0.5.65:** `scripts/lab/build_lab_batch_pack.py` now writes `chain_of_custody.json` and `MANIFEST.json` into lab batch packs. The custody file includes SHA-256 hashes for `panel.csv`, the ordered candidate list, each candidate sequence, and each evidence certificate. `--verify-pack` verifies archive integrity and detects tampering. These hashes prove identity/integrity only, not biological activity, safety, or synthesis success. 5 tests.
@@ -52,7 +54,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
 > **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
 > **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
-> **Pipeline version:** v0.5.67
+> **Pipeline version:** v0.5.72
 > **Branch:** main
 
 ---

docs/research/ROADMAP.md

@@ -1,5 +1,86 @@
 # Roadmap
 
+## v0.5.72 — Loop 71: External Review Packet Schemas (Phase E, E1-E3) ✓ (2026-07-09)
+
+External credibility infrastructure: machine-checkable schemas that let external
+reviewers verify what a "review packet" contains before running any experiments.
+
+Changes:
+- `schemas/external_review_packet.schema.json` (E1) — JSON Schema Draft 2020-12
+  defining what a complete external review packet must contain: packet_id, version,
+  generated_at, pipeline_version, git_sha, candidate_count, candidates (array with
+  per-candidate ids/sequences/scores), benchmark_summary, calibration_summary,
+  limitations (array), safety_attestations, dry_lab_only_attestation (bool, must be
+  true), proof_ladder_level, contact.
+- `examples/external_review_packet_example.json` (E2) — toy/example packet validating
+  against E1 schema, clearly marked EXAMPLE/NOT-REAL-DATA throughout.
+- `schemas/reviewer_questionnaire.schema.json` (E3) — schema for structured reviewer
+  feedback: reviewer_id, packet_id, reviewed_at, scientific_validity (1-5),
+  benchmark_adequacy (1-5), safety_concerns (bool), safety_concern_description
+  (conditional-required), recommendation (approve/revise/reject), comments,
+  conflict_of_interest_declared, domain_expertise.
+- `tests/evidence/test_external_review_packet_schema.py` — 13 tests covering valid
+  packet, missing fields, dry_lab_only_attestation=false rejection, git_sha validation,
+  candidate_count minimum, empty limitations, example packet validation, proof_ladder
+  bounds, calibration_assessment enum.
+- `tests/evidence/test_reviewer_questionnaire_schema.py` — 14 tests covering valid
+  questionnaire, missing fields, safety_concerns→description conditional,
+  scientific_validity bounds, recommendation enum, conflict_of_interest→detail
+  conditional, domain_expertise, maximal packet.
+- Removed stale `tests/test_simulation_gate.py` duplicate (naming collision with
+  `tests/simulation/test_simulation_gate.py`). The more comprehensive version in
+  `tests/simulation/` is the canonical home.
+- `docs/evidence/METRICS_CURRENT.md` — v0.5.72 changelog. Pipeline version bumped.
+  Test count: 2666.
+- `tests/test_test_count_regression.py` — baseline updated to 2639.
+
+Honest boundaries:
+- These schemas make review packets machine-checkable but do not validate the
+  scientific content of the packet itself.
+- A valid packet can still contain weak science. The schema verifies structure, not
+  truth.
+- The dry_lab_only_attestation is a safety-constraint field that prevents review
+  packets from being mistaken for biological proof, but it relies on the generator's
+  honesty.
+- The example packet contains toy data only and must not be treated as real
+  experimental recommendations.
+
+## v0.5.71 — Loop 60: Dry-lab Infrastructure Complete ✓ (2026-07-08)
+
+Extended to 60 loops. All dry-lab infrastructure requirements are now met:
+
+**Evidence chain (closed end-to-end):** claim scanner → proof_ladder_level →
+quality-tier validator → forbidden-claim cert test → failed-candidate tracker →
+artifact claim boundary doc → model card.
+
+**Benchmark honesty:** calibration benchmark (Brier 0.32, slope 0.43, documented),
+charge distribution, cheap-enemy comparison, deprecation banner system, test count
+regression gate, cross-config determinism tests.
+
+**Simulation honesty:** cheapest-baseline declarations, fail-closed integration tests,
+uncertainty calibration report, simulation gate (weighted mode blocked).
+
+**Agent workflow:** `make pr-ready`, `make doctor`, `make agent-check`, docs-only
+verifier, docs-index coverage checker, CLI help coverage test, `--version` flag,
+stop-condition examples, maintainer prompts.
+
+**Infrastructure:** 10 JSON schemas validated, 16 packages documented in
+ARCHITECTURE.md, 2427+ test baseline, all configs valid YAML, all scripts
+importable, all packages have __init__.py, exit code registry covers key commands.
+
+**Remaining:** External validation (pre-registered pilot, wet-lab partner, real
+assay results) is the next phase. All dry-lab infrastructure is in place.
+
+Changes:
+- Active learning CLI end-to-end test
+- Selectivity proxy negative control
+- Simulation gate weighted-mode-blocked test
+- Cross-dataset benchmark smoke test
+- Cross-reference docs coverage check
+- `make pr-ready` target
+- `make doctor` CLI diagnostic
+- `make bench-calibration` integrated into bench-500
+
 ## v0.5.70 — Loop 50: Reproducibility Report Polish ✓ (2026-07-06)
 
 Fixed lab_batch_pack check in reproducibility report (was checking for

examples/external_review_packet_example.json

@@ -0,0 +1,65 @@
+{
+  "packet_id": "ERP-EXAMPLE-2026-07-09-001",
+  "version": "1.0.0",
+  "generated_at": "2026-07-09T12:00:00Z",
+  "pipeline_version": "v0.5.72",
+  "git_sha": "abcdef1234567890abcdef1234567890abcdef12",
+  "candidate_count": 2,
+  "candidates": [
+    {
+      "candidate_id": "EXAMPLE-001",
+      "sequence": "KLAKLAKKLAKLAK",
+      "ensemble_score": 0.84,
+      "proof_ladder_level": 2,
+      "family": "EXAMPLE-HELIX",
+      "selection_rationale": "EXAMPLE: Strong amphipathic helix, high charge density. This is a TOY EXAMPLE, not a real candidate.",
+      "safety_notes": "EXAMPLE ONLY: No safety assessment performed on this toy candidate."
+    },
+    {
+      "candidate_id": "EXAMPLE-002",
+      "sequence": "RVRIRLVRRLLR",
+      "ensemble_score": 0.79,
+      "proof_ladder_level": 2,
+      "family": "EXAMPLE-ARG-VAL",
+      "selection_rationale": "EXAMPLE: Alternating arginine-valine pattern. This is a TOY EXAMPLE, not a real candidate.",
+      "safety_notes": "EXAMPLE ONLY: No safety assessment performed on this toy candidate."
+    }
+  ],
+  "benchmark_summary": {
+    "auroc": 0.78,
+    "benchmark_name": "EXAMPLE 500-AMP benchmark (toy data)",
+    "n_positives": 500,
+    "n_negatives": 500,
+    "ci_95_lower": 0.75,
+    "ci_95_upper": 0.81,
+    "caveats": [
+      "EXAMPLE: This is a toy/example benchmark summary, not actual benchmark results.",
+      "Benchmark AUROC does not measure real-world usefulness.",
+      "The pipeline underperforms charge density alone on AMP-vs-decoy discrimination."
+    ]
+  },
+  "calibration_summary": {
+    "brier_score": 0.32,
+    "calibration_slope": 0.43,
+    "calibration_assessment": "uninformative",
+    "n_calibration_samples": 1000
+  },
+  "limitations": [
+    "Computational outputs are hypotheses and review aids. They are not biological proof.",
+    "Dry-lab scores do not prove antimicrobial activity.",
+    "Dry-lab safety-risk scores do not prove biological safety.",
+    "Benchmark success does not prove real-world usefulness.",
+    "External review does not equal validation.",
+    "EXAMPLE: This is a toy/example packet created for schema validation purposes only. It does not contain real candidate data or real review recommendations."
+  ],
+  "safety_attestations": {
+    "reviewed_by_human": false,
+    "safety_gate_passed": true,
+    "no_known_toxicity_claim": true,
+    "hemolysis_assay_mandatory": true
+  },
+  "dry_lab_only_attestation": true,
+  "proof_ladder_level": 2,
+  "contact": "EXAMPLE-ONLY: openamp-foundry@example.com (toy data, do not use for actual review)",
+  "notes": "THIS IS A TOY EXAMPLE — NOT REAL DATA. This file demonstrates the external review packet schema format. No candidate sequences, scores, or evaluations in this file should be treated as real experimental recommendations."
+}