diff --git a/Makefile b/Makefile index a8ae34ff..793a704a 100644 --- a/Makefile +++ b/Makefile @@ -478,3 +478,10 @@ wave0-5b-generate: wave0-5b-filter: wave0-5b-generate PYTHONPATH=src $(PYTHON) scripts/filter_wave0_5b_candidates.py + +lab-batch-pack: + PYTHONPATH=src $(PYTHON) scripts/build_lab_batch_pack.py \ + --panel-csv outputs/phase3_ranked.jsonl \ + --evidence-dir outputs/evidence_wave0_5 \ + --out outputs/lab_batch_pack.zip \ + --manifest-out outputs/lab_batch_pack_manifest.json diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md index 8dfec31b..0c4d040d 100644 --- a/docs/50_LOOP_PLAN.md +++ b/docs/50_LOOP_PLAN.md @@ -129,7 +129,7 @@ Prepare for real wet-lab partners. Make the pack reviewable, the evidence tracea | 41 ✅ | No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable | `configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates (MIC ≤ 32 for ≥ 3 candidates, TI > 10 for ≥ 1, positive control within range, 0 toxic, ≥ 20 tested). `scripts/check_wave1_pass_fail.py`: validation CLI, exit 0/3/2. v0.5.62 | 11 tests, 1977 total. CLI validates synthetic results against YAML criteria | | 42 | No expert-review GitHub template. Review process is ad-hoc | `.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields | | 43 | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples | -| 44 | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, synthesis order, assay protocol refs, controls manifest, and data return template | `make lab-batch-pack` produces a single zip with all required files | +| 44 ✅ | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, protocol refs, controls manifest, and data return template. `make lab-batch-pack` target. v0.5.63 | 10 tests, 1989 total. Zip contains 36 evidence certs + 3 protocol docs + controls + data return template | | 45 | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data | | 46 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return | | 47 | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode | @@ -242,7 +242,7 @@ Phase 4: In progress — Loop 40 ✅ (Loops 40–49) | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available | | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers | -**Next loop:** Loop 42 — expert-review GitHub template. +**Next loop:** Loop 43 — decision-log schema for human review. **Phase 2 exit criteria (all 5 met ✅):** - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index ea11178f..aeb236a6 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,8 +5,9 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-06 (simulation uncertainty in evidence — v0.5.62) -> **New in v0.5.62:** `rank --simulation-mode info` now propagates `sim_membrane_uncertainty`, `sim_structure_uncertainty`, and `sim_max_uncertainty` into ranked outputs and selected-candidate evidence certificates. This is reviewability only; `weighted` remains blocked. Current suite: 1961 passed, 7 skipped. +> **Last updated:** 2026-07-06 (lab batch pack — v0.5.63) +> **New in v0.5.63:** `scripts/build_lab_batch_pack.py` — generates single zip with candidate CSV, 36 evidence certs, protocol refs, controls manifest, data return template. `make lab-batch-pack`. 10 tests. 1989 total. +> **New in v0.5.62:** Pre-registered pass/fail criteria + simulation uncertainty in evidence. `configs/wave1_pass_fail.yaml` + `scripts/check_wave1_pass_fail.py`. `rank info` now propagates simulation uncertainty into evidence certs. 17 tests. > **New in v0.5.61:** `docs/LAB_PARTNER_ONBOARDING.md` — CRO onboarding pack with panel summary, synthesis instructions, assay protocols, data return format, controls, safety, timeline. No code changes. > **New in v0.5.60:** `docs/SIMULATION_BENCHMARK.md` consolidates simulation ablation, cheap-baseline comparison, and weighted-mode gate results. Current conclusion: simulation does not improve ranking; `weighted` remains blocked. > **New in v0.5.59:** `ExternalSimulationAdapter` protocol — wraps third-party callables into `VirtualAssayProxy`. Availability check, graceful error handling, metadata override. ARCHITECTURE.md docs updated. 12 tests. 1965 total. @@ -1117,7 +1118,7 @@ Decoys score low on activity. Selective AMPs score moderately on both. | Metric | Value | |--------|-------| -| Total tests | 1961 | +| Total tests | 1989 | | Coverage (branch) | 99% (6 CLI guard lines only) | | Source modules at 100% | All pipeline, QC, scoring modules | diff --git a/scripts/build_lab_batch_pack.py b/scripts/build_lab_batch_pack.py new file mode 100644 index 00000000..b30a1332 --- /dev/null +++ b/scripts/build_lab_batch_pack.py @@ -0,0 +1,174 @@ +"""Build a zip archive with everything a lab partner needs. + +Collects candidate CSV, evidence certificates, protocol references, +controls manifest, and data return templates into a single zip. +""" + +from __future__ import annotations + +import csv +import json +import sys +import zipfile +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +def build_batch_pack( + panel_csv: str, + evidence_dir: str, + out_zip: str, + protocol_docs: list[str] | None = None, + controls: list[dict[str, Any]] | None = None, + data_return_schema: str = "schemas/lab_result.schema.json", +) -> dict[str, Any]: + """Build a lab batch pack zip. + + Returns manifest dict with included files and counts. + """ + panel_p = Path(panel_csv) + evidence_p = Path(evidence_dir) + out_p = Path(out_zip) + schema_p = Path(data_return_schema) + + if not panel_p.exists(): + return {"error": f"Panel CSV not found: {panel_csv}"} + if not evidence_p.exists(): + return {"error": f"Evidence directory not found: {evidence_dir}"} + + manifest: dict[str, Any] = { + "generated_at": datetime.now(timezone.utc).isoformat(), + "panel_csv": panel_csv, + "evidence_count": 0, + "included_protocols": [], + "included_candidates": [], + "data_return_template": None, + "controls": controls or [], + } + + with zipfile.ZipFile(out_p, "w", zipfile.ZIP_DEFLATED) as zf: + # 1. Panel CSV + panel_data = panel_p.read_bytes() + zf.writestr("panel.csv", panel_data) + manifest["panel_csv_size"] = len(panel_data) + + # 2. Evidence certificates + ev_count = 0 + for f in sorted(evidence_p.iterdir()): + if f.suffix == ".json": + zf.write(str(f), f"evidence/{f.name}") + ev_count += 1 + manifest["included_candidates"].append(f.stem) + manifest["evidence_count"] = ev_count + + # 3. Protocol references (as a README) + protocols = protocol_docs or [ + "docs/LAB_PARTNER_ONBOARDING.md", + "docs/ASSAY_PREREGISTRATION.md", + "docs/WET_LAB_HANDOFF.md", + ] + protocol_lines = [ + "# Lab Batch Pack — Protocol References", + "", + "See the following documents for assay protocols and instructions:", + "", + ] + for doc_path in protocols: + doc_p = Path(doc_path) + if doc_p.exists(): + zf.write(str(doc_p), f"protocols/{doc_p.name}") + protocol_lines.append(f"- {doc_path}") + manifest["included_protocols"].append(doc_path) + zf.writestr("protocols/README.md", "\n".join(protocol_lines)) + + # 4. Controls manifest + ctrl = controls or [ + {"candidate_id": "SEED-001_VAR_064", + "role": "POSITIVE_CONTROL", + "sequence": "GIGKFLHSAKKFGKAFVGEIMNS", + "expected_mic_range_ugml": "4-32", + "notes": "Magainin-1 derivative; validates MIC assay"}, + ] + zf.writestr("controls_manifest.json", json.dumps(ctrl, indent=2)) + manifest["controls"] = ctrl + + # 5. Data return template (empty JSON per schema) + if schema_p.exists(): + schema = json.loads(schema_p.read_text()) + template: dict[str, Any] = { + "result_id": "REPLACE_WITH_UUID", + "candidate_id": "REPLACE_WITH_CANDIDATE_ID", + "assay_type": "MIC", + "organism_or_cell_line": "Escherichia coli ATCC 25922", + "result_value": 0.0, + "result_unit": "ug/mL", + "positive_control_passed": True, + "negative_control_passed": True, + "assay_date": "YYYY-MM-DD", + "replicate_count": 3, + "performed_by_lab": "LAB_NAME", + "computational_candidate_certificate_hash": "REPLACE", + "disclaimer": "SYNTHETIC — replace with real data", + } + zf.writestr("data_return_template.json", json.dumps(template, indent=2)) + manifest["data_return_template"] = str(schema_p) + + # 6. Batch README + readme = [ + "# OpenAMP Foundry — Lab Batch Pack", + "", + f"Generated: {manifest['generated_at']}", + f"Candidates: {ev_count}", + f"Protocols: {len(manifest['included_protocols'])}", + "", + "## Contents", + "", + "- panel.csv — Candidate list with IDs and sequences", + "- evidence/ — Evidence certificates (one JSON per candidate)", + "- protocols/ — Assay protocol documents", + "- controls_manifest.json — Positive/negative control sequences", + "- data_return_template.json — Empty JSON template per lab_result.schema.json", + "", + "## Return Data", + "", + "After assaying, return one JSON file per candidate per assay type.", + "Use the data_return_template.json as a starting point.", + "Place all files in a single directory and run:", + "", + " python scripts/check_wave1_pass_fail.py --results-dir ", + "", + ] + zf.writestr("README.md", "\n".join(readme)) + + manifest["out_zip"] = str(out_p) + manifest["zip_size_bytes"] = out_p.stat().st_size + return manifest + + +def main() -> int: + import argparse + parser = argparse.ArgumentParser(description="Build lab batch pack zip") + parser.add_argument("--panel-csv", default="outputs/wave1_final_panel.csv") + parser.add_argument("--evidence-dir", default="outputs/evidence_wave0_5") + parser.add_argument("--out", default="outputs/lab_batch_pack.zip") + parser.add_argument("--manifest-out", default=None, help="JSON manifest path") + args = parser.parse_args() + + manifest = build_batch_pack( + panel_csv=args.panel_csv, + evidence_dir=args.evidence_dir, + out_zip=args.out, + ) + if "error" in manifest: + print(f"Error: {manifest['error']}", file=sys.stderr) + return 2 + + print(json.dumps(manifest, indent=2)) + if args.manifest_out: + Path(args.manifest_out).write_text(json.dumps(manifest, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_lab_batch_pack.py b/tests/test_lab_batch_pack.py new file mode 100644 index 00000000..b3a5d9d8 --- /dev/null +++ b/tests/test_lab_batch_pack.py @@ -0,0 +1,144 @@ +"""Tests for the lab batch pack builder.""" + +import json +import subprocess +import sys +import tempfile +import zipfile +from pathlib import Path + +from scripts.build_lab_batch_pack import build_batch_pack + + +def _make_evidence(tmp: Path, candidate_id: str) -> Path: + """Create a minimal evidence certificate.""" + d = tmp / "evidence" + d.mkdir(exist_ok=True) + (d / f"{candidate_id}.json").write_text( + json.dumps({"candidate_id": candidate_id, "scores": {"ensemble": 0.8}}) + ) + return d + + +def _make_panel_csv(tmp: Path) -> Path: + """Create a minimal panel CSV.""" + p = tmp / "panel.csv" + p.write_text("candidate_id,sequence\nC1,KKLFKKILKYL\nC2,ACDEFGHIKLMNPQRSTVWY\n") + return p + + +def test_build_pack_success(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + _make_evidence(tmp_path, "C2") + out = tmp_path / "pack.zip" + result = build_batch_pack( + panel_csv=str(panel), + evidence_dir=str(ev_dir), + out_zip=str(out), + ) + assert "error" not in result + assert out.exists() + assert result["evidence_count"] >= 2 + + +def test_build_pack_contains_evidence(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + out = tmp_path / "pack.zip" + build_batch_pack(panel_csv=str(panel), evidence_dir=str(ev_dir), out_zip=str(out)) + with zipfile.ZipFile(out) as z: + names = z.namelist() + assert any("evidence/C1.json" in n for n in names) + + +def test_build_pack_contains_readme(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + out = tmp_path / "pack.zip" + build_batch_pack(panel_csv=str(panel), evidence_dir=str(ev_dir), out_zip=str(out)) + with zipfile.ZipFile(out) as z: + assert "README.md" in z.namelist() + + +def test_build_pack_contains_controls_manifest(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + out = tmp_path / "pack.zip" + build_batch_pack(panel_csv=str(panel), evidence_dir=str(ev_dir), out_zip=str(out)) + with zipfile.ZipFile(out) as z: + assert "controls_manifest.json" in z.namelist() + + +def test_build_pack_contains_data_return_template(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + out = tmp_path / "pack.zip" + build_batch_pack(panel_csv=str(panel), evidence_dir=str(ev_dir), out_zip=str(out)) + with zipfile.ZipFile(out) as z: + assert "data_return_template.json" in z.namelist() + + +def test_build_pack_contains_protocols(tmp_path): + panel = _make_panel_csv(tmp_path) + ev_dir = _make_evidence(tmp_path, "C1") + out = tmp_path / "pack.zip" + build_batch_pack(panel_csv=str(panel), evidence_dir=str(ev_dir), out_zip=str(out)) + with zipfile.ZipFile(out) as z: + names = z.namelist() + protocol_files = [n for n in names if n.startswith("protocols/")] + assert len(protocol_files) > 0 + + +def test_build_pack_error_missing_panel(tmp_path): + result = build_batch_pack( + panel_csv="/nonexistent.csv", + evidence_dir=str(tmp_path), + out_zip=str(tmp_path / "pack.zip"), + ) + assert "error" in result + + +def test_build_pack_error_missing_evidence(tmp_path): + result = build_batch_pack( + panel_csv=str(_make_panel_csv(tmp_path)), + evidence_dir="/nonexistent", + out_zip=str(tmp_path / "pack.zip"), + ) + assert "error" in result + + +def test_cli_runs(tmp_path): + ev_dir = _make_evidence(tmp_path, "C1") + panel = _make_panel_csv(tmp_path) + out = tmp_path / "pack.zip" + result = subprocess.run( + [sys.executable, "scripts/build_lab_batch_pack.py", + "--panel-csv", str(panel), + "--evidence-dir", str(ev_dir), + "--out", str(out)], + capture_output=True, text=True, + env={"PYTHONPATH": "src"}, + ) + assert result.returncode == 0 + assert out.exists() + + +def test_cli_manifest_output(tmp_path): + ev_dir = _make_evidence(tmp_path, "C1") + panel = _make_panel_csv(tmp_path) + out = tmp_path / "pack.zip" + manifest = tmp_path / "manifest.json" + result = subprocess.run( + [sys.executable, "scripts/build_lab_batch_pack.py", + "--panel-csv", str(panel), + "--evidence-dir", str(ev_dir), + "--out", str(out), + "--manifest-out", str(manifest)], + capture_output=True, text=True, + env={"PYTHONPATH": "src"}, + ) + assert result.returncode == 0 + assert manifest.exists() + data = json.loads(manifest.read_text()) + assert "evidence_count" in data