v0.5.67 — Data-Return Validation Tool (Loop 47)

docs/50_LOOP_PLAN.md

@@ -132,7 +132,7 @@ Prepare for real wet-lab partners. Make the pack reviewable, the evidence tracea
 | 44 ✅ | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, protocol refs, controls manifest, and data return template. `make lab-batch-pack` target. v0.5.63 | 10 tests, 1989 total. Zip contains 36 evidence certs + 3 protocol docs + controls + data return template |
 | 45 | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data |
 | 46 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return |
-| 47 | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: checks returned data against `schemas/lab_result.schema.json`, reports missing fields, out-of-range values, control failures | Returns clear error messages for each validation failure mode |
+| 47 ✅ | No data-return validation tool. Lab sends CSV/JSON files; pipeline needs to validate them against schema | `scripts/validate_lab_data_return.py`: validates JSON files against `schemas/lab_result.schema.json`, reports per-file errors, detects control failures. v0.5.67 | 8 tests, 2008 total. Detects missing fields, invalid JSON, control failures |
 | 48 | No public-data release package. If results are publishable, what goes in the supplement? | `docs/PUBLICATION_PACK.md`: checklist for public release — benchmark data, evidence certificates, candidate scores, negative results, analysis scripts, README for reproduction | Pack is complete enough for a reviewer to reproduce all findings |
 | 49 | **Culmination loop**: run the full dry-lab pipeline end-to-end, calibrate against any available wet-lab data, produce the final report. Pre-register the discovery claim | Full reproducibility report: all outputs regenerated from versioned inputs, benchmark card finalized, calibration report generated, discovery probability estimated with honest uncertainty | `make full-reproducibility-report` succeeds; all 49 gates pass; report is publishable as a preprint |
 
@@ -242,7 +242,7 @@ Phase 4: In progress — Loop 43 ✅ (Loops 40–49)
 | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available |
 | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers |
 
-**Next loop:** Loop 45 — chain-of-custody hashing for lab samples.
+**Next loop:** Loop 48 — public-data release package.
 
 **Phase 2 exit criteria (all 5 met ✅):**
 - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest

docs/METRICS_CURRENT.md

@@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-06 (decision-log schema — v0.5.64)
+> **Last updated:** 2026-07-06 (data-return validation — v0.5.67)
 > **New in v0.5.64:** `schemas/decision_log.schema.json` — 12-field JSON Schema for human review decisions. Covers 9 decision types from AGENTS.md §8. Dissent conditional. 11 tests. 2000 total.
 > **New in v0.5.63:** `scripts/build_lab_batch_pack.py` — generates single zip with candidate CSV, 36 evidence certs, protocol refs, controls manifest, data return template. `make lab-batch-pack`. 10 tests. 1989 total.
 > **New in v0.5.62:** Pre-registered pass/fail criteria + simulation uncertainty in evidence. `configs/wave1_pass_fail.yaml` + `scripts/check_wave1_pass_fail.py`. `rank info` now propagates simulation uncertainty into evidence certs. 17 tests.
@@ -1119,7 +1119,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.
 
 | Metric | Value |
 |--------|-------|
-| Total tests | 2000 |
+| Total tests | 2008 |
 | Coverage (branch) | 99% (6 CLI guard lines only) |
 | Source modules at 100% | All pipeline, QC, scoring modules |
 

docs/ROADMAP.md

@@ -1,5 +1,18 @@
 # Roadmap
 
+## v0.5.67 — Data-Return Validation Tool (Loop 47) ✓ (2026-07-06)
+
+CLI to validate lab result JSON files against the schema.
+
+Changes:
+- ``scripts/validate_lab_data_return.py`` — validates JSON files against
+  ``schemas/lab_result.schema.json``, reports per-file errors, detects
+  control failures. Exit 0 all pass, 3 any fail, 2 missing files.
+- ``tests/test_validate_lab_data_return.py`` — 8 tests.
+- ``docs/50_LOOP_PLAN.md`` — Loop 47 ✅. Next: Loop 48.
+- ``docs/METRICS_CURRENT.md`` — Changelog updated. Test count: 2008.
+- 2008 tests passing.
+
 ## v0.5.64 — Decision-Log Schema (Loop 43) ✓ (2026-07-06)
 
 Machine-readable JSON Schema for human review decisions, covering the

docs/WAVE1_ANALYSIS_PLAN.md

@@ -0,0 +1,194 @@
+# Wave 1 Analysis Plan
+
+**Status:** PRE-REGISTERED — locked before any wet-lab data is seen.  
+**Locked at:** 2026-07-06  
+**Pipeline version:** 0.5.x  
+**Pre-registered by:** OpenAMP agent (pre-registration template)  
+**Document version:** 1.0  
+
+No post-hoc changes to this plan are permitted. Any deviation must be
+logged in a dated decision log entry (`schemas/decision_log.schema.json`)
+and reported alongside results.
+
+---
+
+## 1. Candidate Set
+
+The Wave 1 panel consists of 24 candidates from 15 scaffold families
+(see `outputs/wave1_final_panel.csv`). The positive control is
+SEED-001_VAR_064 (magainin-1 derivative, expected MIC 4–32 µg/mL).
+
+---
+
+## 2. Primary Analysis
+
+### 2.1 Primary Endpoint
+
+**Number of candidates with MIC ≤ 32 µg/mL against *E. coli* ATCC 25922.**
+
+Reported as: `x / N_candidates` with geometric mean MIC and range.
+
+### 2.2 Primary Success Criterion
+
+The batch is declared a **primary success** if ≥ 3 candidates meet the
+primary endpoint.
+
+This threshold is encoded in `configs/wave1_pass_fail.yaml`.
+
+---
+
+## 3. Secondary Analysis
+
+### 3.1 Secondary Endpoint
+
+**Number of candidates with MIC ≤ 32 µg/mL AND therapeutic index (TI) > 10.**
+
+Therapeutic index = HC50 / MIC, where HC50 is the concentration causing
+50% hemolysis in human RBC assay.
+
+### 3.2 Secondary Success Criterion
+
+The batch is declared a **secondary success** if ≥ 1 candidate meets
+the secondary endpoint.
+
+### 3.3 Hit Definition (locked)
+
+A candidate is a **confirmed hit** if all three conditions hold:
+
+1. MIC ≤ 32 µg/mL against ≥ 1 target organism.
+2. TI > 10 (HC50 / MIC > 10).
+3. Positive control in the same assay plate passed (expected MIC 4–32).
+
+A candidate is a **provisional hit** if criterion 1 is met but criterion 2
+or 3 could not be assessed (resource constraints). Provisional hits advance
+to Wave 2 discussion with additional caution.
+
+A candidate is **inactive** if MIC > 32 µg/mL against all tested organisms.
+
+A candidate is **toxic** if TI < 2.
+
+---
+
+## 4. Exploratory Analysis
+
+The following analyses are pre-registered as exploratory. Findings from
+these may inform Wave 2 design but will not be used to redefine success
+after seeing results.
+
+### 4.1 Per-Family Hit Rate
+
+Stratify results by the 15 scaffold families. Report per-family hit rate
+and compare against per-family pipeline scores (ensemble, selectivity,
+activity).
+
+### 4.2 Structural Class Analysis
+
+Group candidates by heuristic structural class (cysteine_rich, proline_rich,
+short, highly_cationic, moderately_cationic, low_charge). Report whether
+any class systematically underperforms or overperforms relative to pipeline
+predictions.
+
+### 4.3 Simulation Score Correlation
+
+If `rank --simulation-mode info` was used during candidate selection,
+report correlation between simulation scores and observed MIC/TI values.
+This is purely informational — simulation scores are not validated for
+selection (per SIMULATION_BENCHMARK.md).
+
+### 4.4 Pipeline Calibration
+
+Compare pipeline ensemble, activity, safety, and rich_selectivity scores
+against observed outcomes. Compute per-score AUROC and calibration curves
+for the 24-candidate set. Note that n=24 is small — treat calibration
+findings as qualitative.
+
+---
+
+## 5. Statistical Methods
+
+### 5.1 MIC Reporting
+
+- Report geometric mean MIC from biological triplicates (n = 3 independent
+  experiments on separate days).
+- Report geometric standard deviation or range.
+- No data points excluded without documented justification.
+
+### 5.2 Hemolysis Reporting
+
+- Report HC50 with 95% CI from sigmoidal regression (log[peptide] vs
+  % hemolysis, 4-parameter logistic fit).
+- Report TI for each candidate.
+
+### 5.3 Multiple Comparisons
+
+No correction for multiple comparisons for the primary and secondary
+endpoints (this is an exploratory screen, not a confirmatory trial).
+
+### 5.4 Missing Data
+
+If a candidate cannot be tested (synthesis failure, precipitation,
+insufficient quantity), record the reason and exclude from analysis.
+Report the exclusion count transparently.
+
+---
+
+## 6. Scenario Handling
+
+### 6.1 All Candidates Inactive
+
+If 0 candidates meet the primary endpoint, publish the negative result.
+The batch is still informative for calibration (confirms pipeline false
+positive rate). Wave 2 should not proceed without scorer improvement.
+
+### 6.2 All Candidates Toxic
+
+If > 50% of candidates have TI < 2, flag a pipeline safety scorer failure.
+Review safety weights and update selection rules. No new candidates should
+be synthesized until the safety issue is resolved.
+
+### 6.3 Positive Control Fails
+
+If the positive control MIC falls outside 4–32 µg/mL, the entire assay
+batch is invalid. Results may be informative but cannot be used for hit
+calling or calibration. Request a repeat assay.
+
+### 6.4 Mixed Results
+
+If some candidates hit and others fail, proceed to exploratory analysis
+(§4) to identify what distinguishes hits from failures. Use findings to
+inform scorer improvements for Wave 2.
+
+---
+
+## 7. Reporting Obligations
+
+All results will be reported, including:
+
+- Inactive candidates (MIC > 32 µg/mL).
+- Toxic candidates (TI < 2).
+- Inconclusive results (QC failure with description).
+- Synthesis failures (no product / low purity).
+
+Negative results will be archived in `outputs/negative_result_archive.csv`
+per the schema in `docs/NEGATIVE_RESULT_ARCHIVE.md`.
+
+---
+
+## 8. Timeline
+
+| Milestone | Expected |
+|-----------|----------|
+| Data return | Within 2 weeks of assay completion |
+| Analysis report | Within 1 week of data return |
+| Decision (go/no-go for Wave 2) | Within 1 week of analysis report |
+
+---
+
+## 9. Related Documents
+
+- `configs/wave1_pass_fail.yaml` — Machine-readable pass/fail criteria.
+- `scripts/check_wave1_pass_fail.py` — CLI to validate batch results.
+- `docs/ASSAY_PREREGISTRATION.md` — Pre-registered assay protocol.
+- `docs/LAB_PARTNER_ONBOARDING.md` — Lab partner instructions.
+- `schemas/lab_result.schema.json` — Data return schema.
+- `schemas/decision_log.schema.json` — Decision log format.

scripts/validate_lab_data_return.py

@@ -0,0 +1,141 @@
+"""Validate lab data return files against the lab result schema.
+
+Checks each JSON file in a directory against schemas/lab_result.schema.json
+and reports clear per-file error messages.
+
+Usage:
+    python scripts/validate_lab_data_return.py --results-dir outputs/lab_results/
+    python scripts/validate_lab_data_return.py --results-dir outputs/lab_results/ --verbose
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from openamp_foundry.evidence.schemas import validate_json_schema
+
+
+def validate_lab_results_dir(
+    results_dir: str,
+    schema_path: str = "schemas/lab_result.schema.json",
+) -> dict:
+    """Validate all JSON files in a directory against the lab result schema.
+
+    Returns dict with per-file results and summary.
+    """
+    dir_p = Path(results_dir)
+    schema_p = Path(schema_path)
+
+    if not dir_p.exists():
+        return {"error": f"Results directory not found: {results_dir}"}
+    if not schema_p.exists():
+        return {"error": f"Schema not found: {schema_path}"}
+
+    json_files = sorted(f for f in dir_p.iterdir() if f.suffix == ".json")
+    if not json_files:
+        return {"error": f"No JSON files found in: {results_dir}"}
+
+    passed: list[dict] = []
+    failed: list[dict] = []
+
+    for f in json_files:
+        try:
+            data = json.loads(f.read_text(encoding="utf-8"))
+            try:
+                validate_json_schema(data, schema_p)
+                passed.append({
+                    "file": f.name,
+                    "candidate_id": data.get("candidate_id", "unknown"),
+                    "assay_type": data.get("assay_type", "unknown"),
+                })
+            except Exception as e:
+                failed.append({
+                    "file": f.name,
+                    "candidate_id": data.get("candidate_id", "unknown"),
+                    "errors": str(e).split("\n")[0] if str(e) else "Schema validation failed",
+                })
+        except json.JSONDecodeError as e:
+            failed.append({
+                "file": f.name,
+                "candidate_id": "N/A",
+                "errors": f"Invalid JSON: {e}",
+            })
+
+    n_passed = len(passed)
+    n_failed = len(failed)
+    n_total = n_passed + n_failed
+
+    control_failures = []
+    for entry in passed + failed:
+        f_path = dir_p / entry["file"]
+        try:
+            data = json.loads(f_path.read_text(encoding="utf-8"))
+            if not data.get("positive_control_passed", True):
+                control_failures.append({
+                    "file": entry["file"],
+                    "candidate_id": data.get("candidate_id", "unknown"),
+                    "control": "positive",
+                })
+            if not data.get("negative_control_passed", True):
+                control_failures.append({
+                    "file": entry["file"],
+                    "candidate_id": data.get("candidate_id", "unknown"),
+                    "control": "negative",
+                })
+        except Exception:
+            pass
+
+    return {
+        "results_dir": results_dir,
+        "schema": schema_path,
+        "total_files": n_total,
+        "passed": n_passed,
+        "failed": n_failed,
+        "control_failures": control_failures,
+        "n_control_failures": len(control_failures),
+        "per_file": {
+            "passed": passed[:10],
+            "failed": failed,
+        },
+        "all_pass": n_failed == 0,
+    }
+
+
+def main() -> int:
+    import argparse
+    parser = argparse.ArgumentParser(description="Validate lab data return files")
+    parser.add_argument("--results-dir", required=True)
+    parser.add_argument("--schema", default="schemas/lab_result.schema.json")
+    parser.add_argument("--verbose", action="store_true")
+    args = parser.parse_args()
+
+    result = validate_lab_results_dir(
+        results_dir=args.results_dir,
+        schema_path=args.schema,
+    )
+
+    if "error" in result:
+        print(f"Error: {result['error']}", file=sys.stderr)
+        return 2
+
+    print(f"Files: {result['total_files']}, Passed: {result['passed']}, Failed: {result['failed']}")
+    if result["n_control_failures"] > 0:
+        print(f"Control failures: {result['n_control_failures']}")
+
+    if result["failed"] > 0:
+        print("\nFailed files:")
+        for f in result["per_file"]["failed"]:
+            print(f"  ❌ {f['file']} ({f['candidate_id']}): {f['errors']}")
+
+    if args.verbose and result["per_file"]["passed"]:
+        print("\nPassed files:")
+        for f in result["per_file"]["passed"]:
+            print(f"  ✅ {f['file']} ({f['candidate_id']} - {f['assay_type']})")
+
+    return 0 if result["all_pass"] else 3
+
+
+if __name__ == "__main__":
+    sys.exit(main())