diff --git a/docs/50_LOOP_PLAN.md b/docs/50_LOOP_PLAN.md index 0c4d040d..68b23130 100644 --- a/docs/50_LOOP_PLAN.md +++ b/docs/50_LOOP_PLAN.md @@ -128,7 +128,7 @@ Prepare for real wet-lab partners. Make the pack reviewable, the evidence tracea | 40 ✅ | No assay-partner onboarding pack. A new CRO gets no README for how to run the panel | `docs/LAB_PARTNER_ONBOARDING.md`: panel manifest, synthesis instructions, assay protocol references, data return schema, positive/negative control sequences, safety notes, timeline, controls. v0.5.61 | A new CRO can read this and know exactly what to do | | 41 ✅ | No pre-registered pass/fail criteria for the Wave 1 wet-lab batch. ASSAY_PREREGISTRATION exists but criteria are not machine-checkable | `configs/wave1_pass_fail.yaml`: machine-readable pass/fail gates (MIC ≤ 32 for ≥ 3 candidates, TI > 10 for ≥ 1, positive control within range, 0 toxic, ≥ 20 tested). `scripts/check_wave1_pass_fail.py`: validation CLI, exit 0/3/2. v0.5.62 | 11 tests, 1977 total. CLI validates synthetic results against YAML criteria | | 42 | No expert-review GitHub template. Review process is ad-hoc | `.github/ISSUE_TEMPLATE/expert_review.yml`: structured form for expert reviewers covering novelty, safety, mechanism plausibility, experiment design | Template generates a review issue with all required fields | -| 43 | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema for human review decisions. Fields: date, reviewer, decision_type, evidence_refs, reasoning_notes, dissent_flag | Schema validates against real decision examples | +| 43 ✅ | No decision-log format for human review. AGENTS.md requires human review for 6 decision types but no machine-readable log format exists | `schemas/decision_log.schema.json`: JSON Schema (Draft 2020-12) with 12 fields, 9 decision types, dissent-requires-notes conditional. v0.5.64 | 11 tests, 2000 total. Schema validates against real decision examples | | 44 ✅ | No candidate batch pack that a lab partner can download as a single artifact | `scripts/build_lab_batch_pack.py`: generates a zip with candidate CSV, evidence certificates, protocol refs, controls manifest, and data return template. `make lab-batch-pack` target. v0.5.63 | 10 tests, 1989 total. Zip contains 36 evidence certs + 3 protocol docs + controls + data return template | | 45 | No chain-of-custody hashing for lab samples. Lab receives peptides but can't verify identity | Add SHA-256 hash of each candidate sequence and synthesis order to the batch pack. Lab can verify received peptides match | Hash verification script works end-to-end on synthetic data | | 46 | No pre-registered analysis plan for Wave 1 results. What analysis runs when data comes back? | `docs/WAVE1_ANALYSIS_PLAN.md`: pre-registered statistical analysis (primary: MIC ≤ 32 hit rate; secondary: selectivity index; exploratory: per-family correlation). No post-hoc changes allowed | Plan is timestamped and locked before data return | @@ -182,7 +182,7 @@ Phase 0: ✅ Complete (Loops 1–8) Phase 1: ✅ Complete (Loops 9–17) Phase 2: ✅ Complete (Loops 18–29) Phase 3: ✅ Complete (Loops 30–39) -Phase 4: In progress — Loop 40 ✅ (Loops 40–49) +Phase 4: In progress — Loop 43 ✅ (Loops 40–49) ``` ### Completed @@ -242,7 +242,7 @@ Phase 4: In progress — Loop 40 ✅ (Loops 40–49) | 28 ✅ | Policy version bump workflow for when real data arrives | `scripts/bump_recalibration_policy.py`: standalone script with `--dry-run`, decision-log guard, auto-increment + write. CI guard in `ci.yml` validates policy version changes against base branch. v0.5.49. 9 tests. | CI rejects policy PRs without valid decision log; `make bump-policy-version` and `make bump-policy-version-dry-run` available | | 29 ✅ | No public negative-result archive format. If Wave 1 yields all negatives, where do they go? | `docs/NEGATIVE_RESULT_ARCHIVE.md`: full template with entry schema, procedures, automation notes, and limitations. Covers pre-selection rejects, selected-untested, lab inactives, lab toxic, control failures. v0.5.50. | Template complete enough for a lab partner to fill; schema defines 18 fields with required/conditional markers | -**Next loop:** Loop 43 — decision-log schema for human review. +**Next loop:** Loop 45 — chain-of-custody hashing for lab samples. **Phase 2 exit criteria (all 5 met ✅):** - ✅ `make calibration-loop` runs from clean checkout, produces batch-2 manifest diff --git a/docs/METRICS_CURRENT.md b/docs/METRICS_CURRENT.md index aeb236a6..5a52640d 100644 --- a/docs/METRICS_CURRENT.md +++ b/docs/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-06 (lab batch pack — v0.5.63) +> **Last updated:** 2026-07-06 (decision-log schema — v0.5.64) +> **New in v0.5.64:** `schemas/decision_log.schema.json` — 12-field JSON Schema for human review decisions. Covers 9 decision types from AGENTS.md §8. Dissent conditional. 11 tests. 2000 total. > **New in v0.5.63:** `scripts/build_lab_batch_pack.py` — generates single zip with candidate CSV, 36 evidence certs, protocol refs, controls manifest, data return template. `make lab-batch-pack`. 10 tests. 1989 total. > **New in v0.5.62:** Pre-registered pass/fail criteria + simulation uncertainty in evidence. `configs/wave1_pass_fail.yaml` + `scripts/check_wave1_pass_fail.py`. `rank info` now propagates simulation uncertainty into evidence certs. 17 tests. > **New in v0.5.61:** `docs/LAB_PARTNER_ONBOARDING.md` — CRO onboarding pack with panel summary, synthesis instructions, assay protocols, data return format, controls, safety, timeline. No code changes. @@ -1118,7 +1119,7 @@ Decoys score low on activity. Selective AMPs score moderately on both. | Metric | Value | |--------|-------| -| Total tests | 1989 | +| Total tests | 2000 | | Coverage (branch) | 99% (6 CLI guard lines only) | | Source modules at 100% | All pipeline, QC, scoring modules | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 0adf41df..697df434 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,5 +1,18 @@ # Roadmap +## v0.5.64 — Decision-Log Schema (Loop 43) ✓ (2026-07-06) + +Machine-readable JSON Schema for human review decisions, covering the +9 decision types required by AGENTS.md §8. + +Changes: +- ``schemas/decision_log.schema.json`` — Draft 2020-12 JSON Schema with + 12 fields, 9 decision types, conditional dissent-requires-notes. +- ``tests/test_decision_log_schema.py`` — 11 tests. +- ``docs/50_LOOP_PLAN.md`` — Loop 43 ✅. Next: Loop 45. +- ``docs/METRICS_CURRENT.md`` — Changelog updated. Test count: 2000. +- 2000 tests passing. + ## v0.5.62 — Simulation Uncertainty in Evidence ✓ (2026-07-06) Closed the remaining Phase 3 evidence-propagation gap: simulation uncertainty diff --git a/schemas/decision_log.schema.json b/schemas/decision_log.schema.json new file mode 100644 index 00000000..e06c6a8c --- /dev/null +++ b/schemas/decision_log.schema.json @@ -0,0 +1,87 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "OpenAMP Human Review Decision Log", + "description": "Schema for human review decisions required by AGENTS.md §8.", + "type": "object", + "required": [ + "entry_id", + "date", + "reviewer_name", + "decision_type", + "decision", + "evidence_refs", + "reasoning_notes" + ], + "properties": { + "entry_id": { + "type": "string", + "description": "Unique identifier (e.g. 'DR-2026-07-06-001')" + }, + "date": { + "type": "string", + "format": "date", + "description": "ISO 8601 date of the review decision" + }, + "reviewer_name": { + "type": "string", + "description": "Full name of the human reviewer" + }, + "reviewer_affiliation": { + "type": "string", + "description": "Reviewer institution or organisation (optional)" + }, + "decision_type": { + "type": "string", + "enum": [ + "publish_candidate_sequences", + "contact_lab", + "release_generator_weights", + "change_safety_policy", + "external_scientific_claim", + "submit_paper_or_press_release", + "recalibration_approval", + "expert_review_override", + "other" + ], + "description": "Type of decision being logged, matching AGENTS.md §8 categories" + }, + "decision": { + "type": "string", + "enum": ["approved", "rejected", "deferred"], + "description": "The human reviewer's decision" + }, + "evidence_refs": { + "type": "array", + "items": { "type": "string" }, + "description": "References to evidence that informed the decision (PR URLs, document paths, issue numbers)" + }, + "reasoning_notes": { + "type": "string", + "description": "Free-text rationale for the decision" + }, + "dissent_flag": { + "type": "boolean", + "default": false, + "description": "True if any reviewer dissented from the majority decision" + }, + "dissent_notes": { + "type": "string", + "description": "Explanation of dissenting viewpoint, if dissent_flag is true" + }, + "supersedes": { + "type": "string", + "description": "entry_id of a previous decision that this entry overrides" + }, + "related_entries": { + "type": "array", + "items": { "type": "string" }, + "description": "Related entry_ids for tracking chains of decisions" + } + }, + "allOf": [ + { + "if": { "properties": { "dissent_flag": { "const": true } } }, + "then": { "required": ["dissent_notes"] } + } + ] +} diff --git a/tests/test_decision_log_schema.py b/tests/test_decision_log_schema.py new file mode 100644 index 00000000..cbed4bc3 --- /dev/null +++ b/tests/test_decision_log_schema.py @@ -0,0 +1,131 @@ +"""Tests for the decision-log JSON Schema.""" + +import json +from pathlib import Path + +import pytest + +from openamp_foundry.evidence.schemas import validate_json_schema + +SCHEMA_DIR = Path("schemas") +DECISION_LOG_DIR = Path("decision_logs") + + +def _load_schema() -> dict: + path = SCHEMA_DIR / "decision_log.schema.json" + return json.loads(path.read_text(encoding="utf-8")) + + +def test_schema_file_exists(): + assert (SCHEMA_DIR / "decision_log.schema.json").exists() + + +def test_schema_is_valid_json_schema(): + schema = _load_schema() + assert schema["$schema"].startswith("https://json-schema.org/") + assert schema["title"] == "OpenAMP Human Review Decision Log" + + +def test_schema_required_fields(): + schema = _load_schema() + for field in ["entry_id", "date", "reviewer_name", "decision_type", + "decision", "evidence_refs", "reasoning_notes"]: + assert field in schema["required"], f"Missing required field: {field}" + + +def test_schema_decision_types(): + schema = _load_schema() + valid_types = schema["properties"]["decision_type"]["enum"] + for dt in ["publish_candidate_sequences", "contact_lab", + "release_generator_weights", "change_safety_policy", + "external_scientific_claim", "submit_paper_or_press_release", + "recalibration_approval", "expert_review_override", "other"]: + assert dt in valid_types, f"Missing decision type: {dt}" + + +def test_schema_decision_values(): + schema = _load_schema() + valid_decisions = schema["properties"]["decision"]["enum"] + for d in ["approved", "rejected", "deferred"]: + assert d in valid_decisions + + +def test_valid_decision_passes_validation(): + schema = _load_schema() + example = { + "entry_id": "DR-2026-07-06-001", + "date": "2026-07-06", + "reviewer_name": "Dr. Jane Smith", + "reviewer_affiliation": "University of Example", + "decision_type": "contact_lab", + "decision": "approved", + "evidence_refs": ["https://github.com/Open-Problem-Lab/OpenAMP-Foundry/pull/145"], + "reasoning_notes": "Pipeline passes honesty benchmarks. Candidate panel is diverse.", + "dissent_flag": False, + } + validate_json_schema(example, SCHEMA_DIR / "decision_log.schema.json") + + +def test_missing_required_field_fails(): + schema = _load_schema() + example = { + "entry_id": "DR-001", + "date": "2026-07-06", + "reviewer_name": "Dr. Smith", + # missing decision_type, decision, evidence_refs, reasoning_notes + } + with pytest.raises(Exception, match="required"): + validate_json_schema(example, SCHEMA_DIR / "decision_log.schema.json") + + +def test_dissent_requires_notes(): + example = { + "entry_id": "DR-002", + "date": "2026-07-06", + "reviewer_name": "Dr. Smith", + "decision_type": "recalibration_approval", + "decision": "rejected", + "evidence_refs": [], + "reasoning_notes": "Insufficient cohort size.", + "dissent_flag": True, + # missing dissent_notes — should fail + } + with pytest.raises(Exception, match="dissent_notes"): + validate_json_schema(example, SCHEMA_DIR / "decision_log.schema.json") + + +def test_invalid_decision_type_fails(): + example = { + "entry_id": "DR-003", + "date": "2026-07-06", + "reviewer_name": "Dr. Smith", + "decision_type": "invalid_type", + "decision": "approved", + "evidence_refs": [], + "reasoning_notes": "Test", + } + with pytest.raises(Exception): + validate_json_schema(example, SCHEMA_DIR / "decision_log.schema.json") + + +def test_rejected_decision_valid(): + example = { + "entry_id": "DR-004", + "date": "2026-07-06", + "reviewer_name": "Dr. Jones", + "decision_type": "publish_candidate_sequences", + "decision": "rejected", + "evidence_refs": ["PR #150"], + "reasoning_notes": "Novelty analysis incomplete.", + "dissent_flag": True, + "dissent_notes": "I believe novelty is sufficient.", + } + validate_json_schema(example, SCHEMA_DIR / "decision_log.schema.json") + + +def test_existing_decision_log_is_parsable(): + """The existing Markdown decision log is not JSON — this test verifies + the file exists. Real decision logs should use the JSON format for + machine readability.""" + log_files = list(DECISION_LOG_DIR.glob("*.md")) + list(DECISION_LOG_DIR.glob("*.json")) + assert len(log_files) >= 1, "No decision log files found"