Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase Q Q1 — pilot evidence package — v0.10.7)
> **Last updated:** 2026-07-10 (Phase Q Q2 — pre-registration entry — v0.10.8)
> **New in v0.10.8 (Phase Q Q2):** PreRegistrationEntry — pre-experiment plan lock (14 fields, 9 validation rules, 4 warnings, 63 tests, BASELINE 5036→5099)
> **New in v0.10.7 (Phase Q Q1):** PilotEvidencePackageEntry — external export bundle (14 fields, CCS+BSP+PSC+PRE+BCM refs, completeness+safety enforcement, 63 tests, BASELINE 4973→5036).
> **New in v0.10.5 (Phase P P2):** RecalibrationRefusalEntry schema (10 fields: rrf_id, pipeline_version, trigger_id, recalibration_refused, refusal_reason, minimum_batches_required, batches_evaluated, refusal_notes, reviewer, dry_lab_only). recalibration_refused must be True — this schema only records valid refusals. 7 validation rules (RRF- prefix, CPS-/DRM- trigger prefix, recalibration_refused=True enforced, reason in 5 valid values, minimum_batches>=1, batches_evaluated>=0, notes<=400 chars). 2 warnings (insufficient_data but meets minimum; reviewer_override without documentation). Complements CalibrationImprovementRecord (O3). Added validate_recalibration_refusal() and validate_recalibration_refusal_dict(). Added 63 tests. CLI: openamp-foundry recalibration-refusal-check with --entry-json and --format text|json. Corrected inflated BASELINE from 4889 to 4784 (actual count 4721 + 63 = 4784).
> **New in v0.10.4 (Phase P P1):** BatchSelectionProposalEntry schema (12 fields: bsp_id, pipeline_version, gate_id, gate_passed, candidate_ids, selection_strategy, exploitation_fraction, exploration_fraction, max_brier_score_allowed, proposal_notes, reviewer, dry_lab_only). gate_passed must be True — a batch can only be proposed when the calibration readiness gate passes. 9 validation rules (BSP- prefix, CRG- prefix, gate_passed=True enforced, candidate_ids >=1, strategy in 5 valid values, fraction bounds, fraction sum=1.0 to 0.001 tolerance, Brier bounds [0.0,1.0], notes <=400 chars). 3 warnings (pure exploitation >=0.99, single candidate, marginal Brier >=0.25). Added validate_batch_selection_proposal() and validate_batch_selection_proposal_dict(). Added 68 tests. CLI: openamp-foundry batch-selection-proposal-check with --entry-json and --format text|json. Closes the calibration-to-selection loop from Phase O.
Expand Down
54 changes: 54 additions & 0 deletions docs/evidence/PRE_REGISTRATION_ENTRY_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Pre-Registration Entry Guide

## Purpose

A Pre-Registration Entry (PRE) documents the experimental plan that is locked BEFORE
any wet-lab experiment begins. It enforces the pre-commitment discipline that makes the
evidence trail auditable: the hypothesis, primary endpoint, and candidate list must be
committed before results are observed — preventing post-hoc hypothesis adjustment
(HARKing: Hypothesizing After Results are Known).

A PRE entry is referenced by the Pilot Evidence Package (PEP) via `pre_registration_id`.
A PEP cannot claim completeness without a valid PRE reference.

## Required fields

| Field | Type | Constraint |
|---|---|---|
| `pre_id` | str | Must start with `PRE-` |
| `pipeline_version` | str | Non-empty |
| `experiment_title` | str | Non-empty; plain language description |
| `hypothesis` | str | Non-empty; falsifiable claim |
| `primary_endpoint` | str | Non-empty; measurable outcome |
| `candidate_ids` | List[str] | At least 1 candidate |
| `positive_control_id` | Optional[str] | Recommended |
| `negative_control_id` | Optional[str] | Recommended |
| `registered_date` | str | ISO format YYYY-MM-DD |
| `registration_status` | str | `submitted`, `approved`, or `rejected` |
| `registrar` | str | Non-empty; who accepted the registration |
| `notes` | str | Max 400 chars |
| `reviewer` | str | Who prepared this record |
| `dry_lab_only` | bool | Default True |

## Registration statuses

- `submitted` — plan filed, pending review
- `approved` — plan accepted; experiments may begin
- `rejected` — plan rejected; reason in notes; preserved for audit trail

## Honest boundaries

- Pre-registration prevents HARKing but does not guarantee quality.
- `approved` status does not mean results will be positive.
- `dry_lab_only=True` entries document computational screening plans,
not wet-lab commitments.
- Rejected entries MUST be preserved — they are part of the audit trail.

## Warnings

The validator emits warnings (not errors) for:

- `approved` with `dry_lab_only=True` — confirm experiment type
- No positive control — reduces result interpretability
- No negative control — reduces result interpretability
- Single candidate — limited statistical power
1 change: 1 addition & 0 deletions docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ bundle.
| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| Q1 | Add pilot evidence package schema (complete). | External export artifact: bundles CCS+BSP+PSC+PRE+BCM references, enforces completeness and safety clearance before external sharing. | B/D |
| Q2 | Add pre-registration entry schema (complete). | Locks hypothesis, endpoint, and candidate list before wet-lab begins; prevents HARKing; referenced by PEP as pre_registration_id. | B/D |

## Prioritization rule

Expand Down
29 changes: 29 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -3756,3 +3756,32 @@ def _run_pilot_evidence_package_check(args) -> int:
for warn in result.warnings:
print(f" WARN: {warn}")
return 0 if result.passed else 1


def _run_pre_registration_entry_check(args) -> int:
import json
from openamp_foundry.evidence.pre_registration_entry import (
validate_pre_registration_entry_dict,
)
if args.entry_json:
try:
data = json.loads(args.entry_json)
except json.JSONDecodeError as exc:
print(f"Error: invalid JSON: {exc}", file=__import__("sys").stderr)
return 1
else:
data = json.load(__import__("sys").stdin)
result = validate_pre_registration_entry_dict(data)
if args.format == "json":
import dataclasses
print(json.dumps(dataclasses.asdict(result), indent=2))
else:
status = "PASS" if result.passed else "FAIL"
print(f"[{status}] Pre-Registration: {result.pre_id} [{result.registration_status}]")
print(f" Title: {result.experiment_title}")
print(f" Candidates: {result.candidate_count}, Controls: +{result.has_positive_control}/-{result.has_negative_control}")
for err in result.errors:
print(f" ERROR: {err}")
for warn in result.warnings:
print(f" WARN: {warn}")
return 0 if result.passed else 1
12 changes: 12 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
_run_pilot_batch_safety_clearance_check,
_run_calibration_cycle_summary_check,
_run_pilot_evidence_package_check,
_run_pre_registration_entry_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2410,6 +2411,14 @@ def build_parser() -> argparse.ArgumentParser:
p_pep.add_argument("--format", choices=["text", "json"], default="text")
p_pep.set_defaults(func=_run_pilot_evidence_package_check)

p_pre = sub.add_parser(
"pre-registration-check",
help="Validate pre-registration entry",
)
p_pre.add_argument("--entry-json", default=None)
p_pre.add_argument("--format", choices=["text", "json"], default="text")
p_pre.set_defaults(func=_run_pre_registration_entry_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2803,6 +2812,9 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "pilot-evidence-package-check":
return _run_pilot_evidence_package_check(args)

if args.command == "pre-registration-check":
return _run_pre_registration_entry_check(args)

parser.error("unknown command")
return 2

Expand Down
202 changes: 202 additions & 0 deletions src/openamp_foundry/evidence/pre_registration_entry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
"""Pre-registration entry schema — Phase Q Q2.

Documents the experimental plan locked BEFORE wet-lab work begins.
A pre-registration enforces that hypothesis, primary endpoint, and
candidate list are committed before any results are observed — the
foundation of auditable, non-cherry-picked evidence.

An approved PRE entry is a prerequisite for a complete Pilot Evidence
Package (PEP). Rejected entries are preserved for audit trail.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from typing import List, Optional

PRE_PREFIX = "PRE-"
VALID_REGISTRATION_STATUSES = frozenset({"submitted", "approved", "rejected"})
NOTES_MAX_LENGTH = 400
MIN_CANDIDATES = 1


@dataclass
class PreRegistrationEntry:
"""Pre-committed experimental plan for a pilot batch."""

pre_id: str
pipeline_version: str
experiment_title: str
hypothesis: str
primary_endpoint: str
candidate_ids: List[str]
positive_control_id: Optional[str]
negative_control_id: Optional[str]
registered_date: str
registration_status: str
registrar: str
notes: str
reviewer: str
dry_lab_only: bool = True


@dataclass
class PreRegistrationResult:
pre_id: str
pipeline_version: str
experiment_title: str
registration_status: str
candidate_count: int
has_positive_control: bool
has_negative_control: bool
passed: bool
errors: List[str]
warnings: List[str]
dry_lab_only: bool = True


def validate_pre_registration_entry(
entry: PreRegistrationEntry,
) -> PreRegistrationResult:
errors: List[str] = []
warnings: List[str] = []

if not entry.pre_id.startswith(PRE_PREFIX):
errors.append(
f"pre_id must start with '{PRE_PREFIX}', got '{entry.pre_id}'"
)

if not entry.pipeline_version:
errors.append("pipeline_version must not be empty")

if not entry.experiment_title:
errors.append("experiment_title must not be empty")

if not entry.hypothesis:
errors.append("hypothesis must not be empty")

if not entry.primary_endpoint:
errors.append("primary_endpoint must not be empty")

if len(entry.candidate_ids) < MIN_CANDIDATES:
errors.append(
f"candidate_ids must contain at least {MIN_CANDIDATES} entry, "
f"got {len(entry.candidate_ids)}"
)

# registered_date must be ISO format YYYY-MM-DD
import re
if not re.match(r"^\d{4}-\d{2}-\d{2}$", entry.registered_date):
errors.append(
f"registered_date must be ISO format YYYY-MM-DD, "
f"got '{entry.registered_date}'"
)

if entry.registration_status not in VALID_REGISTRATION_STATUSES:
errors.append(
f"registration_status must be one of "
f"{sorted(VALID_REGISTRATION_STATUSES)}, "
f"got '{entry.registration_status}'"
)

if not entry.registrar:
errors.append("registrar must not be empty")

if len(entry.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} characters "
f"(got {len(entry.notes)})"
)

# Warnings
if entry.registration_status == "approved" and entry.dry_lab_only:
warnings.append(
"registration_status=approved with dry_lab_only=True — "
"an approved pre-registration implies wet-lab work is planned; "
"confirm dry_lab_only reflects the actual experiment type"
)

if not entry.positive_control_id and not errors:
warnings.append(
"no positive_control_id — experiments without a positive control "
"reduce interpretability of results"
)

if not entry.negative_control_id and not errors:
warnings.append(
"no negative_control_id — experiments without a negative control "
"reduce interpretability of results"
)

if len(entry.candidate_ids) == 1 and not errors:
warnings.append(
"candidate_ids contains only 1 entry — a single-candidate "
"experiment provides limited statistical power"
)

has_positive_control = entry.positive_control_id is not None and entry.positive_control_id != ""
has_negative_control = entry.negative_control_id is not None and entry.negative_control_id != ""

return PreRegistrationResult(
pre_id=entry.pre_id,
pipeline_version=entry.pipeline_version,
experiment_title=entry.experiment_title,
registration_status=entry.registration_status,
candidate_count=len(entry.candidate_ids),
has_positive_control=has_positive_control,
has_negative_control=has_negative_control,
passed=len(errors) == 0,
errors=errors,
warnings=warnings,
dry_lab_only=entry.dry_lab_only,
)


def validate_pre_registration_entry_dict(d: dict) -> PreRegistrationResult:
missing = []
for k in (
"pre_id",
"pipeline_version",
"experiment_title",
"hypothesis",
"primary_endpoint",
"candidate_ids",
"registered_date",
"registration_status",
"registrar",
"notes",
"reviewer",
):
if k not in d:
missing.append(k)
if missing:
return PreRegistrationResult(
pre_id=d.get("pre_id", ""),
pipeline_version=d.get("pipeline_version", ""),
experiment_title=d.get("experiment_title", ""),
registration_status=d.get("registration_status", ""),
candidate_count=0,
has_positive_control=False,
has_negative_control=False,
passed=False,
errors=[f"missing required fields: {missing}"],
warnings=[],
dry_lab_only=d.get("dry_lab_only", True),
)
entry = PreRegistrationEntry(
pre_id=d["pre_id"],
pipeline_version=d["pipeline_version"],
experiment_title=d["experiment_title"],
hypothesis=d["hypothesis"],
primary_endpoint=d["primary_endpoint"],
candidate_ids=list(d["candidate_ids"]),
positive_control_id=d.get("positive_control_id"),
negative_control_id=d.get("negative_control_id"),
registered_date=d["registered_date"],
registration_status=d["registration_status"],
registrar=d["registrar"],
notes=d["notes"],
reviewer=d["reviewer"],
dry_lab_only=bool(d.get("dry_lab_only", True)),
)
return validate_pre_registration_entry(entry)
Loading
Loading