Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase F F1 — negative result entry — v0.10.14)
> **Last updated:** 2026-07-10 (Phase K K3 — pilot package completeness report — v0.10.15)
> **New in v0.10.15 (Phase K K3):** PilotPackageCompletenessReport — PPC- completeness gate: confirms CCS-+BSP-+PSC-+PRE-+BCM- all present and ESC- cleared before external sharing (12 fields, 12 validation rules, 3 warnings, 63 tests, BASELINE 5540→5603)
> **New in v0.10.14 (Phase F F1):** NegativeResultEntry — atomic NRR- failure record with stage/reason/confidence vocabulary, foundation for NAS- and FCR- chain (12 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5477→5540)
> **New in v0.10.14 (Phase E E9):** DomainReviewOutcome — controlled taxonomy expert verdict on a PEP (11 fields, 9 validation rules, 3 warnings, 63 tests, BASELINE 5414→5477); closes ESC→RVQ→DRO review chain
> **New in v0.10.13 (Phase E E3):** ReviewerQuestionnaire — structured external review feedback with Likert clarity ratings, synthesis recommendation, and comments (13 fields, 10 validation rules, 4 warnings, 63 tests, BASELINE 5351→5414)
Expand Down
94 changes: 94 additions & 0 deletions docs/evidence/PILOT_PACKAGE_COMPLETENESS_REPORT_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Pilot Package Completeness Report Guide (PPC-)

## Purpose

A `PilotPackageCompletenessReport` (PPC-) is the auditable completeness gate for a
`PilotEvidencePackage` (PEP-). It confirms that all five required component IDs are
present and that an External Sharing Clearance (ESC-) exists before the package is
shared with an external lab.

Without a PPC-, a PEP could leave the foundry with missing components. A valid PPC-
means the completeness check was explicitly passed and recorded.

## Required Components

A complete pilot package requires:

| Component | Prefix | Description |
|-----------|--------|-------------|
| Calibration Cycle Summary | `CCS-` | Confirms calibration gate passed |
| Batch Selection Proposal | `BSP-` | Documents how candidates were selected |
| Pilot Safety Clearance | `PSC-` | Safety gate before synthesis |
| Pre-Registration Entry | `PRE-` | Hypothesis locked before wet-lab |
| Baseline Comparison Manifest | `BCM-` | Proves model beats cheap baselines |

An External Sharing Clearance (`ESC-`) must also exist, confirming the package
was cleared for release.

## Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ppc_id` | str | Yes | Unique ID, must start with `PPC-` |
| `pipeline_version` | str | Yes | Version of pipeline that ran the check |
| `pep_id` | str | Yes | The PEP being checked (must start with `PEP-`) |
| `esc_id` | str | Yes | External sharing clearance (must start with `ESC-`) |
| `ccs_id` | str | Yes | Calibration cycle summary (must start with `CCS-`) |
| `bsp_id` | str | Yes | Batch selection proposal (must start with `BSP-`) |
| `psc_id` | str | Yes | Pilot safety clearance (must start with `PSC-`) |
| `pre_id` | str | Yes | Pre-registration entry (must start with `PRE-`) |
| `bcm_id` | str | Yes | Baseline comparison manifest (must start with `BCM-`) |
| `checked_date` | str | Yes | ISO date (YYYY-MM-DD) of completeness check |
| `completeness_confirmed` | bool | Yes | Must be `True`; do not record unless complete |
| `notes` | str | No | Reviewer context (≤300 chars) |

## Validation Rules

1. `ppc_id` must start with `PPC-`
2. `pipeline_version` must be non-empty
3. `pep_id` must start with `PEP-`
4. `esc_id` must start with `ESC-`
5. `ccs_id` must start with `CCS-`
6. `bsp_id` must start with `BSP-`
7. `psc_id` must start with `PSC-`
8. `pre_id` must start with `PRE-`
9. `bcm_id` must start with `BCM-`
10. `checked_date` must match `YYYY-MM-DD`
11. `completeness_confirmed` must be `True`
12. `notes` must be ≤300 chars

## Warnings

| Condition | Warning |
|-----------|---------|
| `notes` is empty | No reviewer context recorded |
| Two or more component IDs are identical | Each component should have a unique ID |
| `pep_id` appears in a component ID slot | PEP- IDs should not be reused as component IDs |

## Boundaries

- **PPC- records completeness, not quality.** A complete package may still have scientific limitations; completeness only means all components are present.
- **`completeness_confirmed` must be True.** Do not record a PPC- unless the package genuinely passes. Incomplete packages do not get a PPC-.
- **PPC- is not a release approval.** An ESC- (External Sharing Clearance) is still required separately. PPC- verifies components; ESC- records who received the package and when.

## CLI

```bash
openamp-foundry pilot-package-completeness-check '{"ppc_id": "PPC-001", ...}'
```

Returns `VALID` or `INVALID` with any errors and warnings.

## Relationship to Other Schemas

```
PEP- ──→ PPC- (completeness gate for the PEP bundle)
ESC- ──→ PPC- (ESC- confirms external sharing cleared)
CCS- ──→ PPC- (calibration gate passed)
BSP- ──→ PPC- (batch selection documented)
PSC- ──→ PPC- (safety clearance on record)
PRE- ──→ PPC- (pre-registration locked)
BCM- ──→ PPC- (baseline comparison confirmed)
```

PPC- is the final internal gate before an external lab receives a package.
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ Build the evidence and accountability artifacts needed for credible external val
|---:|---|---|---|
| K1 | Add candidate selection rationale schema. | Auditable selection decisions. | B |
| K2 | Add batch experiment priority ranker. | Rank candidates for next synthesis wave. | B |
| K3 | Add pilot package completeness checker. | All components present before submission. | B |
| K3 | Add pilot package completeness checker (complete). | PPC- completeness gate: confirms CCS-+BSP-+PSC-+PRE-+BCM- all present and ESC- cleared before external sharing. | B |
| K4 | Add post-experiment calibration intake schema. | Structured result-to-prediction comparison. | B/D |
| K5 | Add uncertainty quantification report schema (complete — Loop 123). | Honest prediction intervals for reviewers. | B |

Expand Down
22 changes: 22 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -1330,6 +1330,28 @@ def _run_domain_review_outcome_check(args):
sys.exit(0 if result.passed else 1)


def _run_pilot_package_completeness_check(args):
import json, sys
from openamp_foundry.evidence.pilot_package_completeness_report import validate_dict
try:
data = json.loads(args.json_input)
except json.JSONDecodeError as exc:
print(f"INVALID: JSON parse error: {exc}", file=sys.stderr)
sys.exit(1)
result = validate_dict(data)
if result.valid:
print("VALID")
for w in result.warnings:
print(f"WARNING: {w}")
else:
print("INVALID")
for e in result.errors:
print(f"ERROR: {e}")
for w in result.warnings:
print(f"WARNING: {w}")
sys.exit(1)


def _run_simulation_registry(args: argparse.Namespace) -> int:
"""Display simulation module registry information."""
from openamp_foundry.simulation.module_registry import (
Expand Down
14 changes: 13 additions & 1 deletion src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
from pathlib import Path

from openamp_foundry.evidence.negative_result_entry import validate_dict as validate_negative_result_entry_dict
from openamp_foundry.evidence.pilot_package_completeness_report import validate_dict as validate_pilot_package_completeness_dict
from openamp_foundry.evidence.quality import assess_certificate_quality
from openamp_foundry.evidence.schemas import validate_json_schema
from openamp_foundry.pipeline import run_ranking_pipeline
Expand Down Expand Up @@ -2486,6 +2487,12 @@ def build_parser() -> argparse.ArgumentParser:
)
negative_result_entry_parser.add_argument("json_input", help="JSON string of the NRR- record")

ppc_parser = sub.add_parser(
"pilot-package-completeness-check",
help="Validate a PilotPackageCompletenessReport (PPC-) record",
)
ppc_parser.add_argument("json_input", help="JSON string of the PPC- record")

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2904,7 +2911,12 @@ def main(argv: list[str] | None = None) -> int:
from openamp_foundry.cli.commands.reports import _run_negative_result_entry_check
_run_negative_result_entry_check(args)

parser.error("unknown command")
elif args.command == "pilot-package-completeness-check":
from openamp_foundry.cli.commands.reports import _run_pilot_package_completeness_check
_run_pilot_package_completeness_check(args)

else:
parser.error("unknown command")
return 2


Expand Down
142 changes: 142 additions & 0 deletions src/openamp_foundry/evidence/pilot_package_completeness_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
"""Pilot package completeness report schema — Phase K K3.

Records the completeness gate for a PilotEvidencePackage before external sharing.
Confirms all five required component IDs are present and an external sharing
clearance exists. Completeness must be explicitly confirmed before this record
is valid.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

PPC_PREFIX = "PPC-"
PEP_PREFIX = "PEP-"
ESC_PREFIX = "ESC-"
CCS_PREFIX = "CCS-"
BSP_PREFIX = "BSP-"
PSC_PREFIX = "PSC-"
PRE_PREFIX = "PRE-"
BCM_PREFIX = "BCM-"

NOTES_MAX_LENGTH = 300
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


@dataclass
class PilotPackageCompletenessReport:
ppc_id: str
pipeline_version: str
pep_id: str
esc_id: str
ccs_id: str
bsp_id: str
psc_id: str
pre_id: str
bcm_id: str
checked_date: str
completeness_confirmed: bool
notes: str = ""


@dataclass
class PilotPackageCompletenessReportResult:
valid: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)


def validate(report: PilotPackageCompletenessReport) -> PilotPackageCompletenessReportResult:
errors: List[str] = []
warnings: List[str] = []

if not report.ppc_id.startswith(PPC_PREFIX):
errors.append(f"ppc_id must start with '{PPC_PREFIX}', got: {report.ppc_id!r}")

if not report.pipeline_version.strip():
errors.append("pipeline_version must be a non-empty string")

if not report.pep_id.startswith(PEP_PREFIX):
errors.append(f"pep_id must start with '{PEP_PREFIX}', got: {report.pep_id!r}")

if not report.esc_id.startswith(ESC_PREFIX):
errors.append(f"esc_id must start with '{ESC_PREFIX}', got: {report.esc_id!r}")

if not report.ccs_id.startswith(CCS_PREFIX):
errors.append(f"ccs_id must start with '{CCS_PREFIX}', got: {report.ccs_id!r}")

if not report.bsp_id.startswith(BSP_PREFIX):
errors.append(f"bsp_id must start with '{BSP_PREFIX}', got: {report.bsp_id!r}")

if not report.psc_id.startswith(PSC_PREFIX):
errors.append(f"psc_id must start with '{PSC_PREFIX}', got: {report.psc_id!r}")

if not report.pre_id.startswith(PRE_PREFIX):
errors.append(f"pre_id must start with '{PRE_PREFIX}', got: {report.pre_id!r}")

if not report.bcm_id.startswith(BCM_PREFIX):
errors.append(f"bcm_id must start with '{BCM_PREFIX}', got: {report.bcm_id!r}")

if not _ISO_DATE_RE.match(report.checked_date):
errors.append(
f"checked_date must be ISO format YYYY-MM-DD, got: {report.checked_date!r}"
)

if not report.completeness_confirmed:
errors.append(
"completeness_confirmed must be True; "
"do not record a PPC- unless the package is confirmed complete"
)

if len(report.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(report.notes)})"
)

# Warnings
if not report.notes.strip():
warnings.append(
"notes is empty; consider documenting the completeness review process"
)

component_ids = [report.ccs_id, report.bsp_id, report.psc_id, report.pre_id, report.bcm_id]
if len(set(component_ids)) < len(component_ids):
warnings.append(
"two or more component IDs are identical; "
"each component (CCS-, BSP-, PSC-, PRE-, BCM-) should have a unique ID"
)

if report.pep_id in component_ids:
warnings.append(
f"pep_id {report.pep_id!r} appears as a component ID; "
"PEP- IDs should not be reused as component IDs"
)

return PilotPackageCompletenessReportResult(
valid=len(errors) == 0, errors=errors, warnings=warnings
)


def validate_dict(data: dict) -> PilotPackageCompletenessReportResult:
try:
report = PilotPackageCompletenessReport(
ppc_id=data.get("ppc_id", ""),
pipeline_version=data.get("pipeline_version", ""),
pep_id=data.get("pep_id", ""),
esc_id=data.get("esc_id", ""),
ccs_id=data.get("ccs_id", ""),
bsp_id=data.get("bsp_id", ""),
psc_id=data.get("psc_id", ""),
pre_id=data.get("pre_id", ""),
bcm_id=data.get("bcm_id", ""),
checked_date=data.get("checked_date", ""),
completeness_confirmed=bool(data.get("completeness_confirmed", False)),
notes=data.get("notes", ""),
)
except Exception as exc:
return PilotPackageCompletenessReportResult(
valid=False, errors=[f"Construction error: {exc}"]
)
return validate(report)
Loading
Loading