Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions docs/evidence/CALIBRATION_CYCLE_SUMMARY_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Calibration Cycle Summary Guide

## Purpose

A Calibration Cycle Summary (CCS) is the index record for one complete calibration
feedback loop. It ties together all Phase O and Phase P schemas into a single
searchable audit record.

## The calibration feedback loop

```
CRG-previous (calibration readiness gate passes)
↓
BSP (batch selection proposal)
↓
PSC (pilot batch safety clearance)
↓
[Wet-lab experiments]
↓
BOS (batch outcome summary)
↓
CPS (calibration performance summary)
↓
CBA (cross-batch aggregator)
↓
CRG-next (new calibration readiness gate)
↓
Next CCS
```

A CCS records the IDs of all artifacts from one pass through this loop. The
`crg_id_previous` and `crg_id_next` must differ — a completed cycle always
opens a new gate.

## Schema fields

| Field | Type | Description |
|---|---|---|
| `ccs_id` | str | Must start with `CCS-` |
| `pipeline_version` | str | Pipeline version for this cycle |
| `bsp_id` | str | Must start with `BSP-` |
| `psc_id` | str | Must start with `PSC-` |
| `bos_id` | str | Must start with `BOS-` |
| `cps_id` | str | Must start with `CPS-` |
| `cba_id` | str | Must start with `CBA-` |
| `crg_id_previous` | str | Must start with `CRG-` — the gate that enabled this cycle |
| `crg_id_next` | str | Must start with `CRG-` — must differ from previous |
| `cycle_outcome` | str | `improved`, `stable`, or `degraded` |
| `cycle_notes` | str | Max 400 chars |
| `reviewer` | str | Who completed and validated this cycle |
| `dry_lab_only` | bool | True if cycle used synthetic/simulated data |

## Artifact ID prefix map

| Prefix | Schema | Phase |
|---|---|---|
| `CCS-` | CalibrationCycleSummary | P5 |
| `BSP-` | BatchSelectionProposal | P1 |
| `PSC-` | PilotBatchSafetyClearance | P4 |
| `BOS-` | BatchOutcomeSummary | P3 |
| `CPS-` | CalibrationPerformanceSummary | O1 |
| `CBA-` | CrossBatchAggregator | O4 |
| `CRG-` | CalibrationReadinessGate | O5 |

## Honest boundaries

- A CCS is an index, not a proof of correctness. The schemas it references may
themselves contain errors or missing data.
- `crg_id_previous != crg_id_next` is enforced, but not whether the two gates
belong to the same pipeline version or have a logical succession.
- `dry_lab_only=True` entries document completed cycles in a simulation context;
they cannot support claims about real experimental outcomes.

## Warnings

The validator emits warnings (not errors) for:

- `cycle_outcome="degraded"` — calibration quality decreased; investigate before
the next selection cycle.
- `dry_lab_only=True` — no real experimental data was used in this cycle.
5 changes: 3 additions & 2 deletions docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase P P2 — recalibration refusal record — v0.10.5)
> **Last updated:** 2026-07-10 (Phase P P5 — calibration cycle summary — v0.10.6)
> **New in v0.10.5 (Phase P P2):** RecalibrationRefusalEntry schema (10 fields: rrf_id, pipeline_version, trigger_id, recalibration_refused, refusal_reason, minimum_batches_required, batches_evaluated, refusal_notes, reviewer, dry_lab_only). recalibration_refused must be True — this schema only records valid refusals. 7 validation rules (RRF- prefix, CPS-/DRM- trigger prefix, recalibration_refused=True enforced, reason in 5 valid values, minimum_batches>=1, batches_evaluated>=0, notes<=400 chars). 2 warnings (insufficient_data but meets minimum; reviewer_override without documentation). Complements CalibrationImprovementRecord (O3). Added validate_recalibration_refusal() and validate_recalibration_refusal_dict(). Added 63 tests. CLI: openamp-foundry recalibration-refusal-check with --entry-json and --format text|json. Corrected inflated BASELINE from 4889 to 4784 (actual count 4721 + 63 = 4784).
> **New in v0.10.4 (Phase P P1):** BatchSelectionProposalEntry schema (12 fields: bsp_id, pipeline_version, gate_id, gate_passed, candidate_ids, selection_strategy, exploitation_fraction, exploration_fraction, max_brier_score_allowed, proposal_notes, reviewer, dry_lab_only). gate_passed must be True — a batch can only be proposed when the calibration readiness gate passes. 9 validation rules (BSP- prefix, CRG- prefix, gate_passed=True enforced, candidate_ids >=1, strategy in 5 valid values, fraction bounds, fraction sum=1.0 to 0.001 tolerance, Brier bounds [0.0,1.0], notes <=400 chars). 3 warnings (pure exploitation >=0.99, single candidate, marginal Brier >=0.25). Added validate_batch_selection_proposal() and validate_batch_selection_proposal_dict(). Added 68 tests. CLI: openamp-foundry batch-selection-proposal-check with --entry-json and --format text|json. Closes the calibration-to-selection loop from Phase O.
> **New in v0.10.3 (Phase O O5):** CalibrationReadinessEntry schema (11 fields, 8 validation rules, 2 warnings). Added validate_calibration_readiness() and validate_calibration_readiness_dict(). Added 68 tests. CLI: openamp-foundry calibration-readiness-check. Phase O (Calibration Quality Assurance) COMPLETE.
Expand Down Expand Up @@ -137,7 +137,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **New in v0.5.31:** Added dipeptide-order features for sequence-order awareness. `dipeptide_order_score` achieves AUROC 0.7861 on AMP-vs-scrambled discrimination — the strongest order-dependent feature in the pipeline. Only 7/31 features survive scrambling (amphipathicity/helix-wheel + dipeptide). All composition features are purely position-independent (exactly 0.5000 AUROC on scrambled test).
> **New in v0.5.30:** Easy baseline benchmark added — charge density alone (AUROC 0.8166) outperforms the full pipeline ensemble (0.7792) on AMP-vs-Swiss-Prot-decoy discrimination. Honest finding documented: expected because pipeline optimizes for safety, not raw discrimination.
> **New in v0.5.29:** Expanded benchmark to 500 AMPs + 500 composition-matched decoys (n=1000). AUROC 0.7792 (CI₉₅: 0.7505–0.8065) confirms signal generalizes. Cluster-aware CI: 0.746–0.8102. Representative AUROC: 0.778. Standard benchmark (n=191) retained for backward comparison.
> **Pipeline version:** v0.9.9 (O1 — calibration performance summary)
> **Pipeline version:** v0.10.6 (P5 — calibration cycle summary — Phase P complete)
> **Branch:** main

---
Expand Down Expand Up @@ -1231,6 +1231,7 @@ Decoys score low on activity. Selective AMPs score moderately on both.

| Date | Change | Author |
|------|--------|--------|
| 2026-07-10 | **Calibration cycle summary schema (Phase P P5 — Phase P complete):** CalibrationCycleSummaryEntry (13 fields: ccs_id, pipeline_version, bsp_id, psc_id, bos_id, cps_id, cba_id, crg_id_previous, crg_id_next, cycle_outcome, cycle_notes, reviewer, dry_lab_only). Index record for one complete calibration loop — CRG→BSP→PSC→Lab→BOS→CPS→CBA→CRG. 11 validation rules (7 artifact prefixes: CCS-/BSP-/PSC-/BOS-/CPS-/CBA-/CRG-, crg_id_previous≠crg_id_next, cycle_outcome in {improved,stable,degraded}, notes≤400 chars). 2 warnings: degraded outcome, dry-lab-only cycle. 63 tests. CLI: openamp-foundry calibration-cycle-summary-check. Completes Phase P: P1 BSP + P2 RRF + P3 BOS + P4 PSC + P5 CCS form the full calibration bridge. BASELINE 4910->4973. | OpenAMP Loop |
| 2026-07-10 | **Pilot batch safety clearance schema (Phase P P4):** PilotBatchSafetyClearanceEntry (13 fields: psc_id, bsp_id, pipeline_version, dual_use_risk_checked, novelty_verified, toxicity_screened, hemolysis_screened, max_safety_risk_tier, cleared_for_synthesis, rejection_ids, safety_notes, reviewer, dry_lab_only). All 4 safety screens mandatory. 8 validation rules: PSC-/BSP- prefixes, 4 screens all required True, risk tier in {low,moderate,high}, high-risk cannot be cleared (cleared_for_synthesis must be False), notes<=400 chars. 3 warnings: moderate-risk cleared, rejection_ids present, dry_lab_only clearance needs human review. 63 tests. CLI: openamp-foundry pilot-batch-safety-clearance-check. Safety constraint: max_safety_risk_tier=high cannot set cleared_for_synthesis=True. BASELINE 4847->4910. | OpenAMP Loop |
| 2026-07-10 | **Batch outcome summary schema (Phase P P3):** BatchOutcomeSummaryEntry (12 fields: bos_id, pipeline_version, bsp_id, batch_id, candidates_proposed, candidates_tested, candidates_active, candidates_inactive, is_synthetic, outcome_notes, reviewer, dry_lab_only). 9 validation rules (BOS-/BSP- prefixes, count constraints, synthetic/real boundary: is_synthetic=True requires dry_lab_only=True, notes<=400 chars). 3 warnings (untested candidates, real results marked dry-lab-only, zero active). 63 tests. CLI: openamp-foundry batch-outcome-summary-check. Closes the BSP-to-outcomes feedback loop. BASELINE 4784->4847. | OpenAMP Loop |
| 2026-07-10 | **Recalibration refusal record schema (Phase P P2):** RecalibrationRefusalEntry (10 fields, recalibration_refused must be True). 7 validation rules, 2 warnings. 63 tests. CLI: openamp-foundry recalibration-refusal-check. Complements CalibrationImprovementRecord (O3) with the rejection path. Corrected inflated BASELINE 4889->4784. | OpenAMP Loop |
Expand Down
1 change: 1 addition & 0 deletions docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,7 @@ preventing the pipeline from selecting candidates under poor calibration.
| P2 | Add recalibration refusal record schema (complete). | Documents when recalibration was correctly rejected; prevents spurious recalibrations and creates audit trail. | B/D |
| P3 | Add batch outcome summary schema (complete). | Closes the BSP→lab→outcomes feedback loop; enforces synthetic/real boundary at the batch level. | B/D |
| P4 | Add pilot batch safety clearance schema (complete). | Safety gate before wet-lab synthesis: all 4 screens required, high-risk batches cannot be cleared. | B/D |
| P5 | Add calibration cycle summary schema (complete). | Index record for one complete BSP→PSC→BOS→CPS→CBA→CRG cycle; crg_id_previous must differ from crg_id_next. | B/D |

## Prioritization rule

Expand Down
29 changes: 29 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -3696,3 +3696,32 @@ def _run_pilot_batch_safety_clearance_check(args) -> int:
for warn in result.warnings:
print(f" WARN: {warn}")
return 0 if result.passed else 1


def _run_calibration_cycle_summary_check(args) -> int:
import json
from openamp_foundry.evidence.calibration_cycle_summary import (
validate_calibration_cycle_summary_dict,
)
if args.entry_json:
try:
data = json.loads(args.entry_json)
except json.JSONDecodeError as exc:
print(f"Error: invalid JSON: {exc}", file=__import__("sys").stderr)
return 1
else:
data = json.load(__import__("sys").stdin)
result = validate_calibration_cycle_summary_dict(data)
if args.format == "json":
import dataclasses
print(json.dumps(dataclasses.asdict(result), indent=2))
else:
status = "PASS" if result.passed else "FAIL"
print(f"[{status}] Calibration Cycle Summary: {result.ccs_id}")
print(f" Outcome: {result.cycle_outcome}")
print(f" CRG: {result.crg_id_previous} → {result.crg_id_next}")
for err in result.errors:
print(f" ERROR: {err}")
for warn in result.warnings:
print(f" WARN: {warn}")
return 0 if result.passed else 1
12 changes: 12 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
_run_recalibration_refusal_check,
_run_batch_outcome_summary_check,
_run_pilot_batch_safety_clearance_check,
_run_calibration_cycle_summary_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2392,6 +2393,14 @@ def build_parser() -> argparse.ArgumentParser:
p_psc.add_argument("--format", choices=["text", "json"], default="text")
p_psc.set_defaults(func=_run_pilot_batch_safety_clearance_check)

p_ccs = sub.add_parser(
"calibration-cycle-summary-check",
help="Validate calibration cycle summary entry",
)
p_ccs.add_argument("--entry-json", default=None)
p_ccs.add_argument("--format", choices=["text", "json"], default="text")
p_ccs.set_defaults(func=_run_calibration_cycle_summary_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2779,6 +2788,9 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "pilot-batch-safety-clearance-check":
return _run_pilot_batch_safety_clearance_check(args)

if args.command == "calibration-cycle-summary-check":
return _run_calibration_cycle_summary_check(args)

parser.error("unknown command")
return 2

Expand Down
213 changes: 213 additions & 0 deletions src/openamp_foundry/evidence/calibration_cycle_summary.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
"""Calibration cycle summary schema — Phase P P5.

Index record for one complete calibration cycle. References all artifacts
from the cycle: the opening gate, the selection proposal, the safety clearance,
the experimental outcomes, the performance summary, the aggregator, and the
closing gate.

One CCS per completed cycle creates an auditable history of the learning loop.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from typing import List

CCS_PREFIX = "CCS-"
BSP_PREFIX = "BSP-"
PSC_PREFIX = "PSC-"
BOS_PREFIX = "BOS-"
CPS_PREFIX = "CPS-"
CBA_PREFIX = "CBA-"
CRG_PREFIX = "CRG-"
VALID_CYCLE_OUTCOMES = frozenset({"improved", "stable", "degraded"})
CYCLE_NOTES_MAX_LENGTH = 400


@dataclass
class CalibrationCycleSummaryEntry:
"""Index record for one complete calibration feedback cycle."""

ccs_id: str
pipeline_version: str
bsp_id: str
psc_id: str
bos_id: str
cps_id: str
cba_id: str
crg_id_previous: str
crg_id_next: str
cycle_outcome: str
cycle_notes: str
reviewer: str
dry_lab_only: bool = True


@dataclass
class CalibrationCycleSummaryResult:
ccs_id: str
pipeline_version: str
bsp_id: str
psc_id: str
bos_id: str
cps_id: str
cba_id: str
crg_id_previous: str
crg_id_next: str
cycle_outcome: str
passed: bool
errors: List[str]
warnings: List[str]
dry_lab_only: bool = True


def validate_calibration_cycle_summary(
entry: CalibrationCycleSummaryEntry,
) -> CalibrationCycleSummaryResult:
errors: List[str] = []
warnings: List[str] = []

if not entry.ccs_id.startswith(CCS_PREFIX):
errors.append(
f"ccs_id must start with '{CCS_PREFIX}', got '{entry.ccs_id}'"
)

if not entry.bsp_id.startswith(BSP_PREFIX):
errors.append(
f"bsp_id must start with '{BSP_PREFIX}', got '{entry.bsp_id}'"
)

if not entry.psc_id.startswith(PSC_PREFIX):
errors.append(
f"psc_id must start with '{PSC_PREFIX}', got '{entry.psc_id}'"
)

if not entry.bos_id.startswith(BOS_PREFIX):
errors.append(
f"bos_id must start with '{BOS_PREFIX}', got '{entry.bos_id}'"
)

if not entry.cps_id.startswith(CPS_PREFIX):
errors.append(
f"cps_id must start with '{CPS_PREFIX}', got '{entry.cps_id}'"
)

if not entry.cba_id.startswith(CBA_PREFIX):
errors.append(
f"cba_id must start with '{CBA_PREFIX}', got '{entry.cba_id}'"
)

if not entry.crg_id_previous.startswith(CRG_PREFIX):
errors.append(
f"crg_id_previous must start with '{CRG_PREFIX}', "
f"got '{entry.crg_id_previous}'"
)

if not entry.crg_id_next.startswith(CRG_PREFIX):
errors.append(
f"crg_id_next must start with '{CRG_PREFIX}', "
f"got '{entry.crg_id_next}'"
)

if entry.crg_id_previous == entry.crg_id_next:
errors.append(
"crg_id_previous and crg_id_next must differ — a completed cycle "
"must open a new calibration gate"
)

if entry.cycle_outcome not in VALID_CYCLE_OUTCOMES:
errors.append(
f"cycle_outcome must be one of {sorted(VALID_CYCLE_OUTCOMES)}, "
f"got '{entry.cycle_outcome}'"
)

if len(entry.cycle_notes) > CYCLE_NOTES_MAX_LENGTH:
errors.append(
f"cycle_notes exceeds {CYCLE_NOTES_MAX_LENGTH} characters "
f"(got {len(entry.cycle_notes)})"
)

# Warnings
if entry.cycle_outcome == "degraded" and not errors:
warnings.append(
"cycle_outcome is 'degraded' — calibration quality decreased this cycle; "
"consider whether a recalibration refusal was the right decision or "
"whether additional data is needed before next selection"
)

if entry.dry_lab_only and not errors:
warnings.append(
"dry_lab_only=True: this cycle summary covers a synthetic or "
"computational-only calibration loop — no real experimental data "
"was used to update calibration"
)

return CalibrationCycleSummaryResult(
ccs_id=entry.ccs_id,
pipeline_version=entry.pipeline_version,
bsp_id=entry.bsp_id,
psc_id=entry.psc_id,
bos_id=entry.bos_id,
cps_id=entry.cps_id,
cba_id=entry.cba_id,
crg_id_previous=entry.crg_id_previous,
crg_id_next=entry.crg_id_next,
cycle_outcome=entry.cycle_outcome,
passed=len(errors) == 0,
errors=errors,
warnings=warnings,
dry_lab_only=entry.dry_lab_only,
)


def validate_calibration_cycle_summary_dict(d: dict) -> CalibrationCycleSummaryResult:
missing = []
for k in (
"ccs_id",
"pipeline_version",
"bsp_id",
"psc_id",
"bos_id",
"cps_id",
"cba_id",
"crg_id_previous",
"crg_id_next",
"cycle_outcome",
"cycle_notes",
"reviewer",
):
if k not in d:
missing.append(k)
if missing:
return CalibrationCycleSummaryResult(
ccs_id=d.get("ccs_id", ""),
pipeline_version=d.get("pipeline_version", ""),
bsp_id=d.get("bsp_id", ""),
psc_id=d.get("psc_id", ""),
bos_id=d.get("bos_id", ""),
cps_id=d.get("cps_id", ""),
cba_id=d.get("cba_id", ""),
crg_id_previous=d.get("crg_id_previous", ""),
crg_id_next=d.get("crg_id_next", ""),
cycle_outcome=d.get("cycle_outcome", ""),
passed=False,
errors=[f"missing required fields: {missing}"],
warnings=[],
dry_lab_only=d.get("dry_lab_only", True),
)
entry = CalibrationCycleSummaryEntry(
ccs_id=d["ccs_id"],
pipeline_version=d["pipeline_version"],
bsp_id=d["bsp_id"],
psc_id=d["psc_id"],
bos_id=d["bos_id"],
cps_id=d["cps_id"],
cba_id=d["cba_id"],
crg_id_previous=d["crg_id_previous"],
crg_id_next=d["crg_id_next"],
cycle_outcome=d["cycle_outcome"],
cycle_notes=d["cycle_notes"],
reviewer=d["reviewer"],
dry_lab_only=bool(d.get("dry_lab_only", True)),
)
return validate_calibration_cycle_summary(entry)
Loading
Loading