Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase F F9 — negative result dashboard — v0.10.20)
> **Last updated:** 2026-07-10 (Phase G G1 — recalibration decision log — v0.10.21)

## v0.10.20 — Phase F F9 — Negative Result Dashboard (NRD-)

Expand Down Expand Up @@ -1261,6 +1261,14 @@ Decoys score low on activity. Selective AMPs score moderately on both.

## Change Log

### v0.10.21 — Phase G G1: RecalibrationDecisionLog schema
- Added `RecalibrationDecisionLog` (RDL-) schema
- 13 fields: rdl_id, pipeline_version, calibration_checkpoint, decision_date, trigger_type, trigger_artifact_id, decision_outcome, decision_authority, evidence_summary, rationale, next_review_date, conditions_if_deferred, notes
- 13 validation rules; 3 warnings; 63 tests
- BASELINE 5918→5981
- CLI: openamp-foundry recalibration-decision-log-check
- Closes Phase G G1 — governance audit trail for calibration decisions

| Date | Change | Author |
|------|--------|--------|
| 2026-07-10 | **Calibration cycle summary schema (Phase P P5 — Phase P complete):** CalibrationCycleSummaryEntry (13 fields: ccs_id, pipeline_version, bsp_id, psc_id, bos_id, cps_id, cba_id, crg_id_previous, crg_id_next, cycle_outcome, cycle_notes, reviewer, dry_lab_only). Index record for one complete calibration loop — CRG→BSP→PSC→Lab→BOS→CPS→CBA→CRG. 11 validation rules (7 artifact prefixes: CCS-/BSP-/PSC-/BOS-/CPS-/CBA-/CRG-, crg_id_previous≠crg_id_next, cycle_outcome in {improved,stable,degraded}, notes≤400 chars). 2 warnings: degraded outcome, dry-lab-only cycle. 63 tests. CLI: openamp-foundry calibration-cycle-summary-check. Completes Phase P: P1 BSP + P2 RRF + P3 BOS + P4 PSC + P5 CCS form the full calibration bridge. BASELINE 4910->4973. | OpenAMP Loop |
Expand Down
100 changes: 100 additions & 0 deletions docs/evidence/RECALIBRATION_DECISION_LOG_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# RecalibrationDecisionLog (RDL-) Guide

## Purpose

`RecalibrationDecisionLog` records every governance decision at a calibration
checkpoint — whether the pipeline was recalibrated, refused, deferred, or
partially updated — and preserves the evidence trail for each choice.

## Schema Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `rdl_id` | str | yes | Unique identifier, must start with `RDL-` |
| `pipeline_version` | str | yes | Pipeline version at decision time |
| `calibration_checkpoint` | str | yes | Named checkpoint (e.g. `batch_3_review`) |
| `decision_date` | str | yes | ISO date (YYYY-MM-DD) of decision |
| `trigger_type` | str | yes | What triggered the review (controlled vocabulary) |
| `trigger_artifact_id` | str | yes | ID of the artifact that triggered the review |
| `decision_outcome` | str | yes | `approved`, `refused`, `deferred`, `partial_approved` |
| `decision_authority` | str | yes | Who/what made the decision (controlled vocabulary) |
| `evidence_summary` | str | yes | Concise summary of evidence considered |
| `rationale` | str | yes | Reasoning behind decision (≤500 chars) |
| `next_review_date` | str | yes | ISO date for next scheduled review |
| `conditions_if_deferred` | str | no | Required when outcome is `deferred` |
| `notes` | str | no | Additional context (≤300 chars) |

## Controlled Vocabularies

**trigger_type:**
- `batch_outcome` — triggered by batch result data
- `calibration_performance_summary` — triggered by CPS- artifact
- `scheduled_review` — routine scheduled checkpoint
- `anomaly_detected` — triggered by anomaly detection
- `human_request` — human reviewer requested review

**decision_outcome:**
- `approved` — recalibration proceeds
- `refused` — recalibration rejected (see RRF-)
- `deferred` — decision postponed pending more data
- `partial_approved` — selective update to specific components

**decision_authority:**
- `automated_gate` — automated pipeline gate
- `pipeline_owner` — pipeline owner
- `external_reviewer` — external domain expert
- `safety_officer` — safety review officer
- `committee` — multi-member review committee

## Validation Rules

1. `rdl_id` starts with `RDL-`
2. `pipeline_version` non-empty
3. `calibration_checkpoint` non-empty
4. `decision_date` is ISO date (YYYY-MM-DD)
5. `trigger_type` in controlled vocabulary (5 values)
6. `trigger_artifact_id` starts with a known prefix (CPS-, DRM-, PCI-, BSP-, CIR-, RRF-)
7. `decision_outcome` in controlled vocabulary (4 values)
8. `decision_authority` in controlled vocabulary (5 values)
9. `evidence_summary` non-empty
10. `rationale` non-empty and ≤500 chars
11. `next_review_date` is ISO date (YYYY-MM-DD)
12. `conditions_if_deferred` required when outcome is `deferred`
13. `notes` ≤300 chars

## Warnings

- `refused` outcome: verify rationale cites a specific RRF- artifact
- `deferred` with no notes: consider documenting deferral context
- `notes` empty: consider adding context for future reviewers

## Usage

```python
from openamp_foundry.evidence.recalibration_decision_log import (
RecalibrationDecisionLog, validate
)

log = RecalibrationDecisionLog(
rdl_id="RDL-20240315-001",
pipeline_version="v2.1.0",
calibration_checkpoint="batch_3_review",
decision_date="2024-03-15",
trigger_type="batch_outcome",
trigger_artifact_id="CPS-20240314-001",
decision_outcome="approved",
decision_authority="automated_gate",
evidence_summary="Brier score improved from 0.28 to 0.21 after batch 3.",
rationale="Metric improvement exceeds minimum threshold and is consistent across 3 cohorts.",
next_review_date="2024-06-15",
)
errors = validate(log)
assert errors == []
```

## Integration

- Triggered by: `CPS-` (calibration performance summary), `PCI-` (post-experiment intake)
- Refused outcomes generate: `RRF-` (recalibration refusal record)
- Approved outcomes generate: `CIR-` (calibration improvement record)
- Feeds into: governance audit trail, decision pattern analysis
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Make learning controlled rather than self-serving.

| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| G1 | Add machine-readable recalibration decision log examples. | Improves governance memory. | B |
| G1 | Add machine-readable recalibration decision log examples (complete). | RDL- schema: 13 fields, 13 validation rules, 4 outcomes, 5 trigger types, deferred-requires-conditions enforced; closes governance audit trail for calibration decisions. | B |
| G2 | Add recalibration rejection examples. | Shows gate success by refusal. | B |
| G3 | Add `make calibration-audit`. | Checks intake/gate/engine/report consistency. | C |
| G4 | Add active-learning strategy comparison report. | Prevents one-selector bias. | C |
Expand Down
17 changes: 17 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -1285,6 +1285,23 @@ def _run_validate_simulation_result(args: argparse.Namespace) -> int:
return 3 if batch_result["any_invalid"] else 0


def _run_recalibration_decision_log_check(args):
"""CLI handler for recalibration-decision-log-check."""
import json, sys
from openamp_foundry.evidence.recalibration_decision_log import validate_dict
data = json.load(open(args.input))
issues = validate_dict(data)
errors = [i for i in issues if not i.startswith("WARNING:")]
warnings = [i for i in issues if i.startswith("WARNING:")]
for w in warnings:
print(w)
if errors:
for e in errors:
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print("OK: RecalibrationDecisionLog is valid.")


def _run_domain_review_outcome_check(args):
from openamp_foundry.evidence.domain_review_outcome import (
validate_domain_review_outcome_dict,
Expand Down
8 changes: 8 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
_run_calibration_cycle_summary_check,
_run_pilot_evidence_package_check,
_run_pre_registration_entry_check,
_run_recalibration_decision_log_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2528,6 +2529,10 @@ def build_parser() -> argparse.ArgumentParser:
)
nrd_parser.add_argument("json_input", help="JSON string of the NRD- record")

p_rdl = subparsers.add_parser("recalibration-decision-log-check", help="Validate a RecalibrationDecisionLog JSON")
p_rdl.add_argument("input", help="Path to JSON file")
p_rdl.set_defaults(func=_run_recalibration_decision_log_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down Expand Up @@ -2970,6 +2975,9 @@ def main(argv: list[str] | None = None) -> int:
from openamp_foundry.cli.commands.reports import _run_negative_result_dashboard_check
_run_negative_result_dashboard_check(args)

elif args.command == "recalibration-decision-log-check":
_run_recalibration_decision_log_check(args)

else:
parser.error("unknown command")
return 2
Expand Down
173 changes: 173 additions & 0 deletions src/openamp_foundry/evidence/recalibration_decision_log.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
"""Recalibration decision log schema — Phase G G1.

Records the governance decision at each recalibration checkpoint:
whether recalibration was approved or refused, the trigger artifact,
the deciding evidence, and who (or what process) made the call.

This schema enables audit trails across multiple calibration cycles
so patterns of approval/refusal can themselves be reviewed.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

RDL_PREFIX = "RDL-"

VALID_DECISION_OUTCOMES = frozenset({
"approved",
"refused",
"deferred",
"partial_approved",
})

VALID_TRIGGER_TYPES = frozenset({
"batch_outcome",
"calibration_performance_summary",
"scheduled_review",
"anomaly_detected",
"human_request",
})

VALID_TRIGGER_ID_PREFIXES = frozenset({
"CPS-", "DRM-", "PCI-", "BSP-", "CIR-", "RRF-",
})

VALID_DECISION_AUTHORITIES = frozenset({
"automated_gate",
"pipeline_owner",
"external_reviewer",
"safety_officer",
"committee",
})

RATIONALE_MAX_LENGTH = 500
NOTES_MAX_LENGTH = 300
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


@dataclass
class RecalibrationDecisionLog:
rdl_id: str
pipeline_version: str
calibration_checkpoint: str
decision_date: str
trigger_type: str
trigger_artifact_id: str
decision_outcome: str
decision_authority: str
evidence_summary: str
rationale: str
next_review_date: str
conditions_if_deferred: str = ""
notes: str = ""


def validate(log: RecalibrationDecisionLog) -> List[str]:
"""Return list of error strings; empty list means valid."""
errors: List[str] = []
warnings: List[str] = []

# Rule 1: ID prefix
if not log.rdl_id.startswith(RDL_PREFIX):
errors.append(f"rdl_id must start with '{RDL_PREFIX}', got: {log.rdl_id!r}")

# Rule 2: pipeline_version non-empty
if not log.pipeline_version.strip():
errors.append("pipeline_version must not be empty")

# Rule 3: calibration_checkpoint non-empty
if not log.calibration_checkpoint.strip():
errors.append("calibration_checkpoint must not be empty")

# Rule 4: decision_date ISO format
if not _ISO_DATE_RE.match(log.decision_date):
errors.append(f"decision_date must be YYYY-MM-DD, got: {log.decision_date!r}")

# Rule 5: trigger_type vocabulary
if log.trigger_type not in VALID_TRIGGER_TYPES:
errors.append(
f"trigger_type must be one of {sorted(VALID_TRIGGER_TYPES)}, "
f"got: {log.trigger_type!r}"
)

# Rule 6: trigger_artifact_id must start with a known prefix
if not any(log.trigger_artifact_id.startswith(p) for p in VALID_TRIGGER_ID_PREFIXES):
errors.append(
f"trigger_artifact_id must start with one of "
f"{sorted(VALID_TRIGGER_ID_PREFIXES)}, got: {log.trigger_artifact_id!r}"
)

# Rule 7: decision_outcome vocabulary
if log.decision_outcome not in VALID_DECISION_OUTCOMES:
errors.append(
f"decision_outcome must be one of {sorted(VALID_DECISION_OUTCOMES)}, "
f"got: {log.decision_outcome!r}"
)

# Rule 8: decision_authority vocabulary
if log.decision_authority not in VALID_DECISION_AUTHORITIES:
errors.append(
f"decision_authority must be one of {sorted(VALID_DECISION_AUTHORITIES)}, "
f"got: {log.decision_authority!r}"
)

# Rule 9: evidence_summary non-empty
if not log.evidence_summary.strip():
errors.append("evidence_summary must not be empty")

# Rule 10: rationale non-empty and length
if not log.rationale.strip():
errors.append("rationale must not be empty")
elif len(log.rationale) > RATIONALE_MAX_LENGTH:
errors.append(
f"rationale exceeds {RATIONALE_MAX_LENGTH} chars "
f"(got {len(log.rationale)})"
)

# Rule 11: next_review_date ISO format
if not _ISO_DATE_RE.match(log.next_review_date):
errors.append(
f"next_review_date must be YYYY-MM-DD, got: {log.next_review_date!r}"
)

# Rule 12: deferred outcome requires conditions
if log.decision_outcome == "deferred" and not log.conditions_if_deferred.strip():
errors.append(
"conditions_if_deferred must not be empty when decision_outcome is 'deferred'"
)

# Rule 13: notes length
if len(log.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(log.notes)})"
)

# Warnings (returned with "WARNING:" prefix)
if log.decision_outcome == "refused":
warnings.append(
"WARNING: decision_outcome is 'refused' — verify refusal rationale "
"cites a specific RRF- artifact or documented reason"
)
if log.decision_outcome == "deferred" and not log.notes.strip():
warnings.append(
"WARNING: deferred decision has no notes — consider documenting "
"the deferral context"
)
if not log.notes.strip():
warnings.append(
"WARNING: notes is empty — consider adding context for future reviewers"
)

return errors + warnings


def validate_dict(data: dict) -> List[str]:
"""Validate a plain dict by constructing RecalibrationDecisionLog first."""
try:
log = RecalibrationDecisionLog(**data)
except TypeError as exc:
return [f"Schema construction error: {exc}"]
return validate(log)
Loading
Loading