feat: Phase O O3 calibration improvement record schema -- CIR- before/after audit with metric direction validation

docs/evidence/CALIBRATION_IMPROVEMENT_RECORD_GUIDE.md

@@ -0,0 +1,96 @@
+# Calibration Improvement Record Guide (CIR-)
+
+## Purpose
+
+A `CalibrationImprovementRecord` (CIR-) documents what changed in the calibration
+model and by how much. It is the "before/after" audit record for calibration updates —
+recording the metric name, before/after values, the data source used, and the rationale
+for the change.
+
+CIR- completes Phase O alongside O1 (calibration performance summary), O2 (prediction
+drift monitor), O4 (cross-batch aggregator), and O5 (calibration readiness gate). Without
+CIR-, there is no structured record of what actually changed between calibration versions.
+
+## Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `cir_id` | str | Yes | Unique ID, must start with `CIR-` |
+| `pipeline_version` | str | Yes | Version of the pipeline performing calibration |
+| `calibration_version_before` | str | Yes | Calibration version before the update |
+| `calibration_version_after` | str | Yes | Calibration version after the update (must differ) |
+| `improvement_date` | str | Yes | ISO date (YYYY-MM-DD) of the improvement |
+| `metric_name` | str | Yes | Controlled vocabulary: which metric improved |
+| `metric_value_before` | float | Yes | Metric value before calibration update |
+| `metric_value_after` | float | Yes | Metric value after calibration update |
+| `improvement_confirmed` | bool | Yes | Must be True; not recorded unless confirmed improvement |
+| `improvement_rationale` | str | Yes | Why this is a genuine improvement (≤400 chars) |
+| `data_source_id` | str | Yes | Batch or dataset used for recalibration |
+| `notes` | str | No | Additional context (≤300 chars) |
+
+## Valid Metric Names
+
+### Higher Is Better
+
+| Metric | Description |
+|--------|-------------|
+| `auroc` | Area under the ROC curve |
+| `auprc` | Area under the precision-recall curve |
+| `precision_at_k` | Precision at top-K candidates |
+| `recall_at_k` | Recall at top-K candidates |
+| `f1_at_threshold` | F1 at a fixed decision threshold |
+
+### Lower Is Better
+
+| Metric | Description |
+|--------|-------------|
+| `brier_score` | Mean squared error of probability predictions |
+| `calibration_error` | Mean absolute calibration error |
+| `expected_calibration_error` | ECE across probability bins |
+
+## Validation Rules
+
+1. `cir_id` must start with `CIR-`
+2. `pipeline_version` must be non-empty
+3. `calibration_version_before` must be non-empty
+4. `calibration_version_after` must be non-empty
+5. `calibration_version_before` and `calibration_version_after` must differ
+6. `improvement_date` must match `YYYY-MM-DD`
+7. `metric_name` must be in valid set (8 values)
+8. `improvement_confirmed` must be True
+9. `improvement_rationale` must be non-empty and ≤400 chars
+10. `data_source_id` must be non-empty
+11. `notes` must be ≤300 chars
+
+## Warnings
+
+| Condition | Warning |
+|-----------|---------|
+| Higher-is-better metric but `after ≤ before` | Metric went wrong direction; verify it's a genuine improvement |
+| Lower-is-better metric but `after ≥ before` | Metric went wrong direction; verify it's a genuine improvement |
+| Improvement magnitude < 0.005 | Very small improvement; verify it's scientifically meaningful |
+| `notes` is empty | No reviewer context recorded |
+
+## Boundaries
+
+- **CIR- records confirmed improvements only.** `improvement_confirmed` must be True. If calibration regressed, do not create a CIR-; instead investigate the regression.
+- **CIR- is per-metric.** One CIR- per metric improved per calibration cycle. If AUROC and ECE both improved, create two CIR- records.
+- **Direction validation is advisory (warning, not error).** The schema warns if the metric moved the wrong way but does not error, because calibration objectives sometimes involve trade-offs between metrics.
+- **`data_source_id` links to the real outcomes used.** This makes calibration updates reproducible.
+
+## CLI
+
+```bash
+openamp-foundry calibration-improvement-record-check '{"cir_id": "CIR-001", ...}'
+```
+
+Returns `VALID` or `INVALID` with any errors and warnings.
+
+## Relationship to Other Schemas
+
+```
+O1 (CPS-)  ──→  CIR-  (performance summary triggers improvement effort)
+O2 (PDM-)  ──→  CIR-  (drift monitor identifies what needs fixing)
+CIR-       ──→  O5    (improved calibration feeds readiness gate)
+CIR-       ──→  P2    (rejected recalibrations are documented separately via P2/CRG-)
+```

docs/evidence/METRICS_CURRENT.md

@@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-10 (Phase K K2 — batch experiment priority ranker — v0.10.17)
+> **Last updated:** 2026-07-10 (Phase O O3 — calibration improvement record — v0.10.18)
+> **New in v0.10.18 (Phase O O3 — Calibration Improvement Record (CIR-)):** CalibrationImprovementRecord — 12 fields, before/after audit record for calibration updates. 11 validation rules — CIR- prefix, pipeline_version non-empty, calibration_version_before/after non-empty and must differ, ISO date, metric_name controlled vocab (8 values), improvement_confirmed=True enforced, improvement_rationale non-empty and ≤400 chars, data_source_id non-empty, notes ≤300 chars. 3 warnings — wrong direction for higher-is-better/lower-is-better metrics, very small improvement (<0.005), notes empty. 63 tests across 9 test classes; BASELINE 5729→5792. CLI: openamp-foundry calibration-improvement-record-check. Completes Phase O: joins O1/O2/O4/O5 with the missing before/after change record.
 > **New in v0.10.17 (Phase K K2 — Batch Experiment Priority Ranker):** BatchExperimentPriorityRanker — 11 fields, synthesis wave priority ordering. 10 validation rules — BPR- prefix, pipeline_version+batch_id non-empty, CSR- prefix, ISO date, priority_method controlled vocab (6 values), top_priority_candidates non-empty, priority_rationale non-empty and ≤400 chars, synthesis_wave ≥1, notes ≤300 chars. 3 warnings — resource_constraint_considered=False, expert_ranked without notes, synthesis_wave >5. 63 tests across 9 test classes; BASELINE 5666→5729. CLI: openamp-foundry batch-experiment-priority-ranker-check. BPR- records which candidates to synthesize first; CSR- records why candidates were selected.
 > **New in v0.10.16 (Phase K K1 — Candidate Selection Rationale):** CandidateSelectionRationale — 12 fields, auditable candidate selection decisions. 12 validation rules: CSR- prefix, pipeline_version+batch_id non-empty, BSP- prefix, ISO date, strategy controlled vocab (4 values), candidate_count ≥1, count/ids consistency, ranking_method controlled vocab (6 values), calibration_gate_passed=True enforced, rationale non-empty and ≤400 chars, notes ≤300 chars. 3 warnings: large batch (>20), random_balanced without notes, expert_review without notes. 63 tests; BASELINE 5603→5666. CLI: openamp-foundry candidate-selection-rationale-check.
 > **New in v0.10.15 (Phase K K3):** PilotPackageCompletenessReport — PPC- completeness gate: confirms CCS-+BSP-+PSC-+PRE-+BCM- all present and ESC- cleared before external sharing (12 fields, 12 validation rules, 3 warnings, 63 tests, BASELINE 5540→5603)

docs/research/NEXT_100_PR_MAP.md

@@ -245,7 +245,7 @@ Track how well predictions match experimental outcomes over time. Detect drift,
 |---:|---|---|---|
 | O1 | Add calibration performance summary schema (complete — Loop 138). | Aggregates prediction accuracy metrics across batches with known outcomes. | B/D |
 | O2 | Add prediction drift monitor schema (complete — Loop 139). | Detects when pipeline predictions are systematically shifting. | B/D |
-| O3 | Add calibration improvement record schema. | Documents what was changed to improve calibration and by how much. | B/D |
+| O3 | Add calibration improvement record schema (complete). | CIR- before/after audit record: metric direction validation (higher/lower-is-better), version diff enforced, improvement_confirmed=True required, per-metric granularity. | B/D |
 | O4 | Add cross-batch performance aggregator schema (complete — Loop 140). | Aggregates results across batches for trend analysis. | B/D |
 | [x] O5 | Add calibration readiness gate schema. | Validates calibration quality is sufficient before releasing the next batch. | B/D |
 

src/openamp_foundry/cli/commands/reports.py

@@ -2948,3 +2948,25 @@ def _run_batch_experiment_priority_ranker_check(args):
         for w in result.warnings:
             print(f"WARNING: {w}")
         sys.exit(1)
+
+
+def _run_calibration_improvement_record_check(args):
+    import json, sys
+    from openamp_foundry.evidence.calibration_improvement_record import validate_dict
+    try:
+        data = json.loads(args.json_input)
+    except json.JSONDecodeError as exc:
+        print(f"INVALID: JSON parse error: {exc}", file=sys.stderr)
+        sys.exit(1)
+    result = validate_dict(data)
+    if result.valid:
+        print("VALID")
+        for w in result.warnings:
+            print(f"WARNING: {w}")
+    else:
+        print("INVALID")
+        for e in result.errors:
+            print(f"ERROR: {e}")
+        for w in result.warnings:
+            print(f"WARNING: {w}")
+        sys.exit(1)

src/openamp_foundry/cli/main.py

@@ -53,6 +53,7 @@
 from pathlib import Path
 
 from openamp_foundry.evidence.batch_experiment_priority_ranker import validate_dict as validate_batch_experiment_priority_ranker_dict
+from openamp_foundry.evidence.calibration_improvement_record import validate_dict as validate_calibration_improvement_record_dict
 from openamp_foundry.evidence.candidate_selection_rationale import validate_dict as validate_candidate_selection_rationale_dict
 from openamp_foundry.evidence.negative_result_entry import validate_dict as validate_negative_result_entry_dict
 from openamp_foundry.evidence.pilot_package_completeness_report import validate_dict as validate_pilot_package_completeness_dict
@@ -2507,6 +2508,12 @@ def build_parser() -> argparse.ArgumentParser:
     )
     bpr_parser.add_argument("json_input", help="JSON string of the BPR- record")
 
+    cir_parser = subparsers.add_parser(
+        "calibration-improvement-record-check",
+        help="Validate a CalibrationImprovementRecord (CIR-) record",
+    )
+    cir_parser.add_argument("json_input", help="JSON string of the CIR- record")
+
     # ── Selection rationale check (Phase K K1) ───────────────────────
     src2 = sub.add_parser(
         "selection-rationale-check",
@@ -2937,6 +2944,10 @@ def main(argv: list[str] | None = None) -> int:
         from openamp_foundry.cli.commands.reports import _run_batch_experiment_priority_ranker_check
         _run_batch_experiment_priority_ranker_check(args)
 
+    elif args.command == "calibration-improvement-record-check":
+        from openamp_foundry.cli.commands.reports import _run_calibration_improvement_record_check
+        _run_calibration_improvement_record_check(args)
+
     else:
         parser.error("unknown command")
     return 2