feat: Phase O O4 cross-batch performance aggregator schema -- trend detection across N batches feeds calibration readiness gate

Makefile

@@ -978,3 +978,7 @@ dataset-release-check:
 .PHONY: calibration-improvement-check
 calibration-improvement-check:
 	openamp-foundry calibration-improvement-check --entry-json '{"improvement_id":"CIR-TEST","pipeline_version_before":"v1.0","pipeline_version_after":"v1.1","trigger_ids":["CPS-001"],"action_taken":"threshold adjustment","action_category":"threshold_adjustment","brier_score_before":0.3,"brier_score_after":0.2,"improvement_confirmed":true,"reviewer":"test"}'
+
+.PHONY: cross-batch-aggregator-check
+cross-batch-aggregator-check:
+	openamp-foundry cross-batch-aggregator-check --entry-json '{"aggregator_id":"CBA-TEST","pipeline_version":"v1.0","batch_ids_included":["A","B"],"summary_ids_included":["CPS-A","CPS-B"],"mean_brier_score":0.15,"min_brier_score":0.10,"max_brier_score":0.20,"trend":"stable","total_candidates_evaluated":100,"aggregation_notes":"","reviewer":"test"}'

docs/evidence/CROSS_BATCH_AGGREGATOR_GUIDE.md

@@ -0,0 +1,54 @@
+# Cross-Batch Performance Aggregator Guide
+
+Schema: `CrossBatchAggregatorEntry` — aggregates calibration performance across multiple batches.
+
+## Purpose
+
+A single batch's calibration metrics (Brier score, FP rate, recall) may be noisy. Tracking
+performance across N batches reveals whether calibration is trending up, down, or holding steady
+— the key input to the calibration readiness gate (O5).
+
+## Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `aggregator_id` | str | Unique ID, must start with "CBA-" |
+| `pipeline_version` | str | Pipeline version being aggregated |
+| `batch_ids_included` | List[str] | Batch IDs included in aggregation (at least 2) |
+| `summary_ids_included` | List[str] | CPS- record IDs included (at least 2, must match batch count) |
+| `mean_brier_score` | float | Mean Brier score across batches (0.0–1.0) |
+| `min_brier_score` | float | Minimum Brier score across batches (0.0–1.0) |
+| `max_brier_score` | float | Maximum Brier score across batches (0.0–1.0) |
+| `trend` | str | One of: improving, stable, degrading |
+| `total_candidates_evaluated` | int | Total candidates across all batches (>= 2) |
+| `aggregation_notes` | str | Max 400 chars |
+| `reviewer` | str | Who reviewed this aggregation |
+| `dry_lab_only` | bool | True (computational aggregation) |
+
+## Validation Rules
+
+1. `aggregator_id` must start with "CBA-"
+2. `batch_ids_included` must have at least 2 entries
+3. `summary_ids_included` must have at least 2 entries and each must start with "CPS-"
+4. `len(batch_ids_included) == len(summary_ids_included)`
+5. `mean_brier_score`, `min_brier_score`, `max_brier_score` must be in [0.0, 1.0]
+6. `min_brier_score <= mean_brier_score <= max_brier_score`
+7. `trend` must be one of: improving, stable, degrading
+8. `total_candidates_evaluated` must be >= 2
+9. `aggregation_notes` must not exceed 400 chars
+
+## Warning Conditions
+
+- `mean_brier_score >= POOR_BRIER_THRESHOLD (0.25)` — calibration below par
+- `max_brier_score - min_brier_score >= HIGH_VARIANCE_THRESHOLD (0.2)` — high variance across batches
+- trend == "degrading" — performance getting worse
+
+## Constants
+
+- `AGGREGATION_NOTES_MAX_LENGTH = 400`
+- `VALID_TRENDS = {"improving", "stable", "degrading"}`
+- `POOR_BRIER_THRESHOLD = 0.25`
+- `HIGH_VARIANCE_THRESHOLD = 0.2`
+- `MIN_BATCHES = 2`
+- `SUMMARY_ID_PREFIX = "CBA-"`
+- `CPS_PREFIX = "CPS-"`

docs/evidence/METRICS_CURRENT.md

@@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
 > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
 > with this file, this file wins. Updated whenever benchmark/benchmark config changes.
 >
-> **Last updated:** 2026-07-10 (Phase O O2 — prediction drift monitor — v0.10.0)
+> **Last updated:** 2026-07-10 (Phase O O4 — cross-batch performance aggregator — v0.10.2)
+> **New in v0.10.2 (Phase O O4):** CrossBatchAggregatorEntry schema (12 fields, 9 validation rules, 3 warnings). Added validate_cross_batch_aggregator() and validate_cross_batch_aggregator_dict(). Added 68 tests. CLI: openamp-foundry cross-batch-aggregator-check.
 > **New in v0.10.0 (Phase O O2):** PredictionDriftEntry schema (15 fields: monitor_id, pipeline_version, reference_batch_id, evaluation_batch_id, reference_mean_score, reference_std_score, evaluation_mean_score, evaluation_std_score, mean_shift_magnitude, population_size_reference, population_size_evaluation, drift_flag, drift_notes, reviewer, dry_lab_only). 8 validation rules (DRM- prefix, batch id must differ, scores 0.0-1.0, std >= 0.0, mean shift tolerance ±0.001, population >=1, notes <=400 chars, drift_flag=True requires non-empty notes). 4 warning conditions (unreported large shift >=0.1, small population <10, variance explosion >2x). 5 constants (DRIFT_NOTES_MAX_LENGTH=400, MEAN_SHIFT_TOLERANCE=0.001, SIGNIFICANT_DRIFT_THRESHOLD=0.1, MIN_POPULATION_FOR_RELIABLE_DRIFT=10, VARIANCE_EXPLOSION_RATIO=2.0). PredictionDriftEntry (15 fields), PredictionDriftResult (8 fields: monitor_id, pipeline_version, mean_shift_magnitude, drift_flag, passed, errors, warnings, dry_lab_only). validate_prediction_drift() (12+ error checks, 4 warning conditions). validate_prediction_drift_dict() (14 required fields guard). CLI (openamp-foundry prediction-drift-check) with --entry-json, --format text|json. make prediction-drift-check target. 68 tests. Phase O O2: prediction drift monitor is the earliest warning signal for pipeline score distribution shifts before wet-lab validation data arrives.
 > **New in v0.9.9:** Calibration performance summary schema (O1 — Loop 138) — docs/evidence/CALIBRATION_PERFORMANCE_GUIDE.md with purpose, schema fields table (13 fields: summary_id, pipeline_version, evaluation_date, batch_ids_evaluated, total_candidates_evaluated, true_positive_count, false_positive_count, true_negative_count, false_negative_count, brier_score, calibration_notes, reviewer, dry_lab_only), 4 derived metrics (precision, recall, specificity, F1), 9 validation rules (CPS- prefix, >=1 batch, >=1 candidates, all counts >=0, total == TP+FP+TN+FN, Brier 0.0-1.0, notes<=400 chars, non-empty reviewer, dry_lab_only must be False), 4 warning conditions (high FP rate >0.5, low recall <0.3, poor Brier >0.25, small sample <10), honest-use boundary. CalibrationPerformanceEntry dataclass (13 fields, dry_lab_only=False), CalibrationPerformanceResult dataclass (8 fields: summary_id, pipeline_version, total_candidates_evaluated, brier_score, passed, errors, warnings, dry_lab_only=False), MAX_CALIBRATION_NOTES_LENGTH=400, MIN_CANDIDATES_FOR_RELIABLE_ESTIMATE=10, HIGH_FP_RATE_THRESHOLD=0.5, LOW_RECALL_THRESHOLD=0.3, POOR_BRIER_SCORE_THRESHOLD=0.25. validate_calibration_performance() (9+ error checks: CPS- prefix, >=1 batch, >=1 candidates, >=0 counts, confusion sum match, Brier bounds, notes<=400, non-empty reviewer, dry_lab_only must be False; 4 warnings: high FP, low recall, poor Brier, small sample). validate_calibration_performance_dict() (12 required fields guard). CLI (openamp-foundry calibration-performance-check) with --entry-json, --format text|json. make calibration-performance-check target. 68 tests. Phase O O1: calibration performance summaries now measure how well predictions match experimental outcomes. Phase O (Calibration Quality Assurance) started.
 > **New in v0.9.8:** Experiment priority justification schema (N5 — Loop 137) — docs/evidence/EXPERIMENT_PRIORITY_GUIDE.md with purpose, schema fields table (12 fields: justification_id, batch_id, pipeline_version, decision_date, selection_criteria, rejected_alternatives, rejection_rationale, resource_constraint, safety_reviewed, pre_specified, decided_by, dry_lab_only), 8 validation rules (EPJ- prefix, >=2 selection criteria, >=1 rejected alternative, non-empty rejection_rationale<=500 chars, resource_constraint<=200 chars, safety_reviewed must be True, non-empty decided_by, dry_lab_only must be True), 4 warning conditions (post-hoc, minimum criteria, no resource constraint, many criteria), honest-use boundary. ExperimentPriorityEntry dataclass (12 fields, dry_lab_only=True), ExperimentPriorityResult dataclass (8 fields: justification_id, batch_id, criteria_count, rejected_alternative_count, safety_reviewed, passed, errors, warnings, dry_lab_only=True), MINIMUM_SELECTION_CRITERIA=2, MAXIMUM_SELECTION_CRITERIA_WARNING=6, MAX_REJECTION_RATIONALE_LENGTH=500, MAX_RESOURCE_CONSTRAINT_LENGTH=200. validate_experiment_priority() (8+ error checks: EPJ- prefix, >=2 criteria, >=1 alternative, non-empty rationale<=500 chars, constraint<=200 chars, safety_reviewed must be True, non-empty decided_by, dry_lab_only must be True; 4 warnings: post-hoc, minimum criteria, no constraint, many criteria). validate_experiment_priority_dict() (11 required fields guard). CLI (openamp-foundry experiment-priority-check) with --entry-json, --format text|json. make experiment-priority-check target. 62 tests. Phase N N5: batch selection decisions are now documented with full audit trail over alternatives. Phase N (Pre-registration & Baseline Honesty) is fully complete with N1-N5.

docs/research/NEXT_100_PR_MAP.md

@@ -246,7 +246,7 @@ Track how well predictions match experimental outcomes over time. Detect drift,
 | O1 | Add calibration performance summary schema (complete — Loop 138). | Aggregates prediction accuracy metrics across batches with known outcomes. | B/D |
 | O2 | Add prediction drift monitor schema (complete — Loop 139). | Detects when pipeline predictions are systematically shifting. | B/D |
 | O3 | Add calibration improvement record schema. | Documents what was changed to improve calibration and by how much. | B/D |
-| O4 | Add cross-batch performance aggregator schema. | Aggregates results across batches for trend analysis. | B/D |
+| O4 | Add cross-batch performance aggregator schema (complete — Loop 140). | Aggregates results across batches for trend analysis. | B/D |
 | O5 | Add calibration readiness gate schema. | Validates calibration quality is sufficient before releasing the next batch. | B/D |
 
 ## Prioritization rule

docs/research/ROADMAP.md

@@ -1,5 +1,10 @@
 # Roadmap
 
+### v0.10.2 - Phase O O4: Cross-Batch Performance Aggregator
+- CrossBatchAggregatorEntry aggregates CalibrationPerformanceEntry results across N>=2 batches
+- trend field (improving/stable/degrading) feeds calibration readiness gate
+- min/mean/max Brier scores, high-variance warning
+
 ### v0.10.1 - Phase O O3: Calibration Improvement Record
 - CalibrationImprovementEntry schema documents recalibration actions taken after drift/quality failures
 - 6 action categories, trigger_ids linked to CPS-/DRM- records

src/openamp_foundry/cli/commands/reports.py

@@ -3488,3 +3488,30 @@ def _run_calibration_improvement_check(args) -> int:
             print(f"  WARN:  {warn}")
 
     return 0 if result.passed else 1
+
+
+def _run_cross_batch_aggregator_check(args) -> int:
+    import json
+    from openamp_foundry.evidence.cross_batch_aggregator import (
+        validate_cross_batch_aggregator_dict,
+    )
+    if args.entry_json:
+        try:
+            data = json.loads(args.entry_json)
+        except json.JSONDecodeError as exc:
+            print(f"Error: invalid JSON: {exc}", file=__import__("sys").stderr)
+            return 1
+    else:
+        data = json.load(__import__("sys").stdin)
+    result = validate_cross_batch_aggregator_dict(data)
+    if args.format == "json":
+        import dataclasses
+        print(json.dumps(dataclasses.asdict(result), indent=2))
+    else:
+        status = "PASS" if result.passed else "FAIL"
+        print(f"[{status}] Cross-Batch Aggregator: {result.aggregator_id}")
+        for err in result.errors:
+            print(f"  ERROR: {err}")
+        for warn in result.warnings:
+            print(f"  WARN:  {warn}")
+    return 0 if result.passed else 1

src/openamp_foundry/cli/main.py

@@ -21,6 +21,7 @@
     _run_hypothesis_outcome_check, _run_baseline_comparison_check, _run_negative_result_check,
     _run_experiment_priority_check, _run_calibration_performance_check, _run_prediction_drift_check,
     _run_calibration_improvement_check,
+    _run_cross_batch_aggregator_check,
 )
 from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
 from openamp_foundry.cli.commands.reports import _run_contribution_check
@@ -2338,6 +2339,14 @@ def build_parser() -> argparse.ArgumentParser:
     p_cir.add_argument("--format", choices=["text", "json"], default="text")
     p_cir.set_defaults(func=_run_calibration_improvement_check)
 
+    p_cba = sub.add_parser(
+        "cross-batch-aggregator-check",
+        help="Validate cross-batch performance aggregator entry",
+    )
+    p_cba.add_argument("--entry-json", default=None)
+    p_cba.add_argument("--format", choices=["text", "json"], default="text")
+    p_cba.set_defaults(func=_run_cross_batch_aggregator_check)
+
     # ── Selection rationale check (Phase K K1) ───────────────────────
     src2 = sub.add_parser(
         "selection-rationale-check",
@@ -2707,6 +2716,9 @@ def main(argv: list[str] | None = None) -> int:
     if args.command == "prediction-drift-check":
         return _run_prediction_drift_check(args)
 
+    if args.command == "cross-batch-aggregator-check":
+        return _run_cross_batch_aggregator_check(args)
+
     parser.error("unknown command")
     return 2
 

src/openamp_foundry/evidence/cross_batch_aggregator.py

@@ -0,0 +1,190 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import List, Set
+
+AGGREGATION_NOTES_MAX_LENGTH: int = 400
+VALID_TRENDS: Set[str] = {"improving", "stable", "degrading"}
+POOR_BRIER_THRESHOLD: float = 0.25
+HIGH_VARIANCE_THRESHOLD: float = 0.2
+MIN_BATCHES: int = 2
+SUMMARY_ID_PREFIX: str = "CBA-"
+CPS_PREFIX: str = "CPS-"
+
+
+@dataclass
+class CrossBatchAggregatorEntry:
+    aggregator_id: str
+    pipeline_version: str
+    batch_ids_included: List[str]
+    summary_ids_included: List[str]
+    mean_brier_score: float
+    min_brier_score: float
+    max_brier_score: float
+    trend: str
+    total_candidates_evaluated: int
+    aggregation_notes: str
+    reviewer: str
+    dry_lab_only: bool = True
+
+
+@dataclass
+class CrossBatchAggregatorResult:
+    aggregator_id: str
+    pipeline_version: str
+    mean_brier_score: float
+    trend: str
+    passed: bool
+    errors: List[str] = field(default_factory=list)
+    warnings: List[str] = field(default_factory=list)
+    dry_lab_only: bool = True
+
+
+def validate_cross_batch_aggregator(
+    entry: CrossBatchAggregatorEntry,
+) -> CrossBatchAggregatorResult:
+    errors: List[str] = []
+    warnings: List[str] = []
+
+    if not entry.aggregator_id.startswith(SUMMARY_ID_PREFIX):
+        errors.append(
+            f"aggregator_id must start with '{SUMMARY_ID_PREFIX}', "
+            f"got '{entry.aggregator_id}'"
+        )
+
+    if len(entry.batch_ids_included) < MIN_BATCHES:
+        errors.append(
+            f"batch_ids_included must have at least {MIN_BATCHES} entries, "
+            f"got {len(entry.batch_ids_included)}"
+        )
+
+    if len(entry.summary_ids_included) < MIN_BATCHES:
+        errors.append(
+            f"summary_ids_included must have at least {MIN_BATCHES} entries, "
+            f"got {len(entry.summary_ids_included)}"
+        )
+    else:
+        bad_summaries = [
+            sid for sid in entry.summary_ids_included
+            if not sid.startswith(CPS_PREFIX)
+        ]
+        if bad_summaries:
+            errors.append(
+                f"summary_ids_included entries must start with '{CPS_PREFIX}': "
+                f"{bad_summaries}"
+            )
+
+    if (
+        len(entry.batch_ids_included) >= MIN_BATCHES
+        and len(entry.summary_ids_included) >= MIN_BATCHES
+        and len(entry.batch_ids_included) != len(entry.summary_ids_included)
+    ):
+        errors.append(
+            f"batch_ids_included ({len(entry.batch_ids_included)}) and "
+            f"summary_ids_included ({len(entry.summary_ids_included)}) must have equal length"
+        )
+
+    for name, score in [
+        ("mean_brier_score", entry.mean_brier_score),
+        ("min_brier_score", entry.min_brier_score),
+        ("max_brier_score", entry.max_brier_score),
+    ]:
+        if not (0.0 <= score <= 1.0):
+            errors.append(
+                f"{name} must be in [0.0, 1.0], got {score}"
+            )
+
+    if (
+        0.0 <= entry.min_brier_score <= 1.0
+        and 0.0 <= entry.mean_brier_score <= 1.0
+        and 0.0 <= entry.max_brier_score <= 1.0
+    ):
+        if not (entry.min_brier_score <= entry.mean_brier_score <= entry.max_brier_score):
+            errors.append(
+                f"min_brier_score ({entry.min_brier_score}) <= "
+                f"mean_brier_score ({entry.mean_brier_score}) <= "
+                f"max_brier_score ({entry.max_brier_score}) must hold"
+            )
+
+    if entry.trend not in VALID_TRENDS:
+        errors.append(
+            f"trend must be one of {sorted(VALID_TRENDS)}, got '{entry.trend}'"
+        )
+
+    if entry.total_candidates_evaluated < MIN_BATCHES:
+        errors.append(
+            f"total_candidates_evaluated must be >= {MIN_BATCHES}, "
+            f"got {entry.total_candidates_evaluated}"
+        )
+
+    if len(entry.aggregation_notes) > AGGREGATION_NOTES_MAX_LENGTH:
+        errors.append(
+            f"aggregation_notes exceeds {AGGREGATION_NOTES_MAX_LENGTH} chars "
+            f"(got {len(entry.aggregation_notes)})"
+        )
+
+    # Warnings
+    if entry.mean_brier_score >= POOR_BRIER_THRESHOLD:
+        warnings.append(
+            f"mean_brier_score ({entry.mean_brier_score}) >= {POOR_BRIER_THRESHOLD} "
+            "-- calibration below par"
+        )
+
+    brier_range = entry.max_brier_score - entry.min_brier_score
+    if brier_range >= HIGH_VARIANCE_THRESHOLD:
+        warnings.append(
+            f"Brier score range ({brier_range:.3f}) >= {HIGH_VARIANCE_THRESHOLD} "
+            "-- high variance across batches"
+        )
+
+    if entry.trend == "degrading":
+        warnings.append("trend=degrading -- performance getting worse")
+
+    return CrossBatchAggregatorResult(
+        aggregator_id=entry.aggregator_id,
+        pipeline_version=entry.pipeline_version,
+        mean_brier_score=entry.mean_brier_score,
+        trend=entry.trend,
+        passed=len(errors) == 0,
+        errors=errors,
+        warnings=warnings,
+        dry_lab_only=entry.dry_lab_only,
+    )
+
+
+def validate_cross_batch_aggregator_dict(data: dict) -> CrossBatchAggregatorResult:
+    required = [
+        "aggregator_id", "pipeline_version", "batch_ids_included",
+        "summary_ids_included", "mean_brier_score", "min_brier_score",
+        "max_brier_score", "trend", "total_candidates_evaluated",
+        "aggregation_notes", "reviewer",
+    ]
+    errors: List[str] = []
+    for key in required:
+        if key not in data:
+            errors.append(f"Missing required field: {key}")
+    if errors:
+        return CrossBatchAggregatorResult(
+            aggregator_id=data.get("aggregator_id", ""),
+            pipeline_version=data.get("pipeline_version", ""),
+            mean_brier_score=float(data.get("mean_brier_score", 0.0)),
+            trend=data.get("trend", ""),
+            passed=False,
+            errors=errors,
+            warnings=[],
+        )
+    entry = CrossBatchAggregatorEntry(
+        aggregator_id=str(data["aggregator_id"]),
+        pipeline_version=str(data["pipeline_version"]),
+        batch_ids_included=list(data["batch_ids_included"]),
+        summary_ids_included=list(data["summary_ids_included"]),
+        mean_brier_score=float(data["mean_brier_score"]),
+        min_brier_score=float(data["min_brier_score"]),
+        max_brier_score=float(data["max_brier_score"]),
+        trend=str(data["trend"]),
+        total_candidates_evaluated=int(data["total_candidates_evaluated"]),
+        aggregation_notes=str(data["aggregation_notes"]),
+        reviewer=str(data["reviewer"]),
+        dry_lab_only=bool(data.get("dry_lab_only", True)),
+    )
+    return validate_cross_batch_aggregator(entry)