Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/evidence/METRICS_CURRENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,17 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak
> **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees
> with this file, this file wins. Updated whenever benchmark/benchmark config changes.
>
> **Last updated:** 2026-07-10 (Phase G G1 — recalibration decision log — v0.10.21)
> **Last updated:** 2026-07-10 (Phase G G2 — recalibration rejection summary — v0.10.22)

## Changelog

### v0.10.22 — Phase G G2: RecalibrationRejectionSummary schema
- Added `RecalibrationRejectionSummary` (RRS-) schema
- 13 fields: rrs_id, pipeline_version, period_start, period_end, total_checkpoints_reviewed, total_refused, total_approved, refusal_rate, top_refusal_reason, gate_status, all_refusals_have_rrf, summary, notes
- 15 validation rules; 3 warnings; 63 tests
- BASELINE 5981→6044
- CLI: openamp-foundry recalibration-rejection-summary-check
- Closes Phase G G2 — aggregate refusal audit trail showing gate effectiveness

## v0.10.20 — Phase F F9 — Negative Result Dashboard (NRD-)

Expand Down
66 changes: 66 additions & 0 deletions docs/evidence/RECALIBRATION_REJECTION_SUMMARY_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# RecalibrationRejectionSummary (RRS-) Guide

## Purpose

`RecalibrationRejectionSummary` aggregates recalibration refusal events across
a pipeline period to show that the calibration gate is working correctly —
that the right recalibrations were correctly refused, not rubber-stamped.

## Schema Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `rrs_id` | str | yes | Unique identifier, must start with `RRS-` |
| `pipeline_version` | str | yes | Pipeline version during this period |
| `period_start` | str | yes | ISO date (YYYY-MM-DD) for period start |
| `period_end` | str | yes | ISO date (YYYY-MM-DD) for period end |
| `total_checkpoints_reviewed` | int | yes | Total calibration checkpoints reviewed (≥1) |
| `total_refused` | int | yes | Number refused (in [0, total_checkpoints_reviewed]) |
| `total_approved` | int | yes | Number approved (refused+approved=total) |
| `refusal_rate` | float | yes | Fraction refused (must match total_refused/total, tol 0.01) |
| `top_refusal_reason` | str | yes | Most common refusal reason (controlled vocabulary) |
| `gate_status` | str | yes | Current gate status (`active`, `suspended`, `under_review`) |
| `all_refusals_have_rrf` | bool | yes | Must be True — all refusals have RRF- artifacts |
| `summary` | str | yes | Narrative summary (≤400 chars) |
| `notes` | str | no | Additional context (≤300 chars) |

## Controlled Vocabularies

**top_refusal_reason (7 values):**
- `insufficient_data`, `effect_within_noise`, `recent_recalibration`
- `conflicting_signals`, `reviewer_override`, `multiple_reasons`, `none_refused`

**gate_status (3 values):**
- `active` — gate operating normally
- `suspended` — gate temporarily suspended (requires human approval)
- `under_review` — gate configuration under review

## Validation Rules

1. `rrs_id` starts with `RRS-`
2. `pipeline_version` non-empty
3. `period_start` is ISO date (YYYY-MM-DD)
4. `period_end` is ISO date (YYYY-MM-DD)
5. `total_checkpoints_reviewed` ≥ 1
6. `total_refused` in [0, total_checkpoints_reviewed]
7. `total_approved` in [0, total_checkpoints_reviewed]
8. `total_refused + total_approved == total_checkpoints_reviewed`
9. `refusal_rate` in [0.0, 1.0]
10. `refusal_rate` consistent with `total_refused / total_checkpoints_reviewed` (tolerance 0.01)
11. `top_refusal_reason` in controlled vocabulary (7 values)
12. `gate_status` in controlled vocabulary (3 values)
13. `all_refusals_have_rrf` must be True
14. `summary` non-empty and ≤400 chars
15. `notes` ≤300 chars

## Warnings

- `refusal_rate > 0.9`: investigate whether gate thresholds are too strict
- `total_refused == 0`: verify gate is operating (not silently approving all)
- `notes` empty: consider adding context for reviewers

## Integration

- Each refusal references: `RRF-` (recalibration refusal record)
- Triggered by: `RDL-` (recalibration decision log) showing refused outcomes
- Feeds into: governance audit trail, calibration health monitoring
2 changes: 1 addition & 1 deletion docs/research/NEXT_100_PR_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Make learning controlled rather than self-serving.
| PR | Task | Why it matters | Review class |
|---:|---|---|---|
| G1 | Add machine-readable recalibration decision log examples (complete). | RDL- schema: 13 fields, 13 validation rules, 4 outcomes, 5 trigger types, deferred-requires-conditions enforced; closes governance audit trail for calibration decisions. | B |
| G2 | Add recalibration rejection examples. | Shows gate success by refusal. | B |
| G2 | Add recalibration rejection examples (complete). | RRS- schema: 13 fields, 15 validation rules, refusal_rate consistency check (tol 0.01), all_refusals_have_rrf=True enforced, gate status controlled vocabulary; shows gate success by audit-ready refusal aggregation. | B |
| G3 | Add `make calibration-audit`. | Checks intake/gate/engine/report consistency. | C |
| G4 | Add active-learning strategy comparison report. | Prevents one-selector bias. | C |
| G5 | Add batch-2 rationale report explaining exploit/explore/diversity roles. | Reviewers understand next batch. | C |
Expand Down
17 changes: 17 additions & 0 deletions src/openamp_foundry/cli/commands/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -1302,6 +1302,23 @@ def _run_recalibration_decision_log_check(args):
print("OK: RecalibrationDecisionLog is valid.")


def _run_recalibration_rejection_summary_check(args):
"""CLI handler for recalibration-rejection-summary-check."""
import json, sys
from openamp_foundry.evidence.recalibration_rejection_summary import validate_dict
data = json.load(open(args.input))
issues = validate_dict(data)
errors = [i for i in issues if not i.startswith("WARNING:")]
warnings = [i for i in issues if i.startswith("WARNING:")]
for w in warnings:
print(w)
if errors:
for e in errors:
print(f"ERROR: {e}", file=sys.stderr)
sys.exit(1)
print("OK: RecalibrationRejectionSummary is valid.")


def _run_domain_review_outcome_check(args):
from openamp_foundry.evidence.domain_review_outcome import (
validate_domain_review_outcome_dict,
Expand Down
5 changes: 5 additions & 0 deletions src/openamp_foundry/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
_run_pilot_evidence_package_check,
_run_pre_registration_entry_check,
_run_recalibration_decision_log_check,
_run_recalibration_rejection_summary_check,
)
from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check
from openamp_foundry.cli.commands.reports import _run_contribution_check
Expand Down Expand Up @@ -2533,6 +2534,10 @@ def build_parser() -> argparse.ArgumentParser:
p_rdl.add_argument("input", help="Path to JSON file")
p_rdl.set_defaults(func=_run_recalibration_decision_log_check)

p_rrs = subparsers.add_parser("recalibration-rejection-summary-check", help="Validate a RecalibrationRejectionSummary JSON")
p_rrs.add_argument("input", help="Path to JSON file")
p_rrs.set_defaults(func=_run_recalibration_rejection_summary_check)

# ── Selection rationale check (Phase K K1) ───────────────────────
src2 = sub.add_parser(
"selection-rationale-check",
Expand Down
197 changes: 197 additions & 0 deletions src/openamp_foundry/evidence/recalibration_rejection_summary.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
"""Recalibration rejection summary schema — Phase G G2.

Aggregates multiple recalibration refusal events across a pipeline period
to show that the calibration gate is working correctly — that the right
recalibrations were correctly refused, not rubber-stamped.

A healthy rejection summary shows refusals happening for documented reasons,
not silence or blanket approval.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, field
from typing import List

RRS_PREFIX = "RRS-"

VALID_TOP_REFUSAL_REASONS = frozenset({
"insufficient_data",
"effect_within_noise",
"recent_recalibration",
"conflicting_signals",
"reviewer_override",
"multiple_reasons",
"none_refused",
})

VALID_GATE_STATUSES = frozenset({
"active",
"suspended",
"under_review",
})

REFUSAL_RATE_TOLERANCE = 0.01
HIGH_REFUSAL_RATE_WARNING = 0.9
ZERO_REFUSAL_WARNING_THRESHOLD = 0 # warn when zero refusals observed
SUMMARY_MAX_LENGTH = 400
NOTES_MAX_LENGTH = 300
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


@dataclass
class RecalibrationRejectionSummary:
rrs_id: str
pipeline_version: str
period_start: str
period_end: str
total_checkpoints_reviewed: int
total_refused: int
total_approved: int
refusal_rate: float
top_refusal_reason: str
gate_status: str
all_refusals_have_rrf: bool
summary: str
notes: str = ""


def validate(summary: RecalibrationRejectionSummary) -> List[str]:
"""Return list of error strings; empty list means valid."""
errors: List[str] = []

# Rule 1: ID prefix
if not summary.rrs_id.startswith(RRS_PREFIX):
errors.append(
f"rrs_id must start with '{RRS_PREFIX}', got: {summary.rrs_id!r}"
)

# Rule 2: pipeline_version non-empty
if not summary.pipeline_version.strip():
errors.append("pipeline_version must not be empty")

# Rule 3: period_start ISO format
if not _ISO_DATE_RE.match(summary.period_start):
errors.append(
f"period_start must be YYYY-MM-DD, got: {summary.period_start!r}"
)

# Rule 4: period_end ISO format
if not _ISO_DATE_RE.match(summary.period_end):
errors.append(
f"period_end must be YYYY-MM-DD, got: {summary.period_end!r}"
)

# Rule 5: total_checkpoints_reviewed >= 1
if summary.total_checkpoints_reviewed < 1:
errors.append(
f"total_checkpoints_reviewed must be >= 1, "
f"got: {summary.total_checkpoints_reviewed}"
)

# Rule 6: total_refused in [0, total_checkpoints_reviewed]
if not (0 <= summary.total_refused <= summary.total_checkpoints_reviewed):
errors.append(
f"total_refused must be in [0, {summary.total_checkpoints_reviewed}], "
f"got: {summary.total_refused}"
)

# Rule 7: total_approved in [0, total_checkpoints_reviewed]
if not (0 <= summary.total_approved <= summary.total_checkpoints_reviewed):
errors.append(
f"total_approved must be in [0, {summary.total_checkpoints_reviewed}], "
f"got: {summary.total_approved}"
)

# Rule 8: total_refused + total_approved == total_checkpoints_reviewed
if (
summary.total_refused + summary.total_approved
!= summary.total_checkpoints_reviewed
):
errors.append(
f"total_refused ({summary.total_refused}) + "
f"total_approved ({summary.total_approved}) must equal "
f"total_checkpoints_reviewed ({summary.total_checkpoints_reviewed})"
)

# Rule 9: refusal_rate in [0.0, 1.0]
if not (0.0 <= summary.refusal_rate <= 1.0):
errors.append(
f"refusal_rate must be in [0.0, 1.0], got: {summary.refusal_rate}"
)

# Rule 10: refusal_rate consistency check (tolerance 0.01)
if summary.total_checkpoints_reviewed >= 1:
computed = summary.total_refused / summary.total_checkpoints_reviewed
if abs(computed - summary.refusal_rate) > REFUSAL_RATE_TOLERANCE:
errors.append(
f"refusal_rate ({summary.refusal_rate:.4f}) inconsistent with "
f"total_refused/total_checkpoints_reviewed "
f"({summary.total_refused}/{summary.total_checkpoints_reviewed} "
f"= {computed:.4f}); tolerance is {REFUSAL_RATE_TOLERANCE}"
)

# Rule 11: top_refusal_reason vocabulary
if summary.top_refusal_reason not in VALID_TOP_REFUSAL_REASONS:
errors.append(
f"top_refusal_reason must be one of {sorted(VALID_TOP_REFUSAL_REASONS)}, "
f"got: {summary.top_refusal_reason!r}"
)

# Rule 12: gate_status vocabulary
if summary.gate_status not in VALID_GATE_STATUSES:
errors.append(
f"gate_status must be one of {sorted(VALID_GATE_STATUSES)}, "
f"got: {summary.gate_status!r}"
)

# Rule 13: all_refusals_have_rrf must be True
if not summary.all_refusals_have_rrf:
errors.append(
"all_refusals_have_rrf must be True — every refusal must "
"have a corresponding RRF- artifact"
)

# Rule 14: summary non-empty and length
if not summary.summary.strip():
errors.append("summary must not be empty")
elif len(summary.summary) > SUMMARY_MAX_LENGTH:
errors.append(
f"summary exceeds {SUMMARY_MAX_LENGTH} chars "
f"(got {len(summary.summary)})"
)

# Rule 15: notes length
if len(summary.notes) > NOTES_MAX_LENGTH:
errors.append(
f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(summary.notes)})"
)

# Warnings
if summary.refusal_rate >= HIGH_REFUSAL_RATE_WARNING:
errors.append(
f"WARNING: refusal_rate ({summary.refusal_rate:.2f}) > "
f"{HIGH_REFUSAL_RATE_WARNING} — investigate whether gate thresholds "
"are too strict"
)
if summary.total_refused == ZERO_REFUSAL_WARNING_THRESHOLD:
errors.append(
"WARNING: total_refused is 0 — verify that the gate is operating "
"and not silently approving all recalibrations"
)
if not summary.notes.strip():
errors.append(
"WARNING: notes is empty — consider adding context for reviewers"
)

return errors


def validate_dict(data: dict) -> List[str]:
"""Validate a plain dict by constructing RecalibrationRejectionSummary first."""
try:
s = RecalibrationRejectionSummary(**data)
except TypeError as exc:
return [f"Schema construction error: {exc}"]
return validate(s)
Loading
Loading