diff --git a/docs/evidence/METRICS_CURRENT.md b/docs/evidence/METRICS_CURRENT.md index 638424b3..e2f91d53 100644 --- a/docs/evidence/METRICS_CURRENT.md +++ b/docs/evidence/METRICS_CURRENT.md @@ -5,7 +5,7 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-10 (Phase F F9 — negative result dashboard — v0.10.20) +> **Last updated:** 2026-07-10 (Phase G G1 — recalibration decision log — v0.10.21) ## v0.10.20 — Phase F F9 — Negative Result Dashboard (NRD-) @@ -1261,6 +1261,14 @@ Decoys score low on activity. Selective AMPs score moderately on both. ## Change Log +### v0.10.21 — Phase G G1: RecalibrationDecisionLog schema +- Added `RecalibrationDecisionLog` (RDL-) schema +- 13 fields: rdl_id, pipeline_version, calibration_checkpoint, decision_date, trigger_type, trigger_artifact_id, decision_outcome, decision_authority, evidence_summary, rationale, next_review_date, conditions_if_deferred, notes +- 13 validation rules; 3 warnings; 63 tests +- BASELINE 5918→5981 +- CLI: openamp-foundry recalibration-decision-log-check +- Closes Phase G G1 — governance audit trail for calibration decisions + | Date | Change | Author | |------|--------|--------| | 2026-07-10 | **Calibration cycle summary schema (Phase P P5 — Phase P complete):** CalibrationCycleSummaryEntry (13 fields: ccs_id, pipeline_version, bsp_id, psc_id, bos_id, cps_id, cba_id, crg_id_previous, crg_id_next, cycle_outcome, cycle_notes, reviewer, dry_lab_only). Index record for one complete calibration loop — CRG→BSP→PSC→Lab→BOS→CPS→CBA→CRG. 11 validation rules (7 artifact prefixes: CCS-/BSP-/PSC-/BOS-/CPS-/CBA-/CRG-, crg_id_previous≠crg_id_next, cycle_outcome in {improved,stable,degraded}, notes≤400 chars). 2 warnings: degraded outcome, dry-lab-only cycle. 63 tests. CLI: openamp-foundry calibration-cycle-summary-check. Completes Phase P: P1 BSP + P2 RRF + P3 BOS + P4 PSC + P5 CCS form the full calibration bridge. BASELINE 4910->4973. | OpenAMP Loop | diff --git a/docs/evidence/RECALIBRATION_DECISION_LOG_GUIDE.md b/docs/evidence/RECALIBRATION_DECISION_LOG_GUIDE.md new file mode 100644 index 00000000..9fa97196 --- /dev/null +++ b/docs/evidence/RECALIBRATION_DECISION_LOG_GUIDE.md @@ -0,0 +1,100 @@ +# RecalibrationDecisionLog (RDL-) Guide + +## Purpose + +`RecalibrationDecisionLog` records every governance decision at a calibration +checkpoint — whether the pipeline was recalibrated, refused, deferred, or +partially updated — and preserves the evidence trail for each choice. + +## Schema Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `rdl_id` | str | yes | Unique identifier, must start with `RDL-` | +| `pipeline_version` | str | yes | Pipeline version at decision time | +| `calibration_checkpoint` | str | yes | Named checkpoint (e.g. `batch_3_review`) | +| `decision_date` | str | yes | ISO date (YYYY-MM-DD) of decision | +| `trigger_type` | str | yes | What triggered the review (controlled vocabulary) | +| `trigger_artifact_id` | str | yes | ID of the artifact that triggered the review | +| `decision_outcome` | str | yes | `approved`, `refused`, `deferred`, `partial_approved` | +| `decision_authority` | str | yes | Who/what made the decision (controlled vocabulary) | +| `evidence_summary` | str | yes | Concise summary of evidence considered | +| `rationale` | str | yes | Reasoning behind decision (≤500 chars) | +| `next_review_date` | str | yes | ISO date for next scheduled review | +| `conditions_if_deferred` | str | no | Required when outcome is `deferred` | +| `notes` | str | no | Additional context (≤300 chars) | + +## Controlled Vocabularies + +**trigger_type:** +- `batch_outcome` — triggered by batch result data +- `calibration_performance_summary` — triggered by CPS- artifact +- `scheduled_review` — routine scheduled checkpoint +- `anomaly_detected` — triggered by anomaly detection +- `human_request` — human reviewer requested review + +**decision_outcome:** +- `approved` — recalibration proceeds +- `refused` — recalibration rejected (see RRF-) +- `deferred` — decision postponed pending more data +- `partial_approved` — selective update to specific components + +**decision_authority:** +- `automated_gate` — automated pipeline gate +- `pipeline_owner` — pipeline owner +- `external_reviewer` — external domain expert +- `safety_officer` — safety review officer +- `committee` — multi-member review committee + +## Validation Rules + +1. `rdl_id` starts with `RDL-` +2. `pipeline_version` non-empty +3. `calibration_checkpoint` non-empty +4. `decision_date` is ISO date (YYYY-MM-DD) +5. `trigger_type` in controlled vocabulary (5 values) +6. `trigger_artifact_id` starts with a known prefix (CPS-, DRM-, PCI-, BSP-, CIR-, RRF-) +7. `decision_outcome` in controlled vocabulary (4 values) +8. `decision_authority` in controlled vocabulary (5 values) +9. `evidence_summary` non-empty +10. `rationale` non-empty and ≤500 chars +11. `next_review_date` is ISO date (YYYY-MM-DD) +12. `conditions_if_deferred` required when outcome is `deferred` +13. `notes` ≤300 chars + +## Warnings + +- `refused` outcome: verify rationale cites a specific RRF- artifact +- `deferred` with no notes: consider documenting deferral context +- `notes` empty: consider adding context for future reviewers + +## Usage + +```python +from openamp_foundry.evidence.recalibration_decision_log import ( + RecalibrationDecisionLog, validate +) + +log = RecalibrationDecisionLog( + rdl_id="RDL-20240315-001", + pipeline_version="v2.1.0", + calibration_checkpoint="batch_3_review", + decision_date="2024-03-15", + trigger_type="batch_outcome", + trigger_artifact_id="CPS-20240314-001", + decision_outcome="approved", + decision_authority="automated_gate", + evidence_summary="Brier score improved from 0.28 to 0.21 after batch 3.", + rationale="Metric improvement exceeds minimum threshold and is consistent across 3 cohorts.", + next_review_date="2024-06-15", +) +errors = validate(log) +assert errors == [] +``` + +## Integration + +- Triggered by: `CPS-` (calibration performance summary), `PCI-` (post-experiment intake) +- Refused outcomes generate: `RRF-` (recalibration refusal record) +- Approved outcomes generate: `CIR-` (calibration improvement record) +- Feeds into: governance audit trail, decision pattern analysis diff --git a/docs/research/NEXT_100_PR_MAP.md b/docs/research/NEXT_100_PR_MAP.md index f058d936..a4f055f7 100644 --- a/docs/research/NEXT_100_PR_MAP.md +++ b/docs/research/NEXT_100_PR_MAP.md @@ -126,7 +126,7 @@ Make learning controlled rather than self-serving. | PR | Task | Why it matters | Review class | |---:|---|---|---| -| G1 | Add machine-readable recalibration decision log examples. | Improves governance memory. | B | +| G1 | Add machine-readable recalibration decision log examples (complete). | RDL- schema: 13 fields, 13 validation rules, 4 outcomes, 5 trigger types, deferred-requires-conditions enforced; closes governance audit trail for calibration decisions. | B | | G2 | Add recalibration rejection examples. | Shows gate success by refusal. | B | | G3 | Add `make calibration-audit`. | Checks intake/gate/engine/report consistency. | C | | G4 | Add active-learning strategy comparison report. | Prevents one-selector bias. | C | diff --git a/src/openamp_foundry/cli/commands/reports.py b/src/openamp_foundry/cli/commands/reports.py index b6a610ec..b0bec099 100644 --- a/src/openamp_foundry/cli/commands/reports.py +++ b/src/openamp_foundry/cli/commands/reports.py @@ -1285,6 +1285,23 @@ def _run_validate_simulation_result(args: argparse.Namespace) -> int: return 3 if batch_result["any_invalid"] else 0 +def _run_recalibration_decision_log_check(args): + """CLI handler for recalibration-decision-log-check.""" + import json, sys + from openamp_foundry.evidence.recalibration_decision_log import validate_dict + data = json.load(open(args.input)) + issues = validate_dict(data) + errors = [i for i in issues if not i.startswith("WARNING:")] + warnings = [i for i in issues if i.startswith("WARNING:")] + for w in warnings: + print(w) + if errors: + for e in errors: + print(f"ERROR: {e}", file=sys.stderr) + sys.exit(1) + print("OK: RecalibrationDecisionLog is valid.") + + def _run_domain_review_outcome_check(args): from openamp_foundry.evidence.domain_review_outcome import ( validate_domain_review_outcome_dict, diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index d3ea6581..a9da318a 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -36,6 +36,7 @@ _run_calibration_cycle_summary_check, _run_pilot_evidence_package_check, _run_pre_registration_entry_check, + _run_recalibration_decision_log_check, ) from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check from openamp_foundry.cli.commands.reports import _run_contribution_check @@ -2528,6 +2529,10 @@ def build_parser() -> argparse.ArgumentParser: ) nrd_parser.add_argument("json_input", help="JSON string of the NRD- record") + p_rdl = subparsers.add_parser("recalibration-decision-log-check", help="Validate a RecalibrationDecisionLog JSON") + p_rdl.add_argument("input", help="Path to JSON file") + p_rdl.set_defaults(func=_run_recalibration_decision_log_check) + # ── Selection rationale check (Phase K K1) ─────────────────────── src2 = sub.add_parser( "selection-rationale-check", @@ -2970,6 +2975,9 @@ def main(argv: list[str] | None = None) -> int: from openamp_foundry.cli.commands.reports import _run_negative_result_dashboard_check _run_negative_result_dashboard_check(args) + elif args.command == "recalibration-decision-log-check": + _run_recalibration_decision_log_check(args) + else: parser.error("unknown command") return 2 diff --git a/src/openamp_foundry/evidence/recalibration_decision_log.py b/src/openamp_foundry/evidence/recalibration_decision_log.py new file mode 100644 index 00000000..569615c0 --- /dev/null +++ b/src/openamp_foundry/evidence/recalibration_decision_log.py @@ -0,0 +1,173 @@ +"""Recalibration decision log schema — Phase G G1. + +Records the governance decision at each recalibration checkpoint: +whether recalibration was approved or refused, the trigger artifact, +the deciding evidence, and who (or what process) made the call. + +This schema enables audit trails across multiple calibration cycles +so patterns of approval/refusal can themselves be reviewed. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import List + +RDL_PREFIX = "RDL-" + +VALID_DECISION_OUTCOMES = frozenset({ + "approved", + "refused", + "deferred", + "partial_approved", +}) + +VALID_TRIGGER_TYPES = frozenset({ + "batch_outcome", + "calibration_performance_summary", + "scheduled_review", + "anomaly_detected", + "human_request", +}) + +VALID_TRIGGER_ID_PREFIXES = frozenset({ + "CPS-", "DRM-", "PCI-", "BSP-", "CIR-", "RRF-", +}) + +VALID_DECISION_AUTHORITIES = frozenset({ + "automated_gate", + "pipeline_owner", + "external_reviewer", + "safety_officer", + "committee", +}) + +RATIONALE_MAX_LENGTH = 500 +NOTES_MAX_LENGTH = 300 +_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") + + +@dataclass +class RecalibrationDecisionLog: + rdl_id: str + pipeline_version: str + calibration_checkpoint: str + decision_date: str + trigger_type: str + trigger_artifact_id: str + decision_outcome: str + decision_authority: str + evidence_summary: str + rationale: str + next_review_date: str + conditions_if_deferred: str = "" + notes: str = "" + + +def validate(log: RecalibrationDecisionLog) -> List[str]: + """Return list of error strings; empty list means valid.""" + errors: List[str] = [] + warnings: List[str] = [] + + # Rule 1: ID prefix + if not log.rdl_id.startswith(RDL_PREFIX): + errors.append(f"rdl_id must start with '{RDL_PREFIX}', got: {log.rdl_id!r}") + + # Rule 2: pipeline_version non-empty + if not log.pipeline_version.strip(): + errors.append("pipeline_version must not be empty") + + # Rule 3: calibration_checkpoint non-empty + if not log.calibration_checkpoint.strip(): + errors.append("calibration_checkpoint must not be empty") + + # Rule 4: decision_date ISO format + if not _ISO_DATE_RE.match(log.decision_date): + errors.append(f"decision_date must be YYYY-MM-DD, got: {log.decision_date!r}") + + # Rule 5: trigger_type vocabulary + if log.trigger_type not in VALID_TRIGGER_TYPES: + errors.append( + f"trigger_type must be one of {sorted(VALID_TRIGGER_TYPES)}, " + f"got: {log.trigger_type!r}" + ) + + # Rule 6: trigger_artifact_id must start with a known prefix + if not any(log.trigger_artifact_id.startswith(p) for p in VALID_TRIGGER_ID_PREFIXES): + errors.append( + f"trigger_artifact_id must start with one of " + f"{sorted(VALID_TRIGGER_ID_PREFIXES)}, got: {log.trigger_artifact_id!r}" + ) + + # Rule 7: decision_outcome vocabulary + if log.decision_outcome not in VALID_DECISION_OUTCOMES: + errors.append( + f"decision_outcome must be one of {sorted(VALID_DECISION_OUTCOMES)}, " + f"got: {log.decision_outcome!r}" + ) + + # Rule 8: decision_authority vocabulary + if log.decision_authority not in VALID_DECISION_AUTHORITIES: + errors.append( + f"decision_authority must be one of {sorted(VALID_DECISION_AUTHORITIES)}, " + f"got: {log.decision_authority!r}" + ) + + # Rule 9: evidence_summary non-empty + if not log.evidence_summary.strip(): + errors.append("evidence_summary must not be empty") + + # Rule 10: rationale non-empty and length + if not log.rationale.strip(): + errors.append("rationale must not be empty") + elif len(log.rationale) > RATIONALE_MAX_LENGTH: + errors.append( + f"rationale exceeds {RATIONALE_MAX_LENGTH} chars " + f"(got {len(log.rationale)})" + ) + + # Rule 11: next_review_date ISO format + if not _ISO_DATE_RE.match(log.next_review_date): + errors.append( + f"next_review_date must be YYYY-MM-DD, got: {log.next_review_date!r}" + ) + + # Rule 12: deferred outcome requires conditions + if log.decision_outcome == "deferred" and not log.conditions_if_deferred.strip(): + errors.append( + "conditions_if_deferred must not be empty when decision_outcome is 'deferred'" + ) + + # Rule 13: notes length + if len(log.notes) > NOTES_MAX_LENGTH: + errors.append( + f"notes exceeds {NOTES_MAX_LENGTH} chars (got {len(log.notes)})" + ) + + # Warnings (returned with "WARNING:" prefix) + if log.decision_outcome == "refused": + warnings.append( + "WARNING: decision_outcome is 'refused' — verify refusal rationale " + "cites a specific RRF- artifact or documented reason" + ) + if log.decision_outcome == "deferred" and not log.notes.strip(): + warnings.append( + "WARNING: deferred decision has no notes — consider documenting " + "the deferral context" + ) + if not log.notes.strip(): + warnings.append( + "WARNING: notes is empty — consider adding context for future reviewers" + ) + + return errors + warnings + + +def validate_dict(data: dict) -> List[str]: + """Validate a plain dict by constructing RecalibrationDecisionLog first.""" + try: + log = RecalibrationDecisionLog(**data) + except TypeError as exc: + return [f"Schema construction error: {exc}"] + return validate(log) diff --git a/tests/evidence/test_recalibration_decision_log.py b/tests/evidence/test_recalibration_decision_log.py new file mode 100644 index 00000000..61af2788 --- /dev/null +++ b/tests/evidence/test_recalibration_decision_log.py @@ -0,0 +1,497 @@ +"""Tests for RecalibrationDecisionLog schema — Phase G G1. + +Exactly 63 tests: valid baseline + each validation rule + edge cases + warnings. +""" + +from __future__ import annotations + +import pytest + +from openamp_foundry.evidence.recalibration_decision_log import ( + NOTES_MAX_LENGTH, + RATIONALE_MAX_LENGTH, + RDL_PREFIX, + VALID_DECISION_AUTHORITIES, + VALID_DECISION_OUTCOMES, + VALID_TRIGGER_ID_PREFIXES, + VALID_TRIGGER_TYPES, + RecalibrationDecisionLog, + validate, + validate_dict, +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _valid_log(**overrides) -> RecalibrationDecisionLog: + defaults = dict( + rdl_id="RDL-20240315-001", + pipeline_version="v2.1.0", + calibration_checkpoint="batch_3_review", + decision_date="2024-03-15", + trigger_type="batch_outcome", + trigger_artifact_id="CPS-20240314-001", + decision_outcome="approved", + decision_authority="automated_gate", + evidence_summary="Brier score improved from 0.28 to 0.21 after batch 3.", + rationale="Metric improvement exceeds minimum threshold (0.005) and is " + "statistically consistent across 3 cohorts.", + next_review_date="2024-06-15", + conditions_if_deferred="", + notes="", + ) + defaults.update(overrides) + return RecalibrationDecisionLog(**defaults) + + +def _valid() -> RecalibrationDecisionLog: + return _valid_log() + + +# --------------------------------------------------------------------------- +# Group 1: Valid baseline (3 tests) +# --------------------------------------------------------------------------- + +class TestValidBaseline: + def test_valid_returns_no_errors(self): + log = _valid_log(notes="Standard log entry.") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_valid_with_notes(self): + log = _valid_log(notes="Reviewed by safety officer.") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_valid_deferred_with_conditions(self): + log = _valid_log( + decision_outcome="deferred", + conditions_if_deferred="Await batch 4 results before deciding.", + notes="Deferred pending further data.", + ) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 2: Rule 1 — rdl_id prefix (4 tests) +# --------------------------------------------------------------------------- + +class TestRdlIdPrefix: + def test_wrong_prefix_rejected(self): + log = _valid_log(rdl_id="CIR-001") + assert any("rdl_id" in e for e in validate(log)) + + def test_lowercase_prefix_rejected(self): + log = _valid_log(rdl_id="rdl-001") + assert any("rdl_id" in e for e in validate(log)) + + def test_no_prefix_rejected(self): + log = _valid_log(rdl_id="001") + assert any("rdl_id" in e for e in validate(log)) + + def test_correct_prefix_accepted(self): + log = _valid_log(rdl_id="RDL-2024-001") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 3: Rule 2 — pipeline_version (3 tests) +# --------------------------------------------------------------------------- + +class TestPipelineVersion: + def test_empty_rejected(self): + log = _valid_log(pipeline_version="") + assert any("pipeline_version" in e for e in validate(log)) + + def test_whitespace_only_rejected(self): + log = _valid_log(pipeline_version=" ") + assert any("pipeline_version" in e for e in validate(log)) + + def test_valid_version(self): + log = _valid_log(pipeline_version="v3.0.1") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 4: Rule 3 — calibration_checkpoint (3 tests) +# --------------------------------------------------------------------------- + +class TestCalibrationCheckpoint: + def test_empty_rejected(self): + log = _valid_log(calibration_checkpoint="") + assert any("calibration_checkpoint" in e for e in validate(log)) + + def test_whitespace_only_rejected(self): + log = _valid_log(calibration_checkpoint=" ") + assert any("calibration_checkpoint" in e for e in validate(log)) + + def test_valid_checkpoint(self): + log = _valid_log(calibration_checkpoint="monthly_review_2024_03") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 5: Rule 4 — decision_date ISO format (4 tests) +# --------------------------------------------------------------------------- + +class TestDecisionDate: + def test_invalid_format_rejected(self): + log = _valid_log(decision_date="March 15, 2024") + assert any("decision_date" in e for e in validate(log)) + + def test_wrong_separator_rejected(self): + log = _valid_log(decision_date="2024/03/15") + assert any("decision_date" in e for e in validate(log)) + + def test_empty_rejected(self): + log = _valid_log(decision_date="") + assert any("decision_date" in e for e in validate(log)) + + def test_valid_iso_date(self): + log = _valid_log(decision_date="2025-01-01") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 6: Rule 5 — trigger_type vocabulary (4 tests) +# --------------------------------------------------------------------------- + +class TestTriggerType: + def test_invalid_trigger_type_rejected(self): + log = _valid_log(trigger_type="unknown_trigger") + assert any("trigger_type" in e for e in validate(log)) + + def test_empty_trigger_type_rejected(self): + log = _valid_log(trigger_type="") + assert any("trigger_type" in e for e in validate(log)) + + @pytest.mark.parametrize("tt", sorted(VALID_TRIGGER_TYPES)) + def test_all_valid_trigger_types_accepted(self, tt): + log = _valid_log(trigger_type=tt) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 7: Rule 6 — trigger_artifact_id prefix (4 tests) +# --------------------------------------------------------------------------- + +class TestTriggerArtifactId: + def test_unknown_prefix_rejected(self): + log = _valid_log(trigger_artifact_id="XYZ-001") + assert any("trigger_artifact_id" in e for e in validate(log)) + + def test_empty_rejected(self): + log = _valid_log(trigger_artifact_id="") + assert any("trigger_artifact_id" in e for e in validate(log)) + + @pytest.mark.parametrize("prefix", sorted(VALID_TRIGGER_ID_PREFIXES)) + def test_all_valid_prefixes_accepted(self, prefix): + log = _valid_log(trigger_artifact_id=f"{prefix}001") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 8: Rule 7 — decision_outcome vocabulary (4 tests) +# --------------------------------------------------------------------------- + +class TestDecisionOutcome: + def test_invalid_outcome_rejected(self): + log = _valid_log(decision_outcome="maybe") + assert any("decision_outcome" in e for e in validate(log)) + + def test_empty_outcome_rejected(self): + log = _valid_log(decision_outcome="") + assert any("decision_outcome" in e for e in validate(log)) + + @pytest.mark.parametrize("outcome", sorted(VALID_DECISION_OUTCOMES)) + def test_all_valid_outcomes_accepted(self, outcome): + extra = {} + if outcome == "deferred": + extra["conditions_if_deferred"] = "Pending batch 4 results." + log = _valid_log(decision_outcome=outcome, **extra) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 9: Rule 8 — decision_authority vocabulary (4 tests) +# --------------------------------------------------------------------------- + +class TestDecisionAuthority: + def test_invalid_authority_rejected(self): + log = _valid_log(decision_authority="random_person") + assert any("decision_authority" in e for e in validate(log)) + + def test_empty_authority_rejected(self): + log = _valid_log(decision_authority="") + assert any("decision_authority" in e for e in validate(log)) + + @pytest.mark.parametrize("auth", sorted(VALID_DECISION_AUTHORITIES)) + def test_all_valid_authorities_accepted(self, auth): + log = _valid_log(decision_authority=auth) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 10: Rule 9 — evidence_summary (3 tests) +# --------------------------------------------------------------------------- + +class TestEvidenceSummary: + def test_empty_rejected(self): + log = _valid_log(evidence_summary="") + assert any("evidence_summary" in e for e in validate(log)) + + def test_whitespace_only_rejected(self): + log = _valid_log(evidence_summary=" ") + assert any("evidence_summary" in e for e in validate(log)) + + def test_valid_summary(self): + log = _valid_log(evidence_summary="AUROC improved from 0.72 to 0.78.") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 11: Rule 10 — rationale (4 tests) +# --------------------------------------------------------------------------- + +class TestRationale: + def test_empty_rejected(self): + log = _valid_log(rationale="") + assert any("rationale" in e for e in validate(log)) + + def test_whitespace_only_rejected(self): + log = _valid_log(rationale=" ") + assert any("rationale" in e for e in validate(log)) + + def test_too_long_rejected(self): + log = _valid_log(rationale="x" * (RATIONALE_MAX_LENGTH + 1)) + assert any("rationale" in e for e in validate(log)) + + def test_exactly_at_limit_accepted(self): + log = _valid_log(rationale="x" * RATIONALE_MAX_LENGTH) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 12: Rule 11 — next_review_date ISO format (3 tests) +# --------------------------------------------------------------------------- + +class TestNextReviewDate: + def test_invalid_format_rejected(self): + log = _valid_log(next_review_date="Q2 2024") + assert any("next_review_date" in e for e in validate(log)) + + def test_empty_rejected(self): + log = _valid_log(next_review_date="") + assert any("next_review_date" in e for e in validate(log)) + + def test_valid_iso_date(self): + log = _valid_log(next_review_date="2025-06-30") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 13: Rule 12 — deferred requires conditions (3 tests) +# --------------------------------------------------------------------------- + +class TestDeferredConditions: + def test_deferred_without_conditions_rejected(self): + log = _valid_log(decision_outcome="deferred", conditions_if_deferred="") + assert any("conditions_if_deferred" in e for e in validate(log)) + + def test_deferred_with_whitespace_conditions_rejected(self): + log = _valid_log(decision_outcome="deferred", conditions_if_deferred=" ") + assert any("conditions_if_deferred" in e for e in validate(log)) + + def test_approved_without_conditions_accepted(self): + log = _valid_log(decision_outcome="approved", conditions_if_deferred="") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 14: Rule 13 — notes length (3 tests) +# --------------------------------------------------------------------------- + +class TestNotesLength: + def test_too_long_rejected(self): + log = _valid_log(notes="n" * (NOTES_MAX_LENGTH + 1)) + assert any("notes" in e for e in validate(log)) + + def test_exactly_at_limit_accepted(self): + log = _valid_log(notes="n" * NOTES_MAX_LENGTH) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_empty_notes_accepted(self): + log = _valid_log(notes="") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 15: Warnings (6 tests) +# --------------------------------------------------------------------------- + +class TestWarnings: + def test_refused_outcome_triggers_warning(self): + log = _valid_log(decision_outcome="refused") + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert any("refused" in w for w in warns) + + def test_deferred_no_notes_triggers_warning(self): + log = _valid_log( + decision_outcome="deferred", + conditions_if_deferred="Await batch 4.", + notes="", + ) + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert any("deferred" in w for w in warns) + + def test_empty_notes_triggers_warning(self): + log = _valid_log(notes="") + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert any("notes" in w.lower() for w in warns) + + def test_notes_present_suppresses_notes_warning(self): + log = _valid_log(notes="Full context provided.") + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert not any("notes is empty" in w for w in warns) + + def test_approved_has_no_refused_warning(self): + log = _valid_log(decision_outcome="approved") + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert not any("refused" in w for w in warns) + + def test_deferred_with_notes_suppresses_deferred_warning(self): + log = _valid_log( + decision_outcome="deferred", + conditions_if_deferred="Pending batch 5.", + notes="Safety officer requested additional review.", + ) + warns = [e for e in validate(log) if e.startswith("WARNING:")] + assert not any("deferred decision has no notes" in w for w in warns) + + +# --------------------------------------------------------------------------- +# Group 16: validate_dict (4 tests) +# --------------------------------------------------------------------------- + +class TestValidateDict: + def test_valid_dict_returns_no_errors(self): + data = dict( + rdl_id="RDL-20240315-001", + pipeline_version="v2.1.0", + calibration_checkpoint="batch_3_review", + decision_date="2024-03-15", + trigger_type="batch_outcome", + trigger_artifact_id="CPS-20240314-001", + decision_outcome="approved", + decision_authority="automated_gate", + evidence_summary="Brier score improved from 0.28 to 0.21.", + rationale="Improvement exceeds minimum threshold across 3 cohorts.", + next_review_date="2024-06-15", + ) + errs = [e for e in validate_dict(data) if not e.startswith("WARNING:")] + assert errs == [] + + def test_missing_required_field_returns_error(self): + data = dict( + rdl_id="RDL-001", + pipeline_version="v1.0", + ) + result = validate_dict(data) + assert any("Schema construction error" in e for e in result) + + def test_invalid_field_caught_by_dict_validator(self): + data = dict( + rdl_id="WRONG-001", + pipeline_version="v1.0", + calibration_checkpoint="batch_1", + decision_date="2024-01-01", + trigger_type="batch_outcome", + trigger_artifact_id="CPS-001", + decision_outcome="approved", + decision_authority="automated_gate", + evidence_summary="Evidence here.", + rationale="Rationale here.", + next_review_date="2024-04-01", + ) + result = validate_dict(data) + assert any("rdl_id" in e for e in result) + + def test_extra_notes_in_dict(self): + data = dict( + rdl_id="RDL-20240315-002", + pipeline_version="v2.1.0", + calibration_checkpoint="quarterly_review", + decision_date="2024-03-15", + trigger_type="scheduled_review", + trigger_artifact_id="DRM-20240314-001", + decision_outcome="approved", + decision_authority="pipeline_owner", + evidence_summary="All metrics stable.", + rationale="Quarterly check passed with no regressions.", + next_review_date="2024-06-15", + notes="Reviewed by J. Smith.", + ) + errs = [e for e in validate_dict(data) if not e.startswith("WARNING:")] + assert errs == [] + + +# --------------------------------------------------------------------------- +# Group 17: Edge cases (8 tests) +# --------------------------------------------------------------------------- + +class TestEdgeCases: + def test_partial_approved_no_conditions_accepted(self): + log = _valid_log(decision_outcome="partial_approved", conditions_if_deferred="") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_all_valid_trigger_prefixes_individually(self): + for prefix in sorted(VALID_TRIGGER_ID_PREFIXES): + log = _valid_log(trigger_artifact_id=f"{prefix}test-001") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [], f"Prefix {prefix} should be valid" + + def test_human_request_trigger_type(self): + log = _valid_log(trigger_type="human_request") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_committee_authority(self): + log = _valid_log(decision_authority="committee") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_safety_officer_authority(self): + log = _valid_log(decision_authority="safety_officer") + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_rationale_one_below_limit(self): + log = _valid_log(rationale="r" * (RATIONALE_MAX_LENGTH - 1)) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_notes_one_below_limit(self): + log = _valid_log(notes="n" * (NOTES_MAX_LENGTH - 1)) + errs = [e for e in validate(log) if not e.startswith("WARNING:")] + assert errs == [] + + def test_rdl_prefix_constant(self): + assert RDL_PREFIX == "RDL-" diff --git a/tests/test_test_count_regression.py b/tests/test_test_count_regression.py index 99e4db8e..8445695b 100644 --- a/tests/test_test_count_regression.py +++ b/tests/test_test_count_regression.py @@ -6,7 +6,7 @@ # Note: 3 pre-existing collection errors in test_benchmark_calibration.py, # test_benchmark_charge_distribution.py, test_benchmark_cheap_enemies.py. # Raise threshold if legitimate tests are removed; lower threshold is okay. -BASELINE = 5918 +BASELINE = 5981 TOLERANCE = 0.05 # allow 5% fluctuation