diff --git a/docs/evidence/METRICS_CURRENT.md b/docs/evidence/METRICS_CURRENT.md index 4f33ba16..6c0d24a1 100644 --- a/docs/evidence/METRICS_CURRENT.md +++ b/docs/evidence/METRICS_CURRENT.md @@ -5,7 +5,8 @@ Machine-readable snapshot: `outputs/metrics_snapshot.json` regenerated with `mak > **Purpose:** One authoritative table of current pipeline metrics. If any doc disagrees > with this file, this file wins. Updated whenever benchmark/benchmark config changes. > -> **Last updated:** 2026-07-10 (Phase P P1 — batch selection proposal — v0.10.4) +> **Last updated:** 2026-07-10 (Phase P P2 — recalibration refusal record — v0.10.5) +> **New in v0.10.5 (Phase P P2):** RecalibrationRefusalEntry schema (10 fields: rrf_id, pipeline_version, trigger_id, recalibration_refused, refusal_reason, minimum_batches_required, batches_evaluated, refusal_notes, reviewer, dry_lab_only). recalibration_refused must be True — this schema only records valid refusals. 7 validation rules (RRF- prefix, CPS-/DRM- trigger prefix, recalibration_refused=True enforced, reason in 5 valid values, minimum_batches>=1, batches_evaluated>=0, notes<=400 chars). 2 warnings (insufficient_data but meets minimum; reviewer_override without documentation). Complements CalibrationImprovementRecord (O3). Added validate_recalibration_refusal() and validate_recalibration_refusal_dict(). Added 63 tests. CLI: openamp-foundry recalibration-refusal-check with --entry-json and --format text|json. Corrected inflated BASELINE from 4889 to 4784 (actual count 4721 + 63 = 4784). > **New in v0.10.4 (Phase P P1):** BatchSelectionProposalEntry schema (12 fields: bsp_id, pipeline_version, gate_id, gate_passed, candidate_ids, selection_strategy, exploitation_fraction, exploration_fraction, max_brier_score_allowed, proposal_notes, reviewer, dry_lab_only). gate_passed must be True — a batch can only be proposed when the calibration readiness gate passes. 9 validation rules (BSP- prefix, CRG- prefix, gate_passed=True enforced, candidate_ids >=1, strategy in 5 valid values, fraction bounds, fraction sum=1.0 to 0.001 tolerance, Brier bounds [0.0,1.0], notes <=400 chars). 3 warnings (pure exploitation >=0.99, single candidate, marginal Brier >=0.25). Added validate_batch_selection_proposal() and validate_batch_selection_proposal_dict(). Added 68 tests. CLI: openamp-foundry batch-selection-proposal-check with --entry-json and --format text|json. Closes the calibration-to-selection loop from Phase O. > **New in v0.10.3 (Phase O O5):** CalibrationReadinessEntry schema (11 fields, 8 validation rules, 2 warnings). Added validate_calibration_readiness() and validate_calibration_readiness_dict(). Added 68 tests. CLI: openamp-foundry calibration-readiness-check. Phase O (Calibration Quality Assurance) COMPLETE. > **New in v0.10.2 (Phase O O4):** CrossBatchAggregatorEntry schema (12 fields, 9 validation rules, 3 warnings). Added validate_cross_batch_aggregator() and validate_cross_batch_aggregator_dict(). Added 68 tests. CLI: openamp-foundry cross-batch-aggregator-check. @@ -1230,6 +1231,7 @@ Decoys score low on activity. Selective AMPs score moderately on both. | Date | Change | Author | |------|--------|--------| +| 2026-07-10 | **Recalibration refusal record schema (Phase P P2):** RecalibrationRefusalEntry (10 fields, recalibration_refused must be True). 7 validation rules, 2 warnings. 63 tests. CLI: openamp-foundry recalibration-refusal-check. Complements CalibrationImprovementRecord (O3) with the rejection path. Corrected inflated BASELINE 4889->4784. | OpenAMP Loop | | 2026-07-09 | **Release checklist and gate validator (J1, starts Phase J):** `docs/governance/RELEASE_CHECKLIST.md` with structured checklist. `src/openamp_foundry/governance/release_gate.py` with `RELEASE_TYPES` (5), `UNIVERSAL_GATES` (7), `EXTRA_GATES_BY_TYPE`, `ReleaseGateResult`, `validate_release_gate()`. CLI `release-gate-check`. `make release-gate-check`. 18 tests. **Phase J (Governance and release maturity) started. 3478 total.** | OpenAMP Loop 109 | | 2026-07-09 | **Adoption scorecard dashboard added (I10):** `src/openamp_foundry/adoption/scorecard.py` with `SCORECARD_DIMENSIONS` (5, weights sum 1.0), `ADOPTION_TIERS` (4), `DimensionScore`, `AdoptionScorecard`, `build_scorecard()`, `compute_adoption_tier()`. CLI `adoption-scorecard` with `--scores-json` and `--format`. `make adoption-scorecard`. 17 tests. **Phase I (Interoperability and Adoption) is now complete** — all 10 items I1–I10 implemented. **3446 total.** | OpenAMP Loop 108 | | 2026-07-09 | **Adapter author validator added (I6):** `src/openamp_foundry/adapters/adapter_validator.py` with `AdapterDeclaration` (14 fields), `AdapterValidationResult` (5 fields), `validate_adapter_declaration()` (10 checks enforcing ADAPTER_AUTHOR_GUIDE contract), `validate_adapter_dict()` (dict input with missing-fields guard). 4 valid-value sets (VALID_ADAPTER_MODES, VALID_OUTPUT_STATUSES, VALID_RANKING_EFFECTS, VALID_RELEASE_STATUSES). CLI (`openamp-foundry adapter-check`) with `--adapter-json`, `--format text|json`. `make adapter-author-check` target. 31 tests. **3387 total.** | OpenAMP Loop 104 | diff --git a/docs/evidence/RECALIBRATION_REFUSAL_GUIDE.md b/docs/evidence/RECALIBRATION_REFUSAL_GUIDE.md new file mode 100644 index 00000000..f1698499 --- /dev/null +++ b/docs/evidence/RECALIBRATION_REFUSAL_GUIDE.md @@ -0,0 +1,68 @@ +# Recalibration Refusal Record Guide + +## Purpose + +A Recalibration Refusal Record (RRF) documents when a recalibration evaluation was +conducted and the decision was to **not** recalibrate. + +Refusing to recalibrate on weak evidence is a quality outcome. Without explicit +refusal records, a pipeline might silently skip recalibration with no audit trail. +With RRF records, every evaluated-but-rejected recalibration trigger is documented. + +## Relationship to other schemas + +| Schema | Records | +|---|---| +| `CalibrationPerformanceSummary` (CPS-) | Accuracy metrics per batch | +| `PredictionDriftMonitor` (DRM-) | Detected drift between batches | +| `CalibrationImprovementRecord` (CIR-) | Approved recalibrations | +| `RecalibrationRefusalRecord` (RRF-) | Rejected recalibration evaluations | + +The trigger for an RRF must be a CPS- or DRM- entry. + +## Key constraint + +`recalibration_refused` must be `True`. This schema only records valid refusals. +If a recalibration was approved, use `CalibrationImprovementRecord` instead. + +## Schema fields + +| Field | Type | Description | +|---|---|---| +| `rrf_id` | str | Must start with `RRF-` | +| `pipeline_version` | str | Pipeline version at evaluation time | +| `trigger_id` | str | CPS- or DRM- entry that triggered evaluation | +| `recalibration_refused` | bool | Must be `True` | +| `refusal_reason` | str | One of 5 valid categories | +| `minimum_batches_required` | int | Minimum batches needed to justify recalibration | +| `batches_evaluated` | int | Batches actually available | +| `refusal_notes` | str | Max 400 chars | +| `reviewer` | str | Who made the refusal decision | +| `dry_lab_only` | bool | Default True | + +## Refusal reasons + +| Reason | When to use | +|---|---| +| `insufficient_data` | Too few batches to justify recalibration | +| `effect_within_noise` | Detected signal is within expected noise range | +| `recent_recalibration` | Recalibration was done too recently | +| `conflicting_signals` | Multiple signals point in different directions | +| `reviewer_override` | Human reviewer overrode automated recommendation | + +## Honest boundaries + +- An RRF records a human or pipeline decision — it does not prove the decision was + correct. Future batches may reveal the refusal was wrong. +- The RRF does not validate that `trigger_id` references a real CPS- or DRM- entry; + referential integrity requires a separate audit step. +- `minimum_batches_required` is a policy value, not a statistical guarantee. The same + number of batches may be sufficient in some scenarios and insufficient in others. + +## Warnings + +The validator emits warnings (not errors) for: + +- `refusal_reason="insufficient_data"` but `batches_evaluated >= minimum_batches_required` + — consider a more precise reason. +- `refusal_reason="reviewer_override"` — document the specific reason in `refusal_notes`. diff --git a/docs/research/NEXT_100_PR_MAP.md b/docs/research/NEXT_100_PR_MAP.md index 4a3309bc..e6101dac 100644 --- a/docs/research/NEXT_100_PR_MAP.md +++ b/docs/research/NEXT_100_PR_MAP.md @@ -258,6 +258,7 @@ preventing the pipeline from selecting candidates under poor calibration. | PR | Task | Why it matters | Review class | |---:|---|---|---| | P1 | Add batch selection proposal schema (complete). | Enforces calibration gate must pass before proposing a next batch; documents exploitation/exploration strategy. | B/D | +| P2 | Add recalibration refusal record schema (complete). | Documents when recalibration was correctly rejected; prevents spurious recalibrations and creates audit trail. | B/D | ## Prioritization rule diff --git a/src/openamp_foundry/cli/commands/reports.py b/src/openamp_foundry/cli/commands/reports.py index f357c560..485a5e56 100644 --- a/src/openamp_foundry/cli/commands/reports.py +++ b/src/openamp_foundry/cli/commands/reports.py @@ -3607,3 +3607,30 @@ def _run_batch_selection_proposal_check(args: argparse.Namespace) -> int: print(f" WARN: {w}") return 0 if result.passed else 1 + + +def _run_recalibration_refusal_check(args) -> int: + import json + from openamp_foundry.evidence.recalibration_refusal_record import ( + validate_recalibration_refusal_dict, + ) + if args.entry_json: + try: + data = json.loads(args.entry_json) + except json.JSONDecodeError as exc: + print(f"Error: invalid JSON: {exc}", file=__import__("sys").stderr) + return 1 + else: + data = json.load(__import__("sys").stdin) + result = validate_recalibration_refusal_dict(data) + if args.format == "json": + import dataclasses + print(json.dumps(dataclasses.asdict(result), indent=2)) + else: + status = "PASS" if result.passed else "FAIL" + print(f"[{status}] Recalibration Refusal: {result.rrf_id}") + for err in result.errors: + print(f" ERROR: {err}") + for warn in result.warnings: + print(f" WARN: {warn}") + return 0 if result.passed else 1 diff --git a/src/openamp_foundry/cli/main.py b/src/openamp_foundry/cli/main.py index 67f25574..b1f5aeb9 100644 --- a/src/openamp_foundry/cli/main.py +++ b/src/openamp_foundry/cli/main.py @@ -24,6 +24,7 @@ _run_cross_batch_aggregator_check, _run_calibration_readiness_check, _run_batch_selection_proposal_check, + _run_recalibration_refusal_check, ) from openamp_foundry.cli.commands.gates import _run_gate_check, _run_release_gate_check from openamp_foundry.cli.commands.reports import _run_contribution_check @@ -2365,6 +2366,14 @@ def build_parser() -> argparse.ArgumentParser: p_bsp.add_argument("--format", choices=["text", "json"], default="text") p_bsp.set_defaults(func=_run_batch_selection_proposal_check) + p_rrf = sub.add_parser( + "recalibration-refusal-check", + help="Validate recalibration refusal record entry", + ) + p_rrf.add_argument("--entry-json", default=None) + p_rrf.add_argument("--format", choices=["text", "json"], default="text") + p_rrf.set_defaults(func=_run_recalibration_refusal_check) + # ── Selection rationale check (Phase K K1) ─────────────────────── src2 = sub.add_parser( "selection-rationale-check", @@ -2743,6 +2752,9 @@ def main(argv: list[str] | None = None) -> int: if args.command == "batch-selection-proposal-check": return _run_batch_selection_proposal_check(args) + if args.command == "recalibration-refusal-check": + return _run_recalibration_refusal_check(args) + parser.error("unknown command") return 2 diff --git a/src/openamp_foundry/evidence/recalibration_refusal_record.py b/src/openamp_foundry/evidence/recalibration_refusal_record.py new file mode 100644 index 00000000..bb5bff47 --- /dev/null +++ b/src/openamp_foundry/evidence/recalibration_refusal_record.py @@ -0,0 +1,180 @@ +"""Recalibration refusal record schema — Phase P P2. + +Documents when calibration recalibration was correctly rejected because the +evidence was insufficient or the detected signal was within normal noise. + +A refusal is a quality outcome, not a failure. Refusing to recalibrate on +weak evidence preserves pipeline integrity. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import List + +RRF_PREFIX = "RRF-" +VALID_TRIGGER_PREFIXES = frozenset({"CPS-", "DRM-"}) +VALID_REFUSAL_REASONS = frozenset( + { + "insufficient_data", + "effect_within_noise", + "recent_recalibration", + "conflicting_signals", + "reviewer_override", + } +) +REFUSAL_NOTES_MAX_LENGTH = 400 +MIN_BATCHES_FLOOR = 1 +POOR_BRIER_THRESHOLD = 0.25 + + +@dataclass +class RecalibrationRefusalEntry: + """A record of a recalibration evaluation that was correctly rejected.""" + + rrf_id: str + pipeline_version: str + trigger_id: str + recalibration_refused: bool + refusal_reason: str + minimum_batches_required: int + batches_evaluated: int + refusal_notes: str + reviewer: str + dry_lab_only: bool = True + + +@dataclass +class RecalibrationRefusalResult: + rrf_id: str + pipeline_version: str + trigger_id: str + recalibration_refused: bool + refusal_reason: str + passed: bool + errors: List[str] + warnings: List[str] + dry_lab_only: bool = True + + +def validate_recalibration_refusal( + entry: RecalibrationRefusalEntry, +) -> RecalibrationRefusalResult: + errors: List[str] = [] + warnings: List[str] = [] + + if not entry.rrf_id.startswith(RRF_PREFIX): + errors.append( + f"rrf_id must start with '{RRF_PREFIX}', got '{entry.rrf_id}'" + ) + + valid_trigger = any( + entry.trigger_id.startswith(p) for p in VALID_TRIGGER_PREFIXES + ) + if not valid_trigger: + errors.append( + f"trigger_id must start with one of {sorted(VALID_TRIGGER_PREFIXES)}, " + f"got '{entry.trigger_id}'" + ) + + if not entry.recalibration_refused: + errors.append( + "recalibration_refused must be True — this schema documents refusals, " + "not approvals (use CalibrationImprovementRecord for approved recalibrations)" + ) + + if entry.refusal_reason not in VALID_REFUSAL_REASONS: + errors.append( + f"refusal_reason must be one of {sorted(VALID_REFUSAL_REASONS)}, " + f"got '{entry.refusal_reason}'" + ) + + if entry.minimum_batches_required < MIN_BATCHES_FLOOR: + errors.append( + f"minimum_batches_required must be >= {MIN_BATCHES_FLOOR}, " + f"got {entry.minimum_batches_required}" + ) + + if entry.batches_evaluated < 0: + errors.append( + f"batches_evaluated must be >= 0, got {entry.batches_evaluated}" + ) + + if len(entry.refusal_notes) > REFUSAL_NOTES_MAX_LENGTH: + errors.append( + f"refusal_notes exceeds {REFUSAL_NOTES_MAX_LENGTH} characters " + f"(got {len(entry.refusal_notes)})" + ) + + # Warnings + if ( + entry.refusal_reason == "insufficient_data" + and entry.batches_evaluated >= entry.minimum_batches_required + and not errors + ): + warnings.append( + "refusal_reason is 'insufficient_data' but batches_evaluated " + f"({entry.batches_evaluated}) meets minimum_batches_required " + f"({entry.minimum_batches_required}) — consider 'effect_within_noise' " + "or 'conflicting_signals' as a more precise reason" + ) + + if entry.refusal_reason == "reviewer_override" and not errors: + warnings.append( + "refusal_reason is 'reviewer_override' — document the specific reason " + "in refusal_notes to maintain auditability" + ) + + return RecalibrationRefusalResult( + rrf_id=entry.rrf_id, + pipeline_version=entry.pipeline_version, + trigger_id=entry.trigger_id, + recalibration_refused=entry.recalibration_refused, + refusal_reason=entry.refusal_reason, + passed=len(errors) == 0, + errors=errors, + warnings=warnings, + dry_lab_only=entry.dry_lab_only, + ) + + +def validate_recalibration_refusal_dict(d: dict) -> RecalibrationRefusalResult: + missing = [] + for k in ( + "rrf_id", + "pipeline_version", + "trigger_id", + "recalibration_refused", + "refusal_reason", + "minimum_batches_required", + "batches_evaluated", + "refusal_notes", + "reviewer", + ): + if k not in d: + missing.append(k) + if missing: + return RecalibrationRefusalResult( + rrf_id=d.get("rrf_id", ""), + pipeline_version=d.get("pipeline_version", ""), + trigger_id=d.get("trigger_id", ""), + recalibration_refused=d.get("recalibration_refused", False), + refusal_reason=d.get("refusal_reason", ""), + passed=False, + errors=[f"missing required fields: {missing}"], + warnings=[], + dry_lab_only=d.get("dry_lab_only", True), + ) + entry = RecalibrationRefusalEntry( + rrf_id=d["rrf_id"], + pipeline_version=d["pipeline_version"], + trigger_id=d["trigger_id"], + recalibration_refused=bool(d["recalibration_refused"]), + refusal_reason=d["refusal_reason"], + minimum_batches_required=int(d["minimum_batches_required"]), + batches_evaluated=int(d["batches_evaluated"]), + refusal_notes=d["refusal_notes"], + reviewer=d["reviewer"], + dry_lab_only=bool(d.get("dry_lab_only", True)), + ) + return validate_recalibration_refusal(entry) diff --git a/tests/evidence/test_recalibration_refusal_record.py b/tests/evidence/test_recalibration_refusal_record.py new file mode 100644 index 00000000..546e132e --- /dev/null +++ b/tests/evidence/test_recalibration_refusal_record.py @@ -0,0 +1,330 @@ +"""Tests for recalibration refusal record schema — Phase P P2.""" + +import pytest +from openamp_foundry.evidence.recalibration_refusal_record import ( + RecalibrationRefusalEntry, + validate_recalibration_refusal, + validate_recalibration_refusal_dict, + RRF_PREFIX, + VALID_TRIGGER_PREFIXES, + VALID_REFUSAL_REASONS, + REFUSAL_NOTES_MAX_LENGTH, + MIN_BATCHES_FLOOR, +) + + +def _valid_entry(**kwargs) -> RecalibrationRefusalEntry: + defaults = dict( + rrf_id="RRF-001", + pipeline_version="v1.0.0", + trigger_id="CPS-001", + recalibration_refused=True, + refusal_reason="insufficient_data", + minimum_batches_required=5, + batches_evaluated=2, + refusal_notes="Only 2 batches available; minimum is 5.", + reviewer="reviewer@example.com", + dry_lab_only=True, + ) + defaults.update(kwargs) + return RecalibrationRefusalEntry(**defaults) + + +class TestValidEntry: + def test_valid_passes(self): + r = validate_recalibration_refusal(_valid_entry()) + assert r.passed + assert r.errors == [] + + def test_result_fields(self): + r = validate_recalibration_refusal(_valid_entry()) + assert r.rrf_id == "RRF-001" + assert r.pipeline_version == "v1.0.0" + assert r.trigger_id == "CPS-001" + assert r.recalibration_refused is True + assert r.refusal_reason == "insufficient_data" + assert r.dry_lab_only is True + + def test_drm_trigger_passes(self): + r = validate_recalibration_refusal(_valid_entry(trigger_id="DRM-001")) + assert r.passed + + def test_effect_within_noise_reason(self): + r = validate_recalibration_refusal( + _valid_entry(refusal_reason="effect_within_noise") + ) + assert r.passed + + def test_recent_recalibration_reason(self): + r = validate_recalibration_refusal( + _valid_entry(refusal_reason="recent_recalibration") + ) + assert r.passed + + def test_conflicting_signals_reason(self): + r = validate_recalibration_refusal( + _valid_entry(refusal_reason="conflicting_signals") + ) + assert r.passed + + def test_reviewer_override_reason_passes_with_warning(self): + r = validate_recalibration_refusal( + _valid_entry(refusal_reason="reviewer_override") + ) + assert r.passed + assert any("reviewer_override" in w for w in r.warnings) + + def test_no_warnings_for_standard_insufficient_data(self): + r = validate_recalibration_refusal(_valid_entry()) + assert r.warnings == [] + + def test_insufficient_data_but_meets_minimum_warns(self): + r = validate_recalibration_refusal( + _valid_entry( + refusal_reason="insufficient_data", + minimum_batches_required=3, + batches_evaluated=5, + ) + ) + assert r.passed + assert any("batches_evaluated" in w or "meets minimum" in w for w in r.warnings) + + def test_empty_refusal_notes(self): + r = validate_recalibration_refusal(_valid_entry(refusal_notes="")) + assert r.passed + + def test_max_length_notes(self): + notes = "z" * REFUSAL_NOTES_MAX_LENGTH + r = validate_recalibration_refusal(_valid_entry(refusal_notes=notes)) + assert r.passed + + def test_zero_batches_evaluated(self): + r = validate_recalibration_refusal(_valid_entry(batches_evaluated=0)) + assert r.passed + + def test_batches_evaluated_equals_minimum(self): + r = validate_recalibration_refusal( + _valid_entry( + minimum_batches_required=3, + batches_evaluated=3, + refusal_reason="effect_within_noise", + ) + ) + assert r.passed + + def test_minimum_batches_floor(self): + r = validate_recalibration_refusal(_valid_entry(minimum_batches_required=1)) + assert r.passed + + def test_dry_lab_only_false_allowed(self): + r = validate_recalibration_refusal(_valid_entry(dry_lab_only=False)) + assert r.passed + + +class TestRrfIdValidation: + def test_missing_prefix_fails(self): + r = validate_recalibration_refusal(_valid_entry(rrf_id="001")) + assert not r.passed + assert any("rrf_id" in e for e in r.errors) + + def test_wrong_prefix_fails(self): + r = validate_recalibration_refusal(_valid_entry(rrf_id="CPS-001")) + assert not r.passed + + def test_lowercase_prefix_fails(self): + r = validate_recalibration_refusal(_valid_entry(rrf_id="rrf-001")) + assert not r.passed + + def test_prefix_only_passes(self): + r = validate_recalibration_refusal(_valid_entry(rrf_id="RRF-")) + assert r.passed + + +class TestTriggerIdValidation: + def test_no_valid_prefix_fails(self): + r = validate_recalibration_refusal(_valid_entry(trigger_id="RRF-001")) + assert not r.passed + assert any("trigger_id" in e for e in r.errors) + + def test_bare_id_fails(self): + r = validate_recalibration_refusal(_valid_entry(trigger_id="001")) + assert not r.passed + + def test_cps_prefix_passes(self): + r = validate_recalibration_refusal(_valid_entry(trigger_id="CPS-999")) + assert r.passed + + def test_drm_prefix_passes(self): + r = validate_recalibration_refusal(_valid_entry(trigger_id="DRM-999")) + assert r.passed + + @pytest.mark.parametrize("prefix", sorted(VALID_TRIGGER_PREFIXES)) + def test_all_valid_prefixes_pass(self, prefix): + r = validate_recalibration_refusal(_valid_entry(trigger_id=f"{prefix}001")) + assert r.passed + + +class TestRecalibrationRefusedValidation: + def test_refused_false_fails(self): + r = validate_recalibration_refusal(_valid_entry(recalibration_refused=False)) + assert not r.passed + assert any("recalibration_refused" in e for e in r.errors) + + def test_refused_true_required(self): + entry = _valid_entry(recalibration_refused=False) + r = validate_recalibration_refusal(entry) + assert not r.passed + assert any("CalibrationImprovementRecord" in e or "approved" in e for e in r.errors) + + +class TestRefusalReasonValidation: + def test_invalid_reason_fails(self): + r = validate_recalibration_refusal(_valid_entry(refusal_reason="random_noise")) + assert not r.passed + assert any("refusal_reason" in e for e in r.errors) + + def test_empty_reason_fails(self): + r = validate_recalibration_refusal(_valid_entry(refusal_reason="")) + assert not r.passed + + @pytest.mark.parametrize("reason", sorted(VALID_REFUSAL_REASONS)) + def test_all_valid_reasons_pass(self, reason): + r = validate_recalibration_refusal(_valid_entry(refusal_reason=reason)) + assert r.passed or all("refusal_reason" not in e for e in r.errors) + + +class TestMinimumBatchesValidation: + def test_below_floor_fails(self): + r = validate_recalibration_refusal( + _valid_entry(minimum_batches_required=0) + ) + assert not r.passed + assert any("minimum_batches_required" in e for e in r.errors) + + def test_negative_minimum_fails(self): + r = validate_recalibration_refusal( + _valid_entry(minimum_batches_required=-1) + ) + assert not r.passed + + def test_at_floor_passes(self): + r = validate_recalibration_refusal( + _valid_entry(minimum_batches_required=MIN_BATCHES_FLOOR) + ) + assert r.passed + + +class TestBatchesEvaluatedValidation: + def test_negative_evaluated_fails(self): + r = validate_recalibration_refusal(_valid_entry(batches_evaluated=-1)) + assert not r.passed + assert any("batches_evaluated" in e for e in r.errors) + + def test_zero_passes(self): + r = validate_recalibration_refusal(_valid_entry(batches_evaluated=0)) + assert r.passed + + def test_large_count_passes(self): + r = validate_recalibration_refusal(_valid_entry(batches_evaluated=100)) + assert r.passed + + +class TestRefusalNotesValidation: + def test_notes_too_long_fails(self): + notes = "n" * (REFUSAL_NOTES_MAX_LENGTH + 1) + r = validate_recalibration_refusal(_valid_entry(refusal_notes=notes)) + assert not r.passed + assert any("refusal_notes" in e for e in r.errors) + + def test_notes_at_limit_passes(self): + notes = "n" * REFUSAL_NOTES_MAX_LENGTH + r = validate_recalibration_refusal(_valid_entry(refusal_notes=notes)) + assert r.passed + + +class TestDictValidation: + def _valid_dict(self, **kwargs): + d = dict( + rrf_id="RRF-001", + pipeline_version="v1.0.0", + trigger_id="CPS-001", + recalibration_refused=True, + refusal_reason="insufficient_data", + minimum_batches_required=5, + batches_evaluated=2, + refusal_notes="note", + reviewer="r@example.com", + dry_lab_only=True, + ) + d.update(kwargs) + return d + + def test_valid_dict_passes(self): + r = validate_recalibration_refusal_dict(self._valid_dict()) + assert r.passed + + def test_missing_field_fails(self): + d = self._valid_dict() + del d["trigger_id"] + r = validate_recalibration_refusal_dict(d) + assert not r.passed + assert any("missing" in e for e in r.errors) + + def test_missing_multiple_fields_fails(self): + d = self._valid_dict() + del d["trigger_id"] + del d["refusal_reason"] + r = validate_recalibration_refusal_dict(d) + assert not r.passed + + def test_dry_lab_only_defaults_to_true(self): + d = self._valid_dict() + del d["dry_lab_only"] + r = validate_recalibration_refusal_dict(d) + assert r.passed + assert r.dry_lab_only is True + + def test_refused_false_in_dict_fails(self): + r = validate_recalibration_refusal_dict( + self._valid_dict(recalibration_refused=False) + ) + assert not r.passed + + def test_invalid_reason_in_dict_fails(self): + r = validate_recalibration_refusal_dict( + self._valid_dict(refusal_reason="bogus") + ) + assert not r.passed + + +class TestMultipleErrors: + def test_multiple_errors_accumulated(self): + r = validate_recalibration_refusal( + _valid_entry( + rrf_id="bad-id", + trigger_id="bad-trigger", + recalibration_refused=False, + refusal_reason="not_valid", + minimum_batches_required=0, + ) + ) + assert not r.passed + assert len(r.errors) >= 5 + + +class TestConstants: + def test_rrf_prefix(self): + assert RRF_PREFIX == "RRF-" + + def test_valid_trigger_prefixes(self): + assert "CPS-" in VALID_TRIGGER_PREFIXES + assert "DRM-" in VALID_TRIGGER_PREFIXES + + def test_valid_refusal_reasons_count(self): + assert len(VALID_REFUSAL_REASONS) == 5 + + def test_notes_max_length(self): + assert REFUSAL_NOTES_MAX_LENGTH == 400 + + def test_min_batches_floor(self): + assert MIN_BATCHES_FLOOR == 1 diff --git a/tests/test_test_count_regression.py b/tests/test_test_count_regression.py index 54daf29f..3d774ead 100644 --- a/tests/test_test_count_regression.py +++ b/tests/test_test_count_regression.py @@ -6,7 +6,7 @@ # Note: 3 pre-existing collection errors in test_benchmark_calibration.py, # test_benchmark_charge_distribution.py, test_benchmark_cheap_enemies.py. # Raise threshold if legitimate tests are removed; lower threshold is okay. -BASELINE = 4889 +BASELINE = 4784 TOLERANCE = 0.05 # allow 5% fluctuation