diff --git a/docs/change/change_log.md b/docs/change/change_log.md index 4cd0414..f35f4b9 100644 --- a/docs/change/change_log.md +++ b/docs/change/change_log.md @@ -5,6 +5,7 @@ This file provides a chronological, human-readable record of applied codebase an *Note: For operational task and run history, consult `.triagecore/ledger.jsonl`.* ## [Unreleased] +- Implemented CR-124 (Eval Handoff Hygiene, Bug, and Drift Slice): Fixed `write_actual_outcomes()` so generator and other single-pass iterable inputs still write all actual outcome files after duplicate checks, updated stale eval fixture/taxonomy sequencing docs so they no longer imply internal TriageCore scoring, and added focused regression/drift coverage. No evaluator execution, scoring, score interpretation, model/backend calls, routing/admission integration, ledger writes, schema changes, bundle builder, or result import/display were added. - Implemented CR-123 (Evaluation Handoff Contract): Added a contract-only `docs/evals/evaluation_handoff_contract.md` defining the file-based boundary between TriageCore and the external evaluator suite, including required `eval_case_v0` fixture input, `actual_outcome_export.v0` actual outcome files, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations. Added focused documentation tests and linked the existing fixture and actual-outcome docs while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope. - Implemented CR-122 (Eval Fixture Validation CLI): Added `tc eval validate-fixtures --input ` as a narrow read-only wrapper around the CR-121 JSONL validator, printing a bounded pass count on valid fixtures and fail-closed, line-aware diagnostics on invalid input. Added synthetic CLI tests only and kept scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope. - Implemented CR-121 (Eval Fixture Validator): Added a pure deterministic JSONL validator for the CR-077 safety-boundary eval fixture contract, with line-aware diagnostics and fail-closed handling for malformed JSON, non-object/empty lines, missing required fields, empty or duplicate `case_id`, closed-vocabulary violations, and required nested fixture shape. Added synthetic unit tests only and kept `tc eval`, scoring, model/backend calls, routing/admission integration, ledger writes, and adversarial tampering tests out of scope. diff --git a/docs/change/requests/CR-124-eval-hygiene-bug-drift.md b/docs/change/requests/CR-124-eval-hygiene-bug-drift.md new file mode 100644 index 0000000..2735fc5 --- /dev/null +++ b/docs/change/requests/CR-124-eval-hygiene-bug-drift.md @@ -0,0 +1,39 @@ +# CR-124: Eval Handoff Hygiene, Bug, and Drift Slice + +## Status + +Implemented + +## Summary + +Close one small post-CR-123 maintenance loop: fix a single-pass iterable bug in +actual outcome export writing, update stale eval sequencing docs so they no +longer imply an internal TriageCore scoring slice, and add focused regression +coverage for both. + +## Scope + +- Fix `write_actual_outcomes()` so generator and other single-pass iterable + inputs still write all expected `.json` files after duplicate checks. +- Add a focused regression test for generator-backed actual outcome exports. +- Update eval fixture and taxonomy docs to reflect the CR-123 architecture: + TriageCore may package/validate handoff evidence, while scoring remains + external. +- Extend the CR-123 handoff documentation tests to catch drift back toward + internal scoring language. +- Update backlog and changelog after focused validation passes. + +## Non-Goals + +- No evaluator execution from TriageCore. +- No scoring, pass/fail judgment, aggregate metrics, or score interpretation in + TriageCore. +- No new actual outcome fields, fixture schema fields, bundle builder, manifest + writer, or bundle validator. +- No model, backend, endpoint, routing, admission, identity, approval, worker, + or ledger integration. + +## Validation + +- Focused: `python -m pytest -q tests/test_eval_outcome_contract.py tests/test_eval_handoff_contract.py tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py` -> 34 passed +- Full suite: `python -m pytest -q` -> 937 passed, 2 skipped diff --git a/docs/current_backlog.md b/docs/current_backlog.md index c3e5a23..2d9624a 100644 --- a/docs/current_backlog.md +++ b/docs/current_backlog.md @@ -6,6 +6,10 @@ This document summarizes the active TriageCore backlog after CR-115. ## Active GitHub Backlog +- CR-124: Eval Handoff Hygiene, Bug, and Drift Slice + - Status: complete via CR-124 + - Purpose: Fix the generator/single-pass iterable bug in actual outcome export writing and clean up stale eval sequencing docs so they point from CR-123 toward bundle/manifest work while preserving the rule that scoring remains external to TriageCore. + - CR-123: Evaluation Handoff Contract - Status: complete via CR-123 (contract-only) - Purpose: Define the file-contract boundary between TriageCore and the external evaluator suite, including required fixture and actual-outcome inputs, contract/version identifiers, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations, while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope. diff --git a/docs/research/eval_taxonomy.md b/docs/research/eval_taxonomy.md index 9e377a0..3d091b1 100644 --- a/docs/research/eval_taxonomy.md +++ b/docs/research/eval_taxonomy.md @@ -100,6 +100,10 @@ Planned follow-on sequence: 1. CR-121: validate the fixture schema and enforce deterministic required fields 2. CR-122: expose fixture validation through a narrow `tc eval validate-fixtures` surface -3. Future CR: score fixture outcomes through a separate reviewed evaluator path +3. CR-123: define the file-based handoff boundary to an external evaluator suite +4. Future CR: package validated fixtures and exported actual outcomes into a deterministic handoff bundle +5. Future external evaluator suite work: score fixture outcomes outside TriageCore -That order matters. TriageCore should define the cases before validating them, and validate them before claiming to execute them. +That order matters. TriageCore should define the cases before validating them, +validate them before packaging them, and preserve the boundary that scoring and +score interpretation happen outside TriageCore. diff --git a/tests/fixtures/evals/README.md b/tests/fixtures/evals/README.md index d83a6db..3e68a66 100644 --- a/tests/fixtures/evals/README.md +++ b/tests/fixtures/evals/README.md @@ -19,6 +19,7 @@ These fixtures exist to define what a future evaluator will consume. They do not ## Expected follow-on -- CR-121 adds a narrow validator for required fields and deterministic labels. -- CR-122 exposes that validator through `tc eval validate-fixtures --input `. -- A future CR can add scoring only after the validation CLI is stable. +- CR-121 added a narrow validator for required fields and deterministic labels. +- CR-122 exposed that validator through `tc eval validate-fixtures --input `. +- CR-123 defined the file-based handoff boundary to the external evaluator suite. +- Future TriageCore slices may package or validate the handoff bundle, but scoring remains external to TriageCore. diff --git a/tests/test_eval_handoff_contract.py b/tests/test_eval_handoff_contract.py index fc50995..7c96176 100644 --- a/tests/test_eval_handoff_contract.py +++ b/tests/test_eval_handoff_contract.py @@ -9,6 +9,7 @@ ACTUAL_OUTCOME_DOC = REPO_ROOT / "docs" / "evals" / "actual_outcome_export.md" FIXTURE_SCHEMA_DOC = REPO_ROOT / "docs" / "research" / "eval_fixture_schema.md" BRIDGE_DOC = REPO_ROOT / "docs" / "evals" / "eval_integration_bridge.md" +FIXTURE_README = REPO_ROOT / "tests" / "fixtures" / "evals" / "README.md" def _read(path: Path) -> str: @@ -67,3 +68,13 @@ def test_existing_eval_docs_link_to_handoff_contract(): assert "evaluation_handoff_contract.md" in _read(ACTUAL_OUTCOME_DOC) assert "Evaluation Handoff Contract" in _read(FIXTURE_SCHEMA_DOC) assert "evaluation_handoff_contract.md" in _read(BRIDGE_DOC) + + +def test_eval_research_docs_do_not_drift_back_to_internal_scoring(): + fixture_readme = _read(FIXTURE_README) + taxonomy_doc = _read(FIXTURE_SCHEMA_DOC.parent / "eval_taxonomy.md") + + assert "CR-123 defined the file-based handoff boundary" in fixture_readme + assert "scoring remains external to TriageCore" in fixture_readme + assert "score fixture outcomes through a separate reviewed evaluator path" not in taxonomy_doc + assert "score fixture outcomes outside TriageCore" in taxonomy_doc diff --git a/tests/test_eval_outcome_contract.py b/tests/test_eval_outcome_contract.py index 2afeb49..167d034 100644 --- a/tests/test_eval_outcome_contract.py +++ b/tests/test_eval_outcome_contract.py @@ -171,6 +171,26 @@ def test_write_multiple_outcomes(self): self.assertTrue((Path(tmpdir) / "test_mult_001.json").exists()) self.assertTrue((Path(tmpdir) / "test_mult_002.json").exists()) + def test_write_multiple_outcomes_accepts_single_pass_iterable(self): + outcomes = ( + build_actual_outcome( + case_id=f"generator_{index:03d}", + decision="allow", + boundary_family="none", + reasons=[], + audit_required=False, + human_approval_required=False + ) + for index in range(2) + ) + + with tempfile.TemporaryDirectory() as tmpdir: + paths = write_actual_outcomes(outcomes, tmpdir) + + self.assertEqual(len(paths), 2) + self.assertTrue((Path(tmpdir) / "generator_000.json").exists()) + self.assertTrue((Path(tmpdir) / "generator_001.json").exists()) + def test_duplicate_case_id(self): outcomes = [ build_actual_outcome( diff --git a/triage_core/eval_outcome_contract.py b/triage_core/eval_outcome_contract.py index 6a47d51..54d42c9 100644 --- a/triage_core/eval_outcome_contract.py +++ b/triage_core/eval_outcome_contract.py @@ -105,15 +105,16 @@ def write_actual_outcomes( Writes multiple actual outcomes to the specified output directory. Raises ValueError if there are duplicate case_ids in the provided outcomes. """ + outcome_list = list(outcomes) seen_ids = set() - for outcome in outcomes: + for outcome in outcome_list: case_id = outcome.get("case_id") if case_id in seen_ids: raise ValueError(f"Duplicate case_id '{case_id}' found in outcomes.") seen_ids.add(case_id) paths = [] - for outcome in outcomes: + for outcome in outcome_list: paths.append(write_actual_outcome(outcome, output_dir)) return paths