Define CR-123 evaluation handoff contract

docs/change/change_log.md

@@ -5,6 +5,7 @@ This file provides a chronological, human-readable record of applied codebase an
 *Note: For operational task and run history, consult `.triagecore/ledger.jsonl`.*
 
 ## [Unreleased]
+- Implemented CR-123 (Evaluation Handoff Contract): Added a contract-only `docs/evals/evaluation_handoff_contract.md` defining the file-based boundary between TriageCore and the external evaluator suite, including required `eval_case_v0` fixture input, `actual_outcome_export.v0` actual outcome files, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations. Added focused documentation tests and linked the existing fixture and actual-outcome docs while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope.
 - Implemented CR-122 (Eval Fixture Validation CLI): Added `tc eval validate-fixtures --input <path>` as a narrow read-only wrapper around the CR-121 JSONL validator, printing a bounded pass count on valid fixtures and fail-closed, line-aware diagnostics on invalid input. Added synthetic CLI tests only and kept scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope.
 - Implemented CR-121 (Eval Fixture Validator): Added a pure deterministic JSONL validator for the CR-077 safety-boundary eval fixture contract, with line-aware diagnostics and fail-closed handling for malformed JSON, non-object/empty lines, missing required fields, empty or duplicate `case_id`, closed-vocabulary violations, and required nested fixture shape. Added synthetic unit tests only and kept `tc eval`, scoring, model/backend calls, routing/admission integration, ledger writes, and adversarial tampering tests out of scope.
 - Implemented CR-120 (Telemetry Lane Release Hygiene): Added a docs-only post-merge hygiene note for the completed CR-117 through CR-119 lane, marked the reviewer checkpoint/release-hygiene candidate complete, and corrected stale telemetry brief wording that could confuse the CR-114 reviewer checkpoint with the CR-119 probe validation gate. No runtime code, schemas, tests, CLI behavior, routing, ledger writes, probe execution, endpoint calls, tags, or model/backend calls were changed.

docs/change/requests/CR-123-evaluation-handoff-contract.md

@@ -0,0 +1,48 @@
+# CR-123: Evaluation Handoff Contract
+
+## Status
+
+Implemented (contract-only)
+
+## Summary
+
+Define the file-contract boundary between TriageCore and the external evaluator
+suite after CR-122. TriageCore validates expected fixtures and exports actual
+outcome evidence; the external evaluator suite owns scoring, pass/fail
+judgment, findings, and aggregate metrics.
+
+## Scope
+
+- Add `docs/evals/evaluation_handoff_contract.md`.
+- Define required inputs and outputs for the handoff:
+  - `eval_case_v0` expected fixture JSONL.
+  - `actual_outcome_export.v0` actual outcome JSON files.
+- Pin deterministic path vocabulary for a future bundle:
+  - `fixtures/safety_boundaries_v0.jsonl`
+  - `actuals/<case_id>.json`
+  - `manifest/evaluation_handoff_manifest.json`
+- Document TriageCore-side exit-code expectations for fixture validation and
+  existing actual-outcome smoke export commands.
+- Link the existing fixture and actual-outcome docs to the handoff contract.
+- Add focused documentation tests that keep the contract aligned with the
+  existing fixture validator and actual outcome writer.
+- Update backlog and changelog after focused tests pass.
+
+## Non-Goals
+
+- No scoring, pass/fail judgment, aggregate metrics, partial credit, or score
+  interpretation inside TriageCore.
+- No evaluator execution from TriageCore.
+- No model, completion, chat, embedding, backend, endpoint, or network calls.
+- No routing, admission, approval, identity, worker, or ledger integration.
+- No ledger writes or durable evaluation-run evidence.
+- No bundle builder, manifest writer, bundle validator, result importer, or
+  external result display.
+- No changes to `eval_case_v0`, actual outcome JSON fields, or the current
+  fixture family.
+- No adversarial or tampering expansion.
+
+## Validation
+
+- Focused: `python -m pytest -q tests/test_eval_handoff_contract.py tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py tests/test_eval_outcome_contract.py` -> 32 passed
+- Full suite: `python -m pytest -q` -> 935 passed, 2 skipped

docs/current_backlog.md

@@ -6,6 +6,10 @@ This document summarizes the active TriageCore backlog after CR-115.
 
 ## Active GitHub Backlog
 
+- CR-123: Evaluation Handoff Contract
+  - Status: complete via CR-123 (contract-only)
+  - Purpose: Define the file-contract boundary between TriageCore and the external evaluator suite, including required fixture and actual-outcome inputs, contract/version identifiers, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations, while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope.
+
 - CR-122: Eval Fixture Validation CLI
   - Status: complete via CR-122 (validator CLI only)
   - Purpose: Expose the CR-121 safety-boundary eval fixture validator through `tc eval validate-fixtures --input <path>`, with bounded pass output and fail-closed line-aware diagnostics, while keeping scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope.
@@ -120,7 +124,7 @@ This document summarizes the active TriageCore backlog after CR-115.
 
 - Empirical AI safety evaluation track
   - Source: CR-076 and CR-077 research framing/eval taxonomy docs
-  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; scoring/evaluator execution, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
+  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; bundle/manifest building, bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
   - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure.
 
 - Operator UX implementation path
@@ -212,7 +216,7 @@ Keep three work lanes distinct:
 
 For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness.
 
-For the empirical AI safety evaluation track, CR-121 completes fixture validation and CR-122 exposes it through a narrow CLI. Keep the next slices sequential: scoring/evaluator execution only after the validation CLI remains stable, and broader adversarial/tampering studies only after scoring is separately reviewed.
+For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, and CR-123 defines the external-evaluator handoff contract. Keep the next slices sequential: bundle/manifest builder, bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
 
 For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path.
 

docs/evals/actual_outcome_export.md

@@ -7,10 +7,12 @@ By exporting evidence as files:
 - TriageCore does not need to self-score.
 - There are no cross-repository Python imports, preventing tight coupling.
 
-> **Reviewer Note:** For the conceptual explanation of how these actual outcome exports can be scored by an independent harness without importing TriageCore, see [Eval Integration Bridge](eval_integration_bridge.md).
+> **Reviewer Note:** For the conceptual explanation of how these actual outcome exports can be scored by an independent harness without importing TriageCore, see [Eval Integration Bridge](eval_integration_bridge.md). For the CR-123 file handoff boundary, required inputs/outputs, deterministic path vocabulary, and exit-code expectations, see [Evaluation Handoff Contract](evaluation_handoff_contract.md).
 
 ## JSON Contract Shape
 
+Contract identifier: `actual_outcome_export.v0`
+
 TriageCore generates one JSON file per evaluation scenario. The exported file must match the following shape:
 
 ```json

docs/evals/eval_integration_bridge.md

@@ -3,6 +3,8 @@
 ## Purpose
 This document provides a reviewer-facing explanation of how TriageCore interfaces with the independent `agent-control-evals` repository. It clarifies how actual outcome exports produced by TriageCore can be deterministically scored by an external harness without importing or coupling the repositories.
 
+The normative CR-123 handoff boundary is documented in [Evaluation Handoff Contract](evaluation_handoff_contract.md). This bridge remains a conceptual integration note.
+
 ## Integration Boundary
 The integration between TriageCore and the evaluation harness is strictly **file-contract-based**.
 There are no shared runtime dependencies, no cross-repository Python imports, and no hidden coupling. The boundary is entirely maintained by writing static JSON files to disk, which are then independently read and scored.

docs/evals/evaluation_handoff_contract.md

@@ -0,0 +1,123 @@
+# Evaluation Handoff Contract
+
+Contract identifier: `evaluation_handoff_contract.v0`
+
+## Purpose
+
+This document defines the CR-123 handoff boundary between TriageCore and an
+external evaluator suite. TriageCore may validate and export deterministic
+evidence. The external evaluator suite owns scoring, pass/fail judgment,
+findings, and aggregate metrics.
+
+The contract is intentionally file-based. It avoids cross-repository imports,
+runtime callbacks, hidden model calls, and any requirement that TriageCore know
+the evaluator's scoring implementation.
+
+## Required TriageCore Inputs
+
+The TriageCore side of the handoff requires explicit operator-named paths:
+
+| Input | Required | Contract identifier | Notes |
+| --- | --- | --- | --- |
+| Expected fixture JSONL | Yes | `eval_case_v0` | One JSON object per line. Validate with `tc eval validate-fixtures --input <path>`. |
+| Actual outcome directory | Yes | `actual_outcome_export.v0` | Directory containing one `<case_id>.json` file per observed case. |
+
+The current repository fixture is:
+
+```text
+tests/fixtures/evals/safety_boundaries_v0.jsonl
+```
+
+The current documented actual-output directories are:
+
+```text
+.triagecore/eval_actuals/<run_id>/
+actuals/triagecore_smoke/
+```
+
+TriageCore must not infer default evaluator inputs from the ledger, route
+history, admission state, or local runtime state. The operator or a future
+bundle builder must pass file paths explicitly.
+
+## Required TriageCore Outputs
+
+TriageCore produces static files only:
+
+| Output | Required | Deterministic path rule | Producer |
+| --- | --- | --- | --- |
+| Validated expected fixture JSONL | Yes | Preserve the operator-provided filename; current fixture path is `tests/fixtures/evals/safety_boundaries_v0.jsonl`. | Human-authored fixture plus CR-121/CR-122 validation |
+| Actual outcome JSON files | Yes | Write one path-safe `<case_id>.json` file under the operator-provided actuals directory. | Existing actual outcome export helpers |
+
+Actual outcome files use the existing JSON shape documented in
+`docs/evals/actual_outcome_export.md`. The required fields are:
+
+- `case_id`
+- `decision`
+- `boundary_family`
+- `reasons`
+- `audit_required`
+- `human_approval_required`
+
+Optional diagnostic fields may exist, but external scoring must treat them as
+diagnostic evidence rather than primary oracle fields unless the external suite
+explicitly adopts them.
+
+## External Evaluator Outputs
+
+The external evaluator suite, not TriageCore, produces scored artifacts such as:
+
+```text
+reports/<run_id>.jsonl
+```
+
+Those reports may include pass/fail outcomes, partial-credit findings,
+aggregate metrics, or reviewer-facing summaries. TriageCore does not define the
+scored report schema in this CR. TriageCore must treat evaluator findings as external artifacts unless a later CR adds an explicit import or display contract.
+
+## Deterministic Handoff Layout
+
+If a future CR materializes a handoff bundle, it should use these relative
+paths inside the bundle directory:
+
+```text
+fixtures/safety_boundaries_v0.jsonl
+actuals/<case_id>.json
+manifest/evaluation_handoff_manifest.json
+```
+
+CR-123 does not create that bundle, manifest, or builder. These names reserve a
+stable path vocabulary so CR-124+ can package the same already-defined evidence
+without changing the scoring boundary.
+
+## Exit-Code Expectations
+
+TriageCore commands used before handoff must follow these expectations:
+
+| Command | Success | Failure | CLI usage error |
+| --- | --- | --- | --- |
+| `tc eval validate-fixtures --input <path>` | Exit `0` when the JSONL fixture is valid. | Exit `1` for missing, unreadable, malformed, structurally invalid, or duplicate-case fixtures. | Exit `2` for argparse usage errors. |
+| `tc eval export-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
+| `tc eval export-privacy-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
+| `tc eval export-forbidden-tool-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
+
+The external evaluator suite owns its own exit-code contract. TriageCore may
+document how to invoke that suite, but this CR does not add an evaluator runner
+or interpret evaluator exit statuses.
+
+## Non-Goals
+
+CR-123 explicitly excludes:
+
+- scoring, pass/fail judgment, aggregate metrics, partial credit, or score
+  interpretation inside TriageCore
+- no evaluator execution from TriageCore or any TriageCore CLI command
+- model, completion, chat, embedding, backend, endpoint, or network calls
+- routing, admission, approval, identity, or worker integration
+- no ledger writes or durable recording that an evaluation occurred
+- a bundle builder, manifest writer, bundle validator, or result importer
+- changes to `eval_case_v0` or actual outcome JSON fields
+- new fixture families or adversarial/tampering expansion
+
+The next safe slice after this contract is a deterministic bundle or manifest
+builder that packages already-validated fixtures and already-exported actuals
+without executing or scoring the evaluator.

docs/research/eval_fixture_schema.md

@@ -140,4 +140,7 @@ CR-121 treats this document as the contract for a narrow validator. The validato
 
 The validator checks fixture integrity only, and CR-122 exposes that validation
 through `tc eval validate-fixtures --input <path>`. These fixtures still are not
-scored or executed until a separate scoring/evaluator slice exists.
+scored or executed by TriageCore. CR-123 defines the file-based
+[Evaluation Handoff Contract](../evals/evaluation_handoff_contract.md) for
+passing validated fixtures and actual outcome exports to an external evaluator
+suite, while scoring remains outside TriageCore.

tests/test_eval_handoff_contract.py

@@ -0,0 +1,69 @@
+from pathlib import Path
+
+from triage_core.eval_fixture_validator import SCHEMA_VERSION
+from triage_core.eval_outcome_contract import build_actual_outcome, write_actual_outcome
+
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+CONTRACT_DOC = REPO_ROOT / "docs" / "evals" / "evaluation_handoff_contract.md"
+ACTUAL_OUTCOME_DOC = REPO_ROOT / "docs" / "evals" / "actual_outcome_export.md"
+FIXTURE_SCHEMA_DOC = REPO_ROOT / "docs" / "research" / "eval_fixture_schema.md"
+BRIDGE_DOC = REPO_ROOT / "docs" / "evals" / "eval_integration_bridge.md"
+
+
+def _read(path: Path) -> str:
+    return path.read_text(encoding="utf-8")
+
+
+def test_eval_handoff_contract_pins_versions_paths_and_exit_codes():
+    doc = _read(CONTRACT_DOC)
+
+    assert "evaluation_handoff_contract.v0" in doc
+    assert SCHEMA_VERSION in doc
+    assert "actual_outcome_export.v0" in doc
+    assert "tests/fixtures/evals/safety_boundaries_v0.jsonl" in doc
+    assert ".triagecore/eval_actuals/<run_id>/" in doc
+    assert "actuals/triagecore_smoke/" in doc
+    assert "fixtures/safety_boundaries_v0.jsonl" in doc
+    assert "actuals/<case_id>.json" in doc
+    assert "manifest/evaluation_handoff_manifest.json" in doc
+    assert "Exit `0`" in doc
+    assert "Exit `1`" in doc
+    assert "Exit `2`" in doc
+
+
+def test_eval_handoff_contract_preserves_external_scoring_boundary():
+    doc = " ".join(_read(CONTRACT_DOC).lower().split())
+
+    required_phrases = (
+        "the external evaluator suite owns scoring",
+        "treat evaluator findings as external artifacts",
+        "cr-123 does not create that bundle, manifest, or builder",
+        "no evaluator execution from triagecore",
+        "no ledger writes",
+        "scoring, pass/fail judgment, aggregate metrics, partial credit, or score interpretation inside triagecore",
+    )
+    for phrase in required_phrases:
+        assert phrase in doc
+
+
+def test_eval_handoff_contract_matches_actual_outcome_filename_rule(tmp_path):
+    outcome = build_actual_outcome(
+        case_id="privacy-deny-001",
+        decision="block",
+        boundary_family="privacy",
+        reasons=["privacy_check_failed"],
+        audit_required=True,
+        human_approval_required=False,
+    )
+
+    path = write_actual_outcome(outcome, tmp_path)
+
+    assert path.name == "privacy-deny-001.json"
+    assert "<case_id>.json" in _read(CONTRACT_DOC)
+
+
+def test_existing_eval_docs_link_to_handoff_contract():
+    assert "evaluation_handoff_contract.md" in _read(ACTUAL_OUTCOME_DOC)
+    assert "Evaluation Handoff Contract" in _read(FIXTURE_SCHEMA_DOC)
+    assert "evaluation_handoff_contract.md" in _read(BRIDGE_DOC)