Add deterministic evaluation handoff bundle

docs/change/change_log.md

@@ -5,6 +5,8 @@ This file provides a chronological, human-readable record of applied codebase an
 *Note: For operational task and run history, consult `.triagecore/ledger.jsonl`.*
 
 ## [Unreleased]
+
+- Implemented CR-127 (Evaluation Handoff Bundle Builder): Added `tc eval build-handoff --fixture <jsonl> --actuals-dir <dir> --out-dir <new-dir>` and a pure deterministic builder that validates explicit fixture/actual inputs, applies persistent privacy checks, copies evidence byte-for-byte into the fixed handoff layout, writes a sorted `evaluation_handoff_manifest.v0` with SHA-256 hashes, and atomically publishes only after successful staging. Scoring, pass/fail judgment, evaluator execution, approval claims, automatic discovery, overwrite behavior, and existing-bundle integrity validation remain out of scope and external scoring remains authoritative.
 - Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope.
 - Implemented CR-125 (Honor Terminal Resilience Routes): Made `human_handoff` and the currently unimplemented `deterministic` route terminal at the governed `TriageClient.run_task` boundary. Each now records the route decision plus a `worker_result` with `worker_result_status=not_attempted`, returns `handoff_required`, and reaches the existing `tc run` exit-3 contract without invoking a local or cloud backend. This slice does not add approval-and-resume behavior, change the broader `human_review_required` semantics, or alter cloud gating, signatures, reducer behavior, or other execution surfaces.
 - Implemented CR-124 (Eval Handoff Hygiene, Bug, and Drift Slice): Fixed `write_actual_outcomes()` so generator and other single-pass iterable inputs still write all actual outcome files after duplicate checks, updated stale eval fixture/taxonomy sequencing docs so they no longer imply internal TriageCore scoring, and added focused regression/drift coverage. No evaluator execution, scoring, score interpretation, model/backend calls, routing/admission integration, ledger writes, schema changes, bundle builder, or result import/display were added.

docs/change/requests/CR-127-evaluation-handoff-bundle-builder.md

@@ -0,0 +1,41 @@
+# CR-127 — Evaluation Handoff Bundle Builder
+
+## Status
+
+Implemented
+
+## Purpose
+
+Add a deterministic file-only builder that packages an explicit validated
+fixture and explicit actual-outcome directory for an external evaluator while
+preserving the CR-123 boundary that TriageCore does not score or interpret
+evaluation results.
+
+## Scope
+
+- Add `triage_core/evaluation_handoff_bundle.py`.
+- Add `tc eval build-handoff --fixture <jsonl> --actuals-dir <dir> --out-dir <new-dir>`.
+- Emit the fixed fixture, actuals, and manifest layout documented by
+  `evaluation_handoff_manifest.v0`.
+- Copy source evidence byte-for-byte and record deterministic SHA-256 hashes.
+- Validate all inputs before staging, enforce persistent privacy invariants,
+  and publish with a sibling atomic rename.
+- Add focused pure-module and CLI coverage.
+
+## Non-Goals
+
+- No scoring, pass/fail judgment, score interpretation, or approval claim.
+- No evaluator, model, backend, endpoint, or network invocation.
+- No automatic file discovery, defaults, overwrite, or `--force`.
+- No routing, admission, ledger, approval, or worker integration.
+- No new fixture, decision, or boundary-family vocabulary.
+- No validation of an already-created bundle; integrity validation is a
+  separate future CR.
+
+## Validation
+
+- Focused:
+  `python -m pytest -q tests/test_evaluation_handoff_bundle.py tests/test_evaluation_handoff_bundle_cli.py tests/test_eval_handoff_contract.py tests/test_eval_outcome_contract.py tests/test_eval_fixture_validator.py tests/test_eval_fixture_cli.py`
+  -> 60 passed
+- Full suite: `python -m pytest -q` -> 971 passed, 2 skipped
+- `git diff --check`

docs/current_backlog.md

@@ -2,10 +2,14 @@
 
 ## Status
 
-This document summarizes the active TriageCore backlog after CR-126.
+This document summarizes the active TriageCore backlog after CR-127.
 
 ## Active GitHub Backlog
 
+- CR-127: Evaluation Handoff Bundle Builder
+  - Status: complete via CR-127
+  - Purpose: Package an explicit validated fixture and explicit actual-outcome directory into a deterministic, privacy-checked, unscored handoff bundle with byte-preserving copies and a SHA-256 manifest. External scoring remains authoritative; existing-bundle integrity validation is the next separately scoped slice.
+
 - CR-126: Preflight Privacy Before Ledger Persistence
   - Status: complete via CR-126
   - Purpose: Preflight the complete `tc run` packet before any ledger write, replace persisted prompt/data text with fixed metadata and lengths, and extend the persistent-artifact audit to reject forbidden keys plus high-confidence PII, credential, and precise-location value patterns. Historical records are not rewritten, and arbitrary free-text safety classification remains out of scope.
@@ -136,7 +140,7 @@ This document summarizes the active TriageCore backlog after CR-126.
 
 - Empirical AI safety evaluation track
   - Source: CR-076 and CR-077 research framing/eval taxonomy docs
-  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; bundle/manifest building, bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
+  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
   - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure.
 
 - Operator UX implementation path
@@ -228,7 +232,7 @@ Keep three work lanes distinct:
 
 For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness.
 
-For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, and CR-123 defines the external-evaluator handoff contract. Keep the next slices sequential: bundle/manifest builder, bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
+For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, and CR-127 builds the deterministic unscored bundle. Keep the next slices sequential: bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
 
 For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path.
 
@@ -244,4 +248,5 @@ For operator UX, future slices should focus on reviewability, export polish, and
 - **[done] Eval handoff hygiene (CR-124)**: Fixed generator-backed actual-outcome writing and removed stale language that implied internal TriageCore scoring.
 - **[done] Honor terminal resilience routes (CR-125)**: `human_handoff` and currently unimplemented `deterministic` routes now return a governed handoff before backend execution, recording `worker_result_status=not_attempted`; `tc run` reports the valid handoff with exit code 3. Approval-and-resume behavior, broader `human_review_required` semantics, and other execution seams remain future work.
 - **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope.
-- **Next slice requires a new approved CR**: The eval lane's next bounded candidate is a deterministic bundle/manifest builder for already-validated fixtures and already-exported actuals. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.
+- **[done] Evaluation handoff bundle builder (CR-127)**: Builds the deterministic fixed-layout handoff and SHA-256 manifest from explicit inputs without scoring.
+- **Next slice requires a new approved CR**: The eval lane's next bounded candidate is a deterministic integrity validator for an existing handoff bundle. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.

docs/evals/evaluation_handoff_contract.md

@@ -36,8 +36,8 @@ actuals/triagecore_smoke/
 ```
 
 TriageCore must not infer default evaluator inputs from the ledger, route
-history, admission state, or local runtime state. The operator or a future
-bundle builder must pass file paths explicitly.
+history, admission state, or local runtime state. The operator passes all
+paths explicitly to the CR-127 bundle builder.
 
 ## Required TriageCore Outputs
 
@@ -76,18 +76,21 @@ scored report schema in this CR. TriageCore must treat evaluator findings as ext
 
 ## Deterministic Handoff Layout
 
-If a future CR materializes a handoff bundle, it should use these relative
-paths inside the bundle directory:
+CR-127 materializes a handoff bundle using these relative paths:
 
 ```text
 fixtures/safety_boundaries_v0.jsonl
 actuals/<case_id>.json
 manifest/evaluation_handoff_manifest.json
 ```
 
-CR-123 does not create that bundle, manifest, or builder. These names reserve a
-stable path vocabulary so CR-124+ can package the same already-defined evidence
-without changing the scoring boundary.
+The builder is documented in
+`docs/evals/evaluation_handoff_manifest.md`. It copies the already-defined
+evidence byte-for-byte and records deterministic hashes without changing the
+scoring boundary.
+
+Historically, CR-123 does not create that bundle, manifest, or builder; CR-127
+is the separately approved implementation slice that now materializes them.
 
 ## Exit-Code Expectations
 
@@ -99,6 +102,7 @@ TriageCore commands used before handoff must follow these expectations:
 | `tc eval export-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
 | `tc eval export-privacy-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
 | `tc eval export-forbidden-tool-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
+| `tc eval build-handoff --fixture <path> --actuals-dir <dir> --out-dir <new-dir>` | Exit `0` after atomically publishing an unscored deterministic bundle. | Exit `1` with a stable closed reason for input, privacy, path, or write failures. | Exit `2` for argparse usage errors. |
 
 The external evaluator suite owns its own exit-code contract. TriageCore may
 document how to invoke that suite, but this CR does not add an evaluator runner
@@ -118,6 +122,6 @@ CR-123 explicitly excludes:
 - changes to `eval_case_v0` or actual outcome JSON fields
 - new fixture families or adversarial/tampering expansion
 
-The next safe slice after this contract is a deterministic bundle or manifest
-builder that packages already-validated fixtures and already-exported actuals
-without executing or scoring the evaluator.
+CR-127 implements the deterministic bundle/manifest builder. The next safe
+slice is a deterministic integrity validator for an existing bundle, without
+executing or scoring the evaluator.

docs/evals/evaluation_handoff_manifest.md

@@ -0,0 +1,72 @@
+# Evaluation Handoff Manifest
+
+Contract identifier: `evaluation_handoff_manifest.v0`
+
+## Purpose
+
+`tc eval build-handoff` packages an explicitly named `eval_case_v0` fixture
+and an explicitly named directory of `actual_outcome_export.v0` files for an
+external evaluator. The manifest records deterministic relative paths, counts,
+and SHA-256 digests. It does not score cases, declare pass/fail, approve work,
+or invoke an evaluator.
+
+## Command
+
+```text
+tc eval build-handoff \
+  --fixture <fixture.jsonl> \
+  --actuals-dir <actuals-directory> \
+  --out-dir <new-bundle-directory>
+```
+
+All three paths are required. There are no default discovery rules and no
+overwrite or `--force` mode. The output directory must not already exist.
+
+## Fixed Layout
+
+```text
+<bundle>/
+  fixtures/safety_boundaries_v0.jsonl
+  actuals/<case_id>.json
+  manifest/evaluation_handoff_manifest.json
+```
+
+The fixture and actual files are copied byte-for-byte. Actual files are direct
+children of the input directory; nested paths, symlinks, and non-JSON entries
+are rejected. Missing actuals for fixture cases are allowed because a handoff
+may represent partial observation. Every included actual must name a unique,
+path-safe fixture `case_id`, and its filename must be `<case_id>.json`.
+
+## Manifest
+
+The sorted-key, newline-terminated JSON contains:
+
+- `schema_version: evaluation_handoff_manifest.v0`
+- `bundle_type: evaluation_handoff`
+- `handoff_contract: evaluation_handoff_contract.v0`
+- `scoring_owner: external_evaluator`
+- `triagecore_scored: false`
+- fixture contract, fixed relative path, case count, and SHA-256
+- actual contract, count, and entries sorted by `case_id`, each with its fixed
+  relative path and SHA-256
+
+It contains no timestamps, absolute source paths, scores, verdicts, approval
+claims, or evaluator commands.
+
+## Validation and Failure
+
+Before writing, the builder validates the fixture with the existing fixture
+loader, checks the six required actual-outcome fields and broad JSON types, and
+applies the persistent privacy invariant to every parsed fixture and actual.
+It does not introduce new decision or boundary-family enums.
+
+Failures exit `1` and print only a stable `reason=<reason>` code. Argparse usage
+errors exit `2`. Construction stages into a temporary sibling and atomically
+renames it only after all output is written; failures leave no bundle.
+
+## Integrity Boundary
+
+SHA-256 fields describe the copied bytes and support external inspection. This
+builder does not validate an existing bundle after creation. A deterministic
+bundle integrity validator is a separate future scope. Scoring and score
+interpretation remain exclusively owned by the external evaluator.