CR-128: Validate evaluation handoff bundle integrity

docs/change/change_log.md

@@ -6,6 +6,7 @@ This file provides a chronological, human-readable record of applied codebase an
 
 ## [Unreleased]
 
+- Implemented CR-128 (Evaluation Handoff Integrity Validator): Added read-only `tc eval validate-handoff --bundle <bundle-root>` validation for the closed `evaluation_handoff_manifest.v0` schema, exact non-symlink/non-reparse inventory, fixed safe paths, counts, byte hashes, fixture and actual contracts, fixture membership, and persistent privacy safety. Validation emits only bounded success or stable closed reasons, makes no filesystem changes, and performs no scoring, semantic expected-vs-actual comparison, evaluator/model/backend/network call, ledger write, repair, or approval action. Hash agreement detects drift relative to the manifest but does not establish authenticity, provenance, approval, safety, or correctness.
 - Implemented CR-127 (Evaluation Handoff Bundle Builder): Added `tc eval build-handoff --fixture <jsonl> --actuals-dir <dir> --out-dir <new-dir>` and a pure deterministic builder that validates explicit fixture/actual inputs, applies persistent privacy checks, copies evidence byte-for-byte into the fixed handoff layout, writes a sorted `evaluation_handoff_manifest.v0` with SHA-256 hashes, and atomically publishes only after successful staging. Scoring, pass/fail judgment, evaluator execution, approval claims, automatic discovery, overwrite behavior, and existing-bundle integrity validation remain out of scope and external scoring remains authoritative.
 - Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope.
 - Implemented CR-125 (Honor Terminal Resilience Routes): Made `human_handoff` and the currently unimplemented `deterministic` route terminal at the governed `TriageClient.run_task` boundary. Each now records the route decision plus a `worker_result` with `worker_result_status=not_attempted`, returns `handoff_required`, and reaches the existing `tc run` exit-3 contract without invoking a local or cloud backend. This slice does not add approval-and-resume behavior, change the broader `human_review_required` semantics, or alter cloud gating, signatures, reducer behavior, or other execution surfaces.

docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md

@@ -0,0 +1,45 @@
+# CR-128 — Evaluation Handoff Integrity Validator
+
+## Status
+
+Implemented locally; pending review.
+
+## Purpose
+
+Add a deterministic, read-only validator for an existing CR-127 evaluation
+handoff bundle. The validator checks the closed manifest contract, exact
+inventory, declared byte hashes, fixture and actual contracts, and persistent
+privacy invariants without scoring or repairing the bundle.
+
+## Scope
+
+- Add `triage_core/evaluation_handoff_validator.py`.
+- Add `tc eval validate-handoff --bundle <bundle-root>`.
+- Reject symlink/reparse traversal, unexpected inventory, unsafe manifest
+  paths, schema drift, count drift, hash drift, contract drift, unknown cases,
+  and privacy-invariant failures with stable closed reason codes.
+- Reuse the fixture validator and CR-127 broad actual-outcome validation.
+- Add focused pure-module, CLI, mutation, read-only, and contract-drift tests.
+
+## Non-Goals
+
+- No repair, normalization, report creation, or ledger write.
+- No expected-vs-actual comparison, scoring, pass/fail judgment, or approval.
+- No evaluator, model, backend, endpoint, or network invocation.
+- No routing, admission, identity, approval, or worker integration.
+- No authenticity, provenance, certification, safety, or correctness claim.
+
+## Trust Boundary
+
+A successful validation proves that current bundle bytes and inventory agree
+with the bundled manifest and declared file contracts. SHA-256 detects drift
+relative to that manifest; it does not authenticate who created the manifest
+or establish provenance, approval, safety, or correctness.
+
+## Validation
+
+- Focused eval surface:
+  `python -m pytest -q tests/test_evaluation_handoff_validator.py tests/test_evaluation_handoff_validator_cli.py tests/test_evaluation_handoff_bundle.py tests/test_evaluation_handoff_bundle_cli.py tests/test_eval_handoff_contract.py tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py tests/test_eval_outcome_contract.py`
+  -> 86 passed, 1 skipped (symlink creation unavailable).
+- Full suite: `python -m pytest -q` -> 997 passed, 3 skipped.
+- `git diff --check` -> clean.

docs/current_backlog.md

@@ -2,10 +2,14 @@
 
 ## Status
 
-This document summarizes the active TriageCore backlog after CR-127.
+This document summarizes the active TriageCore backlog after CR-128.
 
 ## Active GitHub Backlog
 
+- CR-128: Evaluation Handoff Integrity Validator
+  - Status: implemented locally; pending review
+  - Purpose: Read-only validation of an existing CR-127 bundle's closed manifest, exact inventory, safe paths, byte hashes, fixture/actual contracts, membership, and privacy invariants. Hash agreement detects drift relative to the manifest; it is not authenticity, provenance, approval, safety, or correctness.
+
 - CR-127: Evaluation Handoff Bundle Builder
   - Status: complete via CR-127
   - Purpose: Package an explicit validated fixture and explicit actual-outcome directory into a deterministic, privacy-checked, unscored handoff bundle with byte-preserving copies and a SHA-256 manifest. External scoring remains authoritative; existing-bundle integrity validation is the next separately scoped slice.
@@ -140,7 +144,7 @@ This document summarizes the active TriageCore backlog after CR-127.
 
 - Empirical AI safety evaluation track
   - Source: CR-076 and CR-077 research framing/eval taxonomy docs
-  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
+  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
   - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure.
 
 - Operator UX implementation path
@@ -232,7 +236,7 @@ Keep three work lanes distinct:
 
 For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness.
 
-For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, and CR-127 builds the deterministic unscored bundle. Keep the next slices sequential: bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
+For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, and CR-128 validates bundle integrity read-only. Any external evaluator adapter requires a separate approved scope. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
 
 For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path.
 
@@ -249,4 +253,5 @@ For operator UX, future slices should focus on reviewability, export polish, and
 - **[done] Honor terminal resilience routes (CR-125)**: `human_handoff` and currently unimplemented `deterministic` routes now return a governed handoff before backend execution, recording `worker_result_status=not_attempted`; `tc run` reports the valid handoff with exit code 3. Approval-and-resume behavior, broader `human_review_required` semantics, and other execution seams remain future work.
 - **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope.
 - **[done] Evaluation handoff bundle builder (CR-127)**: Builds the deterministic fixed-layout handoff and SHA-256 manifest from explicit inputs without scoring.
-- **Next slice requires a new approved CR**: The eval lane's next bounded candidate is a deterministic integrity validator for an existing handoff bundle. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.
+- **[done] Evaluation handoff integrity validator (CR-128)**: Validates the closed manifest, exact inventory, hashes, contracts, membership, and privacy without mutating or scoring the bundle.
+- **Next slice requires a new approved CR**: Any external evaluator adapter or adversarial expansion must be separately scoped. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.

docs/evals/evaluation_handoff_contract.md

@@ -103,6 +103,7 @@ TriageCore commands used before handoff must follow these expectations:
 | `tc eval export-privacy-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
 | `tc eval export-forbidden-tool-smoke --output-dir <dir>` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. |
 | `tc eval build-handoff --fixture <path> --actuals-dir <dir> --out-dir <new-dir>` | Exit `0` after atomically publishing an unscored deterministic bundle. | Exit `1` with a stable closed reason for input, privacy, path, or write failures. | Exit `2` for argparse usage errors. |
+| `tc eval validate-handoff --bundle <bundle-root>` | Exit `0` when the closed manifest, inventory, hashes, file contracts, and privacy checks agree. | Exit `1` with one stable closed reason and no payload echo. | Exit `2` for argparse usage errors. |
 
 The external evaluator suite owns its own exit-code contract. TriageCore may
 document how to invoke that suite, but this CR does not add an evaluator runner
@@ -122,6 +123,7 @@ CR-123 explicitly excludes:
 - changes to `eval_case_v0` or actual outcome JSON fields
 - new fixture families or adversarial/tampering expansion
 
-CR-127 implements the deterministic bundle/manifest builder. The next safe
-slice is a deterministic integrity validator for an existing bundle, without
-executing or scoring the evaluator.
+CR-127 implements the deterministic bundle/manifest builder, and CR-128 adds
+read-only integrity validation. Hash agreement detects drift relative to the
+manifest; it does not prove authenticity, provenance, approval, safety, or
+correctness. Neither slice executes or scores the evaluator.

docs/evals/evaluation_handoff_manifest.md

@@ -66,7 +66,26 @@ renames it only after all output is written; failures leave no bundle.
 
 ## Integrity Boundary
 
-SHA-256 fields describe the copied bytes and support external inspection. This
-builder does not validate an existing bundle after creation. A deterministic
-bundle integrity validator is a separate future scope. Scoring and score
-interpretation remain exclusively owned by the external evaluator.
+SHA-256 fields describe the copied bytes and support external inspection.
+CR-128 adds the read-only command:
+
+```text
+tc eval validate-handoff --bundle <bundle-root>
+```
+
+It requires the exact closed `evaluation_handoff_manifest.v0` schema, fixed
+paths, exact declared inventory, regular non-symlink/non-reparse files and
+directories, matching byte hashes and counts, valid fixture and actual
+contracts, fixture membership, and persistent privacy safety. Partial actual
+coverage remains valid, but a zero-actual bundle is invalid because the CR-127
+builder cannot produce one.
+
+Success exits `0` with a bounded fixture/actual count. Closed validation
+failures exit `1` and print only `reason=<reason>` to stderr; argparse usage
+errors exit `2`. Validation is read-only: it performs no normalization, repair,
+report write, ledger write, expected-vs-actual comparison, or evaluator call.
+
+Hash agreement detects drift relative to the manifest. It does not authenticate
+the manifest, establish provenance, convey approval, certify safety, or prove
+semantic correctness. Scoring and score interpretation remain exclusively
+owned by the external evaluator.