CR-129: Define external evaluator adapter contract

docs/change/change_log.md

@@ -6,6 +6,7 @@ This file provides a chronological, human-readable record of applied codebase an
 
 ## [Unreleased]
 
+- Implemented CR-129 (External Evaluator Adapter Contract): Added `external_evaluator_adapter_contract.v0` as a documentation-only boundary reserving a future closed-profile `tc eval invoke-external` shape while explicitly prohibiting arbitrary executables and free-form argv. The contract requires authoritative evaluator identity/version, fixed bundle mapping, cwd, output ownership, network posture, exit meanings, encoding/caps, CR-128 pre-launch validation, minimal credential-free environment, bounded timeout, portable process-tree termination, and bounded untrusted output before any implementation. No CLI, subprocess, evaluator output, scoring/interpretation, result handling, writes, ledger, network/model/backend call, routing, admission, approval, identity, or worker integration was added.
 - Implemented CR-128 (Evaluation Handoff Integrity Validator): Added read-only `tc eval validate-handoff --bundle <bundle-root>` validation for the closed `evaluation_handoff_manifest.v0` schema, exact non-symlink/non-reparse inventory, fixed safe paths, counts, byte hashes, fixture and actual contracts, fixture membership, and persistent privacy safety. Validation emits only bounded success or stable closed reasons, makes no filesystem changes, and performs no scoring, semantic expected-vs-actual comparison, evaluator/model/backend/network call, ledger write, repair, or approval action. Hash agreement detects drift relative to the manifest but does not establish authenticity, provenance, approval, safety, or correctness.
 - Implemented CR-127 (Evaluation Handoff Bundle Builder): Added `tc eval build-handoff --fixture <jsonl> --actuals-dir <dir> --out-dir <new-dir>` and a pure deterministic builder that validates explicit fixture/actual inputs, applies persistent privacy checks, copies evidence byte-for-byte into the fixed handoff layout, writes a sorted `evaluation_handoff_manifest.v0` with SHA-256 hashes, and atomically publishes only after successful staging. Scoring, pass/fail judgment, evaluator execution, approval claims, automatic discovery, overwrite behavior, and existing-bundle integrity validation remain out of scope and external scoring remains authoritative.
 - Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope.

docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md

@@ -2,7 +2,7 @@
 
 ## Status
 
-Implemented locally; pending review.
+Implemented.
 
 ## Purpose
 

docs/change/requests/CR-129-external-evaluator-adapter-contract.md

@@ -0,0 +1,37 @@
+# CR-129 — External Evaluator Adapter Contract
+
+## Status
+
+Implemented locally; pending review.
+
+## Purpose
+
+Define the contract and stop conditions required before a future TriageCore
+wrapper may launch a version-pinned external evaluator. This slice is
+documentation and drift enforcement only.
+
+## Scope
+
+- Add `external_evaluator_adapter_contract.v0`.
+- Reserve a closed-profile future CLI shape without implementing it.
+- Define required profile identity, executable/version, argv mapping, cwd,
+  result ownership, exit, output, network, environment, timeout, and
+  process-tree termination requirements.
+- Require CR-128 bundle validation before any future launch.
+- Add focused drift tests that keep CR-129 contract-only.
+
+## Non-Goals
+
+- No CLI, subprocess, executable selection, argv forwarding, or process launch.
+- No result parsing/import/rendering/persistence or evaluator output creation.
+- No scoring, evaluator interpretation, ledger write, network/model/backend
+  call, routing, admission, approval, identity, or worker integration.
+- No external evaluator profile; that authoritative versioned artifact is a
+  prerequisite for a separately approved code-bearing CR.
+
+## Validation
+
+- Focused:
+  `python -m pytest -q tests/test_external_evaluator_adapter_contract.py tests/test_eval_handoff_contract.py`
+  -> 10 passed.
+- `git diff --check` -> clean.

docs/current_backlog.md

@@ -2,12 +2,16 @@
 
 ## Status
 
-This document summarizes the active TriageCore backlog after CR-128.
+This document summarizes the active TriageCore backlog after CR-129.
 
 ## Active GitHub Backlog
 
-- CR-128: Evaluation Handoff Integrity Validator
+- CR-129: External Evaluator Adapter Contract
   - Status: implemented locally; pending review
+  - Purpose: Define the closed-profile, pre-launch validation, process safety, exit, output, network, and trust boundaries required before a future external evaluator wrapper can be implemented. CR-129 adds no CLI or subprocess.
+
+- CR-128: Evaluation Handoff Integrity Validator
+  - Status: complete via CR-128 (PR #101)
   - Purpose: Read-only validation of an existing CR-127 bundle's closed manifest, exact inventory, safe paths, byte hashes, fixture/actual contracts, membership, and privacy invariants. Hash agreement detects drift relative to the manifest; it is not authenticity, provenance, approval, safety, or correctness.
 
 - CR-127: Evaluation Handoff Bundle Builder
@@ -144,7 +148,7 @@ This document summarizes the active TriageCore backlog after CR-128.
 
 - Empirical AI safety evaluation track
   - Source: CR-076 and CR-077 research framing/eval taxonomy docs
-  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
+  - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; external evaluator adapter contract complete via CR-129; adapter implementation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices
   - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure.
 
 - Operator UX implementation path
@@ -236,7 +240,7 @@ Keep three work lanes distinct:
 
 For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness.
 
-For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, and CR-128 validates bundle integrity read-only. Any external evaluator adapter requires a separate approved scope. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable.
+For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, CR-128 validates bundle integrity read-only, and CR-129 defines the contract-only adapter boundary. A code-bearing adapter requires a separately approved CR plus an authoritative versioned external evaluator profile. Scoring and score interpretation remain external to TriageCore; adversarial expansion also requires separate approval.
 
 For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path.
 
@@ -254,4 +258,5 @@ For operator UX, future slices should focus on reviewability, export polish, and
 - **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope.
 - **[done] Evaluation handoff bundle builder (CR-127)**: Builds the deterministic fixed-layout handoff and SHA-256 manifest from explicit inputs without scoring.
 - **[done] Evaluation handoff integrity validator (CR-128)**: Validates the closed manifest, exact inventory, hashes, contracts, membership, and privacy without mutating or scoring the bundle.
-- **Next slice requires a new approved CR**: Any external evaluator adapter or adversarial expansion must be separately scoped. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.
+- **[done] External evaluator adapter contract (CR-129)**: Defines the closed-profile and process-safety prerequisites without adding a CLI or subprocess.
+- **Next slice requires a new approved CR**: A code-bearing adapter requires an authoritative versioned external evaluator profile and separate approval; adversarial expansion also remains separate. Do not add arbitrary executable/argv forwarding, scoring or score interpretation inside TriageCore, approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.

docs/evals/evaluation_handoff_contract.md

@@ -127,3 +127,10 @@ CR-127 implements the deterministic bundle/manifest builder, and CR-128 adds
 read-only integrity validation. Hash agreement detects drift relative to the
 manifest; it does not prove authenticity, provenance, approval, safety, or
 correctness. Neither slice executes or scores the evaluator.
+
+CR-129 defines the contract-only boundary for a possible future external
+evaluator wrapper in
+`docs/evals/external_evaluator_adapter_contract.md`. The reserved
+`tc eval invoke-external` shape does not exist in CR-129. A code-bearing adapter
+requires a separately approved CR and an authoritative versioned closed
+evaluator profile.

docs/evals/evaluation_handoff_manifest.md

@@ -89,3 +89,8 @@ Hash agreement detects drift relative to the manifest. It does not authenticate
 the manifest, establish provenance, convey approval, certify safety, or prove
 semantic correctness. Scoring and score interpretation remain exclusively
 owned by the external evaluator.
+
+CR-129 documents the still-unimplemented external evaluator adapter boundary in
+`external_evaluator_adapter_contract.md`. Bundle integrity validation is a
+required pre-launch gate for any future adapter, but successful validation
+neither selects nor authorizes an evaluator.