diff --git a/docs/change/change_log.md b/docs/change/change_log.md index 79519d0..71a1c01 100644 --- a/docs/change/change_log.md +++ b/docs/change/change_log.md @@ -6,6 +6,7 @@ This file provides a chronological, human-readable record of applied codebase an ## [Unreleased] +- Implemented CR-129 (External Evaluator Adapter Contract): Added `external_evaluator_adapter_contract.v0` as a documentation-only boundary reserving a future closed-profile `tc eval invoke-external` shape while explicitly prohibiting arbitrary executables and free-form argv. The contract requires authoritative evaluator identity/version, fixed bundle mapping, cwd, output ownership, network posture, exit meanings, encoding/caps, CR-128 pre-launch validation, minimal credential-free environment, bounded timeout, portable process-tree termination, and bounded untrusted output before any implementation. No CLI, subprocess, evaluator output, scoring/interpretation, result handling, writes, ledger, network/model/backend call, routing, admission, approval, identity, or worker integration was added. - Implemented CR-128 (Evaluation Handoff Integrity Validator): Added read-only `tc eval validate-handoff --bundle ` validation for the closed `evaluation_handoff_manifest.v0` schema, exact non-symlink/non-reparse inventory, fixed safe paths, counts, byte hashes, fixture and actual contracts, fixture membership, and persistent privacy safety. Validation emits only bounded success or stable closed reasons, makes no filesystem changes, and performs no scoring, semantic expected-vs-actual comparison, evaluator/model/backend/network call, ledger write, repair, or approval action. Hash agreement detects drift relative to the manifest but does not establish authenticity, provenance, approval, safety, or correctness. - Implemented CR-127 (Evaluation Handoff Bundle Builder): Added `tc eval build-handoff --fixture --actuals-dir --out-dir ` and a pure deterministic builder that validates explicit fixture/actual inputs, applies persistent privacy checks, copies evidence byte-for-byte into the fixed handoff layout, writes a sorted `evaluation_handoff_manifest.v0` with SHA-256 hashes, and atomically publishes only after successful staging. Scoring, pass/fail judgment, evaluator execution, approval claims, automatic discovery, overwrite behavior, and existing-bundle integrity validation remain out of scope and external scoring remains authoritative. - Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope. diff --git a/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md b/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md index ea41698..9274b05 100644 --- a/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md +++ b/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md @@ -2,7 +2,7 @@ ## Status -Implemented locally; pending review. +Implemented. ## Purpose diff --git a/docs/change/requests/CR-129-external-evaluator-adapter-contract.md b/docs/change/requests/CR-129-external-evaluator-adapter-contract.md new file mode 100644 index 0000000..88ed42c --- /dev/null +++ b/docs/change/requests/CR-129-external-evaluator-adapter-contract.md @@ -0,0 +1,37 @@ +# CR-129 — External Evaluator Adapter Contract + +## Status + +Implemented locally; pending review. + +## Purpose + +Define the contract and stop conditions required before a future TriageCore +wrapper may launch a version-pinned external evaluator. This slice is +documentation and drift enforcement only. + +## Scope + +- Add `external_evaluator_adapter_contract.v0`. +- Reserve a closed-profile future CLI shape without implementing it. +- Define required profile identity, executable/version, argv mapping, cwd, + result ownership, exit, output, network, environment, timeout, and + process-tree termination requirements. +- Require CR-128 bundle validation before any future launch. +- Add focused drift tests that keep CR-129 contract-only. + +## Non-Goals + +- No CLI, subprocess, executable selection, argv forwarding, or process launch. +- No result parsing/import/rendering/persistence or evaluator output creation. +- No scoring, evaluator interpretation, ledger write, network/model/backend + call, routing, admission, approval, identity, or worker integration. +- No external evaluator profile; that authoritative versioned artifact is a + prerequisite for a separately approved code-bearing CR. + +## Validation + +- Focused: + `python -m pytest -q tests/test_external_evaluator_adapter_contract.py tests/test_eval_handoff_contract.py` + -> 10 passed. +- `git diff --check` -> clean. diff --git a/docs/current_backlog.md b/docs/current_backlog.md index 676d77c..24a2e76 100644 --- a/docs/current_backlog.md +++ b/docs/current_backlog.md @@ -2,12 +2,16 @@ ## Status -This document summarizes the active TriageCore backlog after CR-128. +This document summarizes the active TriageCore backlog after CR-129. ## Active GitHub Backlog -- CR-128: Evaluation Handoff Integrity Validator +- CR-129: External Evaluator Adapter Contract - Status: implemented locally; pending review + - Purpose: Define the closed-profile, pre-launch validation, process safety, exit, output, network, and trust boundaries required before a future external evaluator wrapper can be implemented. CR-129 adds no CLI or subprocess. + +- CR-128: Evaluation Handoff Integrity Validator + - Status: complete via CR-128 (PR #101) - Purpose: Read-only validation of an existing CR-127 bundle's closed manifest, exact inventory, safe paths, byte hashes, fixture/actual contracts, membership, and privacy invariants. Hash agreement detects drift relative to the manifest; it is not authenticity, provenance, approval, safety, or correctness. - CR-127: Evaluation Handoff Bundle Builder @@ -144,7 +148,7 @@ This document summarizes the active TriageCore backlog after CR-128. - Empirical AI safety evaluation track - Source: CR-076 and CR-077 research framing/eval taxonomy docs - - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices + - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; external evaluator adapter contract complete via CR-129; adapter implementation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure. - Operator UX implementation path @@ -236,7 +240,7 @@ Keep three work lanes distinct: For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness. -For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, and CR-128 validates bundle integrity read-only. Any external evaluator adapter requires a separate approved scope. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable. +For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, CR-128 validates bundle integrity read-only, and CR-129 defines the contract-only adapter boundary. A code-bearing adapter requires a separately approved CR plus an authoritative versioned external evaluator profile. Scoring and score interpretation remain external to TriageCore; adversarial expansion also requires separate approval. For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path. @@ -254,4 +258,5 @@ For operator UX, future slices should focus on reviewability, export polish, and - **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope. - **[done] Evaluation handoff bundle builder (CR-127)**: Builds the deterministic fixed-layout handoff and SHA-256 manifest from explicit inputs without scoring. - **[done] Evaluation handoff integrity validator (CR-128)**: Validates the closed manifest, exact inventory, hashes, contracts, membership, and privacy without mutating or scoring the bundle. -- **Next slice requires a new approved CR**: Any external evaluator adapter or adversarial expansion must be separately scoped. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR. +- **[done] External evaluator adapter contract (CR-129)**: Defines the closed-profile and process-safety prerequisites without adding a CLI or subprocess. +- **Next slice requires a new approved CR**: A code-bearing adapter requires an authoritative versioned external evaluator profile and separate approval; adversarial expansion also remains separate. Do not add arbitrary executable/argv forwarding, scoring or score interpretation inside TriageCore, approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR. diff --git a/docs/evals/evaluation_handoff_contract.md b/docs/evals/evaluation_handoff_contract.md index 229ded6..1e84a02 100644 --- a/docs/evals/evaluation_handoff_contract.md +++ b/docs/evals/evaluation_handoff_contract.md @@ -127,3 +127,10 @@ CR-127 implements the deterministic bundle/manifest builder, and CR-128 adds read-only integrity validation. Hash agreement detects drift relative to the manifest; it does not prove authenticity, provenance, approval, safety, or correctness. Neither slice executes or scores the evaluator. + +CR-129 defines the contract-only boundary for a possible future external +evaluator wrapper in +`docs/evals/external_evaluator_adapter_contract.md`. The reserved +`tc eval invoke-external` shape does not exist in CR-129. A code-bearing adapter +requires a separately approved CR and an authoritative versioned closed +evaluator profile. diff --git a/docs/evals/evaluation_handoff_manifest.md b/docs/evals/evaluation_handoff_manifest.md index 746976a..41201b8 100644 --- a/docs/evals/evaluation_handoff_manifest.md +++ b/docs/evals/evaluation_handoff_manifest.md @@ -89,3 +89,8 @@ Hash agreement detects drift relative to the manifest. It does not authenticate the manifest, establish provenance, convey approval, certify safety, or prove semantic correctness. Scoring and score interpretation remain exclusively owned by the external evaluator. + +CR-129 documents the still-unimplemented external evaluator adapter boundary in +`external_evaluator_adapter_contract.md`. Bundle integrity validation is a +required pre-launch gate for any future adapter, but successful validation +neither selects nor authorizes an evaluator. diff --git a/docs/evals/external_evaluator_adapter_contract.md b/docs/evals/external_evaluator_adapter_contract.md new file mode 100644 index 0000000..0d5ab28 --- /dev/null +++ b/docs/evals/external_evaluator_adapter_contract.md @@ -0,0 +1,166 @@ +# External Evaluator Adapter Contract + +Contract identifier: `external_evaluator_adapter_contract.v0` + +## Status and Purpose + +CR-129 is contract-only. It defines the minimum information and safety +properties required before TriageCore may launch a separately installed +external evaluator. It adds no subprocess implementation and no CLI command. +Scoring, pass/fail judgment, findings, and score interpretation remain owned by +the external evaluator. + +Implementation stops here because the authoritative evaluator executable or +package and version, bundle-to-argument mapping, working directory, result +location and ownership, evaluator exit meanings, output sensitivity, +network posture, and portable process-tree timeout behavior are not yet pinned +by a versioned profile. Guessing any of them would turn a file-contract +boundary into an implicit execution and trust boundary. + +## Reserved Future CLI + +A later, separately approved code-bearing CR may use this exact shape: + +```text +tc eval invoke-external \ + --bundle \ + --evaluator-profile \ + --timeout-seconds +``` + +This command does not exist in CR-129. The future command must not accept an +arbitrary `--executable`, free-form evaluator arguments, generic argv +forwarding, shell snippets, or operator-provided command templates. A closed +profile selects the entire executable and argument contract. + +The suggested default timeout is 300 seconds. A future implementation must +accept only whole seconds in the inclusive range 1 through 3600. + +## Required Versioned Evaluator Profile + +No adapter may be implemented until an authoritative, reviewed profile pins: + +- a closed profile identifier and profile schema version +- executable identity and an exact or otherwise closed executable path rule +- evaluator package or module identity and version +- a fixed argv-list template mapping the validated bundle's + `fixtures/safety_boundaries_v0.jsonl` and `actuals/` paths +- an explicit working directory rather than ambient caller cwd +- the evaluator result/output contract, output path ownership, overwrite + behavior, cleanup responsibility, and sensitivity classification +- network posture, including whether network is required, prohibited, or + explicitly unverifiable by the wrapper +- evaluator exit meanings, including which evaluator exits constitute wrapper + success or failure +- stdout/stderr encoding, per-stream byte caps, and safe diagnostic treatment + +The profile must be versioned independently of the bundle manifest. It must not +derive executable identity, argv, cwd, outputs, or network posture from bundle +contents. + +## Required Pre-Launch Gate + +A future adapter must run CR-128 +`tc eval validate-handoff --bundle `-equivalent validation before +launch. Any validation failure stops before process creation with +`bundle_invalid`. Validation proves internal bundle consistency only; it does +not authenticate the evaluator or authorize execution. + +## Future Process Safety Requirements + +A future implementation must: + +- construct an argv list and launch with `shell=False` +- set stdin to `DEVNULL` +- use an allowlisted minimal environment +- exclude credentials, proxy settings, model configuration, endpoint + configuration, model/runtime configuration, runtime secrets, and unrelated + TriageCore configuration by default +- enforce the bounded timeout selected within the 1–3600 second range +- terminate the evaluator's full process tree on timeout or interruption using + tested Windows and POSIX behavior, then wait for confirmed termination +- treat stdout and stderr as untrusted bytes, enforce independent output caps, + decode only with the profile-pinned encoding, and never echo unbounded or + sensitive output + +Environment stripping alone does not prove offline execution. Child processes +may still reach the network through operating-system facilities, inherited +machine configuration, or evaluator behavior. A profile must state its network +posture, and `network_posture_unverified` remains a valid stop reason when the +wrapper cannot establish that posture. + +## Reserved Wrapper Exit Semantics + +The future wrapper reserves: + +| Wrapper exit | Meaning | +| --- | --- | +| `0` | The pinned evaluator profile launched and completed according to the profile's success contract. This is not a TriageCore score or approval. | +| `1` | Wrapper validation, launch, timeout, evaluator, output, or interruption failure. | +| `2` | Argparse usage error. | + +Raw evaluator exit codes must never be propagated as wrapper exit codes or +interpreted without the pinned profile. Stable future failure reasons may +include: + +- `bundle_invalid` +- `evaluator_profile_missing` +- `evaluator_profile_invalid` +- `evaluator_not_found` +- `evaluator_version_mismatch` +- `evaluator_launch_failed` +- `evaluator_timeout` +- `evaluator_nonzero_exit` +- `evaluator_output_invalid_encoding` +- `evaluator_output_limit_exceeded` +- `evaluator_interrupted` +- `network_posture_unverified` + +These names reserve a bounded diagnostic vocabulary; CR-129 does not emit them. + +## Output and Ownership Boundary + +The external evaluator, under its future versioned profile, owns any scored +result artifact. TriageCore does not create evaluator output, parse it, import +it, render it, persist it, write a report, or record it in the ledger. The +profile must define result ownership and sensitivity before launch behavior can +be approved. Captured stdout or stderr is diagnostic process output, not a +scored result contract. + +## Stop Conditions + +A future adapter must stop before or terminate launch when: + +- CR-128 bundle validation fails +- the profile is missing, unknown, malformed, or version-mismatched +- executable identity, package/module version, argv mapping, cwd, result + ownership, exit meanings, encoding, caps, or network posture is unresolved +- the executable is missing or does not match the profile +- safe cross-platform process-tree termination cannot be established +- timeout, interruption, invalid encoding, output overflow, or an unrecognized + evaluator exit occurs + +No fallback may switch to an arbitrary executable, shell command, free-form +argv, model/backend call, or network service. + +## Trust and Uncertainty Limits + +A valid bundle plus a pinned profile would establish deterministic wrapper +inputs, not evaluator authenticity, evaluator correctness, scoring validity, +offline execution, sandboxing, approval, safety certification, or provenance. +Process isolation and network containment require controls outside this file +contract. Evaluator output remains untrusted and externally interpreted. + +## CR-129 Non-Goals + +- no `tc eval invoke-external` implementation +- no subprocess or process-tree execution code +- no CLI parser or runtime adapter module +- no result parsing, importing, rendering, persistence, report creation, or + ledger write +- no TriageCore filesystem writes of any kind +- no evaluator output creation by TriageCore +- no scoring, pass/fail interpretation, approval, or safety claim +- no network, model, backend, routing, admission, approval, identity, or worker + integration +- no adversarial fixture expansion or evaluator profile selection diff --git a/tests/test_external_evaluator_adapter_contract.py b/tests/test_external_evaluator_adapter_contract.py new file mode 100644 index 0000000..302ac2c --- /dev/null +++ b/tests/test_external_evaluator_adapter_contract.py @@ -0,0 +1,115 @@ +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[1] +CONTRACT = REPO_ROOT / "docs" / "evals" / "external_evaluator_adapter_contract.md" +CR_RECORD = ( + REPO_ROOT + / "docs" + / "change" + / "requests" + / "CR-129-external-evaluator-adapter-contract.md" +) +TC_CLI = REPO_ROOT / "triage_core" / "tc_cli.py" +ADAPTER_MODULE = REPO_ROOT / "triage_core" / "external_evaluator_adapter.py" + + +def _normalized(path): + return " ".join(path.read_text(encoding="utf-8").lower().split()) + + +def test_adapter_contract_pins_future_closed_profile_boundary(): + doc = _normalized(CONTRACT) + + required = ( + "external_evaluator_adapter_contract.v0", + "this command does not exist in cr-129", + "--bundle ", + "--evaluator-profile ", + "--timeout-seconds ", + "arbitrary `--executable`", + "free-form evaluator arguments", + "fixed argv-list template", + "explicit working directory", + "result/output contract", + "network posture", + "stdout/stderr encoding", + "1 through 3600", + "suggested default timeout is 300 seconds", + ) + for phrase in required: + assert phrase in doc + + +def test_adapter_contract_pins_process_safety_and_prelaunch_validation(): + doc = _normalized(CONTRACT) + + required = ( + "validate-handoff --bundle ", + "argv list", + "`shell=false`", + "stdin to `devnull`", + "allowlisted minimal environment", + "exclude credentials", + "proxy settings", + "process tree", + "windows and posix", + "untrusted bytes", + "environment stripping alone does not prove offline execution", + "network_posture_unverified", + ) + for phrase in required: + assert phrase in doc + + +def test_adapter_contract_pins_wrapper_exits_reasons_and_external_ownership(): + doc = _normalized(CONTRACT) + + assert "raw evaluator exit codes must never be propagated" in doc + assert "| `0` |" in CONTRACT.read_text(encoding="utf-8") + assert "| `1` |" in CONTRACT.read_text(encoding="utf-8") + assert "| `2` |" in CONTRACT.read_text(encoding="utf-8") + for reason in ( + "bundle_invalid", + "evaluator_profile_missing", + "evaluator_profile_invalid", + "evaluator_not_found", + "evaluator_version_mismatch", + "evaluator_launch_failed", + "evaluator_timeout", + "evaluator_nonzero_exit", + "evaluator_output_invalid_encoding", + "evaluator_output_limit_exceeded", + "evaluator_interrupted", + "network_posture_unverified", + ): + assert reason in doc + assert "external evaluator" in doc + assert "owns any scored result artifact" in doc + assert "does not create evaluator output" in doc + assert "does not create evaluator output, parse it, import it, render it, persist it" in doc + + +def test_cr129_remains_contract_only_without_generic_execution_surface(): + contract = _normalized(CONTRACT) + cli_source = TC_CLI.read_text(encoding="utf-8") + + assert CR_RECORD.exists() + assert "no subprocess or process-tree execution code" in contract + assert "no cli parser or runtime adapter module" in contract + assert "invoke-external" not in cli_source + assert not ADAPTER_MODULE.exists() + + +def test_handoff_docs_link_to_adapter_contract_without_claiming_implementation(): + handoff = _normalized( + REPO_ROOT / "docs" / "evals" / "evaluation_handoff_contract.md" + ) + manifest = _normalized( + REPO_ROOT / "docs" / "evals" / "evaluation_handoff_manifest.md" + ) + + assert "external_evaluator_adapter_contract.md" in handoff + assert "does not exist in cr-129" in handoff + assert "external_evaluator_adapter_contract.md" in manifest + assert "required pre-launch gate" in manifest