diff --git a/docs/change/change_log.md b/docs/change/change_log.md index 71558ba..4cd0414 100644 --- a/docs/change/change_log.md +++ b/docs/change/change_log.md @@ -5,6 +5,7 @@ This file provides a chronological, human-readable record of applied codebase an *Note: For operational task and run history, consult `.triagecore/ledger.jsonl`.* ## [Unreleased] +- Implemented CR-123 (Evaluation Handoff Contract): Added a contract-only `docs/evals/evaluation_handoff_contract.md` defining the file-based boundary between TriageCore and the external evaluator suite, including required `eval_case_v0` fixture input, `actual_outcome_export.v0` actual outcome files, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations. Added focused documentation tests and linked the existing fixture and actual-outcome docs while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope. - Implemented CR-122 (Eval Fixture Validation CLI): Added `tc eval validate-fixtures --input ` as a narrow read-only wrapper around the CR-121 JSONL validator, printing a bounded pass count on valid fixtures and fail-closed, line-aware diagnostics on invalid input. Added synthetic CLI tests only and kept scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope. - Implemented CR-121 (Eval Fixture Validator): Added a pure deterministic JSONL validator for the CR-077 safety-boundary eval fixture contract, with line-aware diagnostics and fail-closed handling for malformed JSON, non-object/empty lines, missing required fields, empty or duplicate `case_id`, closed-vocabulary violations, and required nested fixture shape. Added synthetic unit tests only and kept `tc eval`, scoring, model/backend calls, routing/admission integration, ledger writes, and adversarial tampering tests out of scope. - Implemented CR-120 (Telemetry Lane Release Hygiene): Added a docs-only post-merge hygiene note for the completed CR-117 through CR-119 lane, marked the reviewer checkpoint/release-hygiene candidate complete, and corrected stale telemetry brief wording that could confuse the CR-114 reviewer checkpoint with the CR-119 probe validation gate. No runtime code, schemas, tests, CLI behavior, routing, ledger writes, probe execution, endpoint calls, tags, or model/backend calls were changed. diff --git a/docs/change/requests/CR-123-evaluation-handoff-contract.md b/docs/change/requests/CR-123-evaluation-handoff-contract.md new file mode 100644 index 0000000..66f3d0c --- /dev/null +++ b/docs/change/requests/CR-123-evaluation-handoff-contract.md @@ -0,0 +1,48 @@ +# CR-123: Evaluation Handoff Contract + +## Status + +Implemented (contract-only) + +## Summary + +Define the file-contract boundary between TriageCore and the external evaluator +suite after CR-122. TriageCore validates expected fixtures and exports actual +outcome evidence; the external evaluator suite owns scoring, pass/fail +judgment, findings, and aggregate metrics. + +## Scope + +- Add `docs/evals/evaluation_handoff_contract.md`. +- Define required inputs and outputs for the handoff: + - `eval_case_v0` expected fixture JSONL. + - `actual_outcome_export.v0` actual outcome JSON files. +- Pin deterministic path vocabulary for a future bundle: + - `fixtures/safety_boundaries_v0.jsonl` + - `actuals/.json` + - `manifest/evaluation_handoff_manifest.json` +- Document TriageCore-side exit-code expectations for fixture validation and + existing actual-outcome smoke export commands. +- Link the existing fixture and actual-outcome docs to the handoff contract. +- Add focused documentation tests that keep the contract aligned with the + existing fixture validator and actual outcome writer. +- Update backlog and changelog after focused tests pass. + +## Non-Goals + +- No scoring, pass/fail judgment, aggregate metrics, partial credit, or score + interpretation inside TriageCore. +- No evaluator execution from TriageCore. +- No model, completion, chat, embedding, backend, endpoint, or network calls. +- No routing, admission, approval, identity, worker, or ledger integration. +- No ledger writes or durable evaluation-run evidence. +- No bundle builder, manifest writer, bundle validator, result importer, or + external result display. +- No changes to `eval_case_v0`, actual outcome JSON fields, or the current + fixture family. +- No adversarial or tampering expansion. + +## Validation + +- Focused: `python -m pytest -q tests/test_eval_handoff_contract.py tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py tests/test_eval_outcome_contract.py` -> 32 passed +- Full suite: `python -m pytest -q` -> 935 passed, 2 skipped diff --git a/docs/current_backlog.md b/docs/current_backlog.md index 7aa9ac2..c3e5a23 100644 --- a/docs/current_backlog.md +++ b/docs/current_backlog.md @@ -6,6 +6,10 @@ This document summarizes the active TriageCore backlog after CR-115. ## Active GitHub Backlog +- CR-123: Evaluation Handoff Contract + - Status: complete via CR-123 (contract-only) + - Purpose: Define the file-contract boundary between TriageCore and the external evaluator suite, including required fixture and actual-outcome inputs, contract/version identifiers, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations, while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope. + - CR-122: Eval Fixture Validation CLI - Status: complete via CR-122 (validator CLI only) - Purpose: Expose the CR-121 safety-boundary eval fixture validator through `tc eval validate-fixtures --input `, with bounded pass output and fail-closed line-aware diagnostics, while keeping scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope. @@ -120,7 +124,7 @@ This document summarizes the active TriageCore backlog after CR-115. - Empirical AI safety evaluation track - Source: CR-076 and CR-077 research framing/eval taxonomy docs - - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; scoring/evaluator execution, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices + - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; bundle/manifest building, bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure. - Operator UX implementation path @@ -212,7 +216,7 @@ Keep three work lanes distinct: For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness. -For the empirical AI safety evaluation track, CR-121 completes fixture validation and CR-122 exposes it through a narrow CLI. Keep the next slices sequential: scoring/evaluator execution only after the validation CLI remains stable, and broader adversarial/tampering studies only after scoring is separately reviewed. +For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, and CR-123 defines the external-evaluator handoff contract. Keep the next slices sequential: bundle/manifest builder, bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable. For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path. diff --git a/docs/evals/actual_outcome_export.md b/docs/evals/actual_outcome_export.md index 7b5327f..9530842 100644 --- a/docs/evals/actual_outcome_export.md +++ b/docs/evals/actual_outcome_export.md @@ -7,10 +7,12 @@ By exporting evidence as files: - TriageCore does not need to self-score. - There are no cross-repository Python imports, preventing tight coupling. -> **Reviewer Note:** For the conceptual explanation of how these actual outcome exports can be scored by an independent harness without importing TriageCore, see [Eval Integration Bridge](eval_integration_bridge.md). +> **Reviewer Note:** For the conceptual explanation of how these actual outcome exports can be scored by an independent harness without importing TriageCore, see [Eval Integration Bridge](eval_integration_bridge.md). For the CR-123 file handoff boundary, required inputs/outputs, deterministic path vocabulary, and exit-code expectations, see [Evaluation Handoff Contract](evaluation_handoff_contract.md). ## JSON Contract Shape +Contract identifier: `actual_outcome_export.v0` + TriageCore generates one JSON file per evaluation scenario. The exported file must match the following shape: ```json diff --git a/docs/evals/eval_integration_bridge.md b/docs/evals/eval_integration_bridge.md index 5f0a20c..9dc7f42 100644 --- a/docs/evals/eval_integration_bridge.md +++ b/docs/evals/eval_integration_bridge.md @@ -3,6 +3,8 @@ ## Purpose This document provides a reviewer-facing explanation of how TriageCore interfaces with the independent `agent-control-evals` repository. It clarifies how actual outcome exports produced by TriageCore can be deterministically scored by an external harness without importing or coupling the repositories. +The normative CR-123 handoff boundary is documented in [Evaluation Handoff Contract](evaluation_handoff_contract.md). This bridge remains a conceptual integration note. + ## Integration Boundary The integration between TriageCore and the evaluation harness is strictly **file-contract-based**. There are no shared runtime dependencies, no cross-repository Python imports, and no hidden coupling. The boundary is entirely maintained by writing static JSON files to disk, which are then independently read and scored. diff --git a/docs/evals/evaluation_handoff_contract.md b/docs/evals/evaluation_handoff_contract.md new file mode 100644 index 0000000..3c770fb --- /dev/null +++ b/docs/evals/evaluation_handoff_contract.md @@ -0,0 +1,123 @@ +# Evaluation Handoff Contract + +Contract identifier: `evaluation_handoff_contract.v0` + +## Purpose + +This document defines the CR-123 handoff boundary between TriageCore and an +external evaluator suite. TriageCore may validate and export deterministic +evidence. The external evaluator suite owns scoring, pass/fail judgment, +findings, and aggregate metrics. + +The contract is intentionally file-based. It avoids cross-repository imports, +runtime callbacks, hidden model calls, and any requirement that TriageCore know +the evaluator's scoring implementation. + +## Required TriageCore Inputs + +The TriageCore side of the handoff requires explicit operator-named paths: + +| Input | Required | Contract identifier | Notes | +| --- | --- | --- | --- | +| Expected fixture JSONL | Yes | `eval_case_v0` | One JSON object per line. Validate with `tc eval validate-fixtures --input `. | +| Actual outcome directory | Yes | `actual_outcome_export.v0` | Directory containing one `.json` file per observed case. | + +The current repository fixture is: + +```text +tests/fixtures/evals/safety_boundaries_v0.jsonl +``` + +The current documented actual-output directories are: + +```text +.triagecore/eval_actuals// +actuals/triagecore_smoke/ +``` + +TriageCore must not infer default evaluator inputs from the ledger, route +history, admission state, or local runtime state. The operator or a future +bundle builder must pass file paths explicitly. + +## Required TriageCore Outputs + +TriageCore produces static files only: + +| Output | Required | Deterministic path rule | Producer | +| --- | --- | --- | --- | +| Validated expected fixture JSONL | Yes | Preserve the operator-provided filename; current fixture path is `tests/fixtures/evals/safety_boundaries_v0.jsonl`. | Human-authored fixture plus CR-121/CR-122 validation | +| Actual outcome JSON files | Yes | Write one path-safe `.json` file under the operator-provided actuals directory. | Existing actual outcome export helpers | + +Actual outcome files use the existing JSON shape documented in +`docs/evals/actual_outcome_export.md`. The required fields are: + +- `case_id` +- `decision` +- `boundary_family` +- `reasons` +- `audit_required` +- `human_approval_required` + +Optional diagnostic fields may exist, but external scoring must treat them as +diagnostic evidence rather than primary oracle fields unless the external suite +explicitly adopts them. + +## External Evaluator Outputs + +The external evaluator suite, not TriageCore, produces scored artifacts such as: + +```text +reports/.jsonl +``` + +Those reports may include pass/fail outcomes, partial-credit findings, +aggregate metrics, or reviewer-facing summaries. TriageCore does not define the +scored report schema in this CR. TriageCore must treat evaluator findings as external artifacts unless a later CR adds an explicit import or display contract. + +## Deterministic Handoff Layout + +If a future CR materializes a handoff bundle, it should use these relative +paths inside the bundle directory: + +```text +fixtures/safety_boundaries_v0.jsonl +actuals/.json +manifest/evaluation_handoff_manifest.json +``` + +CR-123 does not create that bundle, manifest, or builder. These names reserve a +stable path vocabulary so CR-124+ can package the same already-defined evidence +without changing the scoring boundary. + +## Exit-Code Expectations + +TriageCore commands used before handoff must follow these expectations: + +| Command | Success | Failure | CLI usage error | +| --- | --- | --- | --- | +| `tc eval validate-fixtures --input ` | Exit `0` when the JSONL fixture is valid. | Exit `1` for missing, unreadable, malformed, structurally invalid, or duplicate-case fixtures. | Exit `2` for argparse usage errors. | +| `tc eval export-smoke --output-dir ` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. | +| `tc eval export-privacy-smoke --output-dir ` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. | +| `tc eval export-forbidden-tool-smoke --output-dir ` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. | + +The external evaluator suite owns its own exit-code contract. TriageCore may +document how to invoke that suite, but this CR does not add an evaluator runner +or interpret evaluator exit statuses. + +## Non-Goals + +CR-123 explicitly excludes: + +- scoring, pass/fail judgment, aggregate metrics, partial credit, or score + interpretation inside TriageCore +- no evaluator execution from TriageCore or any TriageCore CLI command +- model, completion, chat, embedding, backend, endpoint, or network calls +- routing, admission, approval, identity, or worker integration +- no ledger writes or durable recording that an evaluation occurred +- a bundle builder, manifest writer, bundle validator, or result importer +- changes to `eval_case_v0` or actual outcome JSON fields +- new fixture families or adversarial/tampering expansion + +The next safe slice after this contract is a deterministic bundle or manifest +builder that packages already-validated fixtures and already-exported actuals +without executing or scoring the evaluator. diff --git a/docs/research/eval_fixture_schema.md b/docs/research/eval_fixture_schema.md index 484eea0..f38f4cb 100644 --- a/docs/research/eval_fixture_schema.md +++ b/docs/research/eval_fixture_schema.md @@ -140,4 +140,7 @@ CR-121 treats this document as the contract for a narrow validator. The validato The validator checks fixture integrity only, and CR-122 exposes that validation through `tc eval validate-fixtures --input `. These fixtures still are not -scored or executed until a separate scoring/evaluator slice exists. +scored or executed by TriageCore. CR-123 defines the file-based +[Evaluation Handoff Contract](../evals/evaluation_handoff_contract.md) for +passing validated fixtures and actual outcome exports to an external evaluator +suite, while scoring remains outside TriageCore. diff --git a/tests/test_eval_handoff_contract.py b/tests/test_eval_handoff_contract.py new file mode 100644 index 0000000..fc50995 --- /dev/null +++ b/tests/test_eval_handoff_contract.py @@ -0,0 +1,69 @@ +from pathlib import Path + +from triage_core.eval_fixture_validator import SCHEMA_VERSION +from triage_core.eval_outcome_contract import build_actual_outcome, write_actual_outcome + + +REPO_ROOT = Path(__file__).resolve().parents[1] +CONTRACT_DOC = REPO_ROOT / "docs" / "evals" / "evaluation_handoff_contract.md" +ACTUAL_OUTCOME_DOC = REPO_ROOT / "docs" / "evals" / "actual_outcome_export.md" +FIXTURE_SCHEMA_DOC = REPO_ROOT / "docs" / "research" / "eval_fixture_schema.md" +BRIDGE_DOC = REPO_ROOT / "docs" / "evals" / "eval_integration_bridge.md" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def test_eval_handoff_contract_pins_versions_paths_and_exit_codes(): + doc = _read(CONTRACT_DOC) + + assert "evaluation_handoff_contract.v0" in doc + assert SCHEMA_VERSION in doc + assert "actual_outcome_export.v0" in doc + assert "tests/fixtures/evals/safety_boundaries_v0.jsonl" in doc + assert ".triagecore/eval_actuals//" in doc + assert "actuals/triagecore_smoke/" in doc + assert "fixtures/safety_boundaries_v0.jsonl" in doc + assert "actuals/.json" in doc + assert "manifest/evaluation_handoff_manifest.json" in doc + assert "Exit `0`" in doc + assert "Exit `1`" in doc + assert "Exit `2`" in doc + + +def test_eval_handoff_contract_preserves_external_scoring_boundary(): + doc = " ".join(_read(CONTRACT_DOC).lower().split()) + + required_phrases = ( + "the external evaluator suite owns scoring", + "treat evaluator findings as external artifacts", + "cr-123 does not create that bundle, manifest, or builder", + "no evaluator execution from triagecore", + "no ledger writes", + "scoring, pass/fail judgment, aggregate metrics, partial credit, or score interpretation inside triagecore", + ) + for phrase in required_phrases: + assert phrase in doc + + +def test_eval_handoff_contract_matches_actual_outcome_filename_rule(tmp_path): + outcome = build_actual_outcome( + case_id="privacy-deny-001", + decision="block", + boundary_family="privacy", + reasons=["privacy_check_failed"], + audit_required=True, + human_approval_required=False, + ) + + path = write_actual_outcome(outcome, tmp_path) + + assert path.name == "privacy-deny-001.json" + assert ".json" in _read(CONTRACT_DOC) + + +def test_existing_eval_docs_link_to_handoff_contract(): + assert "evaluation_handoff_contract.md" in _read(ACTUAL_OUTCOME_DOC) + assert "Evaluation Handoff Contract" in _read(FIXTURE_SCHEMA_DOC) + assert "evaluation_handoff_contract.md" in _read(BRIDGE_DOC)