diff --git a/docs/change/change_log.md b/docs/change/change_log.md index 3d22bf8..79519d0 100644 --- a/docs/change/change_log.md +++ b/docs/change/change_log.md @@ -6,6 +6,7 @@ This file provides a chronological, human-readable record of applied codebase an ## [Unreleased] +- Implemented CR-128 (Evaluation Handoff Integrity Validator): Added read-only `tc eval validate-handoff --bundle ` validation for the closed `evaluation_handoff_manifest.v0` schema, exact non-symlink/non-reparse inventory, fixed safe paths, counts, byte hashes, fixture and actual contracts, fixture membership, and persistent privacy safety. Validation emits only bounded success or stable closed reasons, makes no filesystem changes, and performs no scoring, semantic expected-vs-actual comparison, evaluator/model/backend/network call, ledger write, repair, or approval action. Hash agreement detects drift relative to the manifest but does not establish authenticity, provenance, approval, safety, or correctness. - Implemented CR-127 (Evaluation Handoff Bundle Builder): Added `tc eval build-handoff --fixture --actuals-dir --out-dir ` and a pure deterministic builder that validates explicit fixture/actual inputs, applies persistent privacy checks, copies evidence byte-for-byte into the fixed handoff layout, writes a sorted `evaluation_handoff_manifest.v0` with SHA-256 hashes, and atomically publishes only after successful staging. Scoring, pass/fail judgment, evaluator execution, approval claims, automatic discovery, overwrite behavior, and existing-bundle integrity validation remain out of scope and external scoring remains authoritative. - Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope. - Implemented CR-125 (Honor Terminal Resilience Routes): Made `human_handoff` and the currently unimplemented `deterministic` route terminal at the governed `TriageClient.run_task` boundary. Each now records the route decision plus a `worker_result` with `worker_result_status=not_attempted`, returns `handoff_required`, and reaches the existing `tc run` exit-3 contract without invoking a local or cloud backend. This slice does not add approval-and-resume behavior, change the broader `human_review_required` semantics, or alter cloud gating, signatures, reducer behavior, or other execution surfaces. diff --git a/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md b/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md new file mode 100644 index 0000000..ea41698 --- /dev/null +++ b/docs/change/requests/CR-128-evaluation-handoff-integrity-validator.md @@ -0,0 +1,45 @@ +# CR-128 — Evaluation Handoff Integrity Validator + +## Status + +Implemented locally; pending review. + +## Purpose + +Add a deterministic, read-only validator for an existing CR-127 evaluation +handoff bundle. The validator checks the closed manifest contract, exact +inventory, declared byte hashes, fixture and actual contracts, and persistent +privacy invariants without scoring or repairing the bundle. + +## Scope + +- Add `triage_core/evaluation_handoff_validator.py`. +- Add `tc eval validate-handoff --bundle `. +- Reject symlink/reparse traversal, unexpected inventory, unsafe manifest + paths, schema drift, count drift, hash drift, contract drift, unknown cases, + and privacy-invariant failures with stable closed reason codes. +- Reuse the fixture validator and CR-127 broad actual-outcome validation. +- Add focused pure-module, CLI, mutation, read-only, and contract-drift tests. + +## Non-Goals + +- No repair, normalization, report creation, or ledger write. +- No expected-vs-actual comparison, scoring, pass/fail judgment, or approval. +- No evaluator, model, backend, endpoint, or network invocation. +- No routing, admission, identity, approval, or worker integration. +- No authenticity, provenance, certification, safety, or correctness claim. + +## Trust Boundary + +A successful validation proves that current bundle bytes and inventory agree +with the bundled manifest and declared file contracts. SHA-256 detects drift +relative to that manifest; it does not authenticate who created the manifest +or establish provenance, approval, safety, or correctness. + +## Validation + +- Focused eval surface: + `python -m pytest -q tests/test_evaluation_handoff_validator.py tests/test_evaluation_handoff_validator_cli.py tests/test_evaluation_handoff_bundle.py tests/test_evaluation_handoff_bundle_cli.py tests/test_eval_handoff_contract.py tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py tests/test_eval_outcome_contract.py` + -> 86 passed, 1 skipped (symlink creation unavailable). +- Full suite: `python -m pytest -q` -> 997 passed, 3 skipped. +- `git diff --check` -> clean. diff --git a/docs/current_backlog.md b/docs/current_backlog.md index 290052f..676d77c 100644 --- a/docs/current_backlog.md +++ b/docs/current_backlog.md @@ -2,10 +2,14 @@ ## Status -This document summarizes the active TriageCore backlog after CR-127. +This document summarizes the active TriageCore backlog after CR-128. ## Active GitHub Backlog +- CR-128: Evaluation Handoff Integrity Validator + - Status: implemented locally; pending review + - Purpose: Read-only validation of an existing CR-127 bundle's closed manifest, exact inventory, safe paths, byte hashes, fixture/actual contracts, membership, and privacy invariants. Hash agreement detects drift relative to the manifest; it is not authenticity, provenance, approval, safety, or correctness. + - CR-127: Evaluation Handoff Bundle Builder - Status: complete via CR-127 - Purpose: Package an explicit validated fixture and explicit actual-outcome directory into a deterministic, privacy-checked, unscored handoff bundle with byte-preserving copies and a SHA-256 manifest. External scoring remains authoritative; existing-bundle integrity validation is the next separately scoped slice. @@ -140,7 +144,7 @@ This document summarizes the active TriageCore backlog after CR-127. - Empirical AI safety evaluation track - Source: CR-076 and CR-077 research framing/eval taxonomy docs - - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; bundle integrity validation, evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices + - Status: research question, threat model, eval taxonomy, fixture schema, toy boundary fixtures, **TC-EVAL-001 (Export Actual Outcome Contract Files)**, **TC-EVAL-002 (Actual Outcome Export CLI Smoke)**, **[x] TC-EVAL-003 (Map One Real Internal Decision Path Into the Export Contract)**, **[x] TC-EVAL-004 (Export One Real Privacy Scanner Actual)**, **[x] TC-EVAL-005 / 006 / 007 (Privacy Reason Normalization)** documented; fixture validator complete via CR-121; fixture validation CLI complete via CR-122; external-evaluator handoff contract complete via CR-123; deterministic bundle/manifest builder complete via CR-127; read-only bundle integrity validation complete via CR-128; evaluator invocation, adversarial tests, toy audit tampering eval, behavioral route diffing, **[x] TC-EVAL-008 (Structured Privacy Scanner Finding Codes)**, **[x] TC-EVAL-009 (Shared Internal Reason-Code Constants for Privacy Findings)**, **[x] TC-EVAL-010 (Export One Forbidden Tool-Call Actual)** and technical report remain future slices - Purpose: make TriageCore legible as a reproducible local-first AI control and evaluation harness for testing privacy, routing, identity, provenance, audit, and human-approval boundaries under controlled adversarial pressure. - Operator UX implementation path @@ -232,7 +236,7 @@ Keep three work lanes distinct: For signed ledger coverage, the reviewer-facing `validation_result` path and the signed `route_decision` path are now in place, including a smoke example, a capability-targeted doctor check, and a consolidated reviewer checkpoint for the latter. The current safe lane is packaging/stabilization, reviewer entrypoint maintenance, smoke-runbook clarity, video-first submission packaging, and release-readiness documentation. Deeper signing, cryptographic lifecycle work, and Issue #73 runtime key rotation should remain separate CRs. Do not treat a valid signature as approval, safety, or correctness. -For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, and CR-127 builds the deterministic unscored bundle. Keep the next slices sequential: bundle integrity validator, then a narrow external evaluator adapter. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable. +For the empirical AI safety evaluation track, CR-121 completes fixture validation, CR-122 exposes it through a narrow CLI, CR-123 defines the external-evaluator handoff contract, CR-127 builds the deterministic unscored bundle, and CR-128 validates bundle integrity read-only. Any external evaluator adapter requires a separate approved scope. Scoring and score interpretation must remain external to TriageCore, and broader adversarial/tampering studies should wait until the handoff path is stable. For external runtime interoperability, the next approved slice should be policy tests or execution-path validation for the bounded adapter path. @@ -249,4 +253,5 @@ For operator UX, future slices should focus on reviewability, export polish, and - **[done] Honor terminal resilience routes (CR-125)**: `human_handoff` and currently unimplemented `deterministic` routes now return a governed handoff before backend execution, recording `worker_result_status=not_attempted`; `tc run` reports the valid handoff with exit code 3. Approval-and-resume behavior, broader `human_review_required` semantics, and other execution seams remain future work. - **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope. - **[done] Evaluation handoff bundle builder (CR-127)**: Builds the deterministic fixed-layout handoff and SHA-256 manifest from explicit inputs without scoring. -- **Next slice requires a new approved CR**: The eval lane's next bounded candidate is a deterministic integrity validator for an existing handoff bundle. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR. +- **[done] Evaluation handoff integrity validator (CR-128)**: Validates the closed manifest, exact inventory, hashes, contracts, membership, and privacy without mutating or scoring the bundle. +- **Next slice requires a new approved CR**: Any external evaluator adapter or adversarial expansion must be separately scoped. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR. diff --git a/docs/evals/evaluation_handoff_contract.md b/docs/evals/evaluation_handoff_contract.md index 38fa58b..229ded6 100644 --- a/docs/evals/evaluation_handoff_contract.md +++ b/docs/evals/evaluation_handoff_contract.md @@ -103,6 +103,7 @@ TriageCore commands used before handoff must follow these expectations: | `tc eval export-privacy-smoke --output-dir ` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. | | `tc eval export-forbidden-tool-smoke --output-dir ` | Exit `0` after writing contract-shaped actual JSON. | Exit `1` for contract or write failures. | Exit `2` for argparse usage errors. | | `tc eval build-handoff --fixture --actuals-dir --out-dir ` | Exit `0` after atomically publishing an unscored deterministic bundle. | Exit `1` with a stable closed reason for input, privacy, path, or write failures. | Exit `2` for argparse usage errors. | +| `tc eval validate-handoff --bundle ` | Exit `0` when the closed manifest, inventory, hashes, file contracts, and privacy checks agree. | Exit `1` with one stable closed reason and no payload echo. | Exit `2` for argparse usage errors. | The external evaluator suite owns its own exit-code contract. TriageCore may document how to invoke that suite, but this CR does not add an evaluator runner @@ -122,6 +123,7 @@ CR-123 explicitly excludes: - changes to `eval_case_v0` or actual outcome JSON fields - new fixture families or adversarial/tampering expansion -CR-127 implements the deterministic bundle/manifest builder. The next safe -slice is a deterministic integrity validator for an existing bundle, without -executing or scoring the evaluator. +CR-127 implements the deterministic bundle/manifest builder, and CR-128 adds +read-only integrity validation. Hash agreement detects drift relative to the +manifest; it does not prove authenticity, provenance, approval, safety, or +correctness. Neither slice executes or scores the evaluator. diff --git a/docs/evals/evaluation_handoff_manifest.md b/docs/evals/evaluation_handoff_manifest.md index 7ef1dc9..746976a 100644 --- a/docs/evals/evaluation_handoff_manifest.md +++ b/docs/evals/evaluation_handoff_manifest.md @@ -66,7 +66,26 @@ renames it only after all output is written; failures leave no bundle. ## Integrity Boundary -SHA-256 fields describe the copied bytes and support external inspection. This -builder does not validate an existing bundle after creation. A deterministic -bundle integrity validator is a separate future scope. Scoring and score -interpretation remain exclusively owned by the external evaluator. +SHA-256 fields describe the copied bytes and support external inspection. +CR-128 adds the read-only command: + +```text +tc eval validate-handoff --bundle +``` + +It requires the exact closed `evaluation_handoff_manifest.v0` schema, fixed +paths, exact declared inventory, regular non-symlink/non-reparse files and +directories, matching byte hashes and counts, valid fixture and actual +contracts, fixture membership, and persistent privacy safety. Partial actual +coverage remains valid, but a zero-actual bundle is invalid because the CR-127 +builder cannot produce one. + +Success exits `0` with a bounded fixture/actual count. Closed validation +failures exit `1` and print only `reason=` to stderr; argparse usage +errors exit `2`. Validation is read-only: it performs no normalization, repair, +report write, ledger write, expected-vs-actual comparison, or evaluator call. + +Hash agreement detects drift relative to the manifest. It does not authenticate +the manifest, establish provenance, convey approval, certify safety, or prove +semantic correctness. Scoring and score interpretation remain exclusively +owned by the external evaluator. diff --git a/tests/test_evaluation_handoff_validator.py b/tests/test_evaluation_handoff_validator.py new file mode 100644 index 0000000..16a1795 --- /dev/null +++ b/tests/test_evaluation_handoff_validator.py @@ -0,0 +1,449 @@ +import hashlib +import json +import os +from pathlib import Path + +import pytest + +from triage_core.evaluation_handoff_bundle import build_evaluation_handoff_bundle +from triage_core.evaluation_handoff_validator import ( + EvaluationHandoffValidationError, + validate_evaluation_handoff_bundle, +) + + +def _fixture_case(case_id, boundary_family="privacy"): + return { + "schema_version": "eval_case_v0", + "case_id": case_id, + "boundary_family": boundary_family, + "title": "Boundary case", + "description": "A deterministic safety-boundary fixture.", + "task_packet": { + "summary": "Process a local record.", + "declared_risk": "high", + "relevant_metadata": {}, + }, + "policy_expectation": { + "boundary_rule": "Apply the declared boundary.", + "reason": "The control plane must remain inspectable.", + }, + "simulated_behavior": { + "actor_type": "review_bundle", + "proposed_action": "Produce bounded evidence.", + "notable_conditions": [], + }, + "expected_control_plane_decision": "deny", + "expected_audit_outcome": { + "required_artifacts": [], + "forbidden_artifacts": [], + "notes": "Keep only bounded evidence.", + }, + "expected_eval_outcome": "pass", + } + + +def _actual(case_id, decision="block", **extra): + payload = { + "case_id": case_id, + "decision": decision, + "boundary_family": "privacy", + "reasons": [], + "audit_required": True, + "human_approval_required": False, + } + payload.update(extra) + return payload + + +def _build_bundle(tmp_path, *, partial=False, decisions=None): + tmp_path.mkdir(parents=True, exist_ok=True) + case_ids = ("privacy-001", "routing-002") + fixture = tmp_path / "fixture.jsonl" + fixture.write_text( + "\n".join( + json.dumps( + _fixture_case( + case_id, + "privacy" if case_id.startswith("privacy") else "routing", + ), + sort_keys=True, + ) + for case_id in case_ids + ) + + "\n", + encoding="utf-8", + ) + actuals = tmp_path / "source-actuals" + actuals.mkdir() + selected = case_ids[:1] if partial else case_ids + for case_id in selected: + decision = (decisions or {}).get(case_id, "block") + (actuals / f"{case_id}.json").write_text( + json.dumps(_actual(case_id, decision=decision), sort_keys=True) + "\n", + encoding="utf-8", + ) + bundle = tmp_path / "bundle" + build_evaluation_handoff_bundle(fixture, actuals, bundle) + return bundle + + +def _manifest_path(bundle): + return bundle / "manifest" / "evaluation_handoff_manifest.json" + + +def _read_manifest(bundle): + return json.loads(_manifest_path(bundle).read_text(encoding="utf-8")) + + +def _write_manifest(bundle, manifest): + _manifest_path(bundle).write_text( + json.dumps(manifest, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + + +def _sha(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _assert_reason(reason, action): + with pytest.raises(EvaluationHandoffValidationError) as exc_info: + action() + assert exc_info.value.reason == reason + + +def _snapshot(root): + return { + path.relative_to(root).as_posix(): ( + path.is_dir(), + path.stat().st_mtime_ns, + None if path.is_dir() else path.read_bytes(), + ) + for path in [root, *sorted(root.rglob("*"))] + } + + +def test_pristine_bundle_validates_without_mutating_inventory_bytes_or_mtimes(tmp_path): + bundle = _build_bundle(tmp_path) + before = _snapshot(bundle) + + result = validate_evaluation_handoff_bundle(bundle) + + assert result.fixture_count == 2 + assert result.actual_count == 2 + assert _snapshot(bundle) == before + + +def test_partial_actual_coverage_and_non_enum_decision_are_valid(tmp_path): + bundle = _build_bundle( + tmp_path, + partial=True, + decisions={"privacy-001": "novel_external_decision"}, + ) + + result = validate_evaluation_handoff_bundle(bundle) + + assert result.fixture_count == 2 + assert result.actual_count == 1 + + +def test_root_failures(tmp_path): + _assert_reason( + "bundle_missing", + lambda: validate_evaluation_handoff_bundle(tmp_path / "missing"), + ) + file_path = tmp_path / "file" + file_path.write_text("not a bundle", encoding="utf-8") + _assert_reason( + "bundle_not_directory", + lambda: validate_evaluation_handoff_bundle(file_path), + ) + + +def test_manifest_missing_invalid_json_and_unsafe_case_id_fail_closed(tmp_path): + missing_bundle = _build_bundle(tmp_path / "missing") + _manifest_path(missing_bundle).unlink() + _assert_reason( + "manifest_missing", + lambda: validate_evaluation_handoff_bundle(missing_bundle), + ) + + json_bundle = _build_bundle(tmp_path / "json") + _manifest_path(json_bundle).write_text("{", encoding="utf-8") + _assert_reason( + "manifest_invalid_json", + lambda: validate_evaluation_handoff_bundle(json_bundle), + ) + + unsafe_bundle = _build_bundle(tmp_path / "unsafe") + manifest = _read_manifest(unsafe_bundle) + manifest["actuals"]["entries"][0]["case_id"] = "../unsafe" + _write_manifest(unsafe_bundle, manifest) + _assert_reason( + "unsafe_case_id", + lambda: validate_evaluation_handoff_bundle(unsafe_bundle), + ) + + +def test_zero_actuals_is_rejected(tmp_path): + bundle = _build_bundle(tmp_path, partial=True) + manifest = _read_manifest(bundle) + manifest["actuals"]["count"] = 0 + manifest["actuals"]["entries"] = [] + _write_manifest(bundle, manifest) + next((bundle / "actuals").iterdir()).unlink() + + _assert_reason( + "actual_count_mismatch", + lambda: validate_evaluation_handoff_bundle(bundle), + ) + + +def test_missing_extra_and_tampered_files_fail_closed(tmp_path): + missing_bundle = _build_bundle(tmp_path / "missing-case") + (missing_bundle / "actuals" / "privacy-001.json").unlink() + _assert_reason( + "declared_file_missing", + lambda: validate_evaluation_handoff_bundle(missing_bundle), + ) + + extra_bundle = _build_bundle(tmp_path / "extra-case") + (extra_bundle / "actuals" / ".hidden").write_text("x", encoding="utf-8") + _assert_reason( + "unexpected_bundle_entry", + lambda: validate_evaluation_handoff_bundle(extra_bundle), + ) + + tampered_bundle = _build_bundle(tmp_path / "tamper-case") + with (tampered_bundle / "actuals" / "privacy-001.json").open( + "ab" + ) as handle: + handle.write(b" ") + _assert_reason( + "hash_mismatch", + lambda: validate_evaluation_handoff_bundle(tampered_bundle), + ) + + +@pytest.mark.parametrize( + ("mutation", "reason"), + [ + (lambda value: value.update({"extra": True}), "manifest_invalid_schema"), + ( + lambda value: value.update({"schema_version": "wrong"}), + "manifest_contract_mismatch", + ), + ( + lambda value: value["fixture"].update({"case_count": True}), + "manifest_invalid_schema", + ), + ( + lambda value: value["fixture"].update({"sha256": "A" * 64}), + "manifest_invalid_schema", + ), + ( + lambda value: value["actuals"].update({"count": 99}), + "actual_count_mismatch", + ), + ( + lambda value: value["actuals"]["entries"].reverse(), + "manifest_invalid_schema", + ), + ], +) +def test_manifest_schema_types_counts_order_and_hash_format( + tmp_path, + mutation, + reason, +): + bundle = _build_bundle(tmp_path) + manifest = _read_manifest(bundle) + mutation(manifest) + _write_manifest(bundle, manifest) + + _assert_reason( + reason, + lambda: validate_evaluation_handoff_bundle(bundle), + ) + + +def test_manifest_duplicate_ids_and_paths_are_rejected(tmp_path): + duplicate_id_bundle = _build_bundle(tmp_path / "duplicate-id") + manifest = _read_manifest(duplicate_id_bundle) + manifest["actuals"]["entries"][1]["case_id"] = manifest["actuals"]["entries"][0][ + "case_id" + ] + _write_manifest(duplicate_id_bundle, manifest) + _assert_reason( + "duplicate_case_id", + lambda: validate_evaluation_handoff_bundle(duplicate_id_bundle), + ) + + duplicate_path_bundle = _build_bundle(tmp_path / "duplicate-path") + manifest = _read_manifest(duplicate_path_bundle) + manifest["actuals"]["entries"][1]["path"] = manifest["actuals"]["entries"][0][ + "path" + ] + _write_manifest(duplicate_path_bundle, manifest) + _assert_reason( + "duplicate_actual_path", + lambda: validate_evaluation_handoff_bundle(duplicate_path_bundle), + ) + + +@pytest.mark.parametrize( + "attack", + [ + "../privacy-001.json", + "actuals/../privacy-001.json", + "/actuals/privacy-001.json", + "C:/actuals/privacy-001.json", + "actuals\\privacy-001.json", + "actuals//privacy-001.json", + "actuals/./privacy-001.json", + "", + ], +) +def test_manifest_path_attacks_are_rejected_without_normalization(tmp_path, attack): + bundle = _build_bundle(tmp_path) + manifest = _read_manifest(bundle) + manifest["actuals"]["entries"][0]["path"] = attack + _write_manifest(bundle, manifest) + + _assert_reason( + "manifest_path_invalid", + lambda: validate_evaluation_handoff_bundle(bundle), + ) + + +def test_fixture_invalid_count_and_privacy_drift(tmp_path): + invalid_bundle = _build_bundle(tmp_path / "invalid") + fixture = invalid_bundle / "fixtures" / "safety_boundaries_v0.jsonl" + fixture.write_text("{", encoding="utf-8") + manifest = _read_manifest(invalid_bundle) + manifest["fixture"]["sha256"] = _sha(fixture) + _write_manifest(invalid_bundle, manifest) + _assert_reason( + "fixture_invalid", + lambda: validate_evaluation_handoff_bundle(invalid_bundle), + ) + + count_bundle = _build_bundle(tmp_path / "count") + manifest = _read_manifest(count_bundle) + manifest["fixture"]["case_count"] = 3 + _write_manifest(count_bundle, manifest) + _assert_reason( + "fixture_count_mismatch", + lambda: validate_evaluation_handoff_bundle(count_bundle), + ) + + privacy_bundle = _build_bundle(tmp_path / "privacy") + fixture = privacy_bundle / "fixtures" / "safety_boundaries_v0.jsonl" + cases = [json.loads(line) for line in fixture.read_text().splitlines()] + cases[0]["raw_content"] = "123-45-6789" + fixture.write_text( + "\n".join(json.dumps(case, sort_keys=True) for case in cases) + "\n", + encoding="utf-8", + ) + manifest = _read_manifest(privacy_bundle) + manifest["fixture"]["sha256"] = _sha(fixture) + _write_manifest(privacy_bundle, manifest) + _assert_reason( + "privacy_invariant_failed", + lambda: validate_evaluation_handoff_bundle(privacy_bundle), + ) + + +def test_actual_json_contract_filename_unknown_and_privacy_drift(tmp_path): + invalid_json_bundle = _build_bundle(tmp_path / "json", partial=True) + actual = invalid_json_bundle / "actuals" / "privacy-001.json" + actual.write_text("{", encoding="utf-8") + manifest = _read_manifest(invalid_json_bundle) + manifest["actuals"]["entries"][0]["sha256"] = _sha(actual) + _write_manifest(invalid_json_bundle, manifest) + _assert_reason( + "actual_invalid_json", + lambda: validate_evaluation_handoff_bundle(invalid_json_bundle), + ) + + contract_bundle = _build_bundle(tmp_path / "contract", partial=True) + actual = contract_bundle / "actuals" / "privacy-001.json" + actual.write_text("{}", encoding="utf-8") + manifest = _read_manifest(contract_bundle) + manifest["actuals"]["entries"][0]["sha256"] = _sha(actual) + _write_manifest(contract_bundle, manifest) + _assert_reason( + "actual_invalid_contract", + lambda: validate_evaluation_handoff_bundle(contract_bundle), + ) + + filename_bundle = _build_bundle(tmp_path / "filename", partial=True) + actual = filename_bundle / "actuals" / "privacy-001.json" + payload = _actual("routing-002") + actual.write_text(json.dumps(payload), encoding="utf-8") + manifest = _read_manifest(filename_bundle) + manifest["actuals"]["entries"][0]["sha256"] = _sha(actual) + _write_manifest(filename_bundle, manifest) + _assert_reason( + "actual_filename_mismatch", + lambda: validate_evaluation_handoff_bundle(filename_bundle), + ) + + unknown_bundle = _build_bundle(tmp_path / "unknown", partial=True) + old_actual = unknown_bundle / "actuals" / "privacy-001.json" + new_actual = unknown_bundle / "actuals" / "unknown-003.json" + new_actual.write_text(json.dumps(_actual("unknown-003")), encoding="utf-8") + old_actual.unlink() + manifest = _read_manifest(unknown_bundle) + entry = manifest["actuals"]["entries"][0] + entry.update( + { + "case_id": "unknown-003", + "path": "actuals/unknown-003.json", + "sha256": _sha(new_actual), + } + ) + _write_manifest(unknown_bundle, manifest) + _assert_reason( + "unknown_case_id", + lambda: validate_evaluation_handoff_bundle(unknown_bundle), + ) + + privacy_bundle = _build_bundle(tmp_path / "privacy-actual", partial=True) + actual = privacy_bundle / "actuals" / "privacy-001.json" + actual.write_text( + json.dumps(_actual("privacy-001", raw_content="123-45-6789")), + encoding="utf-8", + ) + manifest = _read_manifest(privacy_bundle) + manifest["actuals"]["entries"][0]["sha256"] = _sha(actual) + _write_manifest(privacy_bundle, manifest) + _assert_reason( + "privacy_invariant_failed", + lambda: validate_evaluation_handoff_bundle(privacy_bundle), + ) + + +def test_bundle_and_component_symlinks_are_rejected_when_supported(tmp_path): + bundle = _build_bundle(tmp_path / "root-link-source") + root_link = tmp_path / "bundle-link" + try: + root_link.symlink_to(bundle, target_is_directory=True) + except (OSError, NotImplementedError): + pytest.skip("symlink creation is unavailable") + _assert_reason( + "bundle_root_symlink", + lambda: validate_evaluation_handoff_bundle(root_link), + ) + + component_bundle = _build_bundle(tmp_path / "component-link-source") + actual = component_bundle / "actuals" / "privacy-001.json" + target = tmp_path / "actual-target.json" + target.write_bytes(actual.read_bytes()) + actual.unlink() + actual.symlink_to(target) + _assert_reason( + "bundle_symlink", + lambda: validate_evaluation_handoff_bundle(component_bundle), + ) diff --git a/tests/test_evaluation_handoff_validator_cli.py b/tests/test_evaluation_handoff_validator_cli.py new file mode 100644 index 0000000..64c7077 --- /dev/null +++ b/tests/test_evaluation_handoff_validator_cli.py @@ -0,0 +1,109 @@ +import json +import sys + +from triage_core.evaluation_handoff_bundle import build_evaluation_handoff_bundle + + +def _run_cli(monkeypatch, args): + from triage_core.tc_cli import main + + monkeypatch.setattr(sys, "argv", ["tc", *args]) + try: + main() + except SystemExit as exc: + return exc.code + return 0 + + +def _bundle(tmp_path): + case_id = "privacy-001" + fixture_case = { + "schema_version": "eval_case_v0", + "case_id": case_id, + "boundary_family": "privacy", + "title": "Boundary", + "description": "A deterministic boundary case.", + "task_packet": { + "summary": "Process a local record.", + "declared_risk": "high", + "relevant_metadata": {}, + }, + "policy_expectation": { + "boundary_rule": "Keep evidence bounded.", + "reason": "The boundary must be inspectable.", + }, + "simulated_behavior": { + "actor_type": "review_bundle", + "proposed_action": "Write bounded evidence.", + "notable_conditions": [], + }, + "expected_control_plane_decision": "deny", + "expected_audit_outcome": { + "required_artifacts": [], + "forbidden_artifacts": [], + "notes": "Keep bounded evidence.", + }, + "expected_eval_outcome": "pass", + } + actual = { + "case_id": case_id, + "decision": "block", + "boundary_family": "privacy", + "reasons": [], + "audit_required": True, + "human_approval_required": False, + } + fixture = tmp_path / "fixture.jsonl" + fixture.write_text(json.dumps(fixture_case) + "\n", encoding="utf-8") + actuals = tmp_path / "actuals-source" + actuals.mkdir() + (actuals / f"{case_id}.json").write_text( + json.dumps(actual) + "\n", + encoding="utf-8", + ) + bundle = tmp_path / "bundle" + build_evaluation_handoff_bundle(fixture, actuals, bundle) + return bundle + + +def test_cli_validate_handoff_success(monkeypatch, capsys, tmp_path): + bundle = _bundle(tmp_path) + + code = _run_cli( + monkeypatch, + ["eval", "validate-handoff", "--bundle", str(bundle)], + ) + + captured = capsys.readouterr() + assert code == 0 + assert captured.out.strip() == ( + "Evaluation handoff bundle valid: " + "1 fixture case(s), 1 actual outcome(s)" + ) + assert captured.err == "" + + +def test_cli_failure_is_reason_only_and_never_echoes_payload( + monkeypatch, + capsys, + tmp_path, +): + bundle = _bundle(tmp_path) + sensitive = "123-45-6789" + actual = bundle / "actuals" / "privacy-001.json" + actual.write_text(sensitive, encoding="utf-8") + + code = _run_cli( + monkeypatch, + ["eval", "validate-handoff", "--bundle", str(bundle)], + ) + + captured = capsys.readouterr() + assert code == 1 + assert captured.out == "" + assert captured.err.strip() == "reason=hash_mismatch" + assert sensitive not in captured.out + captured.err + + +def test_cli_missing_bundle_argument_is_argparse_exit_two(monkeypatch): + assert _run_cli(monkeypatch, ["eval", "validate-handoff"]) == 2 diff --git a/triage_core/evaluation_handoff_bundle.py b/triage_core/evaluation_handoff_bundle.py index 0221a41..3607829 100644 --- a/triage_core/evaluation_handoff_bundle.py +++ b/triage_core/evaluation_handoff_bundle.py @@ -26,7 +26,7 @@ ACTUAL_CONTRACT = "actual_outcome_export.v0" FIXTURE_BUNDLE_PATH = "fixtures/safety_boundaries_v0.jsonl" MANIFEST_BUNDLE_PATH = "manifest/evaluation_handoff_manifest.json" -_SAFE_CASE_ID = re.compile(r"^[A-Za-z0-9_-]+$") +SAFE_CASE_ID_PATTERN = re.compile(r"^[A-Za-z0-9_-]+$") _ACTUAL_REQUIRED_FIELDS = ( "case_id", "decision", @@ -91,7 +91,7 @@ def build_evaluation_handoff_bundle( raise EvaluationHandoffBundleError("privacy_invariant_failed") from exc if any( - not _SAFE_CASE_ID.fullmatch(str(case["case_id"])) + not is_safe_case_id(str(case["case_id"])) for case in fixture_cases ): raise EvaluationHandoffBundleError("unsafe_case_id") @@ -228,9 +228,9 @@ def _load_actuals( except (UnicodeError, json.JSONDecodeError) as exc: raise EvaluationHandoffBundleError("actual_invalid_json") from exc - _validate_actual_contract(payload) + validate_actual_outcome_contract(payload) case_id = payload["case_id"] - if not _SAFE_CASE_ID.fullmatch(case_id): + if not is_safe_case_id(case_id): raise EvaluationHandoffBundleError("unsafe_case_id") if case_id in seen_case_ids: raise EvaluationHandoffBundleError("duplicate_case_id") @@ -255,7 +255,13 @@ def _load_actuals( return tuple(sorted(loaded, key=lambda actual: actual.case_id)) -def _validate_actual_contract(payload: Any) -> None: +def is_safe_case_id(case_id: str) -> bool: + return bool(SAFE_CASE_ID_PATTERN.fullmatch(case_id)) + + +def validate_actual_outcome_contract(payload: Any) -> None: + """Validate the CR-127 broad actual-outcome shape without adding enums.""" + if not isinstance(payload, Mapping): raise EvaluationHandoffBundleError("actual_invalid_contract") if any(field not in payload for field in _ACTUAL_REQUIRED_FIELDS): diff --git a/triage_core/evaluation_handoff_validator.py b/triage_core/evaluation_handoff_validator.py new file mode 100644 index 0000000..7a00cfd --- /dev/null +++ b/triage_core/evaluation_handoff_validator.py @@ -0,0 +1,411 @@ +from __future__ import annotations + +import hashlib +import json +import re +import stat +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Mapping + +from triage_core.eval_fixture_validator import ( + EvalFixtureValidationError, + load_eval_fixture_jsonl, +) +from triage_core.evaluation_handoff_bundle import ( + ACTUAL_CONTRACT, + BUNDLE_TYPE, + FIXTURE_BUNDLE_PATH, + FIXTURE_CONTRACT, + HANDOFF_CONTRACT, + MANIFEST_BUNDLE_PATH, + MANIFEST_SCHEMA_VERSION, + EvaluationHandoffBundleError, + is_safe_case_id, + validate_actual_outcome_contract, +) +from triage_core.privacy_invariants import ( + PersistentPrivacyInvariantError, + assert_persistent_privacy_safe, +) + + +_HEX_SHA256 = re.compile(r"^[0-9a-f]{64}$") +_TOP_LEVEL_KEYS = { + "schema_version", + "bundle_type", + "handoff_contract", + "scoring_owner", + "triagecore_scored", + "fixture", + "actuals", +} +_FIXTURE_KEYS = {"contract_identifier", "path", "sha256", "case_count"} +_ACTUALS_KEYS = {"contract_identifier", "count", "entries"} +_ACTUAL_ENTRY_KEYS = {"case_id", "path", "sha256"} + + +class EvaluationHandoffValidationError(ValueError): + """Fail-closed validation error with one stable public reason.""" + + def __init__(self, reason: str) -> None: + self.reason = reason + super().__init__(reason) + + +@dataclass(frozen=True) +class EvaluationHandoffValidationResult: + fixture_count: int + actual_count: int + + +class _DuplicateObjectKey(ValueError): + pass + + +def validate_evaluation_handoff_bundle( + bundle: str | Path, +) -> EvaluationHandoffValidationResult: + """Validate an existing CR-127 bundle without mutating it.""" + + root = Path(bundle) + _validate_root(root) + manifest_path = root / MANIFEST_BUNDLE_PATH + _validate_manifest_file(manifest_path) + manifest = _load_manifest(manifest_path) + fixture_spec, actual_specs = _validate_manifest_schema(manifest) + _validate_inventory(root, actual_specs) + + fixture_path = root / FIXTURE_BUNDLE_PATH + fixture_bytes = _read_bytes(fixture_path) + if _sha256(fixture_bytes) != fixture_spec["sha256"]: + raise EvaluationHandoffValidationError("hash_mismatch") + + try: + fixture_cases = load_eval_fixture_jsonl(fixture_path) + except EvalFixtureValidationError as exc: + raise EvaluationHandoffValidationError("fixture_invalid") from exc + except UnicodeError as exc: + raise EvaluationHandoffValidationError("fixture_invalid") from exc + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + + if len(fixture_cases) != fixture_spec["case_count"]: + raise EvaluationHandoffValidationError("fixture_count_mismatch") + try: + for case in fixture_cases: + assert_persistent_privacy_safe(case, artifact_name="eval fixture") + except PersistentPrivacyInvariantError as exc: + raise EvaluationHandoffValidationError("privacy_invariant_failed") from exc + + fixture_case_ids = {str(case["case_id"]) for case in fixture_cases} + if any(not is_safe_case_id(case_id) for case_id in fixture_case_ids): + raise EvaluationHandoffValidationError("unsafe_case_id") + + for spec in actual_specs: + actual_path = root.joinpath(*spec["path"].split("/")) + content = _read_bytes(actual_path) + if _sha256(content) != spec["sha256"]: + raise EvaluationHandoffValidationError("hash_mismatch") + try: + payload = json.loads( + content.decode("utf-8"), + parse_constant=_reject_json_constant, + ) + except (UnicodeError, json.JSONDecodeError, ValueError) as exc: + raise EvaluationHandoffValidationError("actual_invalid_json") from exc + try: + validate_actual_outcome_contract(payload) + except EvaluationHandoffBundleError as exc: + raise EvaluationHandoffValidationError("actual_invalid_contract") from exc + case_id = payload["case_id"] + if not is_safe_case_id(case_id): + raise EvaluationHandoffValidationError("unsafe_case_id") + if actual_path.name != f"{case_id}.json" or spec["case_id"] != case_id: + raise EvaluationHandoffValidationError("actual_filename_mismatch") + if case_id not in fixture_case_ids: + raise EvaluationHandoffValidationError("unknown_case_id") + try: + assert_persistent_privacy_safe(payload, artifact_name="actual outcome") + except PersistentPrivacyInvariantError as exc: + raise EvaluationHandoffValidationError("privacy_invariant_failed") from exc + + if len(actual_specs) != manifest["actuals"]["count"]: + raise EvaluationHandoffValidationError("actual_count_mismatch") + return EvaluationHandoffValidationResult( + fixture_count=len(fixture_cases), + actual_count=len(actual_specs), + ) + + +def _validate_root(root: Path) -> None: + try: + if _has_link_or_reparse_ancestor(root): + raise EvaluationHandoffValidationError("bundle_root_symlink") + try: + root.lstat() + except FileNotFoundError: + raise EvaluationHandoffValidationError("bundle_missing") + if _is_link_or_reparse(root): + raise EvaluationHandoffValidationError("bundle_root_symlink") + if not root.is_dir(): + raise EvaluationHandoffValidationError("bundle_not_directory") + except EvaluationHandoffValidationError: + raise + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + + +def _validate_manifest_file(path: Path) -> None: + try: + try: + path.parent.lstat() + except FileNotFoundError: + raise EvaluationHandoffValidationError("manifest_missing") + if _is_link_or_reparse(path.parent): + raise EvaluationHandoffValidationError("bundle_symlink") + try: + path.lstat() + except FileNotFoundError: + raise EvaluationHandoffValidationError("manifest_missing") + if _is_link_or_reparse(path): + raise EvaluationHandoffValidationError("bundle_symlink") + if not path.is_file(): + raise EvaluationHandoffValidationError("manifest_missing") + except EvaluationHandoffValidationError: + raise + except OSError as exc: + raise EvaluationHandoffValidationError("manifest_unreadable") from exc + + +def _load_manifest(path: Path) -> Mapping[str, Any]: + try: + content = path.read_bytes() + except OSError as exc: + raise EvaluationHandoffValidationError("manifest_unreadable") from exc + try: + payload = json.loads( + content.decode("utf-8"), + object_pairs_hook=_strict_object, + parse_constant=_reject_json_constant, + ) + except (UnicodeError, json.JSONDecodeError, _DuplicateObjectKey, ValueError) as exc: + raise EvaluationHandoffValidationError("manifest_invalid_json") from exc + if not isinstance(payload, Mapping): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + return payload + + +def _strict_object(pairs): + result = {} + for key, value in pairs: + if key in result: + raise _DuplicateObjectKey(key) + result[key] = value + return result + + +def _reject_json_constant(value: str): + raise ValueError(value) + + +def _validate_manifest_schema( + manifest: Mapping[str, Any], +) -> tuple[Mapping[str, Any], tuple[Mapping[str, Any], ...]]: + if set(manifest) != _TOP_LEVEL_KEYS: + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if ( + manifest["schema_version"] != MANIFEST_SCHEMA_VERSION + or manifest["bundle_type"] != BUNDLE_TYPE + or manifest["handoff_contract"] != HANDOFF_CONTRACT + or manifest["scoring_owner"] != "external_evaluator" + or manifest["triagecore_scored"] is not False + ): + raise EvaluationHandoffValidationError("manifest_contract_mismatch") + + fixture = manifest["fixture"] + actuals = manifest["actuals"] + if not isinstance(fixture, Mapping) or set(fixture) != _FIXTURE_KEYS: + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if not isinstance(actuals, Mapping) or set(actuals) != _ACTUALS_KEYS: + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if ( + fixture["contract_identifier"] != FIXTURE_CONTRACT + or actuals["contract_identifier"] != ACTUAL_CONTRACT + ): + raise EvaluationHandoffValidationError("manifest_contract_mismatch") + + _validate_manifest_path(fixture["path"], expected=FIXTURE_BUNDLE_PATH) + if not _is_sha256(fixture["sha256"]): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if not _is_count(fixture["case_count"]) or fixture["case_count"] < 1: + raise EvaluationHandoffValidationError("manifest_invalid_schema") + + entries = actuals["entries"] + if not _is_count(actuals["count"]): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if not isinstance(entries, list): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + if actuals["count"] == 0 or not entries: + raise EvaluationHandoffValidationError("actual_count_mismatch") + if actuals["count"] != len(entries): + raise EvaluationHandoffValidationError("actual_count_mismatch") + + seen_ids: set[str] = set() + seen_paths: set[str] = set() + validated_entries: list[Mapping[str, Any]] = [] + for entry in entries: + if not isinstance(entry, Mapping) or set(entry) != _ACTUAL_ENTRY_KEYS: + raise EvaluationHandoffValidationError("manifest_invalid_schema") + case_id = entry["case_id"] + if not isinstance(case_id, str) or not is_safe_case_id(case_id): + raise EvaluationHandoffValidationError("unsafe_case_id") + if case_id in seen_ids: + raise EvaluationHandoffValidationError("duplicate_case_id") + if not isinstance(entry["path"], str): + raise EvaluationHandoffValidationError("manifest_path_invalid") + if entry["path"] in seen_paths: + raise EvaluationHandoffValidationError("duplicate_actual_path") + expected_path = f"actuals/{case_id}.json" + _validate_manifest_path(entry["path"], expected=expected_path) + if not _is_sha256(entry["sha256"]): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + seen_ids.add(case_id) + seen_paths.add(entry["path"]) + validated_entries.append(entry) + + case_ids = [entry["case_id"] for entry in validated_entries] + if case_ids != sorted(case_ids): + raise EvaluationHandoffValidationError("manifest_invalid_schema") + return fixture, tuple(validated_entries) + + +def _validate_manifest_path(value: Any, *, expected: str) -> None: + if not isinstance(value, str) or not value: + raise EvaluationHandoffValidationError("manifest_path_invalid") + if ( + value != expected + or "\\" in value + or value.startswith("/") + or re.match(r"^[A-Za-z]:", value) + ): + raise EvaluationHandoffValidationError("manifest_path_invalid") + components = value.split("/") + if any(component in {"", ".", ".."} for component in components): + raise EvaluationHandoffValidationError("manifest_path_invalid") + + +def _validate_inventory( + root: Path, + actual_specs: tuple[Mapping[str, Any], ...], +) -> None: + expected_root = {"fixtures", "actuals", "manifest"} + _require_exact_directory_entries(root, expected_root) + _require_real_directory(root / "fixtures") + _require_real_directory(root / "actuals") + _require_real_directory(root / "manifest") + _require_exact_directory_entries( + root / "fixtures", + {"safety_boundaries_v0.jsonl"}, + ) + _require_exact_directory_entries( + root / "manifest", + {"evaluation_handoff_manifest.json"}, + ) + expected_actual_names = { + f"{spec['case_id']}.json" + for spec in actual_specs + } + _require_exact_directory_entries(root / "actuals", expected_actual_names) + + for path in ( + root / FIXTURE_BUNDLE_PATH, + root / MANIFEST_BUNDLE_PATH, + *(root.joinpath(*spec["path"].split("/")) for spec in actual_specs), + ): + _require_real_file(path) + + +def _require_exact_directory_entries(path: Path, expected: set[str]) -> None: + try: + entries = list(path.iterdir()) + if any(_is_link_or_reparse(entry) for entry in entries): + raise EvaluationHandoffValidationError("bundle_symlink") + actual = {entry.name for entry in entries} + except EvaluationHandoffValidationError: + raise + except FileNotFoundError as exc: + raise EvaluationHandoffValidationError("declared_file_missing") from exc + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + if actual != expected: + missing = expected - actual + if missing: + raise EvaluationHandoffValidationError("declared_file_missing") + raise EvaluationHandoffValidationError("unexpected_bundle_entry") + + +def _require_real_directory(path: Path) -> None: + try: + if _is_link_or_reparse(path): + raise EvaluationHandoffValidationError("bundle_symlink") + if not path.is_dir(): + raise EvaluationHandoffValidationError("declared_file_missing") + except EvaluationHandoffValidationError: + raise + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + + +def _require_real_file(path: Path) -> None: + try: + if _is_link_or_reparse(path): + raise EvaluationHandoffValidationError("bundle_symlink") + if not path.is_file(): + raise EvaluationHandoffValidationError("declared_file_missing") + except EvaluationHandoffValidationError: + raise + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + + +def _is_link_or_reparse(path: Path) -> bool: + metadata = path.lstat() + if stat.S_ISLNK(metadata.st_mode): + return True + attributes = getattr(metadata, "st_file_attributes", 0) + reparse_flag = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400) + return bool(attributes & reparse_flag) + + +def _has_link_or_reparse_ancestor(path: Path) -> bool: + absolute = path.absolute() + for parent in reversed(absolute.parents): + if parent == Path(absolute.anchor): + continue + try: + is_link = _is_link_or_reparse(parent) + except FileNotFoundError: + continue + if is_link: + return True + return False + + +def _read_bytes(path: Path) -> bytes: + try: + return path.read_bytes() + except OSError as exc: + raise EvaluationHandoffValidationError("input_unreadable") from exc + + +def _sha256(content: bytes) -> str: + return hashlib.sha256(content).hexdigest() + + +def _is_sha256(value: Any) -> bool: + return isinstance(value, str) and bool(_HEX_SHA256.fullmatch(value)) + + +def _is_count(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value >= 0 diff --git a/triage_core/tc_cli.py b/triage_core/tc_cli.py index ec1bb92..34b2bf7 100644 --- a/triage_core/tc_cli.py +++ b/triage_core/tc_cli.py @@ -1769,6 +1769,25 @@ def tc_eval_build_handoff( ) +def tc_eval_validate_handoff(bundle: str) -> None: + from triage_core.evaluation_handoff_validator import ( + EvaluationHandoffValidationError, + validate_evaluation_handoff_bundle, + ) + + try: + result = validate_evaluation_handoff_bundle(bundle) + except EvaluationHandoffValidationError as exc: + print(f"reason={exc.reason}", file=sys.stderr) + sys.exit(1) + + print( + "Evaluation handoff bundle valid: " + f"{result.fixture_count} fixture case(s), " + f"{result.actual_count} actual outcome(s)" + ) + + def tc_eval_review( submission_path: str, context_packet_path: str, @@ -2604,6 +2623,17 @@ def main(): help="New directory where the handoff bundle will be written", ) + eval_validate_handoff_parser = eval_subparsers.add_parser( + "validate-handoff", + help="Validate an existing evaluation handoff bundle without scoring", + ) + eval_validate_handoff_parser.add_argument( + "--bundle", + required=True, + type=str, + help="Existing evaluation handoff bundle root", + ) + eval_review_parser = eval_subparsers.add_parser( "review", help="Validate a review submission and run the deterministic checker against a context packet", @@ -2967,6 +2997,8 @@ def main(): tc_eval_validate_fixtures(args.input) elif args.eval_command == "build-handoff": tc_eval_build_handoff(args.fixture, args.actuals_dir, args.out_dir) + elif args.eval_command == "validate-handoff": + tc_eval_validate_handoff(args.bundle) elif args.eval_command == "review": tc_eval_review( args.submission, @@ -2977,7 +3009,7 @@ def main(): args.fail_on_gate, ) else: - eval_parser.error("eval requires a subcommand: export-smoke, export-privacy-smoke, export-forbidden-tool-smoke, validate-fixtures, build-handoff, or review") + eval_parser.error("eval requires a subcommand: export-smoke, export-privacy-smoke, export-forbidden-tool-smoke, validate-fixtures, build-handoff, validate-handoff, or review") elif args.command == "context": if args.context_command == "plan": tc_context_plan(args.input, args.model)