Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ Expected outputs:
- `tc audit --self-test` writes one privacy-safe `route_audit` event.
- `tc audit --kind route_audit --last 10` shows routing metadata without raw prompt/data leakage.
- `tc audit --kind demo_dry_run --last 5` shows the deterministic demo evidence without raw request or proposed-output content.
- `tc audit --privacy-invariants` scans the persistent ledger for forbidden raw-content keys and confirms the CR-021 invariant still holds.
- `tc audit --privacy-invariants` scans the persistent ledger for forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns; it does not classify arbitrary free text.
- `triagecore benchmark --list-only` shows the benchmark fixture set without contacting a backend.
- `tc model check` validates the documented manifest example locally.
- `tc model warn` provides warning-only route/manifest comparison visibility and
Expand Down
15 changes: 8 additions & 7 deletions docs/architecture/daily_driver_orchestrator_spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ privacy, routing, and evidence layers.

| Dimension | Readiness | Basis |
|---|---|---|
| Local-first daily driver | ~60% | Governed loop exists in the library (`TriageClient.run_task`); no operator run surface, live probe, or enforced budgets. |
| Local-first daily driver | ~60% | Governed loop is available through `tc run`; live capability signals and enforced budgets remain absent. |
| Frontier-cloud orchestrator | ~35% | Only live cloud backend is `QwenCloudBackend`; Claude/GPT/Gemini exist solely as after-the-fact file handoffs (Codex/Antigravity). |
| Token efficiency | Measured, not enforced | `context_budget.py` / context packs record usage; budgets are advisory, no pre-send compaction is bound into the run path. |

Expand All @@ -40,11 +40,12 @@ after M0 (below) produces daily-use evidence. See **Evidence Requirements**.

## What Already Works (verified in-repo)

- `TriageClient.run_task` runs a full governed loop: verify packet → privacy scan
- `TriageClient.run_task` runs a governed loop: verify packet → privacy scan
(fail-closed) → external-safe packet gate → classify → specialist route → resilience
route (6-way) → local execute, with a real cloud branch (`_execute_cloud_task`) that
executes `QwenCloudBackend` when the route is `cloud_primary/secondary` and the packet is
external-safe.
route. `local_heavy` and `local_fast` execute locally; `cloud_primary/secondary` use the
real cloud branch (`_execute_cloud_task`) only for external-safe packets; `human_handoff`
and the currently unimplemented `deterministic` route return `handoff_required` without
model execution.
- `routing/resilience_router.py` encodes local-first ordering: deterministic → local_fast /
local_heavy → cloud, with high-sensitivity forced to human handoff and cloud blocked for
local-only privacy.
Expand All @@ -54,8 +55,8 @@ after M0 (below) produces daily-use evidence. See **Evidence Requirements**.

## Gaps (planning targets)

- **G1 — No unified operator run surface.** `run_task` is library-only; `tc` has no
`run`/`orchestrate`; `triagecore run-pipeline` is local-only and bypasses the router.
- **G1 — No unified operator run surface.** `tc run` exposes the governed loop, but
`triagecore run-pipeline` remains local-only and bypasses the router.
- **G2 — Cloud is Qwen, not frontier.** No live Claude/GPT/Gemini backends, no provider
abstraction beyond OpenAI-compatible, no per-provider cost/credit model.
- **G3 — Route decisions outrun execution bindings.** `local_heavy`/`local_fast` and
Expand Down
2 changes: 2 additions & 0 deletions docs/change/change_log.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ This file provides a chronological, human-readable record of applied codebase an
*Note: For operational task and run history, consult `.triagecore/ledger.jsonl`.*

## [Unreleased]
- Implemented CR-126 (Preflight Privacy Before Ledger Persistence): Moved `tc run` packet verification ahead of ledger construction and task-event creation, so privacy-blocked input exits before a ledger event is written. Successful runs now persist fixed content-withheld task metadata plus prompt/data lengths rather than raw input. Extended the persistent-artifact invariant and its audit command to reject forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns without echoing detected values. Historical records are not rewritten, and arbitrary free-text classification remains out of scope.
- Implemented CR-125 (Honor Terminal Resilience Routes): Made `human_handoff` and the currently unimplemented `deterministic` route terminal at the governed `TriageClient.run_task` boundary. Each now records the route decision plus a `worker_result` with `worker_result_status=not_attempted`, returns `handoff_required`, and reaches the existing `tc run` exit-3 contract without invoking a local or cloud backend. This slice does not add approval-and-resume behavior, change the broader `human_review_required` semantics, or alter cloud gating, signatures, reducer behavior, or other execution surfaces.
- Implemented CR-124 (Eval Handoff Hygiene, Bug, and Drift Slice): Fixed `write_actual_outcomes()` so generator and other single-pass iterable inputs still write all actual outcome files after duplicate checks, updated stale eval fixture/taxonomy sequencing docs so they no longer imply internal TriageCore scoring, and added focused regression/drift coverage. No evaluator execution, scoring, score interpretation, model/backend calls, routing/admission integration, ledger writes, schema changes, bundle builder, or result import/display were added.
- Implemented CR-123 (Evaluation Handoff Contract): Added a contract-only `docs/evals/evaluation_handoff_contract.md` defining the file-based boundary between TriageCore and the external evaluator suite, including required `eval_case_v0` fixture input, `actual_outcome_export.v0` actual outcome files, deterministic future bundle path vocabulary, and TriageCore-side exit-code expectations. Added focused documentation tests and linked the existing fixture and actual-outcome docs while keeping scoring, evaluator execution, model/backend calls, routing/admission integration, ledger writes, result import/display, and score interpretation out of scope.
- Implemented CR-122 (Eval Fixture Validation CLI): Added `tc eval validate-fixtures --input <path>` as a narrow read-only wrapper around the CR-121 JSONL validator, printing a bounded pass count on valid fixtures and fail-closed, line-aware diagnostics on invalid input. Added synthetic CLI tests only and kept scoring, observed-behavior comparison, model/backend calls, routing/admission integration, ledger writes, runtime behavior, and adversarial/tampering expansion out of scope.
Expand Down
35 changes: 35 additions & 0 deletions docs/change/requests/CR-125-honor-terminal-resilience-routes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# CR-125: Honor Terminal Resilience Routes

## Status

Implemented

## Summary

Make the terminal resilience routes selected by `TriageClient.run_task` control
execution. A `human_handoff` route and the currently unimplemented
`deterministic` route now stop before any backend call, record a metadata-only
`worker_result` with `worker_result_status=not_attempted`, and return the
existing governed handoff result consumed by `tc run` as exit code 3.

## Scope

- Treat `human_handoff` as terminal at the governed execution boundary.
- Treat `deterministic` as terminal until a real deterministic executor exists.
- Preserve route-decision and worker-result evidence for terminal routes.
- Add offline sentinel-backend coverage proving terminal routes do not execute.
- Correct the daily-driver architecture note to match the enforced behavior.

## Non-Goals

- No approval-and-resume workflow.
- No change making every `human_review_required` flag pre-execution blocking.
- No deterministic executor implementation.
- No changes to cloud gating, privacy scanning, signatures, reducer behavior,
other execution surfaces, or historical ledger records.

## Validation

- `python -m pytest -q tests/test_tc_run_cli.py tests/test_local_only_routing.py`
- `python -m pytest -q`
- `git diff --check`
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# CR-126: Preflight Privacy Before Ledger Persistence

## Status

Implemented

## Summary

Preflight the complete `tc run` `TaskPacket` before constructing or writing to
the ledger. Privacy-blocked input now exits with code 2 before a ledger event is
created. Successful runs store a fixed content-withheld task marker and input
lengths instead of raw prompt or data text.

## Scope

- Verify the `tc run` packet before ledger construction and task-event writes.
- Replace persisted `tc run` prompt/data text with fixed metadata and lengths.
- Extend the persistent-artifact invariant with high-confidence PII, credential,
and precise-location value patterns from the privacy scanner.
- Make `tc audit --privacy-invariants` report those value-pattern findings
without echoing content.
- Add offline tests for prompt/data absence and blocked-run no-residue behavior.

## Non-Goals

- No historical ledger rewrite or migration.
- No full DLP engine or arbitrary free-text safety classifier.
- No hash or content fingerprint of prompt/data input.
- No routing, approval, signature, or external-runtime behavior change.

## Validation

- `python -m pytest -q tests/test_tc_run_cli.py tests/test_privacy_invariants.py tests/test_audit_cli.py`
- `python -m pytest -q`
- `git diff --check`
16 changes: 14 additions & 2 deletions docs/current_backlog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,18 @@

## Status

This document summarizes the active TriageCore backlog after CR-115.
This document summarizes the active TriageCore backlog after CR-126.

## Active GitHub Backlog

- CR-126: Preflight Privacy Before Ledger Persistence
- Status: complete via CR-126
- Purpose: Preflight the complete `tc run` packet before any ledger write, replace persisted prompt/data text with fixed metadata and lengths, and extend the persistent-artifact audit to reject forbidden keys plus high-confidence PII, credential, and precise-location value patterns. Historical records are not rewritten, and arbitrary free-text safety classification remains out of scope.

- CR-125: Honor Terminal Resilience Routes
- Status: complete via CR-125
- Purpose: Make `human_handoff` and the currently unimplemented `deterministic` resilience routes terminal at `TriageClient.run_task`: record the route decision and a `worker_result` with `not_attempted`, return `handoff_required`, and ensure `tc run` exits 3 without invoking any backend. This slice does not add an approval-and-resume workflow or make every `human_review_required` flag pre-execution blocking.

- CR-124: Eval Handoff Hygiene, Bug, and Drift Slice
- Status: complete via CR-124
- Purpose: Fix the generator/single-pass iterable bug in actual outcome export writing and clean up stale eval sequencing docs so they point from CR-123 toward bundle/manifest work while preserving the rule that scoring remains external to TriageCore.
Expand Down Expand Up @@ -232,4 +240,8 @@ For operator UX, future slices should focus on reviewability, export polish, and
- **[done] Telemetry schema and synthetic-fixture validation (CR-118)**: Hardened the existing local backend probe's serialized record contract before any further probe work — strict schema, pure validator, synthetic fixtures only, no endpoint calls, no routing integration, no ledger writes, and no CLI behavior changes.
- **[done] Local backend telemetry probe validation gate (CR-119)**: Every emitted local backend probe result now validates against the CR-118 record contract before it is treated as an observation; validation failure raises a fail-closed `ProbeInputError`, with no generation calls, routing integration, or ledger writes.
- **[done] Reviewer checkpoint or release-hygiene slice (CR-120, docs-only)**: Froze the completed CR-117 through CR-119 lane in a concise operations note, corrected stale CR-114 telemetry wording, and kept future telemetry work behind a new explicit scope pass instead of adding more features.
- **Next slice TBD by scope pass**: Do not start routing integration, ledger integration, circuit breakers, automatic discovery, background polling, daily-driver enforcement, or additional telemetry behavior without a new approved CR.
- **[done] Evaluation handoff contract (CR-123, contract-only)**: Defined the file-based boundary between TriageCore-produced fixtures/actuals and external scoring, including deterministic future bundle path vocabulary.
- **[done] Eval handoff hygiene (CR-124)**: Fixed generator-backed actual-outcome writing and removed stale language that implied internal TriageCore scoring.
- **[done] Honor terminal resilience routes (CR-125)**: `human_handoff` and currently unimplemented `deterministic` routes now return a governed handoff before backend execution, recording `worker_result_status=not_attempted`; `tc run` reports the valid handoff with exit code 3. Approval-and-resume behavior, broader `human_review_required` semantics, and other execution seams remain future work.
- **[done] Preflight privacy before ledger persistence (CR-126)**: `tc run` now scans its complete packet before opening the ledger, persists only metadata and input lengths, and extends the persistent artifact audit with high-confidence sensitive-value detection. Historical records remain unchanged; arbitrary free-text safety classification and a full DLP engine remain out of scope.
- **Next slice requires a new approved CR**: The eval lane's next bounded candidate is a deterministic bundle/manifest builder for already-validated fixtures and already-exported actuals. Do not add scoring or score interpretation inside TriageCore, and do not start approval-and-resume behavior, routing integration beyond the governed path, ledger integration, circuit breakers, automatic discovery, background polling, or additional telemetry behavior without a new approved CR.
2 changes: 1 addition & 1 deletion docs/evidence_schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The schema supports two project goals:
| `created_at` | Timestamp when the task entered the ledger. |
| `updated_at` | Timestamp of the most recent ledger event applied to the task record. |
| `title` | Human-readable task title. |
| `description` | Prompt or task summary. |
| `description` | Operator-safe task summary. `tc run` stores a fixed content-withheld marker rather than prompt text. |
| `target_files` | Files or artifacts in scope. |
| `runner` | Execution path, such as `local_benchmark`, `local_llm`, `council`, `codex`, or `antigravity`. |
| `status` | Current state: `pending`, `local_draft_generated`, `handoff_generated`, `reviewed`, or `blocked`. |
Expand Down
8 changes: 4 additions & 4 deletions docs/operations/reviewer-traceability.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,17 @@ A completed trace shows that the recorded evidence is present, linked by `task_i

## The Trace Chain

1. **Input.** A task enters as a `TaskPacket` ([task_packet.py](../../triage_core/task_packet.py)) with explicit privacy metadata, or as a task envelope ([task_envelope.py](../../triage_core/task_envelope.py)). Raw prompt and data content are never persisted to the ledger.
1. **Input.** A task enters as a `TaskPacket` ([task_packet.py](../../triage_core/task_packet.py)) with explicit privacy metadata, or as a task envelope ([task_envelope.py](../../triage_core/task_envelope.py)). `tc run` preflights the packet before opening the ledger and persists prompt/data lengths rather than their contents.
2. **Route decision and evidence record.** `TriageClient.run_task` ([client.py](../../triage_core/client.py)) privacy-scans the packet and appends metadata-only events to the append-only ledger at `.triagecore/ledger.jsonl` via `TaskLedger` ([task_ledger.py](../../triage_core/task_ledger.py)): a `route_audit` event for the allow/block outcome, a `route_decision` event for the routing choice (optionally signed), then `worker_result` and `validator_completed` events for execution and validation outcomes.
3. **Reduced record.** `TaskLedger._apply_event` folds all events for one `task_id` into a single `TaskRecord`. The field meanings are documented in [evidence_schema.md](../evidence_schema.md).
4. **Approval or rejection state.** When classification, routing, or a handoff sets `human_review_required`, the task appears in the review queue ([review_queue.py](../../triage_core/review_queue.py), `tc review list`) until a `review_completed` event records `review_decision` (`accepted`, `accepted_with_minor_edits`, or `rejected`) and `task_outcome`.
5. **Verification and test evidence.** `tc audit --privacy-invariants` checks that persisted events contain no forbidden raw-content keys; `tc audit --verify-signatures` checks signed event provenance; the pytest suite (including the traceability regression test below) is the repeatable test evidence.
5. **Verification and test evidence.** `tc audit --privacy-invariants` checks persisted events for forbidden raw-content keys and high-confidence PII, credential, and precise-location value patterns; it does not prove that arbitrary free text is safe. `tc audit --verify-signatures` checks signed event provenance; the pytest suite (including the traceability regression test below) is the repeatable test evidence.

## Ledger Event Types That Carry the Trace

| Event type | Typically appended by | Trace evidence it contributes |
| --- | --- | --- |
| `task_created` | `TriageClient.run_task`, benchmark/pipeline runners | `created_at`, `title`, `description`, `target_files` |
| `task_created` | `TriageClient.run_task`, benchmark/pipeline runners | `created_at`, metadata-only title/description, prompt/data lengths, `target_files` |
| `task_classified` | classification step in `run_task` | `risk_level`, `permission_profile`; medium/high risk sets `human_review_required` |
| `route_audit` | privacy/route gate in `run_task` ([route_audit.py](../../triage_core/route_audit.py)) | allow/block `decision` and `reason_code`; inspected as raw events via `tc audit --kind route_audit`, not reduced into named `TaskRecord` fields |
| `route_decision` | `run_task` via [routing/route_events.py](../../triage_core/routing/route_events.py); optionally signed | `selected_route`, `route_reason`, `route_source`, `fallback_depth`, `selected_backend`, model; can set `human_review_required` |
Expand Down Expand Up @@ -61,7 +61,7 @@ tc audit --kind route_decision --last 10 # routing choices and route source
tc audit --kind worker_result --last 10 # execution outcome metadata
tc audit --kind review_completed --last 10 # recorded approval/rejection state
tc review list # tasks still awaiting a review decision
tc audit --privacy-invariants # persisted evidence contains no raw content
tc audit --privacy-invariants # checks prohibited keys and high-confidence sensitive values
tc audit --verify-signatures --kind route_decision # provenance check for signed events, when present
tc task show <task_id> # show one task's complete evidence chain (displays ledger evidence, does not verify signatures)
```
Expand Down
27 changes: 27 additions & 0 deletions tests/test_audit_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -559,6 +559,33 @@ def test_audit_privacy_invariants_fails_without_echoing_sensitive_values(tmp_pat
assert sensitive_value not in out


def test_audit_privacy_invariants_flags_sensitive_value_without_echoing_it(tmp_path, capsys):
ledger_path = tmp_path / "ledger.jsonl"
sensitive_value = "Contact jane@example.com"
ledger_path.write_text(
json.dumps(
{
"timestamp": "2026-06-12T00:00:00+00:00",
"task_id": "unsafe-task",
"event_type": "task_created",
"payload": {"description": sensitive_value},
}
)
+ "\n",
encoding="utf-8",
)

with patch("triage_core.tc_cli._ledger_path", return_value=ledger_path):
with pytest.raises(SystemExit) as exc:
tc_audit_privacy_invariants()

assert exc.value.code == 1
out = capsys.readouterr().out
assert "$.payload.description" in out
assert "reason=email_pattern" in out
assert sensitive_value not in out


def test_audit_privacy_invariants_flags_malformed_json(tmp_path, capsys):
ledger_path = tmp_path / "ledger.jsonl"
ledger_path.write_text("{not json\n", encoding="utf-8")
Expand Down
Loading
Loading