Ship historical evals

.github/workflows/evals.yml

@@ -0,0 +1,38 @@
+name: Evals
+
+on:
+  pull_request:
+    paths:
+      - "procedures/**"
+      - "src/agents/specialists/**"
+      - "src/agents/supervisor/**"
+      - "src/evals/**"
+      - "src/llm/**"
+      - "src/procedures/**"
+      - "scripts/ranse.ts"
+      - ".github/workflows/evals.yml"
+
+jobs:
+  procedure-evals:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: oven-sh/setup-bun@v2
+      - run: bun install --frozen-lockfile
+      - run: bun run typecheck
+      - name: Run procedure evals
+        run: |
+          shopt -s nullglob
+          for file in procedures/*.yaml procedures/*.yml procedures/*.json; do
+            bun scripts/ranse.ts eval "$file"
+          done
+      - name: Run hosted historical evals when configured
+        env:
+          RANSE_APP_URL: ${{ secrets.RANSE_APP_URL }}
+          RANSE_COOKIE: ${{ secrets.RANSE_COOKIE }}
+        run: |
+          if [ -n "$RANSE_APP_URL" ] && [ -n "$RANSE_COOKIE" ]; then
+            bun scripts/ranse.ts eval --app-url "$RANSE_APP_URL" --cookie "$RANSE_COOKIE" --ci
+          else
+            echo "::notice::Set RANSE_APP_URL and RANSE_COOKIE secrets to gate PRs on hosted historical evals."
+          fi

README.md

@@ -15,6 +15,7 @@ Ranse turns support email into a real-time, multi-agent support workspace built
 - **Multi-agent assist** — triage, summarization, knowledge retrieval, reply drafting, escalation, SLA.
 - **Human approval gates** for every outbound reply, with edit-before-send.
 - **Multi-provider LLM** — Workers AI out of the box; drop-in Anthropic, OpenAI, Google, Grok, OpenRouter via AI Gateway.
+- **Historical evals** — resolved conversations become anonymized replay cases; `ranse eval` catches prompt/procedure regressions before they ship.
 - **One-click deploy** to your own Cloudflare account — customer-owned from day one.
 - **Open source** (Apache-2.0).
 
@@ -83,7 +84,7 @@ Then open http://localhost:5173 (or http://localhost:8787 for the Worker directl
 
 Ranse is heading from "AI-assisted shared inbox" to a full autonomous customer-service agent — but not as an OSS clone of [Fin](https://fin.ai/) or [Decagon](https://decagon.ai/). The goal is the agent those products *structurally cannot become*: sovereign by construction, per-step model choice, procedures-as-code, MCP-native actions, eval-first against your own ticket history, and a forkable procedure library.
 
-The shape, in short: **retrieval → workspace management → agentic retrieval → autonomous resolution → procedures → MCP actions → evals → procedure library → insights → multi-channel.** Phase 0 (bootstrap, inbound email, supervisor DO, draft + approval), Phase 1 (retrieval foundations), Phase 1.5 (workspace management & tenant isolation), Phase 2 (agentic multi-hop retrieval), Phase 3 (autonomous resolution), Phase 4 (procedures as code), and Phase 5 (MCP-native actions) are shipped.
+The shape, in short: **retrieval → workspace management → agentic retrieval → autonomous resolution → procedures → MCP actions → evals → procedure library → insights → multi-channel.** Phase 0 (bootstrap, inbound email, supervisor DO, draft + approval), Phase 1 (retrieval foundations), Phase 1.5 (workspace management & tenant isolation), Phase 2 (agentic multi-hop retrieval), Phase 3 (autonomous resolution), Phase 4 (procedures as code), Phase 5 (MCP-native actions), and Phase 6 (historical evals) are shipped.
 
 Full pipeline, principles, and how to contribute to a phase: **[docs/roadmap.md](docs/roadmap.md)**. It's directional, not committed — if you want to work on something further down the list, open a discussion and we'll happily reorder.
 

docs/architecture.md

@@ -52,7 +52,7 @@ triageAndDraft (runs in DO alarm, async)
 | System | Purpose |
 |---|---|
 | DO SQLite | Workspace state, mailbox counters, BYOK-encrypted secrets |
-| D1 | Tickets, messages, audit, approvals, outcomes, feedback, daily rollups, users, sessions, knowledge, LLM config, procedures, MCP registry/tool calls |
+| D1 | Tickets, messages, audit, approvals, outcomes, feedback, daily rollups, users, sessions, knowledge, LLM config, procedures, MCP registry/tool calls, eval cases/runs/results |
 | R2 | Raw MIME, text/html bodies, attachments, exports |
 | KV | Rate limits, idempotency, lightweight flags |
 | Vectorize | Per-workspace knowledge chunk embeddings |
@@ -101,6 +101,31 @@ ProcedureRunnerAgent
 
 MCP server secrets are stored in `UserSecretsStore` under `mcp:<serverId>`. D1 stores endpoint metadata, discovered tool schemas, annotations, guardrail configuration, and immutable call records.
 
+## Eval flow
+
+```
+Ticket is resolved
+  ├─ setTicketStatus(resolved|closed), autonomous outcome, or procedure status step
+  ├─ captureResolvedTicketEvalCase
+  │    ├─ load ticket + inbound/outbound transcript
+  │    ├─ anonymize email, phone, and requester-name fields
+  │    ├─ reject capture if residual PII remains
+  │    └─ upsert eval_case(source = resolved_ticket)
+  └─ audit eval.case_captured
+
+ranse eval / Settings -> Evals
+  ├─ create eval_run
+  ├─ for each active eval_case
+  │    ├─ agenticSearchKnowledge with current retrieval prompts/config
+  │    ├─ runDraft with current draft prompt/model config
+  │    ├─ score reply overlap, required terms, confidence signals
+  │    ├─ compare against latest prior baseline result
+  │    └─ insert eval_result
+  └─ mark eval_run passed/failed
+```
+
+Procedure evals are local and deterministic. `ranse eval <procedure-file>` loads the spec, runs each inline `evals[]` case through `simulateProcedure`, and checks expected status, context paths, and step order before a PR is merged.
+
 ## Scaling model
 
 - One `WorkspaceSupervisorAgent` DO per workspace. The email handler pins by `idFromName(workspaceId)` so all events for a workspace funnel through one instance — consistent state, no cross-DO coordination needed.

docs/operations.md

@@ -37,6 +37,29 @@ Operational checks:
 - Use per-ticket and per-hour limits to protect internal systems from procedure loops.
 - Inspect `mcp_tool_call` for action history and `audit_event` actions prefixed with `mcp.tool_call_` for timeline-level observability.
 
+## Historical evals
+
+Resolved tickets are the regression suite. When an operator marks a ticket `resolved` or `closed`, an autonomous reply records a resolved outcome, or a procedure resolves a ticket, Ranse captures the inbound/outbound transcript into an anonymized `eval_case` if the conversation has both a customer message and a support reply. Operators can also backfill recent cases from **Settings → Evals**.
+
+Capture fails closed if the anonymized payload still contains residual non-placeholder email, phone, or requester-name data. No eval case is written until the redaction rules are safe for that conversation.
+
+CLI workflow:
+
+```bash
+# Run inline evals shipped alongside a procedure spec
+bun scripts/ranse.ts eval procedures/refund-intake.yaml
+
+# Backfill resolved conversations from a deployed workspace
+bun scripts/ranse.ts eval capture-resolved --app-url "$RANSE_APP_URL" --cookie "$RANSE_COOKIE" --limit 100
+
+# Replay active historical cases through current retrieval + drafting
+bun scripts/ranse.ts eval --app-url "$RANSE_APP_URL" --cookie "$RANSE_COOKIE" --threshold 0.35 --score-drop 0.15 --ci
+```
+
+Eval runs write `eval_run` and `eval_result` rows with assertion details. A run fails when any active case fails or regresses. `regression_count` is reserved for cases with a prior baseline that got worse: previous pass → current fail, or a score drop larger than the configured threshold. Archive noisy cases from **Settings → Evals** instead of deleting them.
+
+The bundled GitHub Actions workflow always runs procedure evals for relevant PRs; set `RANSE_APP_URL` and `RANSE_COOKIE` repository secrets to make hosted historical replay part of the gate.
+
 ## Escalations
 
 The `EscalationAgent` runs on demand. It returns `{ should_escalate, severity, route_to }` and the operator (or an automation rule) picks the handoff target.

docs/roadmap.md

@@ -49,7 +49,9 @@ Most "AI agent" tools are chat shaped and bolt email on. Real B2B support lives
 
 **Phase 5 — MCP-native actions** is shipped. Workspaces can register remote Streamable HTTP MCP servers, discover tools, enforce per-tool guardrails, pause destructive calls for operator approval, and execute `call_action` from procedures with audit-backed tool-call records.
 
-That's now a retrieval-grounded early Fin **Copilot** equivalent with workspace isolation, traceable multi-hop retrieval, a conservative autonomous-send path, a procedure-driven agent loop, and external action execution through the open MCP protocol. Everything below continues the path toward evals, procedure sharing, and insights.
+**Phase 6 — Historical evals** is shipped. Resolved tickets are captured as anonymized replay cases, operators can backfill and run evals from Settings, `ranse eval` runs procedure-file and hosted historical suites, and PRs touching prompts/procedures/model logic have an eval workflow.
+
+That's now a retrieval-grounded early Fin **Copilot** equivalent with workspace isolation, traceable multi-hop retrieval, a conservative autonomous-send path, a procedure-driven agent loop, external action execution through the open MCP protocol, and a regression gate against the workspace's own ticket history. Everything below continues the path toward procedure sharing and insights.
 
 ## Phase 1 — Retrieval foundations
 **Status: shipped.**
@@ -117,7 +119,7 @@ Single-pass RAG fails on the hard tickets — the ones where the customer's ques
 - **Per-hop model routing** (Principle 2): shipped. Planning, judging, rewriting, reranking, and drafting each have independent action keys in workspace model settings.
 - **`search` as a procedure primitive** (Principle 3): shipped. Phase 4 procedures can call `search(query, scope, max_hops)` with the same traceable retrieval loop.
 - **Answer Inspection shows the trace**: shipped. Drafts, approval suggestions, and Content Library test searches show hop query → results → judgment → next-query.
-- **Eval cases test the loop, not just the final answer** (Principle 5): partially shipped as behavior tests around the loop contract. Historical replay belongs in Phase 6.
+- **Eval cases test the loop, not just the final answer** (Principle 5): behavior tests cover the loop contract, and Phase 6 historical replay now runs resolved-ticket cases end-to-end.
 
 The `customer_data` search scope still fails closed with an explicit trace; procedures can now retrieve account state by calling registered MCP tools directly through `call_action`.
 
@@ -165,13 +167,19 @@ The `customer_data` search scope still fails closed with an explicit trace; proc
 - First-party templates are in-repo for Stripe, Shopify, GitHub, Linear, and generic webhook MCP servers so workspaces can register their own protocol-native endpoints without adding connector code to Ranse.
 
 ## Phase 6 — Eval against your own ticket history
+**Status: shipped.**
+
 *Principle 5*
 
-- Every resolved conversation captured as a replayable eval case (anonymized, configurable PII rules)
-- `ranse eval` runs all eval cases against current prompts + procedures, reports regressions
-- CI integration: PRs touching `procedures/`, `prompts/`, or model config gate on eval pass
-- Historical replay is the primary signal; synthetic-conversation generation is a complement, not a substitute
-- Per-procedure eval harness: edge cases, refusal cases, escalation cases all live alongside the procedure file
+- Resolved or closed conversations are captured into `eval_case` with transcript, latest customer message, expected reply preview, outcome kinds, and deterministic source fingerprints.
+- Capture runs automatically when an operator marks a ticket `resolved` or `closed`, when autonomous resolution succeeds, and when a procedure resolves a ticket. Backfill uses both ticket status and resolved outcome events.
+- PII anonymization is applied before persistence, with configurable email, phone, and requester-name redaction rules. Residual PII detection fails closed instead of storing a case when redaction leaves sensitive data behind.
+- Hosted eval runs replay active historical cases through current retrieval + draft logic, score overlap and required terms, compare against the previous baseline result, and store per-case assertions in `eval_result`.
+- Regression gates distinguish first-run failures from true regressions: a previous pass that now fails, or a score drop beyond the configured threshold, increments `regression_count`.
+- Operators can archive noisy historical cases from **Settings → Evals** without deleting the audit trail.
+- `ranse eval <procedure-file>` runs inline procedure evals from the YAML/JSON/TS spec. Edge, refusal, escalation, and wait/resume expectations live next to the procedure.
+- `.github/workflows/evals.yml` gates PRs touching procedure, prompt/model, supervisor, or eval code on local procedure evals; with `RANSE_APP_URL` and `RANSE_COOKIE` secrets, the same workflow also gates on hosted historical replay.
+- Historical replay is the primary signal; synthetic-conversation generation remains a future complement, not a substitute.
 
 ## Phase 7 — Procedure library + community
 *Principle 6*

docs/security.md

@@ -53,6 +53,12 @@ Default policy: AI may classify, summarize, search, and draft. It **cannot** sen
 - Guardrails are enforced in the procedure runner before the MCP request leaves Ranse: disabled tools, per-ticket/hour call limits, dollar limits, and allowed customer segments.
 - Every MCP attempt writes a `mcp_tool_call` row plus an audit event. The request includes a Ranse idempotency key in MCP `_meta` so well-behaved tools can suppress duplicate writes.
 
+## Eval data privacy
+
+Historical eval cases are stored in the workspace's own D1 database and never leave the customer's Cloudflare account unless the operator exports or queries them. Before persistence, resolved-ticket capture redacts email addresses, phone numbers, and the requester's display name by default. Capture then scans the anonymized payload for residual non-placeholder email, phone, or requester-name data and skips the case if any remains. The stored `anonymization_json` records rules and counts only; it does not store the original PII mapping.
+
+Eval access is restricted to `owner` and `admin` workspace roles because even anonymized conversations can contain commercially sensitive support context. Hosted replay uses the same provider-key and model-routing path as normal drafting, so self-hosters keep the same BYOK and Cloudflare tenant boundaries.
+
 ## Audit trail
 
 Every state change writes an `audit_event`:

migrations/20260518_000000_eval_history.sql

@@ -0,0 +1,70 @@
+-- Phase 6 historical evals.
+
+CREATE TABLE IF NOT EXISTS eval_case (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  source TEXT NOT NULL CHECK(source IN ('resolved_ticket','procedure_spec','synthetic','api')),
+  ticket_id TEXT,
+  procedure_id TEXT,
+  procedure_version_id TEXT,
+  name TEXT NOT NULL,
+  status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active','archived')),
+  input_json TEXT NOT NULL,
+  expected_json TEXT NOT NULL,
+  anonymization_json TEXT NOT NULL DEFAULT '{}',
+  source_fingerprint TEXT NOT NULL,
+  captured_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  UNIQUE(workspace_id, source, source_fingerprint),
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE,
+  FOREIGN KEY (ticket_id) REFERENCES ticket(id) ON DELETE SET NULL,
+  FOREIGN KEY (procedure_id) REFERENCES procedure(id) ON DELETE SET NULL,
+  FOREIGN KEY (procedure_version_id) REFERENCES procedure_version(id) ON DELETE SET NULL
+);
+
+CREATE INDEX IF NOT EXISTS idx_eval_case_workspace
+  ON eval_case(workspace_id, status, captured_at DESC);
+
+CREATE INDEX IF NOT EXISTS idx_eval_case_ticket
+  ON eval_case(workspace_id, ticket_id);
+
+CREATE TABLE IF NOT EXISTS eval_run (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  source TEXT NOT NULL DEFAULT 'api' CHECK(source IN ('api','cli','ci','scheduled')),
+  status TEXT NOT NULL CHECK(status IN ('running','passed','failed')),
+  case_count INTEGER NOT NULL DEFAULT 0,
+  passed_count INTEGER NOT NULL DEFAULT 0,
+  failed_count INTEGER NOT NULL DEFAULT 0,
+  regression_count INTEGER NOT NULL DEFAULT 0,
+  config_json TEXT NOT NULL DEFAULT '{}',
+  started_at INTEGER NOT NULL,
+  completed_at INTEGER,
+  created_at INTEGER NOT NULL,
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_eval_run_workspace
+  ON eval_run(workspace_id, created_at DESC);
+
+CREATE TABLE IF NOT EXISTS eval_result (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  run_id TEXT NOT NULL,
+  case_id TEXT NOT NULL,
+  status TEXT NOT NULL CHECK(status IN ('passed','failed','skipped')),
+  score REAL,
+  assertions_json TEXT NOT NULL DEFAULT '[]',
+  actual_json TEXT NOT NULL DEFAULT '{}',
+  error TEXT,
+  created_at INTEGER NOT NULL,
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE,
+  FOREIGN KEY (run_id) REFERENCES eval_run(id) ON DELETE CASCADE,
+  FOREIGN KEY (case_id) REFERENCES eval_case(id) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_eval_result_run
+  ON eval_result(run_id, status);
+
+CREATE INDEX IF NOT EXISTS idx_eval_result_case
+  ON eval_result(workspace_id, case_id, created_at DESC);

procedures/refund-intake.yaml

@@ -22,3 +22,6 @@ evals:
         subject: Refund request
     expect:
       status: completed
+      steps:
+        - find_policy
+        - add_context_note