Ship procedure library

.github/workflows/evals.yml

@@ -20,12 +20,16 @@ jobs:
       - uses: oven-sh/setup-bun@v2
       - run: bun install --frozen-lockfile
       - run: bun run typecheck
+      - name: Validate procedure library manifest
+        run: bun scripts/ranse.ts procedure validate-library
       - name: Run procedure evals
         run: |
           shopt -s nullglob
           for file in procedures/*.yaml procedures/*.yml procedures/*.json; do
             bun scripts/ranse.ts eval "$file"
           done
+      - name: Validate procedure library
+        run: bunx vitest run tests/procedure-library.test.ts
       - name: Run hosted historical evals when configured
         env:
           RANSE_APP_URL: ${{ secrets.RANSE_APP_URL }}

CONTRIBUTING.md

@@ -27,6 +27,13 @@ bun run dev
 - Keep commits focused. One logical change per PR.
 - Include a before/after description in the PR body if the change affects UX, APIs, or the setup flow.
 
+## Procedure library contributions
+
+- Start from `procedure-library/README.md`, `src/procedures/library-data.ts`, and `src/procedures/library-mcp-tools.ts`.
+- Every library procedure must include inline `evals`, a generic owner of `ranse-library`, deterministic provenance, and reference MCP tool specs for external system assumptions. Required MCP references must be exercised by `call_action` steps, and write/destructive actions must stay behind approval.
+- Run `bun scripts/ranse.ts procedure validate-library`, `bun scripts/ranse.ts procedure add <slug> --dir /tmp/ranse-procs --force`, `bun scripts/ranse.ts eval /tmp/ranse-procs/<slug>.yaml`, and `bunx vitest run tests/procedure-library.test.ts`.
+- Do not bake customer-specific policy text, private route names, or proprietary tool names into shared library procedures.
+
 ## Commit messages
 
 Use terse Conventional Commit-style subjects:

README.md

@@ -16,6 +16,8 @@ Ranse turns support email into a real-time, multi-agent support workspace built
 - **Human approval gates** for every outbound reply, with edit-before-send.
 - **Multi-provider LLM** — Workers AI out of the box; drop-in Anthropic, OpenAI, Google, Grok, OpenRouter via AI Gateway.
 - **Historical evals** — resolved conversations become anonymized replay cases; `ranse eval` catches prompt/procedure regressions before they ship.
+- **Forkable procedure library** — install vetted support workflows with evals, provenance checksums, and MCP reference contracts, then customize them in your repo.
+- **Insights loop** — score conversations, surface evidence-backed unresolved intents, draft reviewable KB suggestions with lineage, and detect source-specific drift from successful replies.
 - **One-click deploy** to your own Cloudflare account — customer-owned from day one.
 - **Open source** (Apache-2.0).
 
@@ -84,7 +86,7 @@ Then open http://localhost:5173 (or http://localhost:8787 for the Worker directl
 
 Ranse is heading from "AI-assisted shared inbox" to a full autonomous customer-service agent — but not as an OSS clone of [Fin](https://fin.ai/) or [Decagon](https://decagon.ai/). The goal is the agent those products *structurally cannot become*: sovereign by construction, per-step model choice, procedures-as-code, MCP-native actions, eval-first against your own ticket history, and a forkable procedure library.
 
-The shape, in short: **retrieval → workspace management → agentic retrieval → autonomous resolution → procedures → MCP actions → evals → procedure library → insights → multi-channel.** Phase 0 (bootstrap, inbound email, supervisor DO, draft + approval), Phase 1 (retrieval foundations), Phase 1.5 (workspace management & tenant isolation), Phase 2 (agentic multi-hop retrieval), Phase 3 (autonomous resolution), Phase 4 (procedures as code), Phase 5 (MCP-native actions), and Phase 6 (historical evals) are shipped.
+The shape, in short: **retrieval → workspace management → agentic retrieval → autonomous resolution → procedures → MCP actions → evals → procedure library → insights → multi-channel.** Phase 0 (bootstrap, inbound email, supervisor DO, draft + approval), Phase 1 (retrieval foundations), Phase 1.5 (workspace management & tenant isolation), Phase 2 (agentic multi-hop retrieval), Phase 3 (autonomous resolution), Phase 4 (procedures as code), Phase 5 (MCP-native actions), Phase 6 (historical evals), Phase 7 (procedure library), and Phase 8 (insights & auto-improving KB) are shipped.
 
 Full pipeline, principles, and how to contribute to a phase: **[docs/roadmap.md](docs/roadmap.md)**. It's directional, not committed — if you want to work on something further down the list, open a discussion and we'll happily reorder.
 

docs/architecture.md

@@ -16,6 +16,7 @@ Function-based, not DOs. They live in `src/agents/specialists/` and return struc
 - `triage` — category, priority, sentiment, language, spam detection.
 - `summarize` — thread summary + next-step hint.
 - `knowledge` — manual/URL/PDF/resolved-ticket ingestion, Workers AI embeddings, Vectorize search, reranking, and keyword fallback.
+- `insights` — conversation rubric scoring, aggregate operational metrics, unresolved-intent KB suggestions, and knowledge drift detection.
 - `draft` — generate a reply with citations; flag review risks.
 - `escalation` — decide whether to route to a human/team.
 - `sla` — deterministic, no LLM; computes breach status.
@@ -52,7 +53,7 @@ triageAndDraft (runs in DO alarm, async)
 | System | Purpose |
 |---|---|
 | DO SQLite | Workspace state, mailbox counters, BYOK-encrypted secrets |
-| D1 | Tickets, messages, audit, approvals, outcomes, feedback, daily rollups, users, sessions, knowledge, LLM config, procedures, MCP registry/tool calls, eval cases/runs/results |
+| D1 | Tickets, messages, audit, approvals, outcomes, feedback, daily rollups, users, sessions, knowledge, LLM config, procedures, MCP registry/tool calls, eval cases/runs/results, conversation scores, KB suggestions, drift signals |
 | R2 | Raw MIME, text/html bodies, attachments, exports |
 | KV | Rate limits, idempotency, lightweight flags |
 | Vectorize | Per-workspace knowledge chunk embeddings |
@@ -126,6 +127,44 @@ ranse eval / Settings -> Evals
 
 Procedure evals are local and deterministic. `ranse eval <procedure-file>` loads the spec, runs each inline `evals[]` case through `simulateProcedure`, and checks expected status, context paths, and step order before a PR is merged.
 
+## Procedure library flow
+
+```
+Settings -> Procedures
+  ├─ GET /api/procedures/library
+  │    └─ compare library MCP contracts against discovered workspace MCP tools
+  ├─ GET /api/procedures/library/manifest
+  ├─ POST /api/procedures/library/:slug/install
+  └─ upsertProcedureVersion(source_kind = seed, source_ref = library:<slug>@<version>#sha256:<checksum>)
+
+ranse procedure add <slug>
+  ├─ read built-in catalog from src/procedures/library-data.ts
+  ├─ validate inline evals
+  ├─ write procedures/<slug>.yaml
+  ├─ write procedures/<slug>.mcp.json
+  └─ write procedures/<slug>.provenance.json
+```
+
+The built-in catalog is code, not database state, so deploys carry the exact procedure specs, evals, and reference MCP contracts reviewed in git. List/detail responses include deterministic SHA-256 provenance, the Ranse procedure schema version, MCP readiness for the selected workspace, and the MCP schema version used for reference ToolAnnotations. Validation requires each required MCP reference to be exercised by a `call_action` step; write and destructive actions cannot opt out of approval.
+
+## Insights loop
+
+```
+Weekly cron / manual refresh
+  ├─ score recent tickets on groundedness, tone, resolution, and customer effort
+  ├─ aggregate resolution, follow-up, feedback, unresolved-intent, and procedure-latency metrics
+  ├─ cluster repeated unresolved conversations into confidence-scored KB article suggestions
+  └─ compare cited KB sources against successful replies for source-specific drift signals
+
+Insights page
+  ├─ POST /api/insights/scores/run
+  ├─ POST /api/insights/kb-suggestions/run
+  ├─ POST /api/insights/kb-suggestions/:id/accept
+  └─ POST /api/insights/drift/run
+```
+
+Suggestions are review records, not automatic content edits. They require repeated unresolved-ticket evidence, store confidence and source-ticket lineage, and accepted suggestions become terminal records linked to the manual knowledge source created through the same ingestion path as the Content Library.
+
 ## Scaling model
 
 - One `WorkspaceSupervisorAgent` DO per workspace. The email handler pins by `idFromName(workspaceId)` so all events for a workspace funnel through one instance — consistent state, no cross-DO coordination needed.

docs/operations.md

@@ -60,6 +60,28 @@ Eval runs write `eval_run` and `eval_result` rows with assertion details. A run
 
 The bundled GitHub Actions workflow always runs procedure evals for relevant PRs; set `RANSE_APP_URL` and `RANSE_COOKIE` repository secrets to make hosted historical replay part of the gate.
 
+## Procedure library
+
+Owners and admins can install vetted procedure templates from **Settings → Procedures**. The catalog shows whether the selected workspace has the required MCP servers and tools discovered before install. Installed library procedures are published as immutable procedure versions with `source_ref = library:<slug>@<version>#sha256:<checksum>`.
+
+Local fork workflow:
+
+```bash
+bun scripts/ranse.ts procedure list
+bun scripts/ranse.ts procedure manifest
+bun scripts/ranse.ts procedure validate-library
+bun scripts/ranse.ts procedure add shipping-dispute --dir procedures
+bun scripts/ranse.ts eval procedures/shipping-dispute.yaml
+```
+
+The CLI writes the procedure spec, `<slug>.mcp.json`, and `<slug>.provenance.json`. The provenance file records the library version, source ref, procedure SHA-256 checksum, and standards metadata used when the procedure was forked. Treat the MCP specs as contracts: each required reference is exercised by a `call_action` step, read-only tools may run automatically, and write/destructive tools must remain behind an approval gate unless you deliberately rework the procedure and its evals.
+
+## Insights
+
+Owners and admins can open **Insights** to refresh conversation scores, unresolved-intent KB suggestions, and knowledge drift signals. The weekly cron `17 3 * * 1` runs the same maintenance loop automatically inside the worker.
+
+KB suggestions are generated only from repeated unresolved-ticket clusters and include evidence count, confidence, suggested terms, and source-ticket IDs. Accepted suggestions are idempotently published as manual knowledge sources, linked back through `accepted_source_id`, and then treated as terminal audit records. Dismissed suggestions and resolved drift signals remain in D1 for auditability instead of being deleted.
+
 ## Escalations
 
 The `EscalationAgent` runs on demand. It returns `{ should_escalate, severity, route_to }` and the operator (or an automation rule) picks the handoff target.

docs/roadmap.md

@@ -51,7 +51,11 @@ Most "AI agent" tools are chat shaped and bolt email on. Real B2B support lives
 
 **Phase 6 — Historical evals** is shipped. Resolved tickets are captured as anonymized replay cases, operators can backfill and run evals from Settings, `ranse eval` runs procedure-file and hosted historical suites, and PRs touching prompts/procedures/model logic have an eval workflow.
 
-That's now a retrieval-grounded early Fin **Copilot** equivalent with workspace isolation, traceable multi-hop retrieval, a conservative autonomous-send path, a procedure-driven agent loop, external action execution through the open MCP protocol, and a regression gate against the workspace's own ticket history. Everything below continues the path toward procedure sharing and insights.
+**Phase 7 — Procedure library** is shipped. Workspaces can install vetted workflows from Settings, fork them locally with `ranse procedure add`, and inspect reference MCP tool contracts plus inline evals before customization.
+
+**Phase 8 — Insights & auto-improving KB** is shipped. Workspaces get conversation rubric scoring, aggregate insight dashboards, unresolved-intent KB suggestions, accepted-suggestion publishing into the knowledge base, drift signals against successful replies, and weekly scheduled insight maintenance.
+
+That's now a retrieval-grounded early Fin **Copilot** equivalent with workspace isolation, traceable multi-hop retrieval, a conservative autonomous-send path, a procedure-driven agent loop, external action execution through the open MCP protocol, a regression gate against the workspace's own ticket history, a forkable procedure library, and a sovereign insights loop that turns real support history into reviewed KB improvements. Everything below continues the path toward multi-channel surfaces.
 
 ## Phase 1 — Retrieval foundations
 **Status: shipped.**
@@ -182,20 +186,29 @@ The `customer_data` search scope still fails closed with an explicit trace; proc
 - Historical replay is the primary signal; synthetic-conversation generation remains a future complement, not a substitute.
 
 ## Phase 7 — Procedure library + community
+**Status: shipped.**
+
 *Principle 6*
 
-- Public repo `getranse/procedures-library` — refund flow, password reset, shipping dispute, subscription cancellation, fraud triage, GDPR data request, etc.
-- `ranse procedure add <name>` clones from library into workspace repo as a starting point
-- Each library procedure ships with eval cases and a reference MCP tool spec
-- Contribution guidelines for upstreaming generic procedures back from workspaces
+- Built-in catalog ships refund intake, password reset, shipping dispute, and GDPR data request workflows.
+- Settings exposes the catalog so owners/admins can install procedures directly into the selected workspace, with MCP readiness surfaced before install.
+- `ranse procedure list` and `ranse procedure add <slug>` fork procedures into a repo-local `procedures/` directory as YAML or JSON.
+- `ranse procedure manifest` exports the full machine-readable catalog for a standalone community mirror.
+- Each library procedure ships with inline eval cases, deterministic SHA-256 provenance, and reference MCP tool specs written beside the forked procedure as `<slug>.mcp.json` plus `<slug>.provenance.json`.
+- Library procedures now exercise required MCP contracts through `call_action`; read-only lookups can run automatically, while write/destructive actions pause for operator approval.
+- Library validation runs every procedure's inline evals, checksum generation, immutable clone behavior, route permissions, MCP reference matching, and unsafe-action checks in `tests/procedure-library.test.ts`.
+- `procedure-library/README.md` and `CONTRIBUTING.md` define the contribution bar for upstreaming generic workflows. A standalone `getranse/procedures-library` repo can now mirror this catalog when community volume warrants it.
 
 ## Phase 8 — Insights & auto-improving KB
+**Status: shipped.**
+
 *Principle 5 (extends), Principle 1*
 
-- Per-conversation rubric scoring (groundedness, tone, resolution, customer effort)
-- Aggregate dashboards: resolution rate, escalation reasons, top unanswered intents, slowest procedures
-- **Suggestions agent** clusters unresolved conversations weekly, drafts new KB articles **as PRs to the workspace's content repo** — human review preserved, no surprise edits
-- Drift detection: flag KB entries whose answers diverge from recent successful replies
+- Per-conversation rubric scoring is stored in D1 for groundedness, tone, resolution, customer effort, and overall quality, with signals preserved as auditable JSON.
+- Aggregate dashboards are shipped in the operator console for resolution rate, follow-ups, feedback, low-score conversations, top unresolved intents, escalation reasons, and slowest procedures.
+- The suggestions loop clusters repeated unresolved conversations, stores evidence count/confidence/source-ticket lineage, drafts reviewable KB article candidates, and lets an admin accept a suggestion into the workspace knowledge base. Human review is preserved; no content is published silently.
+- Drift detection flags cited knowledge sources that no longer cover terms appearing in successful replies tied back to those source chunks.
+- Weekly scheduled insight maintenance scores recent conversations, refreshes unresolved-intent suggestions, and detects KB drift inside the customer's Cloudflare account.
 
 ## Phase 9 — Multi-channel + voice
 *Principle 7 — email is the wedge; other channels are derivatives*

migrations/20260518_010000_insights.sql

@@ -0,0 +1,63 @@
+-- Phase 8 insights and auto-improving knowledge base.
+
+CREATE TABLE IF NOT EXISTS conversation_score (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  ticket_id TEXT NOT NULL,
+  groundedness_score REAL NOT NULL CHECK(groundedness_score >= 0 AND groundedness_score <= 1),
+  tone_score REAL NOT NULL CHECK(tone_score >= 0 AND tone_score <= 1),
+  resolution_score REAL NOT NULL CHECK(resolution_score >= 0 AND resolution_score <= 1),
+  effort_score REAL NOT NULL CHECK(effort_score >= 0 AND effort_score <= 1),
+  overall_score REAL NOT NULL CHECK(overall_score >= 0 AND overall_score <= 1),
+  signals_json TEXT NOT NULL DEFAULT '{}',
+  scored_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  UNIQUE(workspace_id, ticket_id),
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE,
+  FOREIGN KEY (ticket_id) REFERENCES ticket(id) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_conversation_score_workspace
+  ON conversation_score(workspace_id, overall_score ASC, scored_at DESC);
+
+CREATE TABLE IF NOT EXISTS kb_suggestion (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  cluster_key TEXT NOT NULL,
+  title TEXT NOT NULL,
+  summary TEXT NOT NULL,
+  body_markdown TEXT NOT NULL,
+  source_ticket_ids_json TEXT NOT NULL DEFAULT '[]',
+  suggested_terms_json TEXT NOT NULL DEFAULT '[]',
+  status TEXT NOT NULL DEFAULT 'open' CHECK(status IN ('open','accepted','dismissed')),
+  source TEXT NOT NULL DEFAULT 'unresolved_cluster',
+  created_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  UNIQUE(workspace_id, cluster_key),
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_kb_suggestion_workspace
+  ON kb_suggestion(workspace_id, status, updated_at DESC);
+
+CREATE TABLE IF NOT EXISTS knowledge_drift_signal (
+  id TEXT PRIMARY KEY,
+  workspace_id TEXT NOT NULL,
+  source_id TEXT NOT NULL,
+  signal_hash TEXT NOT NULL,
+  severity TEXT NOT NULL CHECK(severity IN ('low','medium','high')),
+  title TEXT NOT NULL,
+  summary TEXT NOT NULL,
+  successful_reply_count INTEGER NOT NULL DEFAULT 0,
+  divergence_terms_json TEXT NOT NULL DEFAULT '[]',
+  example_ticket_ids_json TEXT NOT NULL DEFAULT '[]',
+  status TEXT NOT NULL DEFAULT 'open' CHECK(status IN ('open','resolved','dismissed')),
+  detected_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  UNIQUE(workspace_id, source_id, signal_hash),
+  FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE,
+  FOREIGN KEY (source_id) REFERENCES knowledge_source(id) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_knowledge_drift_signal_workspace
+  ON knowledge_drift_signal(workspace_id, status, severity, detected_at DESC);

migrations/20260518_020000_insights_hardening.sql

@@ -0,0 +1,10 @@
+-- Phase 8 insights hardening: suggestion confidence and acceptance lineage.
+
+ALTER TABLE kb_suggestion ADD COLUMN evidence_count INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE kb_suggestion ADD COLUMN confidence_score REAL NOT NULL DEFAULT 0;
+ALTER TABLE kb_suggestion ADD COLUMN accepted_source_id TEXT;
+ALTER TABLE kb_suggestion ADD COLUMN accepted_by_user_id TEXT;
+ALTER TABLE kb_suggestion ADD COLUMN accepted_at INTEGER;
+
+CREATE INDEX IF NOT EXISTS idx_kb_suggestion_acceptance
+  ON kb_suggestion(workspace_id, status, accepted_at DESC);