Add the Scribe starter kit and landing page

.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+# Least privilege: the workflow only reads the repo; the junit reporter
+# needs check-run write access, granted at job scope below.
+permissions:
+  contents: read
+
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      checks: write
+    steps:
+      # Actions are pinned to full commit SHAs; the trailing comment is the
+      # tag the SHA was resolved from. Bump deliberately, with a diff.
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: pnpm/action-setup@b906affcce14559ad1aafd4ab0e942779e9f58b1 # v4
+      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        with:
+          node-version: 24
+          cache: pnpm
+      - run: pnpm install --frozen-lockfile
+      - run: pnpm check
+      - run: pnpm test
+      - run: pnpm build
+
+      # The deterministic offline tier — the PR evaluation gate.
+      # No live backends, no credentials, stub judge only.
+      - name: eval gate (deterministic tier)
+        run: |
+          node packages/kit/src/cli/scribe.ts eval run \
+            --input packages/kit/fixtures/records.synthetic.jsonl \
+            --config packages/kit/fixtures/eval.stub.yaml \
+            --junit .scribe-junit.xml
+      - uses: mikepenz/action-junit-report@3585e9575db828022551b4231f165eb59a0e74e3 # v5
+        if: always()
+        with:
+          report_paths: .scribe-junit.xml

PLAN.md

@@ -0,0 +1,100 @@
+# Scribe: 90-Day Agent Observability and Evaluation Plan
+
+## Summary
+
+Create Scribe, an Apache-2.0 starter-kit repository at `boringcomputers/scribe`, then copy versioned recipes into Ruth, Ezekiel, and Bezalel. The core remains vendor-neutral; Dynatrace is an optional exporter and evaluation backend. Full prompts, replies, and tool traces are captured by default and persisted to an owned corpus that will seed future in-house fine-tuning; credentials and payment data are never recorded.
+
+Use [dt-evals](https://github.com/dynatrace-oss/dt-evals) for reusable evaluation logic, the [agent instrumentation examples](https://github.com/dynatrace-oss/dynatrace-ai-agent-instrumentation-examples) for integration patterns, and a self-hosted OpenTelemetry Collector for portable routing and privacy enforcement on the observability fan-out; the corpus path is scrubbed in-app before upload. Do not fork upstream unless wrapper-based integration proves insufficient.
+
+## Implementation Roadmap
+
+### Days 1–30: Shared observability foundation
+
+- Create the starter-kit repo with tagged template releases, attribution, checksums, upgrade scripts, Collector configuration, and concise agent-operations skills inspired by [dynatrace-for-ai](https://github.com/Dynatrace/dynatrace-for-ai).
+- Define one telemetry contract for Eve and Bezalel:
+  - Retain standard OpenTelemetry GenAI attributes, pinned to an exact semantic-conventions version (the GenAI conventions are still marked development upstream), plus the framework-emitted `eve.*` fields.
+  - Keep the existing per-agent namespaces (`ruth.*`, `ezekiel.*`, `bezalel.*`) for agent-specific context; the consumers namespace deliberately and are not migrated. Only shared contract fields (`dataClass`, hashed principal, recipe release, corpus pointers) go under `boringcomputers.*`.
+  - Record models, tools, timings, token counts, outcomes, trace/session identifiers, and hashed principals.
+  - Record full prompts, responses, tool arguments, and message bodies by default (`bodyMode: "full"`); these turn records are the raw material for a future in-house fine-tuning corpus.
+  - Never record credentials, API keys, raw authorization values, or payment-card data. Sensitive capabilities (for example Agentcard card details) force body capture off for affected turns; this override is not configurable.
+  - The agent process writes the corpus itself: it applies the scrub catalog in-app, then persists full-body turn records as `AgentInteractionRecordV1` JSONL to a private Cloudflare R2 bucket through the S3-compatible API. The Collector is not in this path — its stock S3 exporter writes time-partitioned OTLP batches and cannot produce per-turn objects. Reuse Ezekiel’s `aws4fetch` R2 client as the reference implementation; provision buckets and `R2_*` credentials for Ruth and Bezalel, which have none today. Partition keys by service and date (for example `corpus/v1/service=ruth/dt=2026-08-11/<turnId>.jsonl`) with the turn ID as the object key, making retries idempotent. Vendor exporters receive observability copies but are not the corpus of record.
+- Run one self-hosted OpenTelemetry Collector as a standalone container on an owner-operated host; Ruth (Vercel) and Ezekiel (Workers) cannot run sidecars, so they export OTLP over HTTPS to it. Hosting it on Nehemiah remains a post-90-day follow-up. Add Collector-side redaction and filtering based on [Dynatrace’s OTel patterns](https://github.com/Dynatrace/demo-opentelemetry-patterns) — those scenarios are log-focused, so scrub secret patterns, credentials, and payment data inside `gen_ai.*` span-attribute bodies with the transform processor (OTTL) rather than suppressing bodies — then fan out to OTLP-capable destinations (Braintrust ingest, generic OTLP, optional Dynatrace). The Raindrop turn-lifecycle hook stays in-process with its own PII redaction. The Collector protects only this observability fan-out; the corpus path is scrubbed in-app before upload.
+- Apply the Eve recipe to Ruth first, port it to Ezekiel, and apply the Effect/MCP recipe to Bezalel.
+- Establish W3C trace propagation from an Eve turn through MCP to Bezalel. Nothing propagates context across this boundary today, so plan on injecting and extracting `traceparent` at the HTTP transport boundary from the start (Nehemiah’s control plane carries a working W3C reference implementation); never put authorization data in baggage.
+- Record the installed recipe release and file hashes in each consumer’s `.scribe.json`. Verification must detect local drift without imposing a runtime package dependency.
+
+### Days 31–60: Offline evaluation layer
+
+- Keep the existing live Eve scenario suites as the post-merge regression signal, exactly as they run today. They hit live backends and LLM judges, so they are deliberately gated off pull requests for credential security; the PR gate remains typecheck plus unit tests.
+- Add a copied local adapter around an exact pinned version of `@dynatrace-oss/dt-eval-lib` (0.0.15-alpha today; the alpha line will break, and the adapter isolates that risk). Reach the existing Vercel AI Gateway through the library’s `openai` provider with a `baseUrl` override pointed at the [OpenAI-compatible API](https://vercel.com/docs/ai-gateway/sdks-and-apis) — the only custom-endpoint mechanism the library supports.
+- Keep `anthropic/claude-haiku-4.5` as the default judge model initially; it is already the configured judge in both repos’ `evals.config.ts`.
+- Introduce explicit JSONL evaluation records as the adapter’s only input, with a clear boundary against the framework: `eve eval` keeps owning live end-to-end scenario runs (it already ships judges, strict mode, and JUnit output), while the Scribe adapter owns offline scoring of corpus and fixture records. The `eval run` CLI is the adapter’s entry point and neither wraps nor replaces `eve eval`.
+- Add the deterministic tier the live suites are not: fixture and recorded `AgentInteractionRecordV1` JSONL scored through the adapter with stubbed judge responses, no live backends, and no credentials. This offline tier becomes the PR evaluation gate.
+- Run shared quality checks for:
+  - PII leakage and prompt injection.
+  - Relevance and conciseness.
+  - Faithfulness when reference context exists.
+- Implement Bezalel’s agent-in-the-loop evaluation tier through Eve, initially covering synthetic receipt and email workflows with stubbed providers.
+- Produce human-readable summaries plus stable JSON and JUnit artifacts for CI. JSONL input, JUnit output, and the source/sanitizer/sink interfaces all live in the adapter by design — none of them exist upstream.
+- Treat upstream contributions to `dt-evals` as opportunistic rather than load-bearing: it is a months-old alpha with no extension surface and a large open-issue backlog, so plan for zero upstream acceptance and maintain the attributed local wrapper indefinitely.
+
+### Days 61–90: Continuous synthetic canaries
+
+- Run the full evaluation suite nightly from a GitHub Actions cron in each repo, generalizing Bezalel’s existing `canary.yml` pattern (scheduled run that opens or refreshes a tracking issue on failure). A canary run exercises the deployed environment with synthetic principals and fixtures: Ruth against a Vercel preview or production URL, Ezekiel against its Workers deployment, Bezalel through its existing canary command.
+- Bodies are captured everywhere by default; `dataClass` governs evaluation eligibility rather than capture:
+  - Only records marked `dataClass: "synthetic"` or fixture records enter judge evaluation.
+  - Real-user records are persisted to the fine-tuning corpus but are not judged during this phase.
+- Store the pinned baseline and per-run history under `evals/` in the same R2 bucket; compare the latest seven synthetic runs:
+  - Hard-fail any PII or prompt-injection breach.
+  - Soft-alert on a quality-score decline of at least `0.10`.
+  - Hard-fail relevance or faithfulness below `0.50` on an individual eligible case.
+- Add optional Dynatrace source/sink adapters without making Dynatrace necessary for local runs or CI.
+- Add two operational skills: diagnose a failed agent turn by trace ID, and compare an evaluation run against its baseline.
+- Document a post-90-day Nehemiah research track. Treat NetTracer and Koney as design references only: Firecracker networking requires TAP/bridge/conntrack or tc/XDP observation rather than host-socket tracing, while Koney is Kubernetes-specific and AGPL-licensed.
+
+## Interfaces and Repository Contract
+
+- `AgentInteractionRecordV1`:
+  - `schemaVersion`, `recordId`, `capturedAt`, `dataClass`, `service`, `environment`.
+  - Optional `traceId`, `sessionId`, `turnId`, `model`, context, expected output, usage, and duration.
+  - `input` and `output` captured by default under `bodyMode: "full"`; they remain required for fixture or synthetic records entering judge evaluation.
+  - Tool entries containing name, outcome, and optional duration, plus arguments and results under `bodyMode: "full"`.
+  - Outcome and scalar attributes; arbitrary secrets or raw authorization values are forbidden.
+- `TelemetryPolicy`:
+  - `bodyMode: "full" | "structural" | "synthetic"`; the default is `"full"`.
+  - Sensitive capabilities always override body capture to off.
+- `ScrubCatalog`:
+  - A versioned list of secret, credential, and payment-data patterns shipped with the starter kit.
+  - Applied in two places: in-app before every corpus write, and in the Collector for the observability fan-out.
+  - Scrub tests assert against this catalog; it is the tested guarantee behind the privacy acceptance criteria.
+- Evaluation adapters:
+  - `EvaluationSource.read(): AsyncIterable<AgentInteractionRecordV1>`.
+  - `EvaluationSink.write(run): Promise<void>`.
+- Copied command recipe:
+  - `eval validate`.
+  - `eval run --input <jsonl> --config <yaml> --json --junit <path>`.
+  - `doctor`.
+  - `verify-recipes`.
+- Commands must be non-interactive under `--json` and return nonzero exit codes for schema, policy, or gate failures.
+
+## Verification and Acceptance Criteria
+
+- Full-body telemetry is the default in Ruth, Ezekiel, and Bezalel; tests prove every pattern in the versioned scrub catalog is removed before export and before corpus writes, and that sensitive-capability turns drop bodies entirely. Pattern scrubbing is best-effort by nature — the catalog is the tested guarantee, and the non-configurable sensitive-capability override is the hard one. Structural mode remains available as an explicit opt-down.
+- One integration test correlates an Eve turn, MCP request, Bezalel tool invocation, and provider call under one trace.
+- Existing Ruth and Ezekiel scenario suites continue passing on their post-merge schedule.
+- Fixture tests cover malformed records, ineligible real-user records, judge failures, retry exhaustion, redaction, and threshold boundaries.
+- Evaluation jobs retry transient judge errors at most twice and fail closed afterward.
+- PRs run the deterministic offline tier only; live credentials never reach pull-request-controlled code, matching the existing CI stance in both repos. Judge-backed and live scenario suites run post-merge on main and in the nightly canaries.
+- Nightly synthetic canaries emit JSON and JUnit results and never evaluate real-user bodies.
+- All three consumers pass recipe checksum verification against the same tagged starter-kit release.
+- The system remains fully usable without a Dynatrace account.
+
+## Assumptions
+
+- The shared repository is a template and governance source, not a public runtime package.
+- Upstream projects are wrapped and pinned before any fork is considered.
+- No real production interactions are evaluated during this 90-day phase; they are captured and stored for the future fine-tuning corpus.
+- Full-body capture relies on the owner-operated, single-user consent model of these agents. The corpus of record lives in a private Cloudflare R2 bucket — S3-compatible and zero-egress, with lifecycle rules for retention and a bucket-scoped API token — never in vendor SaaS alone.
+- [dtctl](https://github.com/dynatrace-oss/dtctl) informs command ergonomics, while [dynatrace-managed-mcp](https://github.com/dynatrace-oss/dynatrace-managed-mcp) serves as an authentication and transport reference rather than a direct dependency.
+- Ruth and Ezekiel run eve 0.27.7 while Bezalel’s consumer app is on ^0.30.8; recipes declare and are tested against this version range rather than requiring alignment as a precondition.
+- Herdr and Nehemiah integrations are follow-up tracks after the agent stack meets the acceptance criteria.

README.md

@@ -0,0 +1,58 @@
+# Scribe
+
+Observability and evaluation starter kit for the boringcomputers agents (Ruth, Ezekiel, Bezalel).
+
+Scribe is a template and governance source, not a runtime package. It publishes tagged, checksummed recipes — telemetry contract, scrub catalog, corpus writer, Collector configuration, evaluation adapter, and operational skills — that consumers copy into their repos and verify with `verify-recipes` against `.scribe.json`. The core is vendor-neutral OpenTelemetry; Dynatrace is an optional exporter and evaluation backend.
+
+See [PLAN.md](./PLAN.md) for the full 90-day plan, interfaces, and acceptance criteria.
+
+## Layout
+
+- `packages/kit` — all recipe sources
+  - `src/contract` — `AgentInteractionRecordV1`, validation, attribute namespaces, principal hashing
+  - `src/propagation` — W3C `traceparent` helpers for the Eve → MCP → Bezalel boundary
+  - `src/scrub` — versioned scrub catalog (every pattern ships with tested examples) + engine
+  - `src/corpus` — telemetry policy (non-configurable sensitive-capability override), R2 S3 client, corpus writer
+  - `src/evals` — offline evaluation adapter: JSONL sources, judges (pinned `dt-eval-lib` via AI Gateway + deterministic stub), runner with fail-closed retries, gates, JUnit, R2 run store
+  - `src/cli` — the `scribe` command (`eval validate`, `eval run`, `doctor`, `verify-recipes`, `install-recipe`, `gen-collector`)
+  - `fixtures/` — fixture records and eval configs (stub + gateway)
+- `collector/` — self-hosted OpenTelemetry Collector; `otel-collector.yaml` is generated from the scrub catalog
+- `skills/` — agent-operations skills (diagnose a turn by trace ID, compare eval runs)
+- `apps/web` — landing page (Vite, React, [Kumo](https://kumo-ui.com/))
+
+## The command recipe
+
+```sh
+node packages/kit/src/cli/scribe.ts eval validate --input records.jsonl --json
+node packages/kit/src/cli/scribe.ts eval run --input records.jsonl \
+  --config packages/kit/fixtures/eval.stub.yaml --json --junit junit.xml
+node packages/kit/src/cli/scribe.ts doctor --json
+node packages/kit/src/cli/scribe.ts install-recipe kit-core --to ../ruth
+node packages/kit/src/cli/scribe.ts verify-recipes --dir ../ruth
+node packages/kit/src/cli/scribe.ts gen-collector
+```
+
+Exit codes: `0` ok, `1` unexpected, `2` schema, `3` config/policy, `4` gate, `5` recipe drift. Commands are non-interactive under `--json`.
+
+## Evaluation tiers
+
+1. **Deterministic (PR gate)** — stub judge driven by fixture attributes; no live backends, no credentials. Runs in CI on every PR.
+2. **Judge-backed (post-merge / nightly)** — pinned `@dynatrace-oss/dt-eval-lib` reaching the Vercel AI Gateway (`openai` provider + `baseUrl`), default judge `anthropic/claude-haiku-4.5`. Hard-fails PII/prompt-injection breaches and sub-0.50 relevance/faithfulness; soft-alerts on a 0.10 mean decline vs the pinned baseline.
+
+Only `dataClass: "synthetic"` or `"fixture"` records are ever judged. Real-user records go to the corpus and nothing else.
+
+## Development
+
+Requires Node 24 and pnpm 11.
+
+```sh
+pnpm install
+pnpm check    # typecheck everything
+pnpm test     # kit test suite (scrub catalog guarantees, gates, CLI exit codes)
+pnpm build    # landing page build
+pnpm dev      # landing page on http://localhost:5173
+```
+
+## License
+
+Apache-2.0.

apps/web/index.html

@@ -0,0 +1,17 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
+    <title>Scribe — observability and evaluation for personal agents</title>
+    <meta
+      name="description"
+      content="Scribe is an Apache-2.0 starter kit of versioned, checksummed recipes for agent telemetry, an owned interaction corpus, and offline evaluation."
+    />
+  </head>
+  <body>
+    <div id="root"></div>
+    <script type="module" src="/src/main.tsx"></script>
+  </body>
+</html>

apps/web/package.json

@@ -0,0 +1,28 @@
+{
+  "name": "@scribe/web",
+  "private": true,
+  "version": "0.0.1",
+  "type": "module",
+  "scripts": {
+    "dev": "vite",
+    "build": "tsc -b && vite build",
+    "check": "tsc -b",
+    "preview": "vite preview"
+  },
+  "dependencies": {
+    "@cloudflare/kumo": "^2.9.2",
+    "@phosphor-icons/react": "^2.1.10",
+    "react": "^19.2.8",
+    "react-dom": "^19.2.8"
+  },
+  "devDependencies": {
+    "@tailwindcss/vite": "^4.3.3",
+    "@types/node": "^24.13.3",
+    "@types/react": "^19.2.17",
+    "@types/react-dom": "^19.2.3",
+    "@vitejs/plugin-react": "^6.0.4",
+    "tailwindcss": "^4.3.3",
+    "typescript": "~6.0.2",
+    "vite": "^8.2.0"
+  }
+}