ci: fail on committed secrets and on stale generated files

.github/workflows/ci.yml

@@ -1,3 +1,19 @@
+# Three jobs.
+#
+# `check` is the build gate: typecheck, lint, tests, build, and a smoke run of
+# the shipped artifact under node.
+#
+# `secrets` greps the tracked tree for credential shapes — Solari keys, preview
+# tokens, JWTs, Slack, Telegram and Discord tokens, and any tracked `.env` — and
+# fails on a hit. Its allowlist is empty by design and the patterns are written
+# so they do not match their own source, so "we never commit secrets" is checked
+# rather than asserted.
+#
+# `generated` regenerates every generated file — the embedded relay server, the
+# embedded test app, and the measured numbers in README.md and
+# benchmarks/README.md — and then fails on `git diff --exit-code`. A stale embed
+# or a benchmark figure typed by hand therefore cannot merge.
+
 name: ci
 
 on:
@@ -18,7 +34,7 @@ jobs:
         run: bunx playwright-core install --with-deps chromium
       - name: typecheck
         run: bun run typecheck
-      - name: lint (biome + oxlint + relay embed sync)
+      - name: lint (biome + oxlint + embed/number sync)
         run: bun run lint
       - name: unit + integration tests
         run: bun test src/ test-app/ e2e/ui.spec.ts
@@ -38,3 +54,32 @@ jobs:
       # e2e (bun run test:e2e) is intentionally not run in CI: it needs a real
       # SOLARI_API_KEY and spends plan quota. It is the pre-publish gate, run
       # locally against the live API.
+
+  secrets:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      # A gate nobody has watched fire is a gate nobody has tested: this runs
+      # every pattern against a live sample and against known-good lines from
+      # this repo, and fails if a pattern has gone blind.
+      - name: secret scan can still fail
+        run: scripts/scan-secrets.sh --self-test
+      - name: no credential shapes in the tracked tree
+        run: scripts/scan-secrets.sh
+
+  generated:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+      # No `bun install`: all three generators read files and write files, and
+      # import nothing outside node: and bun: builtins. The job is ~15 s.
+      - name: regenerate every generated file
+        run: |
+          bun scripts/embed-guest.ts
+          bun test-app/embed-app.ts
+          bun scripts/render-measured.ts
+      - name: generated files are in sync with their sources
+        run: git diff --exit-code

CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- **CI fails on a committed secret.** `scripts/scan-secrets.sh` greps the
+  tracked tree for Solari live keys, port-preview tokens, JWTs, Slack, Telegram
+  and Discord tokens, and for any tracked `.env`. Its allowlist is empty and the
+  patterns do not match their own source, so nothing is exempt. It ships with a
+  `--self-test` that CI runs first, because a gate nobody has watched fire is a
+  gate nobody has tested.
+- **CI fails on a stale generated file.** A `generated` job regenerates the
+  embedded relay server, the embedded test app and the measured numbers in
+  `README.md` and `benchmarks/README.md`, then runs `git diff --exit-code`.
+- **The measured numbers are generated.** `scripts/render-measured.ts` renders
+  the tables and headline counts in `README.md` and `benchmarks/README.md` from
+  `benchmarks/*.json`, inside `<!-- generated:… -->` blocks. A figure typed by
+  hand no longer merges. `bun run lint` checks the same thing locally.
+
 ## [0.7.0] - 2026-09-03
 
 The handoff now knows whether anybody is on the other side, and the ending is

README.md

@@ -39,9 +39,11 @@ returns. Set `SOLARI_API_KEY`; handraise uses it to create the relay sandbox.
 Measured against the live API, method and raw data in
 [`benchmarks/`](benchmarks/README.md):
 
+<!-- generated:readme-headline — bun scripts/render-measured.ts -->
 - **19/20 blocked workflows rescued** (baseline 0/20).
 - **30/30 handoffs resolved** in the latency benchmark.
 - **3.5 s median** from raise to live on the phone.
+<!-- /generated:readme-headline -->
 
 ## Approval mode
 
@@ -465,20 +467,24 @@ Solari plan, one fresh relay sandbox each, a scripted human on the public
 WebSocket, measured from Germany against the default (us-west) endpoint.
 **30/30 resolved, zero reconnects, zero leaked sandboxes.**
 
+<!-- generated:readme-latency — bun scripts/render-measured.ts -->
 | | p50 | p75 | worst of 30 |
 |---|---|---|---|
 | Agent raises its hand → the phone shows the live page | 3.5s | 3.6s | 3.7s |
 | — of which: relay sandbox cold start | 2.7s | 2.7s | 2.9s |
 | Input round trip through the relay (150 samples) | 186ms | 191ms | 286ms |
+<!-- /generated:readme-latency -->
 
 The number that matters more than any latency is what handraise does to
 workflows that would otherwise fail. `bun run bench:rescue`: 40 runs against a
 live portal with a real TOTP wall, interleaved arms, one completion test:
 
+<!-- generated:readme-rescue — bun scripts/render-measured.ts -->
 | | completed | median human time |
 |---|---|---|
 | baseline agent (no human available) | 0/20 | — |
 | with handraise | **19/20** | 5.5s |
+<!-- /generated:readme-rescue -->
 
 The 0/20 baseline is the design fact, not a crippled agent: it tried, and a
 machine cannot know a TOTP code. The 5.5s is a scripted human — the machine
@@ -491,10 +497,12 @@ side by side: 20 workflows against one live portal, interleaved takeover,
 approval, takeover, on 2026-09-02. A takeover needs the browser driven; an
 approval needs one decision, and every fourth one here was a denial.
 
+<!-- generated:readme-mixed — bun scripts/render-measured.ts -->
 | | completed | to visible | frames | bytes | relay sandbox |
 |---|---|---|---|---|---|
 | takeover — the human drives | 10/10 | 4718ms | 14 | 142 KB | 10.7s |
 | approval — the human decides | 10/10 | 4896ms | 1 | 25 KB | 5.3s |
+<!-- /generated:readme-mixed -->
 
 Medians over the completed runs. A denied approval counts as completed: the
 decision was delivered, and the bench then loads the account page and requires

benchmarks/README.md

@@ -32,20 +32,24 @@ MIXED_N=4 bun run bench:mixed      # a short run
 
 Each writes its JSON back into this directory, overwriting the committed file.
 
+<!-- generated:bench-latency-heading — bun scripts/render-measured.ts -->
 ## Handoff latency — 30 handoffs, 30 resolved
+<!-- /generated:bench-latency-heading -->
 
 30 consecutive real handoffs, run one at a time, one fresh relay sandbox each,
 with a scripted human on the public WebSocket. Measured from Germany against
 the default (us-west) endpoint on 2026-09-01. **30/30 resolved, zero
 reconnects, zero failures, zero leaked sandboxes.**
 
+<!-- generated:bench-latency-table — bun scripts/render-measured.ts -->
 | Metric | p50 | p75 | worst |
 |---|---|---|---|
 | `stuckToVisibleMs` — `raiseHand()` → first frame at the human | 3532 | 3594 | 3746 |
 | `relayColdStartMs` — sandbox create → public URL answering | 2679 | 2731 | 2895 |
 | `firstFrameMs` — same frame, timed agent-side | 644 | 664 | 679 |
 | `inputRttMs` — human → relay → human, 150 samples | 186 | 191 | 286 |
 | `handoffDurationMs` — whole handoff live, scripted human | 2031 | 2065 | 2183 |
+<!-- /generated:bench-latency-table -->
 
 All values in milliseconds. `stuckToVisibleMs` is the honest end-to-end number:
 it is measured on the human's socket, not the agent's. Cold start is ~75% of
@@ -57,17 +61,21 @@ input round trip sits on the network RTT floor from Germany to the us-west edge
 relay with your region. `handoffDurationMs` is the machine floor of a handoff,
 not a human's reading pace.
 
+<!-- generated:bench-rescue-heading — bun scripts/render-measured.ts -->
 ## Rescue rate — 19 of 20 blocked workflows completed
+<!-- /generated:bench-rescue-heading -->
 
 One workflow run 2×20 times against one live portal with a real RFC 6238 TOTP
 wall: sign in, reach the account page. The arms are interleaved (baseline i,
 handraise i, baseline i+1, …) so both see the same browser ages, the same
 network minute and the same app state.
 
+<!-- generated:bench-rescue-table — bun scripts/render-measured.ts -->
 | | completed | median handoff |
 |---|---|---|
 | baseline — no human, no access to the shared secret | 0/20 | — |
 | with handraise | **19/20** | 5490 ms |
+<!-- /generated:bench-rescue-table -->
 
 The 0/20 baseline is a design fact, not a crippled agent: it submits the form,
 scrapes the page for a code, reloads and retries, and it never touches the
@@ -86,7 +94,9 @@ The counting is load-bearing rather than decorative, and that is testable:
 test, and the table must then read 20/20 for the baseline and 0/20 for
 handraise.
 
+<!-- generated:bench-mixed-heading — bun scripts/render-measured.ts -->
 ## Mixed workload — 20 of 20 workflows completed, and what each mode cost
+<!-- /generated:bench-mixed-heading -->
 
 The first two benches ask about one mode. This one runs both against the same
 Aurora Bank instance, interleaved (takeover, approval, takeover, …), on
@@ -100,10 +110,12 @@ Aurora Bank instance, interleaved (takeover, approval, takeover, …), on
   decision. A scripted human sees one screenshot and the action in words, and
   answers. Every fourth approval is denied (2 of 10 here).
 
+<!-- generated:bench-mixed-table — bun scripts/render-measured.ts -->
 | | completed | time to visible p50 / p75 | handoff p50 / p75 | frames | bytes | inputs | relay-sandbox s |
 |---|---|---|---|---|---|---|---|
 | takeover | 10/10 | 4718 / 4808 ms | 6927 / 7202 ms | 14 | 142 KB | 8 | 10.7 |
 | approval | 10/10 | 4896 / 4977 ms | 2063 / 2084 ms | 1 | 25 KB | 0 | 5.3 |
+<!-- /generated:bench-mixed-table -->
 
 All per-handoff figures are medians over the runs that completed, and the two
 time columns measure different spans:

package.json

@@ -37,7 +37,7 @@
   "scripts": {
     "build": "tsup",
     "typecheck": "tsc --noEmit",
-    "lint": "biome check . && oxlint && bun scripts/embed-guest.ts --check",
+    "lint": "biome check . && oxlint && bun scripts/embed-guest.ts --check && bun test-app/embed-app.ts --check && bun scripts/render-measured.ts --check",
     "test": "bun test src/ test-app/ e2e/ui.spec.ts",
     "test:coverage": "bun test --coverage src/ test-app/",
     "test:e2e": "bun --env-file=.env e2e/handoff.e2e.ts",