docs(benchmark): refresh task-aware measurements (#143)

Author: SarthakWade

README.md

@@ -62,8 +62,8 @@ judgments (not lab benchmarks):
   Runs on macOS and Linux (Ubuntu and common CI/test distros); Windows is the
   exception.
 
-**Measured agent surface** (same P2 fixture flow, Docker ARM64, 17 Jul 2026 —
-point-in-time): Headless warm **147** est. tokens vs Selenium **410** /
+**Measured agent surface** (same P2 fixture flow, Docker ARM64, 27 Aug 2026 —
+point-in-time): Headless warm **174** est. tokens vs Selenium **410** /
 Puppeteer **499**. Full method and limits:
 [BENCHMARK.md](apps/headless/docs/BENCHMARK.md).
 
@@ -236,7 +236,6 @@ published GHCR image under Docker Desktop:
 docker run --shm-size=1g ghcr.io/lockintime/headless:latest headless --version
 ```
 
-
 ### npm / npx
 
 JavaScript-based agent harnesses can run the verified launcher without a

apps/headless/docs/BENCHMARK.md

@@ -5,26 +5,26 @@ record through the `Continue` transition, and save a final screenshot.
 
 ## Current snapshot
 
-Five fresh containers were run for each case on 12 August 2026, on Apple
+Five fresh containers were run for each case on 27 August 2026, on Apple
 Silicon with Docker Linux ARM64. The table reports the median of each metric.
 These remain point-in-time measurements; repeat the benchmark before using
 them to compare a change. The generated
 [`results.json`](../../../packages/benchmark-results/results.json) preserves
 all 20 raw samples and the aggregation provenance.
 
-| Workflow | Estimated tokens | Wall time | CPU time | Peak memory |
-| --- | ---: | ---: | ---: | ---: |
-| Headless, cold | 218 | 3,484 ms | 2,028 ms | 368 MiB |
-| Headless, warm | 174 | 3,248 ms | 1,379 ms | 366 MiB |
-| Selenium with Python | 410 | 2,880 ms | 2,010 ms | 363 MiB |
-| Puppeteer | 499 | 2,402 ms | 1,860 ms | 358 MiB |
+| Workflow             | Estimated tokens | Wall time | CPU time | Peak memory |
+| -------------------- | ---------------: | --------: | -------: | ----------: |
+| Headless, cold       |              218 |  3,413 ms | 1,794 ms |     381 MiB |
+| Headless, warm       |              174 |  3,239 ms | 1,218 ms |     379 MiB |
+| Selenium with Python |              410 |  2,828 ms | 1,788 ms |     378 MiB |
+| Puppeteer            |              499 |  2,400 ms | 1,849 ms |     367 MiB |
 
 Estimated tokens are `ceil(workflow source bytes / 4)`. They compare the agent
 workflow surface, not billed LLM tokens, tool schemas, prompts, or responses.
 
 Headless has the smallest measured agent surface: the warm workflow uses about
 58% fewer estimated tokens than Selenium and 65% fewer than Puppeteer. Its
-median CPU time is about 31% lower than Selenium and 26% lower than Puppeteer.
+median CPU time is about 32% lower than Selenium and 34% lower than Puppeteer.
 Puppeteer is fastest and has the lowest median peak memory; Headless does not
 lead those dimensions. The reusable P2 flow command reduces orchestration work
 for real agent-driven repeats, but this benchmark retains the comparable
@@ -38,9 +38,9 @@ lookup.
 
 ## Method
 
-Each workflow uses Chromium 151 and FFmpeg 5.1 to produce the same two
+Each workflow uses Chromium 151.0.7922.173 and FFmpeg 5.1.9 to produce the same two
 artifacts: an MP4 that tours both pages and a final viewport PNG. Selenium 4.8.3
-uses ChromeDriver 151; Puppeteer Core is 22.15.0. All waits use page load or an
+uses ChromeDriver 151.0.7922.173; Puppeteer Core is 22.15.0. All waits use page load or an
 explicit URL condition. Every measured run gets a fresh container. The warm
 Headless case starts its host and session before timing; the cold case
 includes them.

apps/web/lib/repository-content.d.mts

@@ -1,5 +1,6 @@
 export type BenchmarkWorkflow = {
   case: string;
+  label: string;
   workflow: string;
   name: string;
   variant: string;
@@ -19,6 +20,7 @@ export type BenchmarkWorkflow = {
 
 export type BenchmarkContent = {
   sectionLabel: string;
+  methodDate: string;
   headline: string;
   summary: string;
   proofs: Array<{

apps/web/lib/repository-content.mjs

@@ -165,12 +165,19 @@ export function loadBenchmarkContent() {
     year: "numeric",
     timeZone: "UTC",
   }).format(generatedAt);
+  const methodDate = new Intl.DateTimeFormat("en-GB", {
+    day: "numeric",
+    month: "long",
+    year: "numeric",
+    timeZone: "UTC",
+  }).format(generatedAt);
 
   const workflows = BENCHMARK_CASES.map((caseName) => {
     const entry = cases.get(caseName);
     const presentation = PRESENTATION[caseName];
     return {
       case: caseName,
+      label: entry.label,
       workflow: entry.label.replace(",", ""),
       ...presentation,
       tokens: entry.median.estimatedTokens,
@@ -189,6 +196,7 @@ export function loadBenchmarkContent() {
 
   benchmarkCache = {
     sectionLabel: `P2 benchmark / ${date}`,
+    methodDate,
     headline:
       warm.estimatedTokens ===
       Math.min(...workflows.map((workflow) => workflow.tokens))
@@ -431,6 +439,51 @@ export function loadDocumentationContent() {
 export function validateRepositoryContent() {
   const benchmark = loadBenchmarkContent();
   const documentation = loadDocumentationContent();
+  const readme = readRepositoryFile("README.md");
+  const method = readRepositoryFile("apps/headless/docs/BENCHMARK.md");
+  const measuredClaim = readme.match(
+    /\*\*Measured agent surface\*\*[\s\S]*?Docker ARM64, ([0-9]{1,2} [A-Z][a-z]{2} [0-9]{4})[\s\S]*?Headless warm \*\*(\d+)\*\* est\. tokens vs Selenium \*\*(\d+)\*\* \/\s*Puppeteer \*\*(\d+)\*\*/,
+  );
+  if (!measuredClaim) fail("README measured agent-surface claim is missing");
+  const workflows = new Map(
+    benchmark.workflows.map((workflow) => [workflow.case, workflow]),
+  );
+  const expectedClaim = [
+    benchmark.sectionLabel.replace("P2 benchmark / ", ""),
+    String(workflows.get("headless-warm").tokens),
+    String(workflows.get("selenium").tokens),
+    String(workflows.get("puppeteer").tokens),
+  ];
+  if (
+    JSON.stringify(measuredClaim.slice(1)) !== JSON.stringify(expectedClaim)
+  ) {
+    fail(
+      "README measured agent-surface claim does not match generated results",
+    );
+  }
+  if (!method.includes(`for each case on ${benchmark.methodDate},`)) {
+    fail("benchmark method date does not match generated results");
+  }
+  for (const workflow of benchmark.workflows) {
+    const row = method
+      .split("\n")
+      .find((line) => line.startsWith(`| ${workflow.label}`));
+    if (!row) fail(`benchmark method is missing row: ${workflow.label}`);
+    const cells = row
+      .split("|")
+      .slice(1, -1)
+      .map((cell) => cell.trim());
+    const expectedCells = [
+      workflow.label,
+      workflow.tokens.toLocaleString("en-US"),
+      `${workflow.wallMs.toLocaleString("en-US")} ms`,
+      `${workflow.cpuMs.toLocaleString("en-US")} ms`,
+      `${workflow.memoryMiB.toLocaleString("en-US")} MiB`,
+    ];
+    if (JSON.stringify(cells) !== JSON.stringify(expectedCells)) {
+      fail(`benchmark method row is stale: ${workflow.label}`);
+    }
+  }
   return {
     benchmarkCases: benchmark.workflows.length,
     commandGroups: documentation.commandGroups.length,

docs/ROADMAP.md

@@ -81,12 +81,13 @@ of them:
 Honest snapshot, so newcomers know what is real:
 
 **Working and verified**
+
 - 39-verb JSON protocol over a `0600` Unix socket with peer-UID checks
   (`apps/headless/Sources/HeadlessProtocol/`).
 - Two engines behind one CLI: macOS `WKWebView` app, Linux sandboxed Chromium
   over the DevTools fd-3/4 pipe (no debug port).
 - Progressive context pruning (`inspect --context
-  summary|outline|text|actions|full`, `--task`, `--within @rN`, `--budget`)
+summary|outline|text|actions|full`, `--task`, `--within @rN`, `--budget`)
   with a measured **94.5 % token reduction** on the 120-section fixture.
 - Evidence capture: PNG/JPG/PDF screenshots, viewport/section series, MP4/MOV/
   WebM/GIF recordings, visual diffs, flows, QA reports.
@@ -107,6 +108,7 @@ Honest snapshot, so newcomers know what is real:
   command reference, and a Docker sandbox wrapper.
 
 **Not yet real**
+
 - ~~No CI on pull requests or `main`~~ — PR CI landed (`.github/workflows/ci.yml`,
   backlog §D1/§D3). Correctness fixes in §A are still outstanding, and the
   macOS E2E is nightly/label-gated rather than a per-PR gate.
@@ -121,7 +123,7 @@ Honest snapshot, so newcomers know what is real:
   three places each and will drift; the site has no deploy pipeline.
 - Windows is not supported.
 - A list of real code defects (thread-safety on shutdown, oversized `qa
-  report` responses, `@eN` ref invalidation surprises, host code duplication)
+report` responses, `@eN` ref invalidation surprises, host code duplication)
   — all catalogued in the [improvements backlog](roadmap/improvements-backlog.md).
 
 ---
@@ -152,23 +154,23 @@ codebase's existing behavior because they are the product:
 
 ## 4. Platform support
 
-| Platform | Status today | Target |
-| --- | --- | --- |
-| **macOS 13+** (Apple Silicon) | Universal signed release pipeline awaiting tag | Signed + notarized, Homebrew, universal binary |
-| **macOS Intel** | Universal release pipeline awaiting tag | Universal binary in release CI |
-| **Linux** (Debian/Ubuntu, non-Snap Chromium) | Verified installer and GHCR pipeline awaiting tag | curl installer, published Docker image (GHCR), apt guidance |
-| **Linux other distros** | Works where a non-Snap Chromium exists | Documented candidate paths per distro family |
-| **Windows 10/11** | Not supported | **Stretch goal (Phase W)** — Chromium host ported; not required for "done". See [architecture decisions §6](roadmap/architecture-decisions.md). |
-| **Any OS via Docker** | Works (build locally) | `docker run ghcr.io/…/headless` one-liner, including as the practical Windows answer (WSL2/Docker Desktop) until Phase W lands |
+| Platform                                     | Status today                                      | Target                                                                                                                                          |
+| -------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| **macOS 13+** (Apple Silicon)                | Universal signed release pipeline awaiting tag    | Signed + notarized, Homebrew, universal binary                                                                                                  |
+| **macOS Intel**                              | Universal release pipeline awaiting tag           | Universal binary in release CI                                                                                                                  |
+| **Linux** (Debian/Ubuntu, non-Snap Chromium) | Verified installer and GHCR pipeline awaiting tag | curl installer, published Docker image (GHCR), apt guidance                                                                                     |
+| **Linux other distros**                      | Works where a non-Snap Chromium exists            | Documented candidate paths per distro family                                                                                                    |
+| **Windows 10/11**                            | Not supported                                     | **Stretch goal (Phase W)** — Chromium host ported; not required for "done". See [architecture decisions §6](roadmap/architecture-decisions.md). |
+| **Any OS via Docker**                        | Works (build locally)                             | `docker run ghcr.io/…/headless` one-liner, including as the practical Windows answer (WSL2/Docker Desktop) until Phase W lands                  |
 
 Agent-harness support (the other axis of "platform"):
 
-| Harness | Today | Target |
-| --- | --- | --- |
-| MCP clients (Claude Code, Cursor, Codex, …) | `headless-mcp` stdio server | unchanged core; per-client setup docs + `.mcp.json` example |
-| Claude Code | manual skill pointer | root `CLAUDE.md` + discoverable skill |
-| Codex / OpenCode / Amp / others | `AGENTS.md` convention | root `AGENTS.md` (done in this change) |
-| Plain shell agents | CLI + `headless capabilities` | unchanged; capabilities doc kept machine-checked |
+| Harness                                     | Today                         | Target                                                      |
+| ------------------------------------------- | ----------------------------- | ----------------------------------------------------------- |
+| MCP clients (Claude Code, Cursor, Codex, …) | `headless-mcp` stdio server   | unchanged core; per-client setup docs + `.mcp.json` example |
+| Claude Code                                 | manual skill pointer          | root `CLAUDE.md` + discoverable skill                       |
+| Codex / OpenCode / Amp / others             | `AGENTS.md` convention        | root `AGENTS.md` (done in this change)                      |
+| Plain shell agents                          | CLI + `headless capabilities` | unchanged; capabilities doc kept machine-checked            |
 
 ---
 
@@ -184,7 +186,7 @@ every item carries file/line detail.
 Write down what the product is, what must not change, and everything that is
 wrong. Add agent rule files so every harness can work on this repo.
 
-*Exit test:* this document set is merged; `AGENTS.md`/`CLAUDE.md` exist at
+_Exit test:_ this document set is merged; `AGENTS.md`/`CLAUDE.md` exist at
 root.
 
 ### Phase 1 — Trust the build (CI + correctness)
@@ -202,7 +204,7 @@ known races exist.
   `ELEMENT_NOT_FOUND` (backlog §A7).
 - Web app gets `next build` + eslint in the same CI (backlog §D3).
 
-*Exit test:* a PR cannot merge with failing tests; the E2E suites pass on CI
+_Exit test:_ a PR cannot merge with failing tests; the E2E suites pass on CI
 runners, not just laptops; the known-crash list in the backlog §A is empty.
 
 ### Phase 2 — One host, written once (deduplication refactor)
@@ -216,7 +218,7 @@ errors end-to-end (backlog §B).
 This is also the **prerequisite for Windows**: after it, a Windows port is one
 new engine + one new transport backend, not a third copy of everything.
 
-*Exit test:* adding a hypothetical new verb touches one dispatch site; the
+_Exit test:_ adding a hypothetical new verb touches one dispatch site; the
 error-code mapping is a typed enum, not string matching; the capability matrix
 (clipboard, network mock, PDF fidelity, …) is generated from code and asserted
 in tests.
@@ -237,7 +239,7 @@ in tests.
 - **npm wrapper** (`npx headless-browser` style) that downloads the platform
   binary — the cheapest path into JS-centric agent stacks. (Backlog §E.)
 
-*Exit test:* a new user on a clean macOS or Linux machine gets from zero to
+_Exit test:_ a new user on a clean macOS or Linux machine gets from zero to
 `headless start` + first `visit` in under two minutes without touching a
 compiler, and without a Gatekeeper override on macOS.
 
@@ -258,7 +260,7 @@ Make Headless the obvious choice inside every harness:
   implemented so `fill` can type literal `--json`; response pagination for
   large reports remains (backlog §G).
 
-*Exit test:* a fresh Claude Code, Cursor, and Codex session can each discover
+_Exit test:_ a fresh Claude Code, Cursor, and Codex session can each discover
 and drive Headless with zero manual prompting beyond repo checkout.
 
 ### Phase 5 — Website and docs as a product surface
@@ -269,11 +271,11 @@ and drive Headless with zero manual prompting beyond repo checkout.
   generated from the CLI) (backlog §F).
 - Add the missing pages: install, security model, MCP setup, command
   reference, changelog, platform matrix.
-- Re-run the benchmark with the task-aware inspect flow before quoting any
-  token number — the site currently markets `--task` while quoting pre-`--task`
-  measurements (backlog §F4).
+- Keep the generated benchmark current with the task-aware inspect flow before
+  quoting any token number. The site imports the generated medians and carries
+  their point-in-time caveat (backlog §F4).
 
-*Exit test:* site deploys on merge; every number and command on it is
+_Exit test:_ site deploys on merge; every number and command on it is
 generated or test-asserted; the "stale benchmark" warning is gone because the
 benchmark is current.
 
@@ -295,7 +297,7 @@ Phase 2's engine/transport split. Shape of the work (detailed in
 - Until then, the documented Windows answer is Docker Desktop/WSL2 with the
   published image (Phase 3 dependency).
 
-*Exit test:* the Linux E2E scenario passes on a Windows runner with the
+_Exit test:_ the Linux E2E scenario passes on a Windows runner with the
 Chromium engine; `winget install headless` works.
 
 ---
@@ -315,7 +317,7 @@ of the following hold:
    returns an explicit capability error; the matrix is generated and asserted.
 4. **Agent-native:** any MCP-capable harness and any AGENTS.md-reading harness
    can drive Headless from a fresh checkout with no human glue; `headless
-   capabilities` is machine-accurate.
+capabilities` is machine-accurate.
 5. **Evidence current:** benchmark re-run on the shipping workflow; QA
    evidence regenerated for the release; website deployed and drift-free.
 6. **Docs:** this roadmap's Phases 1–5 checked off, with Windows either

docs/roadmap/improvements-backlog.md

@@ -431,9 +431,11 @@ Owner-decided scope: package managers, no hosted service.
   matrix, the README's comparison table (strongest positioning content, not
   on site). Plus `robots.txt`, `sitemap`, OG metadata, per-page `metadata`,
   404 page.
-- **F4. Stale-benchmark honesty:** ([#50](https://github.com/LockInTime/headless/issues/50)) site quotes pre-`--task` numbers while
+- **F4. Stale-benchmark honesty:** ([#50](https://github.com/LockInTime/headless/issues/50)) ~~site quotes pre-`--task` numbers while
   marketing `--task`, and drops BENCHMARK.md's re-run warning; headline says
-  "Measured, not claimed." Fix by refresh (D5) or by carrying the caveat.
+  "Measured, not claimed." Fix by refresh (D5) or by carrying the caveat.~~
+  **Done:** a new five-repeat, task-aware snapshot was generated on 27 August 2026. The website imports its medians and date, preserves the point-in-time
+  warning, and web lint rejects README claims that drift from generated data.
 - **F5. Dead weight:** ([#51](https://github.com/LockInTime/headless/issues/51)) `components/ui/side-rays.tsx` + `ogl` dep (unused),
   `public/scan-dashboard.png` (unreferenced), leftover shadcn `.dark` block in
   `globals.css`, 8 unused button variants; two WebGL stacks (~700 KB) for