feat(evalboard): generate the cost table from pricing.py, and chart in every edition

.claude/harness-candidates.md

@@ -107,3 +107,60 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   caught them. The cleanup plan explicitly deferred this as YAGNI for the
   one-time purge, but any future doc rename/deletion re-opens the same blind
   spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+## From 2026-07-30 evalboard pricing SSOT + multi-harness charts
+
+- [ ] **CI never runs the `evalboard/` test suite — the highest-value gap here.**
+  `grep -rn evalboard .github/workflows/ .pre-commit-config.yaml Makefile` yields
+  exactly one hit: a comment in `pr-checks.yml` saying `evalboard/pnpm-lock.yaml`
+  is *out of scope*. `pnpm verify` (`tsc --noEmit && vitest run && next build`)
+  exists in `evalboard/package.json` but nothing invokes it. Consequence: the
+  pricing drift guard that the whole generated-SSOT design rests on is never
+  executed. A Python dev edits a rate in `src/coder_eval/pricing.py`, never opens
+  `evalboard/`, and the PR goes green while every rendered USD figure keeps the
+  stale rate — precisely how the 7 wrong rates (Opus 4.6/4.7/4.8 at 3× actual)
+  accumulated in the first place. Flagged independently by two reviewers.
+  *Not done here because adding a required check to `pr-checks.yml` is a repo-wide
+  CI-policy change affecting every contributor's PR, outside a plan scoped to
+  `evalboard/`.* Ready to apply: a job mirroring the existing `setup-node` steps
+  (`pnpm install --frozen-lockfile` + `pnpm verify` with `working-directory:
+  evalboard`), ideally non-blocking first. Alternatively a Python-side CExxx rule
+  asserting `lib/pricing.generated.ts` is current, which would put the guard on
+  the side that actually edits rates.
+- [ ] **One shared normalizer fixture for `normalizeModel` (TS) and
+  `_normalize_model` (Python).** Both strip the same routing/region/vendor
+  prefixes, but TS additionally strips a trailing `-YYYYMMDD` and Python does not
+  — so `claude-opus-4-6-20250514` prices in the evalboard and is unpriced in the
+  backend, and the frontend estimate can disagree with the backend's
+  authoritative Cost for dated ids. Neither side's tests can see the other, and
+  the generated table does not close this: generation shares the rate *values*,
+  not the *lookup* logic. Needs a decision on which behaviour is the reference
+  before a fixture can be written. Note also that plugin rates registered via
+  `register_pricing` (e.g. `coder_eval_uipath`) can never reach the generated
+  table at all, since the generator parses `pricing.py`'s literal table only.
+- [ ] **No test can see `app/page.tsx`'s self-link scope threading.** `hParam`,
+  `base` and `buildHref` are module-private inside an async server component and
+  there is no test for that file. A future self-link spelled `buildHref({ tag })`
+  silently resets the user's harness scope with the whole suite green — now that
+  `?h=` scopes the charts, the tiles, the run table *and* the ad-hoc section,
+  dropping it is a bigger jump than it was when it only re-scoped the chart.
+  Guarding it means extracting `buildHref` + a `selfLinkBase` helper into a
+  testable module.
+- [ ] **"Any chart with ≥2 series must have a hover-attribution test."** Nothing
+  in the suite mounted a chart tooltip before the multi-harness work, which is
+  how two real misattribution bugs survived a spec review: recharts silently
+  ignores `shared={false}` on `LineChart`, and with the default
+  `allowDuplicatedCategory` each `<Line>` indexes its own points with an index
+  into the concatenation of all series. `app/_overview/__tests__/harness-legend.test.tsx`
+  now covers the two overview charts (hovered-x attribution, real zeros, unknown
+  series keys), but that is one test file by convention, not a rule — nothing
+  enforces it for the next multi-series chart someone adds.
+- [ ] **Nothing checks that the frontend rate table's deliberate omissions stay
+  deliberate.** `EXCLUDED_MODELS` in `evalboard/scripts/gen-pricing.mjs` withholds
+  the OpenRouter open-weight ids because they are routed per-request and shown at
+  captured actual cost, so a static headline rate would be confidently wrong.
+  That set is guarded in both directions today (a stale entry throws; a
+  reintroduced static rate fails `pricing-parity.test.ts`), but only *within*
+  `evalboard/`. The Python side has no matching notion — `pricing.py` prices them
+  for the `max_usd` fallback with nothing recording that the board must not — so
+  the rationale lives in a JS comment a Python-side reprice will never surface.

evalboard/app/page.tsx

@@ -6,7 +6,7 @@ import {
     listRecentHarnesses,
     type TagCount,
 } from "@/lib/overview";
-import { parseHarnessScope } from "@/lib/harness";
+import { orderHarnesses, parseHarnessScope } from "@/lib/harness";
 import { fmtDuration, fmtRunTime, fmtTimestamp } from "@/lib/format";
 import { passClass } from "@/lib/pass-rate";
 import { type Window } from "@/lib/reviews-types";
@@ -128,7 +128,7 @@ export default async function Page({
     const [overview, listing, adhoc, harnesses] = await Promise.all([
         getOverview(WINDOW, activeTag, q, harness),
         getRunListing(activeTag, q, limit, harness),
-        getAdhocRunListing(adhocLimit),
+        getAdhocRunListing(adhocLimit, harness),
         listRecentHarnesses(),
     ]);
 
@@ -193,7 +193,10 @@ export default async function Page({
                 together. Buried in the chart card it read as a chart control
                 while the numbers above it silently covered every harness. Same
                 position as the selector on Path to GA, trends, and the
-                watchlist. Internal-only, like the analytics block below. */}
+                watchlist. Shown in every edition, like the analytics block it
+                scopes — the charts below name a harness per line, so gating the
+                only control that isolates one would leave an OSS instance able
+                to see a harness but not select it. */}
             <div className="flex flex-wrap items-start justify-between gap-3">
                 <div className="space-y-1">
                     <h1 className="text-xl font-semibold text-gray-900">
@@ -204,13 +207,23 @@ export default async function Page({
                         and logs.
                     </p>
                 </div>
-                {isInternal && (
-                    <HarnessSelector
-                        current={harness}
-                        harnesses={harnesses}
-                        includeAll
-                    />
-                )}
+                <HarnessSelector
+                    current={harness}
+                    // Union the recent-runs discovery list with every harness in
+                    // this window. They cover different slices —
+                    // listRecentHarnesses scans a fixed count of recent runs,
+                    // windowHarnesses is windowed — so a harness that ran once
+                    // three weeks ago gets a chip in the 30d view despite being
+                    // absent from the discovery list, instead of being drawn on
+                    // the chart with no way to isolate it. windowHarnesses is
+                    // captured before the harness filter, so this set does not
+                    // shift when a chip is selected.
+                    harnesses={orderHarnesses([
+                        ...harnesses,
+                        ...overview.windowHarnesses,
+                    ])}
+                    includeAll
+                />
             </div>
 
             <WindowSummary
@@ -221,10 +234,15 @@ export default async function Page({
             />
 
             {/* The analytics block — daily success / turn-budget charts and the
-                colored skill/review/tag rail — is an internal-only surface (see
-                lib/edition.ts). The public OSS edition drops it so the front
-                page is just the run list. */}
-            {isInternal && (
+                colored skill/review/tag rail — renders in EVERY edition. It used
+                to sit behind isInternal, a gate that dated to the initial public
+                release rather than to anything in the block: nothing here is
+                UiPath-specific, it just charts whatever runs the instance is
+                pointed at, so an OSS clone charts its own results and a
+                default-configured instance no longer shows a bare run list.
+                The Harness column in the run table below stays gated separately,
+                so on an OSS instance the chart legend can name a harness that the
+                table does not have a column for. */}
             <section className="border border-gray-200 rounded-lg bg-white p-4 space-y-4">
                 <div>
                     <h2 className="text-sm font-semibold text-gray-900">
@@ -324,7 +342,6 @@ export default async function Page({
                         )}
                 </div>
             </section>
-            )}
 
             <div className="flex items-baseline justify-between gap-3 pt-1">
                 <div className="flex items-baseline gap-3 flex-wrap">

evalboard/lib/__tests__/module-boundaries.test.ts

@@ -0,0 +1,80 @@
+import { readFileSync } from "node:fs";
+import { dirname, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+import { describe, expect, test } from "vitest";
+
+// Two bundle-correctness invariants that are otherwise enforced only by
+// comments, each worth a measured ~105 kB of First Load JS. Both are violated by
+// adding one import, and neither shows up in tsc, the test suite, or a passing
+// `next build` — only in the bundle size, which nobody reads on a green PR.
+
+const here = dirname(fileURLToPath(import.meta.url));
+const root = resolve(here, "../..");
+
+function readSource(rel: string): string {
+    return readFileSync(resolve(root, rel), "utf8");
+}
+
+// Matches `import … from "x"`, `import "x"`, and `export … from "x"`.
+function moduleSpecifiers(src: string): string[] {
+    return [...src.matchAll(/(?:^|\n)\s*(?:import|export)[\s\S]*?from\s+"([^"]+)"/g)]
+        .map((m) => m[1])
+        .concat(
+            [...src.matchAll(/(?:^|\n)\s*import\s+"([^"]+)"/g)].map((m) => m[1]),
+        );
+}
+
+describe("lib/harness.ts is a true leaf", () => {
+    // Client components ("use client" charts, the selector, the badge) all import
+    // this module for values, not just types. lib/overview.ts transitively reaches
+    // node:fs, node:path and @azure/storage-blob, so a single VALUE import from
+    // there would drag that whole graph into the browser bundle. Type-only
+    // imports erase at compile time; values do not.
+    //
+    // This is why the module's own header says it must stay dependency-free, and
+    // why anything that needs both harness knowledge and run data (the chart
+    // pivot in app/_overview/harness-series.ts) imports harness.ts rather than
+    // the other way round.
+    test("has no imports at all", () => {
+        const specs = moduleSpecifiers(readSource("lib/harness.ts"));
+        expect(
+            specs,
+            "lib/harness.ts must stay dependency-free — see the leaf-module note at its top",
+        ).toEqual([]);
+    });
+});
+
+describe("the tag rail stays chart-free", () => {
+    // tag-rail.tsx (ChipLegend / MergedTagRail) renders on pages with no chart at
+    // all — /runs/[id] via run-view and /trends via trends-view — as well as on
+    // the overview. Pulling chart code in here measured +105 kB First Load JS on
+    // /runs/[id] (163 -> 268 kB) and on /trends (119 -> 225 kB).
+    //
+    // The tempting regression is to reuse the overview's swatch/legend
+    // primitives: app/_overview/harness-legend.tsx is a sibling that looks
+    // reusable, but it is chart-side and pulls the chart module's types and, via
+    // the charts, recharts itself. Duplicating a 6-line swatch is the cheaper
+    // trade.
+    const FORBIDDEN = ["recharts", "harness-series", "harness-legend"];
+
+    test("imports neither recharts nor any chart module", () => {
+        const specs = moduleSpecifiers(readSource("app/_overview/tag-rail.tsx"));
+        for (const spec of specs) {
+            for (const bad of FORBIDDEN) {
+                expect(
+                    spec.includes(bad),
+                    `tag-rail.tsx must not import ${spec} — it renders on chart-less pages`,
+                ).toBe(false);
+            }
+        }
+    });
+
+    test("its dependencies stay on a short, reviewed list", () => {
+        // Keeps the guard above honest: a blocklist only catches the imports we
+        // thought of, so any NEW dependency has to be added here deliberately —
+        // at which point that dependency's own graph gets re-checked.
+        expect(
+            moduleSpecifiers(readSource("app/_overview/tag-rail.tsx")).sort(),
+        ).toEqual(["@/lib/overview", "next/link"]);
+    });
+});

evalboard/lib/__tests__/overview.test.ts

@@ -9,7 +9,7 @@ import {
     type PerRun,
     type RunListingRow,
 } from "../overview";
-import { normalizeHarness } from "../harness";
+import { normalizeHarness, orderHarnesses } from "../harness";
 import type { RunOverviewTask } from "../runs";
 
 function task(overrides: Partial<RunOverviewTask>): RunOverviewTask {
@@ -640,3 +640,87 @@ describe("projectRunRow", () => {
         });
     });
 });
+
+// The switcher's chip set must be STABLE under selection. getOverview feeds it
+// `windowHarnesses`, not `harnesses`: the latter is derived from the FILTERED
+// points, so with `?h=codex` active it collapses to ["codex"] and every other
+// chip would vanish the moment one was selected — stranding the user on a scope
+// they cannot leave without hand-editing the URL. These pin the distinction.
+describe("harnesses vs windowHarnesses", () => {
+    function run(harness: string | null, id = "2026-07-30_00-00-00"): PerRun {
+        return {
+            id,
+            overview: {
+                id,
+                harness,
+                tasks: [task({ status: "SUCCESS" })],
+                totalCostUsd: null,
+                taskDurationSeconds: null,
+                componentShas: [],
+            } as unknown as NonNullable<PerRun["overview"]>,
+            reviewTagCounts: {},
+            reviewTagsByTask: {},
+            adhoc: false,
+            title: null,
+        };
+    }
+
+    // The derivation getOverview applies for windowHarnesses: every non-adhoc
+    // run in the window with a readable overview, BEFORE the harness filter.
+    function windowHarnessesOf(runs: PerRun[]): string[] {
+        return orderHarnesses(
+            runs
+                .filter((r) => !r.adhoc && r.overview)
+                .map((r) => normalizeHarness(r.overview!.harness)),
+        );
+    }
+
+    const window = [
+        run("claude-code", "2026-07-28_00-00-00"),
+        run("codex", "2026-07-29_00-00-00"),
+        run("delegate-sdk", "2026-07-30_00-00-00"),
+    ];
+
+    test("windowHarnesses lists every harness in the window", () => {
+        expect(windowHarnessesOf(window)).toEqual([
+            "claude-code",
+            "codex",
+            "delegate-sdk",
+        ]);
+    });
+
+    test("it is IDENTICAL whichever harness is selected — the anti-shift rule", () => {
+        // Contrast the two derivations over the SAME scoped input. The
+        // post-filter one collapses to the selected harness and deletes the
+        // other chips; the pre-filter one is invariant. Comparing the pre-filter
+        // result to itself would be vacuous, so apply the real harness filter
+        // and show only one of the two survives it.
+        const unscoped = windowHarnessesOf(window);
+        expect(unscoped).toHaveLength(3);
+
+        for (const scope of ["claude-code", "codex", "delegate-sdk"]) {
+            const afterHarnessFilter = window.filter(
+                (r) => normalizeHarness(r.overview!.harness) === scope,
+            );
+
+            // What the chips must NOT be built from — collapses to one.
+            expect(windowHarnessesOf(afterHarnessFilter)).toEqual([scope]);
+
+            // What they ARE built from: computed before that filter, so the full
+            // set survives and every chip stays clickable.
+            expect(
+                windowHarnessesOf(window),
+                `chip set shifted when scoped to ${scope}`,
+            ).toEqual(unscoped);
+        }
+    });
+
+    test("a legacy unstamped run folds into claude-code, not a phantom chip", () => {
+        expect(windowHarnessesOf([run(null)])).toEqual(["claude-code"]);
+    });
+
+    test("ad-hoc runs contribute no chip", () => {
+        const adhoc = { ...run("delegate-sdk"), adhoc: true };
+        expect(windowHarnessesOf([run("codex"), adhoc])).toEqual(["codex"]);
+    });
+});