feat(evalboard): generate the cost table from pricing.py, and chart in every edition

[uipreliga]

Rebased onto main and rewritten (2026-07-31). #69 landed the multi-harness overview this branch was independently building, so everything it covers — the harness table and series colours, the chart refactor, the badge/selector work, the All chip, whole-page ?h= scoping — has been dropped from this PR. What remains is the work #69 does not have. 26 files / +2706 across 12 commits → 13 files / +1044 in one commit. Earlier review comments are anchored to commits that no longer exist; a fresh pass is warranted.

Goal

Write the evalboard's per-MTok rates in exactly one file in one language, and stop the analytics block being hidden on a default-configured instance.

The cost bug this fixes

lib/pricing.ts declares itself a hand-copy of src/coder_eval/pricing.py. Measured against main today, that 30-row copy has drifted three ways:

6 models carry WRONG rates	claude-opus-4-8, 4-7, 4-6 and 4-5-20251101 read Opus 4.1's legacy $15/$75 against an actual $5/$25 — overstating cost 3× on every rendered estimate. claude-opus-4-6 appears in real run data. Also claude-haiku-4-5-20251001 ($0.80/$4 vs $1/$5) and gemini-3-flash-preview ($1.5/$9 vs $0.50/$3).
25 priced models are missing	rendered "—" for cost. 8 sat in the DELIBERATELY_UNMIRRORED allowlist; the other 17 drifted past it unnoticed.
2 entries are dead	claude-opus-4-6-20250514, claude-sonnet-4-6-20250514 — absent from the Python table, and redundant given the undated fallback.

The allowlist meant to hold the line had failed twice over: 17 models drifted past it, and 2 of its 8 entries were models real runs use (gpt-5.6-terra is the single most-run model in the recorded data), so it was actively suppressing cost for runs the board does execute.

Rather than hand-copy better — which converts a one-time copy error into a recurring obligation to retype 53 rows × 4 numbers across two languages — rates are generated into a checked-in artifact. pnpm gen:pricing after editing pricing.py. No Python file is modified by this PR; if a rate is wrong there, that is a separate backend change.

The drift guard is not self-blind

pricing-parity.test.ts imports readTable, not parsePythonTable. Comparing two products of the same regex cannot detect a row that regex skipped — the row would be missing from the artifact and from the expectation, so deep-equal passes while the board renders "—" for that model forever. readTable cross-checks matched rows against the count of ModelPricing( constructions in the file and refuses to emit on a mismatch. Generation also makes exact-match affordable, so the allowlist is gone entirely.

Reconciling with #66's per-call costs

One deliberate gap remains, and it is the reason a naive generator would have been a regression. #66 removed the three OpenRouter open-weight models from this table on purpose: OpenRouter routes per request, so a static headline rate is wrong, and the detail view now shows each call's captured actual cost instead. pricing.py still prices them for the Python max_usd fallback — so generating blindly would have silently reintroduced exactly what #66 removed, putting a confidently wrong estimate beside the measured one in the per-message Cost column, the Tokens↔USD toggle and thinkingSim.

EXCLUDED_MODELS in the generator carries that rationale and is guarded both ways: a stale entry (an id pricing.py no longer prices) throws rather than silently outliving its model, and a reintroduced static rate fails the parity test. The Bedrock open-weight models are pinned as not excluded — those run at fixed rates with no per-call capture, so static pricing is correct and required there.

The analytics block is no longer edition-gated

The charts and tag rail sat behind isInternal, so a default-configured instance — including the deployed board — showed only the run list. That gate dates to the initial public release, not to any recent change. Nothing in the block is UiPath-specific: it renders whatever runs the instance is pointed at, so an OSS clone charts its own results.

The harness selector is ungated with it. The charts name a harness per line, so gating the only control that isolates one leaves an OSS instance able to see a harness but not select it. The run table's Harness column stays gated separately, so on an OSS instance the chart legend can name a harness the table has no column for.

Two scoping fixes on top of #69

• windowHarnesses — the chip set came from listRecentHarnesses' fixed-count discovery scan, so a harness that ran inside the 30d window but not recently got a chart line with no chip to select it. getOverview now also returns every harness in the window, captured before the harness filter — invariant under selection by construction, where deriving it after the filter would collapse it to the active harness and delete every other chip on click. OverviewData.harnesses keeps its meaning (the plotted set, so "all N harnesses" still equals the lines drawn). The new test contrasts the two derivations over the same scoped input, so it fails if the chips are ever re-pointed at the filtered set.
• Ad-hoc runs are scoped by ?h= like the rest of the page. They carry a harness stamp like any other run and that section has no Harness column, so leaving it unfiltered showed codex uploads under a Claude Code scope with nothing on screen to explain why.

Bundle-boundary guards

module-boundaries.test.ts pins two invariants previously held by comments alone, each worth a measured ~105 kB First Load JS and invisible to tsc, the suite, and a green next build:

• lib/harness.ts stays a true leaf (zero imports). Client components import it for values, and lib/overview.ts transitively reaches node:fs and @azure/storage-blob — one value import from there drags that graph into the browser bundle. Type-only imports erase; values do not.
• tag-rail.tsx stays chart-free. It renders on /runs/[id] and /trends, which have no chart at all; the tempting regression is reusing the overview's sibling legend primitives, which pull the chart module and recharts with them.

Verification

• 439 tests pass across 37 files
• pnpm tsc --noEmit clean · pnpm next build clean · pnpm gen:pricing idempotent
• Bundles: /runs/[id] 167 kB, /trends 119 kB First Load JS — recharts is not leaking onto the chart-less pages
• The generator's refusal guards were exercised directly: a stale exclusion throws, and a table below the row-count floor or with an unparsed ModelPricing( row refuses to emit rather than writing a partial table

⚠️ CI does not exercise any of this

Nothing in .github/workflows/ runs the evalboard/ suite — grep -rn evalboard .github/workflows/ yields one hit, a comment saying evalboard/pnpm-lock.yaml is out of scope. So the 439 tests and the pricing drift guard are green-by-absence on this PR, and the drift class this PR removes stays undetected going forward: a rate edited in pricing.py without pnpm gen:pricing will pass CI. That is precisely how the 3× Opus rates accumulated.

Deliberately not fixed here — adding a required check to pr-checks.yml is a repo-wide CI policy change affecting every contributor's PR, outside a change scoped to evalboard/. The concrete job to add is the top item in .claude/harness-candidates.md, alongside four other deferred guards recorded by this PR.

Not in scope

Non-Claude rates (gpt-5.x, gemini-3.x, Bedrock open-weight) are generated faithfully but were never verified against their vendors — generation makes the mirror faithful, not the source right. normalizeModel (TS) and _normalize_model (Python) have drifted: TS strips a trailing -YYYYMMDD and Python does not, so a dated id can price on one side and not the other; generation shares rate values, not lookup logic. Plugin rates registered via register_pricing (e.g. coder_eval_uipath) cannot reach the generated table at all, since the generator parses pricing.py's literal table only. Introductory pricing (Sonnet 5 at $2/$10 through 2026-08-31 against a $3/$15 list) would need time-dependent rates in the Python schema. All four are written up in .claude/harness-candidates.md.

[github-actions]

Claude finished @uipreliga's task in 1m 33s —— View job

---

📋 Review Task List

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Examine the full diff (git diff origin/main...HEAD)
• Review key changed files with full context
• Analyze for cross-file consistency issues
• Check "what's missing" per review guidelines
• Perform design-level scrutiny
• Post comprehensive review feedback

Starting review now...

[CarlesUIPath]

@uipreliga Does this have conflicts with #66 ?

[Mihaiii]

This is incorrect or at least depends on the provider. I'd look for some REST endpoint to call that gets the data from a database.