refactor(core): source pricing + context limits from tokenlens registry

[faraa2m]

Summary

• Replace the hand-maintained RATES / MODELS tables in packages/core/src/rates.ts with a registry built at module load from @tokenlens/models (anthropic + openai + google subpaths). 7 curated models → 42 stable models, no extra manual maintenance.
• Every ModelDescriptor now carries contextWindow, maxOutputTokens, and pricingSource: 'local' | 'tokenlens' (from models.dev).
• Surface the new metadata in CLI (Limits: block under the cost table), web Playground (provider-grouped selector with · 128k context chips), and GitHub Action (Limits: line in the sticky PR comment).

Public API (getRate, getModel, KNOWN_MODELS, RateEntry, ModelDescriptor) is byte-compatible — no consumer code outside the touched files needs to change.

Why a hybrid registry, not a full swap?

claude-haiku-4-5 / claude-opus-4-7 / claude-sonnet-4-6 are not yet in tokenlens upstream (its anthropic catalog tops at claude-opus-4-1-20250805). They stay in a small LOCAL_OVERRIDES table that wins on id collision. A weekly cron (.github/workflows/registry-check.yml) runs scripts/check-overrides.mjs:

1. Detects when upstream picks up an overridden id → fail with "drop the override".
2. Diffs tokenlens-sourced pricing/context against packages/core/src/__snapshots__/registry.json → fail on drift, prompt to regenerate the snapshot.

On failure the workflow opens (or comments on) a tracking issue.

Other changes

• benchmarks/run.mjs gains --filter / --models (also honors BENCH_MODELS env) so the regenerate sweep can be scoped — the matrix grew from 7×N×M to 42×N×M cells.
• benchmarks/results.json regenerated against the full sweep.
• Rates tests loosened from exact-dollar assertions → per-model positive-price invariants + canary checks on stable IDs + registry-size floor (>=15) so a broken registry can't silently pass.
• packages/core adds two npm scripts: snapshot:registry, check:overrides.

Test plan

• npm run build (all packages compile)
• npm run lint (biome clean)
• npm run typecheck
• npx vitest run — 48 tests pass (was 46; added 2 for renderModelLimits)
• npm run benchmarks — drift check passes against regenerated results.json
• npm run check:overrides -w @tokenometer/core → OK — 42 models, 3 overrides intact, snapshot in sync.
• CLI smoke: echo "..." | node packages/cli/dist/index.js - --model claude-opus-4-7,gpt-4o produces the expected table + new Limits: block + summary
• First scheduled run of registry-check.yml (Mondays 12:00 UTC) — verify via workflow_dispatch after merge

Open follow-ups (out of scope)

• The web bundle grows by the size of the bundled models.dev provider data (~33KB raw across the 3 providers). For an even smaller browser footprint, tokenlens ships an async fetchModels() path that could be opted into for packages/web only.
• Adding gpt-3.5-turbo / gpt-4 / gpt-4-turbo (now in KNOWN_MODELS) reuses the existing o200k_base openai dispatch in tokenize.ts. Those older models are technically cl100k_base. Pre-existing bug surfaced by the wider catalog; worth a dedicated fix.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
tokenometer	Ready	Preview, Comment	May 9, 2026 3:34am