Skip to content

SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 - #61

Open
DiTo97 wants to merge 17 commits into
mainfrom
claude/gracious-hypatia-hqy7xl
Open

SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1#61
DiTo97 wants to merge 17 commits into
mainfrom
claude/gracious-hypatia-hqy7xl

Conversation

@DiTo97

@DiTo97 DiTo97 commented Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this PR does

Records the architecture decision and lands Phase 0 + Phase 1 of the migration: keep the git-committed JSON as the canonical, diff-reviewable source of truth, build a derived SQLite prices.db in the sync pipeline (never committed), and have the dashboard read from it with the JSON path kept as an airtight fallback.

Tracking issue: #60. Decision record: wiki/decisions/0001-canonical-pricing-database-storage.md.

Why

The canonical data is JSON served over the GitHub raw CDN — there is no backend. Every read materializes the entire ~2.9 MB / 3,244-model dataset to answer one question: the dashboard parses it all on mount and filters O(n) per keystroke; the SDKs parse 2.9 MB on cold start then scan linearly; database/history/ is ~115 MB. The cost is payload + parse, not the pricing math. A derived SQLite artifact enables indexed/FTS queries and a compact history time-series without losing the reviewable JSON diff or committing a binary that rewrites every 6 hours.

Changes

Decision record (docs)

  • wiki/decisions/0001-canonical-pricing-database-storage.md — context, three options with trade-offs, the decision, proposed v1 schema, consequences, rollout.
  • wiki/decisions/README.md — decisions index + format conventions.

Phase 0 — build & publish prices.db (non-breaking)

  • services/sync/src/tokenpricing_sync/build_db.py — materializes prices.json + all history snapshots into SQLite following the v1 schema: meta (PRAGMA user_version = 1), providers, models (indexes on provider/category/model_type), model_sources, price_history, populated models_fts FTS5. Prices as REAL; VACUUM + ANALYZE.
  • services/sync/.../cli.py — new build-db subcommand; sync builds the DB automatically after writing JSON.
  • services/sync/tests/test_build_db.py — 11 JSON↔DB equivalence tests.
  • .gitignore — ignores prices.db (+ wal/shm); the binary is a published artifact, never committed.
  • .github/workflows/database-sync.yml — builds and uploads prices.db as the prices-db workflow artifact.
  • services/sync/README.md, docs/database.md — document the artifact and command.

Phase 1 — dashboard reads SQLite with a guaranteed JSON fallback

  • services/dashboard/src/lib/sqlite-db.ts — opens prices.db via sql.js-httpvfs, validates PRAGMA user_version = 1, runs queries in a worker thread, maps rows back to the existing RawModelInfo/RawPricingData shapes.
  • services/dashboard/src/lib/data.ts — loadPricingData() tries SQLite first and catches every failure mode (DB 404, worker/WASM error, schema mismatch, flag off) to fall through to the original JSON fetch. No user-visible change on fallback.
  • public/sqlite.worker.js + public/sql-wasm.wasm vendored so Vite serves them at the correct Pages base URL; sql.js-httpvfs added to package.json.
  • Feature flags: VITE_SQLITE_ENABLED (default on) and VITE_PRICES_DB_URL (override; defaults to VITE_CANONICAL_DATA_ROOT/current/prices.db).

Verification

  • services/sync: 17 passed, ruff clean. build-db on live data → 3,244 models / 166 providers / 133,441 history rows / user_version = 1 / FTS working.
  • services/dashboard: typecheck 0 errors, lint clean (1 pre-existing unrelated warning), npm run build passes with worker + wasm emitted at the right base.
  • CI on the head commit: Docs Build, Dashboard Build, and tests all green. No .db tracked in git.

Known limitation / follow-up

  • Phase 1 uses whole-file .db download + in-browser SQL (serverMode: "full"), not HTTP range-request streaming. raw.githubusercontent.com does not reliably support Range (flagged in the decision record). True range streaming can be turned on once prices.db is served from GitHub Pages (which supports ranges) with a chunked config — a clean follow-up tracked in Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60. The in-browser-SQL approach already removes the per-keystroke O(n) JS scans.

Remaining phases (tracked in #60)

  • Phase 2: optional SQLite backends in the Python/TS SDKs.
  • Phase 3: history charts + sync diffing on the DB.
  • Phase 4: revisit JSON retention (follow-up decision record).

Open questions for review

  1. REAL vs TEXT-decimal price storage (currently REAL).
  2. Publish target for prices.db — workflow artifact (current), GitHub Release, or GitHub Pages (needed to unlock range streaming).
  3. Keep JSON canonical long-term (recommended) or eventually retire it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d

claude added 4 commits June 27, 2026 08:29
@claude
docs(adr): propose SQLite read layer for canonical pricing database
49c7398 
Add ADR 0001 recording the decision to keep the git-committed JSON as the
canonical, diff-reviewable source of truth while building a derived SQLite
artifact (prices.db) in the sync pipeline for fast, indexed, range-request
reads from the dashboard and SDKs. Includes a proposed v1 schema and the
phased rollout summary. Wire the ADR into the VitePress docs sidebar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
docs(adr): link ADR 0001 to migration tracking issue #60
cc2809b 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
docs: move architecture decision records to wiki/decisions
e5078d2 
Relocate ADR 0001 and the decisions index from docs/adr to wiki/decisions,
and drop the VitePress sidebar wiring (decisions are not part of the
published docs site).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat(sync): add build-db step — SQLite prices.db from canonical JSON …
6102ff6 
…(Phase 0)

Implements ADR 0001 Phase 0:

- Add services/sync/src/tokenpricing_sync/build_db.py: builds a SQLite
  database from prices.json + history snapshots following the v1 schema
  (meta, providers, models with indexes, model_sources, price_history,
  models_fts FTS5 table). VACUUM + ANALYZE at the end.
- Wire a `build-db` CLI subcommand into cli.py (mirrors the existing
  sync/normalize pattern). The main `sync` flow also calls build_db()
  automatically after writing JSON.
- Add database/current/prices.db and *.db-wal/shm to .gitignore so the
  binary is never committed.
- Add tests/test_build_db.py: 11 fixture-based equivalence tests covering
  model count, field values, FTS search, price_history rows, and
  PRAGMA user_version. All 17 tests pass; ruff clean.
- Update .github/workflows/database-sync.yml to run build-db after the
  sync step and upload prices.db as a workflow artifact (prices-db).
- Update services/sync/README.md and docs/database.md to document the
  derived prices.db artifact and build-db command.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@DiTo97 DiTo97 changed the title docs(adr): propose SQLite read layer for the canonical pricing database SQLite read layer for the canonical pricing database — decision record + Phase 0 Jun 27, 2026
@DiTo97
DiTo97 marked this pull request as ready for review June 27, 2026 08:54
claude[bot]
claude Bot reviewed Jun 27, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.

Once credits are available, reopen this pull request to trigger a review.

claude added 2 commits June 27, 2026 09:01
@claude
fix(docs): use absolute URL for ADR link to satisfy VitePress dead-li…
31f60e9 
…nk check

The relative link from docs/database.md into wiki/decisions/ points outside
the VitePress site root, which the build's dead-link checker rejected. Use
the canonical GitHub URL instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat(dashboard): add SQLite read path with airtight JSON fallback (Ph…
dfe3fcc 
…ase 1)

Introduces an optional sql.js-httpvfs read path for prices.db alongside
the existing JSON fetch, keeping the JSON path as a mandatory fallback.

Changes:
- Install sql.js-httpvfs; copy sqlite.worker.js + sql-wasm.wasm to
  public/ so Vite serves them at the correct base URL in both dev and
  the GitHub Pages build (/tokenpricing/).
- New src/lib/sqlite-db.ts: opens prices.db (full-file serverMode),
  checks PRAGMA user_version == 1, queries all models, and maps rows
  back to the existing RawModelInfo/RawPricingData shape.
- src/lib/data.ts: loadPricingData() now tries SQLite first and
  silently falls back to JSON on ANY error (DB 404, worker failure,
  schema version mismatch, VITE_SQLITE_ENABLED=false).
- Feature flags: VITE_SQLITE_ENABLED=false disables SQLite entirely;
  VITE_PRICES_DB_URL overrides the DB URL for reviewer testing.
- All existing component contracts (ModelRow, ExplorerView, CompareView,
  HistoryView) are unchanged.

Quality gates: typecheck PASSED, lint 0 errors (pre-existing warning in
explorer.tsx retained), build PASSED.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@DiTo97 DiTo97 changed the title SQLite read layer for the canonical pricing database — decision record + Phase 0 SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 Jun 27, 2026
claude added 11 commits June 27, 2026 09:23
@claude
ci: publish prices.db to rolling database-latest GitHub Release
7bcd008 
Add a database-release workflow (workflow_dispatch + database-release-* tag
trigger) that builds prices.db from the committed canonical JSON and uploads
it to a rolling GitHub Release under the fixed tag database-latest, giving a
stable download URL. Wire the same publish step into the scheduled
database-sync workflow so the release stays fresh on every sync. Record the
resolved publish target in ADR 0001 and document the URL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
ci: only commit refreshed data on scheduled sync runs
6f17d8b 
Gate the commit step to schedule events so a manual workflow_dispatch can
build and publish the prices.db release/artifact without pushing a data
refresh commit. Scheduled runs remain the canonical data-producing cadence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat(python-sdk): add optional SQLite read backend (Phase 2)
713e3c3 
Adds an experimental opt-in SQLite backend to the Python SDK
(libraries/python). When TOKENPRICING_USE_SQLITE=1 is set, the SDK
downloads prices.db from the rolling 'database-latest' GitHub Release,
caches it on disk at ~/.cache/tokenpricing/prices.db with a 6h TTL
(mirroring the JSON cache), and answers lookups/searches via indexed SQL
and FTS5 instead of parsing the full 2.9 MB JSON. On any failure it
silently falls back to the existing HTTP-JSON path; the public API and
all return types are unchanged. JSON remains the default backend.

Files changed:
- src/tokenpricing/sqlite_backend.py (new): download, cache, schema
  validation (PRAGMA user_version=1), read-only open, row→Pydantic
  mapping, get_model() (PK lookup), search_models() (WHERE on indexed
  cols + optional FTS5 via models_fts), get_all_pricing_data().
- src/tokenpricing/pricing.py: _sqlite_enabled() flag check; when ON,
  runs get_all_pricing_data() via asyncio.to_thread and catches all
  exceptions to fall back to JSON.
- tests/test_sqlite_backend.py (new): 26 tests covering lookup parity,
  search filter parity, FTS search, user_version validation, missing-FTS
  validation, broken-DB fallback, and default-off behavior.
- README.md: "SQLite backend (experimental, opt-in)" section with env
  vars, default-off rationale, and speedup notes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
style: satisfy pre-commit (ruff format + end-of-file newline)
ad5d67a 
Apply ruff-format to the sync build_db module/tests and add the missing
trailing newline to the vendored dashboard SQLite worker, matching the
repo-wide pre-commit hooks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat: publish slim + full SQLite databases (SDK vs web platform)
75b129a 
build-db now emits two artifacts from the canonical JSON:
- prices.db (~30MB): full v1 schema incl. price_history, for the dashboard
- prices-current.db (~1.3MB): same schema minus price_history, for SDKs

Both are published to the rolling database-latest release and uploaded as
workflow artifacts. The Python SDK now defaults to the slim DB; the dashboard
defaults to the full DB (release URL). Docs and ADR 0001 updated to describe
the two assets and the interim static-hosting trade-offs (browser CORS/Range
on release assets, GitHub Pages as the next step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat(typescript): add optional SQLite read backend (Phase 2)
cb69bc4 
Implements the TypeScript SDK half of Phase 2 of the SQLite migration,
mirroring the Python sqlite_backend.py reference implementation.

- New `src/sqlite-backend.ts`: downloads and caches `prices-current.db`
  from the rolling GitHub Release, validates schema version (PRAGMA
  user_version = 1) and FTS table presence, then converts DB rows to
  `RawPricingData` — identical in shape to the JSON path output.
  Activation is opt-in via `TOKENPRICING_USE_SQLITE=1`. Any failure
  raises `SQLiteBackendError` for transparent JSON fallback.
- `src/pricing.ts`: wires in the SQLite path before the HTTP-JSON fetch;
  catches `SQLiteBackendError` and falls back with a `console.warn`.
- `package.json`: adds `better-sqlite3` as optionalDependency and
  `@types/better-sqlite3` as devDependency.
- `tsup.config.ts`: marks `better-sqlite3` as external so it is never
  bundled into browser-targeted output.
- `tests/sqlite-backend.test.ts` + `tests/fixtures/prices-current.db`:
  22 tests covering happy-path data mapping, env-var flag parsing, and
  error paths (404 download, wrong schema version, missing FTS table).
- `README.md`: documents the SQLite backend, env-var overrides, and
  fallback behaviour.

All 73 tests pass; typecheck and lint clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
test(typescript): build SQLite fixture at runtime instead of committi…
fd8b435 
…ng a binary

The SQLite fixture DB is now generated at test time with better-sqlite3 and
the committed tests/fixtures/prices-current.db binary is removed — derived
SQLite databases are never checked into git, matching the migration's premise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
build(typescript): update pnpm lockfile for better-sqlite3 + allow it…
19fa0c1 
…s build

CI installs with pnpm --frozen-lockfile, so the lockfile must include the new
better-sqlite3 / @types/better-sqlite3 entries. Also add pnpm
onlyBuiltDependencies for better-sqlite3 so pnpm runs its native build script
(prebuild-install) in CI; otherwise the native addon is missing and the
SQLite tests cannot load it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
build(typescript): allow better-sqlite3 build via pnpm-workspace.yaml
3688b33 
pnpm v11 (used in CI) no longer reads the package.json "pnpm" field and
hard-errors on ignored build scripts. Move the build allowance to
pnpm-workspace.yaml using the repo's existing allowBuilds convention (already
used for esbuild) so better-sqlite3's native install script runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
docs(adr): add ADR 0002 — data persistence and JSON retention (Phase 4)
b832174 
Resolve ADR 0001's deferred question: JSON stays the canonical, committed
source of truth indefinitely; SQLite remains a derived read layer. Record the
history-representation problem and options (compact time-series JSON vs
out-of-repo vs Pages-served DB vs external store), recommending the infra-free
compact-JSON path as the near-term Phase 3 move and deferring Pages/external
store as the future 'proper persistence mechanism'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
@claude
feat(phase3): emit compact price-history.json, dashboard fetches sing…
d9202b5 
…le artifact

- services/sync/src/tokenpricing_sync/history.py: new module; reads all
  database/history/prices-*.json snapshots and builds a compact per-model
  price time-series {generated_at, models: {id: [{t,in,out,cr,cc}]}}
- services/sync/src/tokenpricing_sync/cli.py: call write_compact_history()
  after snapshots are written; add compact_history key to sync() return dict
- services/sync/tests/test_history.py: 12 unit tests covering ordering,
  null cache fields, single-point models, and empty-directory edge case
- services/dashboard/src/lib/data.ts: loadHistorySnapshots() now fetches
  current/price-history.json in one request; falls back to legacy
  multi-snapshot listing on 404 or parse error (loadHistorySnapshotsLegacy)
- database/current/price-history.json: generated from committed history/
  snapshots (42 × 3,252 models); 17.5 MB raw / ~430 KB gzip vs 12 × 2.8 MB
  (33.6 MB) for the old path — single request replaces up to 12 parallel fetches
- docs/database.md: add price-history.json row to layout table

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vu4DV9Df3uHTsG87YYVE3d
This was referenced Jul 30, 2026
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DiTo97 @claude