♻️ SQLite state foundation with migrations and full metadata parity

[Robdel12]

Why

The filesystem reporter state was hitting scaling limits under heavier local TDD/CLI usage. This switches us to a SQLite-first state model with explicit schema versioning + migrations so we can safely evolve build tracking and related state over time.

What changed

• Added better-sqlite3 and implemented a migration-driven state store
  • schema_migrations tracking
  • versioned schema setup for report/comparison state and metadata state
  • one-time legacy imports from existing JSON files
• Broke up src/tdd/state-store.js into focused modules under src/tdd/state-store/
  • sqlite-store.js (DB backend)
  • file-store.js (legacy/test backend)
  • migrations.js (schema + migration runner)
  • events.js, constants.js, utils.js (shared internals)
  • src/tdd/state-store.js is now a thin public facade
• Fully ported metadata/state reads/writes to DB-backed store APIs
  • baseline metadata
  • hotspot metadata
  • region metadata
  • baseline build metadata (MCP/plugin consumers)
• Updated server/router/service/context callsites to use DB state
  • dashboard/events/health/static report now read baseline metadata from DB
  • TDD handler reset/delete flows now mutate metadata through state store
  • TDD service writes baseline build metadata through DB metadata APIs
• Ported global daemon server registry to SQLite in src/tdd/server-registry.js
  • migration-tracked schema
  • one-time import from legacy ~/.vizzly/servers.json
• Expanded/updated tests for new DB behavior and migration paths
  • metadata suites
  • server registry suite
  • router/context/static-report/TDD service suites

Test plan

• npm run lint
• node --test --test-reporter=spec tests/server/handlers/tdd-handler.test.js tests/server/routers/dashboard.test.js tests/server/routers/events.test.js tests/server/routers/health.test.js tests/services/static-report-generator.test.js tests/tdd/metadata/baseline-metadata.test.js tests/tdd/metadata/hotspot-metadata.test.js tests/tdd/metadata/region-metadata.test.js tests/tdd/server-registry.test.js tests/tdd/tdd-service.test.js tests/utils/context.test.js tests/server/http-server.test.js
• npm run build
• npm test

[claude]

PR Review: SQLite state foundation with migrations and full metadata parity

Overall this is a solid, well-scoped migration. The schema versioning story is clean, the file/SQLite backends share a coherent interface, and the dependency injection pattern preserves testability throughout. A few things worth addressing before merge:

---

Performance: store opened/closed per HTTP request

In dashboard.js and health.js every route handler calls createStateStore(...) / stateStore.close(). Each open runs applySchemaMigrations (queries schema_migrations) plus two legacy-migration guard reads before doing the actual work. Under active TDD polling this adds meaningful overhead.

The events router shows the right pattern — create one store, subscribe, close on disconnect. Dashboard and health should receive the store as a dependency (injected at server startup) the same way the TDD handler does with injectedStateStore.

---

Per-operation store opening in metadata modules

baseline-metadata.js, hotspot-metadata.js, and region-metadata.js all use:

function withStateStore(workingDir, operation) {
  let store = createStateStore({ workingDir });
  try { return operation(store); }
  finally { store.close(); }
}

Every loadBaselineMetadata / saveBaselineMetadata call opens and closes the DB, triggering migration checks each time. During screenshot processing this is called in a hot path. The store should be injected or cached at the call site rather than created per-operation.

---

Module-level stateEmitters Map never cleaned up

let stateEmitters = new Map();   // src/tdd/state-store.js

Entries are added by workingDir but never removed. In tests (and hypothetically in long-running daemon mode cycling across projects) this leaks EventEmitter instances. Adding a stateEmitters.delete(workingDir) in close() — or exposing a clearEmitter(workingDir) helper for tests — would fix this.

---

findAvailablePort fallback silently returns an occupied port

// src/tdd/server-registry.js
for (let i = 0; i < maxAttempts; i++) { ... }
return startPort;   // ← returned even when startPort was occupied

If all 100 candidates are occupied the function returns startPort unconditionally. The server will then fail to bind with a confusing error. Returning null or throwing here would surface the real problem earlier.

---

Legacy migration marks completion even on partial failure

// src/tdd/state-store.js — maybeMigrateLegacyJson
} catch (error) {
  output.debug?.('state', `legacy report JSON migration skipped: ${error.message}`);
} finally {
  setKey('legacy_json_migrated', '1');    // ← set even on error
}

A transient read error (permissions, disk full, malformed JSON mid-write) silently discards the legacy data permanently. Moving the setKey call to the success path, or only marking it done after a successful replaceReportDataInternal, would avoid silent data loss.

---

createSqliteStateStore — no error handling around DB open

let db = new DatabaseImpl(resolvedDbPath);

A locked, corrupted, or read-only DB throws synchronously here with no try/catch. Since this now runs inside HTTP request handlers (dashboard, health), an unhandled throw will crash the request or the process. At minimum, let it propagate with a clear message; ideally surface it as a 500 with a log line.

---

Minor nits

• buildSummary makes four .filter() passes over the same array; a single reduce would be cleaner for larger comparison sets (not a blocker).
• write() in ServerRegistry uses delete-all + re-insert inside a transaction rather than upsert. Correct, but more disruptive than needed for the register use-case it's adjacent to.
• The engines constraint in better-sqlite3 (20.x || 22.x || ...) should be reflected in the root package.json engines field if it isn't already, so consumers get a clear error on unsupported runtimes.

---

What's working well

• Migration versioning with schema_migrations + idempotent IF NOT EXISTS DDL is solid.
• WAL mode, synchronous = NORMAL, busy_timeout = 5000, and foreign_keys = ON are all the right pragmas for this use-case.
• Transactions wrapping every multi-step mutation — good consistency guarantees.
• ON DELETE CASCADE on comparison_details provides a safety net even though the code also deletes explicitly.
• The dual-backend design (SQLite in production, file in tests) keeps the test suite fast without mocking SQLite.
• close() guards (if (!this.db) return) and silent-swallow in the catch are appropriate for cleanup paths.
• The subscribe/unsubscribe pattern through the module-level EventEmitter correctly decouples DB lifecycle from SSE connection lifecycle.

[vizzly-testing]

Vizzly - Visual Test Results

CLI Reporter - Approved

19 comparisons, no changes detected.

View build

CLI TUI - Approved

5 comparisons, no changes detected.

View build

---

_{feat/sqlite-state · 6a9c32e2}