Skip to content

fix(db): guard SQLite store init against concurrent opens - #686

Merged
tobi merged 3 commits into
tobi:mainfrom
brettdavies:fix/sqlite-busy-timeout
Jun 24, 2026
Merged

tobi merged 3 commits into
tobi:mainfrom
brettdavies:fix/sqlite-busy-timeout

Conversation

@brettdavies

@brettdavies brettdavies commented May 26, 2026

Copy link
Copy Markdown
Contributor

Fixes three crashes that hit when multiple qmd processes open the same index at once: the failure mode in #710 (an update or query racing a long embed, an agent fanning out searches, or a first-open migration racing a routine command).

Root cause

Three independent faults on the shared createStoreinitializeDatabase path, all triggered by concurrent opens:

  1. SQLITE_BUSY: database is locked on any contended write. bun:sqlite and better-sqlite3 both default busy_timeout to 0, so a writer that loses the lock throws on contact instead of waiting. WAL lets readers and one writer coexist but does not serialise writers.
  2. trigger documents_ai already exists. The FTS sync triggers were dropped and recreated on every open as separate autocommit statements, so two connections interleave between the DROP and the CREATE (A drops, B drops, A creates, B creates, throw). busy_timeout serialises individual statements but not the DROP/CREATE pair.
  3. database is locked while migrating a cold database to WAL. PRAGMA journal_mode = WAL needs a brief exclusive lock and does not invoke the busy handler, so concurrent first-ever opens throw immediately regardless of busy_timeout.

Fix

  • src/db.ts: openDatabase sets PRAGMA busy_timeout (default 120000, override QMD_SQLITE_BUSY_TIMEOUT; 0 restores fail-fast) and enables WAL with a bounded retry within the same budget, so the cold-database journal migration survives concurrent opens. Connection-level pragmas now live in one place.
  • src/store.ts: the FTS trigger rebuild is gated behind PRAGMA user_version and applied inside one IMMEDIATE transaction with a double-checked read. The DROP/CREATE pair is atomic across connections and runs once per schema version instead of on every open. Bump STORE_SCHEMA_VERSION to reissue changed trigger bodies to existing databases.

Tests

  • test/db.test.ts: busy_timeout default, per-connection application, env override, =0 fail-fast, garbage-falls-back-to-default, and a real-lock-contention timing check.
  • test/store-concurrency.test.ts (new): spawns N processes that open the same database at once (cold and existing), asserting no already exists / database is locked throw and that the triggers, FTS table, and user_version survive. Fails against the pre-fix code, passes after.
  • tsc -p tsconfig.build.json --noEmit clean. Node and Bun suites pass; the LLM-pipeline integration tests fail in this environment on VRAM pressure (same baseline noted in feat(serve): qmd serve shared model server + RemoteQMD client (supersedes #511) #663) and reproduce identically against bare main.

Closes #710.

@brettdavies
brettdavies force-pushed the fix/sqlite-busy-timeout branch from 330fa74 to 158238a Compare May 31, 2026 06:53
…ing SQLITE_BUSY

Running multiple `qmd query` invocations against the same index in parallel (e.g. an agent fanning out searches) caused N-1 of N processes to fail immediately with `SQLiteError: database is locked` from `initializeDatabase`. The first DDL statement (`DROP TRIGGER IF EXISTS documents_ai`) hits the write lock; with `busy_timeout = 0` (the default for both `bun:sqlite` and `better-sqlite3`), the loser throws on contact instead of waiting.

WAL improves read-while-write concurrency but does not serialise concurrent writers. Only the timeout does that. Setting `PRAGMA busy_timeout = 5000` in `openDatabase` makes any connection wait up to 5s for the write lock before failing. Initialization runs in <100ms, so the worst-case wait for typical agent fan-out (5-10 processes) is ~1s and every process eventually succeeds.

Three tests in test/db.test.ts cover the PRAGMA round-trip on a single and multiple connections, plus a behavioural check that SQLite honours the configured timeout under real lock contention.
@brettdavies
brettdavies force-pushed the fix/sqlite-busy-timeout branch from 158238a to 553f607 Compare June 4, 2026 05:00
…Y_TIMEOUT

The initial 5s ceiling was sized for init-time DDL contention (sub-100ms work). On
multi-GB indexes, a single `embed` batch commit can outlast 5s, so an `update` or
`query` that races a long-running `embed` still hits `SQLITE_BUSY` on the first
write. Raise the default to 120000 ms, which outlasts the worst-case batch commit
observed on multi-GB / multi-tens-of-thousands-of-doc indexes.

Add `QMD_SQLITE_BUSY_TIMEOUT` (milliseconds) as an operator escape hatch. Unset,
empty, or unparseable values fall back to the default; `0` restores upstream
fail-fast behaviour for environments that prefer surfacing contention as an
error.

Tests expand from 3 to 6 cases: default value, multi-connection default,
env-override honored, env=0 fail-fast, garbage value falls back to default, and
the existing real-lock-contention timing check (unchanged).

Verified under both `bun:sqlite` and `better-sqlite3` runtimes; `tsc` clean.
@brettdavies brettdavies changed the title fix(db): set busy_timeout so concurrent writers wait instead of throwing SQLITE_BUSY fix(db): guard SQLite store init against concurrent opens Jun 23, 2026
Concurrent processes opening the same index could crash during store
initialization with `trigger documents_ai already exists` or
`database is locked`, even with busy_timeout set.

The FTS sync triggers were dropped and recreated on every open as separate
autocommit statements, so two connections interleaved between the DROP and the
CREATE (A drops, B drops, A creates, B creates -> "already exists");
busy_timeout serialises individual statements but not the DROP/CREATE pair.
Separately, `PRAGMA journal_mode = WAL` needs a brief exclusive lock to migrate
a cold database and does not invoke the busy handler, so concurrent first opens
threw SQLITE_BUSY.

Gate the trigger rebuild behind PRAGMA user_version inside one IMMEDIATE
transaction with a double-checked read, so the DROP/CREATE pair is atomic
across connections and runs once per schema version. Move WAL setup into
openDatabase with a bounded retry within the busy-timeout budget, alongside
busy_timeout. Add a multi-process regression test (cold and existing database)
that fails before and passes after.
@brettdavies
brettdavies force-pushed the fix/sqlite-busy-timeout branch from 77eb395 to 5496df3 Compare June 23, 2026 05:39
@tobi
tobi merged commit 233c680 into tobi:main Jun 24, 2026
2 checks passed
@brettdavies
brettdavies deleted the fix/sqlite-busy-timeout branch June 24, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overlapping qmd processes crash with SQLITE_BUSY (no busy_timeout): update dies at insertContent while an embed runs

2 participants