feat(store): BYODB — bring your own database (15 backends)

[adityak74]

BYODB — Bring Your Own Database

opencontext stored everything in a single JSON file that was rewritten in full on every operation. This adds 15 pluggable backends behind one async adapter interface, without changing how the CLI, HTTP API, or MCP tools behave.

The JSON file stays the zero-config default. Existing installs need to do nothing — with no configuration, opencontext reads the same file it always has, and OPENCONTEXT_STORE_PATH keeps working.

opencontext db adapters                    # what's available, what's installed
opencontext db test    "postgres://…"      # try before committing
opencontext db use     "postgres://…"      # switch
opencontext db migrate --to "postgres://…" # bring history along

Or use the new Database page in the web UI — pick a backend, test, save, migrate.

---

Backends

Family	Backends
File	JSON (default)
Embedded	SQLite, DuckDB, in-memory
SQL	PostgreSQL, MySQL/MariaDB, SQL Server/Azure SQL, Google Cloud SQL, Cloudflare D1, libSQL/Turso
Document / KV	MongoDB, Redis/Valkey, Google Firestore, Amazon DynamoDB
Multi-model	SurrealDB

JSON, in-memory, SQLite (node:sqlite) and Cloudflare D1 (HTTP) need no dependency at all. Everything else is an optional peer dependency — npm does not auto-install those, so the default install and the Docker image stay lean. Ask for a backend whose driver is missing and you get npm install pg, not a stack trace.

Managed services come along for free via wire compatibility: Neon, Supabase, RDS/Aurora, Azure Database for PostgreSQL, CockroachDB and Timescale on postgres://; PlanetScale and Cloud SQL for MySQL on mysql://; Atlas and Cosmos DB's Mongo API on mongodb://; Upstash and ElastiCache on redis://.

---

Design

CRUD is written once per family, not once per backend.

• adapters/sql.ts — one implementation for all 8 SQL engines. Each supplies a Dialect: placeholder style (? / $1 / @p1), DDL, and string concatenation (|| / + / CONCAT()).
• adapters/document.ts — one implementation for all document/KV stores behind a 6-method DocumentDriver.

Adding a SQL engine is a Dialect plus a ~50-line driver. This is a fraction of the code of 15 independent adapters, and more importantly search and ordering semantics cannot drift between backends, because there is only one implementation of them.

The store became async

Every driver is async, so the adapter interface is too. Signatures are otherwise unchanged, so server.ts and mcp/server.ts only gained awaits. src/mcp/store.ts is now a deprecated re-export so existing imports keep resolving.

---

Testing: one conformance suite, every backend

tests/store/conformance.ts is ~50 tests defining the storage contract. Every adapter must pass it unmodified — it is the only definition of correct behaviour.

It runs with no external services against one backend per family (JSON, SQLite, in-memory), so all three shared implementations are covered in ordinary CI. docker-compose.test.yml brings up the rest.

Verified live against real databases: 450 passing = 9 backends × 50 tests — PostgreSQL 17, MySQL 8.4, SQL Server 2022, MongoDB 8, Redis 7, SurrealDB 3.2, DynamoDB Local, DuckDB, in-memory. Re-verified from a clean slate (down -v → up -d → run), so the documented workflow is known to work rather than assumed.

Cross-family migration verified end-to-end: JSON → SQLite → MongoDB, preserving contexts, bubbles, the bubble relationship, and tags.

---

Bugs the conformance suite alone did not catch

Every backend was put through adversarial UAT beyond the green bar. Several backends passed 50/50 on the first run and were still broken:

Backend	Bug
MySQL	sslmode=verify-full silently fell through to no TLS at all — asking for certificate verification got you plaintext
MySQL	Client-side escaping broke under NO_BACKSLASH_ESCAPES (SQL-injection class); now uses server-side prepared statements
MySQL	Inherited charset: emoji failed outright on latin1 databases; default collation made search accent- and case-insensitive, diverging from every other backend
SQL Server	mssql.connect() returns a process-wide singleton pool — two stores on different databases shared one connection and wrote to the wrong one
SQL Server	Search returned zero rows under Turkish collation: LOWER() uses the database's collation, so LOWER('Istanbul') is ıstanbul and never matches JS's dotted i
SQL Server	Concurrent schema bootstrap deadlocked; now serialised with sp_getapplock
DynamoDB	Eventually-consistent reads violated read-your-writes. Invisible on DynamoDB Local, intermittent "not found" on real AWS
DynamoDB	Concurrent cold start crashed with ResourceInUseException; connect() also returned while the table was still CREATING
DuckDB	close() leaked the DuckDB instance (only the connection was closed) — 4.1MB per open/close cycle, 1.3GB over 300 cycles
MongoDB	undefined optional fields resurrected as null, making Mongo the one backend answering null where others answer nothing
SurrealDB	Wrong auth option key and no namespace bootstrap — nothing worked at all
SurrealDB	A single row missing denormalised fields made all search throw permanently, for every call
Redis	No error listener on the client. node-redis emits error on every socket failure, and an EventEmitter with no listener throws — so the MCP/HTTP server died whenever Redis restarted or failed over
Redis	With that fixed, an unreachable host hung createStore forever (node-redis retries the first connection indefinitely); now gives up after 3 attempts until a connection has succeeded once, then reconnects with backoff
Redis	ping() sat in the offline queue during an outage instead of reporting it — a health check that answers "fine" while the database is down
Redis	close() threw on a second call, and a failed connect() leaked a socket and a reconnect timer that kept the process alive after the caller gave up
Redis	One corrupt hash field made every list and search throw an unattributable SyntaxError, unrepairable through the public API; errors now name the offending field
All	Advertised scheme aliases (mongo://, valkey://) could not connect — drivers passed dsn.raw to client libraries that only accept canonical schemes

That last one produced a security-relevant subtlety worth calling out: naive canonicalization would map rediss:// → redis:// and silently disable TLS. dsn.canonical deliberately preserves rediss:// and mongodb+srv://, and there is a test pinning it.

---

Security

• ~/.opencontext/config.json is written 0600 — connection strings carry passwords.
• Credentials are redacted from every API response, log line, and UI field via redactDsn().
• The UI never receives a stored password back.
• Nothing leaves the machine: your process connects directly to your database.
• A bad connection string cannot take the store down — reconnect() opens the replacement before releasing the current one, and keeps the old one on failure.

---

Also in this PR

Docs. The README described BYODB in its own section, but the rest of it still assumed a JSON file — no db commands, no OPENCONTEXT_DB_URL, no /api/db/* endpoints, no src/store in the project tree. All brought up to date.

Vendor logos for all 14 backends that have one, in the README table and on the Database settings page. Vendored under ui/public/db-logos rather than hotlinked, so the README renders offline and the UI makes no third-party request at runtime. Simple Icons has dropped the Microsoft and Amazon marks, so SQL Server and DynamoDB come from Devicon; both licences are recorded alongside the assets.

Landing page. A "Supports all of these" marquee scrolling all 14 marks, between Features and How it works. The track holds two copies of the list and shifts by exactly half its width, which only loops cleanly if the trailing gap matches the gaps between items — verified in a real browser at zero drift. Chips are aria-hidden with the full list on the container's aria-label, since a marquee is unreadable to a screen reader; the animation is dropped entirely under prefers-reduced-motion and pauses on hover.

A test-harness bug this surfaced. docker compose up + npm run test:backends — the workflow the README documents — failed all 50 mssql tests on a clean machine. Postgres and MySQL create their database from an environment variable; the SQL Server image has no equivalent, and it reports the missing database as Login failed for user 'sa', which sends you looking at the password. Fixed with a healthcheck and an init service, and every service's connection string is now recorded in the compose header.

Includes the eslint-plugin-react-hooks bump from #24, since this branch needs a working npm ci to build the UI. Merging #24 first will make that a no-op.

Known limitations, deliberately not addressed

• Schema migration. DDL is CREATE TABLE IF NOT EXISTS, so a future schema change will not retrofit existing tables. Nothing to migrate today — this feature has never shipped — but it needs a story before the schema changes.
• Document/KV search cost. MongoDB, Redis, Firestore and DynamoDB have no portable case-insensitive substring predicate, so filtering happens in memory. Results are identical; cost grows with store size rather than result size. Documented in the README, with a recommendation to use SQL for large stores.
• DynamoDB 400KB item ceiling and MongoDB 16MB document cap are engine limits a driver cannot work around; both documented.
• SQL Server cannot be addressed by IP while encrypting — TLS forbids an IP as the SNI server name, and the resulting servername error is opaque. Documented with the two workarounds (use a hostname, or ?encrypt=false on a trusted network) rather than papered over with a speculative SNI hack.
• @duckdb/node-api peer range. Upstream publishes its releases as 1.5.5-r.4-style prereleases, which no ordinary semver range matches without includePrerelease. ^1.0.0 stays as the honest declaration for that line; users may see a peer warning.

(An earlier revision of this description claimed redis was missing from package.json. It was not — it was declared, but as ^5.0.0 while UAT ran against 6.2.1. That and two others are corrected in this PR.)

• Nothing outstanding in the UI suite. The two failures this branch was carrying (Landing, Layout) turned out to be stale tests rather than code bugs — one expected hero copy rewritten back in 0f24191, the other rendered Layout at /, which is the public landing page and renders outside Layout, so no sidebar item is ever active there. That made one fail and its sibling pass for the wrong reason. Both fixed; the suite is green at 274.

[cloudflare-workers-and-pages]

Deploying open-context with    Cloudflare Pages

Latest commit:	c4c0846
Status:	✅  Deploy successful!
Preview URL:	https://80581422.open-context.pages.dev
Branch Preview URL:	https://feat-byodb.open-context.pages.dev

View logs