feat(duckdb): cross-database federation via derived DuckDB connection

[kevinmessiaen]

Summary

Adds cross-database federation to ktx. When a project declares 2+ attach-compatible databases (postgres, mysql, sqlite), ktx derives a virtual _ktx_federated connection backed by an embedded DuckDB that ATTACHes each member read-only and runs cross-catalog joins. From the semantic layer's view there is one connection; DuckDB fans out to the real databases underneath. Live data, no copy.

Answers the question that motivated this: books in postgres, reviews in sqlite, joined in one query.

A federated query goes from ktx.yaml to returned rows for real configs, raw federated SQL works via the ktx sql CLI, the ingest path, and the MCP sql_execution path, and declared cross-DB joins survive re-scan.

Design (locked decisions)

• Federation IS the connection. A pure deriveFederatedConnection(connections, projectDir) computes a descriptor from declared state. Never persisted — no ktx.yaml entry, no flag, no _ktx_federated/ directory. Recomputed every run.
• Activates only when 2+ attach-compatible members exist (postgres/mysql/sqlite). 0 or 1 → nothing changes. Other drivers (snowflake/bigquery/clickhouse/sqlserver) stay standalone.
• Embedded DuckDB, fresh in-memory instance per query (MCP-driven, sporadic — no warm pool to hold idle connections).
• Member connectionId → DuckDB catalog alias (connectionId.schema.table), quoted so hyphenated ids attach correctly.
• Read-only enforced two ways: ATTACH ... READ_ONLY (physical) + assertReadOnlySql (statement) + caller-side validateReadOnly with the duckdb dialect.
• _ktx_ is a reserved connection-id namespace.
• Member config is resolved through each connector's canonical resolver — federation and standalone scans interpret ktx.yaml identically (sqlite path:/url:/~/project-relative; postgres/mysql discrete-fields-or-URL, SSL, search_path).
• One implementation of read-only SQL execution. Every entry point — ktx sql, ingest, and MCP sql_execution — routes through the shared executeProjectReadOnlySql, so the federated-vs-direct decision follows from the connection id, not from which caller invoked it. The CLI and the agent expose the identical set of choices.
• Cross-DB joins are declared-only in v1. Automatic cross-DB discovery is a follow-up.

What changed

Core federation

• context/connections/federation.ts — derivation + FEDERATED_CONNECTION_ID; FederatedMember carries the full member connection config + projectDir; federatedConnectionListing exposes the virtual connection (id, members, usage hint) for discovery surfaces.
• connectors/duckdb/federated-attach.ts (new) — resolves each member's DuckDB ATTACH target by reusing the canonical connector resolvers (sqliteDatabasePathFromConfig, postgresPoolConfigFromConfig, mysqlConnectionPoolConfigFromConfig). sqlite path: resolves end-to-end; SSL (sslmode=require / ssl_mode=REQUIRED) and postgres search_path are preserved for discrete-field configs.
• connectors/duckdb/federated-executor.ts — ATTACH read-only + execute, targets resolved via federated-attach. DuckDB returns integer columns as JS bigint; the executor coerces them to number once here so every consumer (CLI/MCP/ingest/SL) gets a JSON-safe result.

Unified query execution

• context/connections/project-sql-executor.ts (new) — single shared executeProjectReadOnlySql that owns the _ktx_federated routing decision. The ingest executor (ingest-query-executor.ts), the MCP sql_execution port (context/mcp/local-project-ports.ts), and the ktx sql CLI command (sql.ts) all delegate to it. MCP federated errors are classified via KtxQueryError consistently with non-federated SQL.

ktx sql CLI parity

• sql.ts — ktx sql -c _ktx_federated "<join>" now runs federated cross-database queries, matching MCP. The command's forked connection-lookup + single-scan-connector path is removed and replaced by a call to executeProjectReadOnlySql; the local duplicate dialect helper and the up-front config guard are deleted (the shared connector factory raises the same "not configured" error for unknown ids). Direct -c <member> queries are unchanged. KtxSqlQueryExecutionResult gained an optional headerTypes so --json output is preserved.

Federated-connection discoverability

• connection.ts (CLI ktx connection) and context/mcp/local-project-ports.ts (MCP connection_list) both surface the _ktx_federated entry — id, member connection ids, and a short usage hint — via the one shared federatedConnectionListing builder, so an agent can discover that cross-database querying exists and how to address it. members/hint thread through LocalConnectionInfo, KtxConnectionSummary, and the MCP output schema as optional fields; DUCKDB is a list-only label and is not added to the driver→connection-type map. The connection_list tool description points agents at the federated id for cross-database joins.

Cross-DB join preservation through ingest

• context/ingest/.../manifest.ts + context/scan/local-enrichment-artifacts.ts — declared cross-DB joins to federated siblings survive a re-scan. The sibling-target set is derived from scanned member state at the producer and honored wherever a cross-DB to: is evaluated.

Semantic layer

• context/sl/local-sl.ts — read-time union of member dirs for _ktx_federated, with member-namespaced source names (pg_books.books) so two members owning a same-named table don't collide. Physical source.table is unchanged.
• context/sl/local-query.ts — duckdb dialect + federated id resolution; federated executed-plan metadata reports duckdb.
• context/sl/source-files.ts — reserve _ktx_ prefix.

Setup / deps / docs

• setup-databases.ts — informational federation notice (no prompt, no persisted state).
• @duckdb/node-api dependency.
• docs-site/.../concepts/cross-database-federation.mdx + nav — documents the config shape, fully-qualified table naming, member-namespaced federated source names, and querying _ktx_federated directly via ktx sql and the MCP sql_execution tool.

Test plan

• pnpm --filter @kaelio/ktx run type-check — clean
• Dead-code (Biome + Knip default + production) — clean on packages/cli
• Federation tests pass, including live DuckDB integration tests: cross-catalog sqlite join with correct per-book averages; INSERT rejected by read-only; hyphenated catalog ids attach and join; sqlite path: resolved end-to-end; the shared executor's federated path against real DuckDB; the MCP sql_execution path running a real _ktx_federated join; a production-path test proving a manual cross-DB join survives re-scan.
• ktx sql parity tests: member-direct execution, _ktx_federated routing to the shared executor (member connector not used), unknown-id error, --json headerTypes preserved; an end-to-end ktx sql -c _ktx_federated cross-file sqlite join through the real executor.
• Discoverability tests: _ktx_federated appears in ktx connection and MCP connection_list with members + hint when 2+ attach-compatible members exist, and is absent otherwise.
• bigint regression guards: a federated query selecting an integer column returns JSON-safe numbers and survives JSON.stringify on both the executor result and the MCP sql_execution path (previously the MCP federated path threw Do not know how to serialize a BigInt on any integer column).
• Reviewer: confirm CI's full suite is green (the local broad suite has pre-existing GPG-signing failures in git-init test fixtures, unrelated to this feature — they fail at project init before the test body; federation tests using the file-store seed pattern are git-free).

Follow-ups (not blocking)

• Automatic cross-DB relationship discovery (v1 is declared-only).
• Quote user-authored three-part table:/to: references in generated SQL where a reserved identifier could appear.
• Federated results carry no column types (headerTypes); the DuckDB executor produces none. Direct member queries still report types.
• bigint values above 2^53 lose precision when coerced to number (consistent with the existing plain/pretty CLI output); a string-fallback for out-of-range integers is a possible follow-up.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
ktx-docs-site	Ready	Preview, Comment	Jun 14, 2026 6:36am

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.71429% with 28 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ackages/cli/src/context/mcp/local-project-ports.ts	52.00%	5 Missing and 7 partials ⚠️
...ages/cli/src/connectors/duckdb/federated-attach.ts	81.48%	0 Missing and 5 partials ⚠️
...cli/src/context/scan/local-enrichment-artifacts.ts	75.00%	2 Missing and 2 partials ⚠️
packages/cli/src/setup-databases.ts	55.55%	4 Missing ⚠️
...ages/cli/src/connectors/shared/string-reference.ts	85.71%	0 Missing and 1 partial ⚠️
.../context/ingest/adapters/live-database/manifest.ts	91.66%	0 Missing and 1 partial ⚠️
packages/cli/src/context/sl/local-query.ts	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!