CockroachDB metadata-store support report + e2e test harness - #24
Open
fuziontech wants to merge 1 commit into
Open
CockroachDB metadata-store support report + e2e test harness#24fuziontech wants to merge 1 commit into
fuziontech wants to merge 1 commit into
Conversation
Experiment: audit + full end-to-end test of CockroachDB v25.2 as a DuckLake metadata catalog (docker compose, single node, Postgres 17 as control baseline). Verdict: near-supportable. Core feature surface passes with pg_use_text_protocol=true / pg_use_ctid_scan=false. Two real gaps: - ducklake_expire_snapshots() / ducklake_flush_inlined_data() fail (ctid-based DELETE path in postgres_scanner; CRDB has no ctid) - concurrent writers lose ~24% of commits (CRDB 40001 "restart transaction" errors not matched by RetryOnError); a one-line patch fixes it, verified 0/45 failures See experiments/crdb/REPORT.md for details and repro scripts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An experiment: how well does CockroachDB work as a DuckLake metadata store? This PR adds the resulting support report (
experiments/crdb/REPORT.md) plus the docker-compose setup, test scripts, and raw outputs to reproduce it.Tested: DuckDB v1.5.3 + ducklake against CockroachDB v25.2.19 (single node, docker compose), with Postgres 17 as a control baseline.
TL;DR of the findings
Verdict: near-supportable. With two scanner settings, the entire core feature surface passes (DDL, DML, deletion vectors, snapshots, time travel, schema evolution, transactions, views, partitioning, compaction, data inlining, change feeds, concurrent readers):
Two real gaps remain, both fixable upstream:
ducklake_expire_snapshots()/ducklake_flush_inlined_data()fail — these route metadata DELETEs through DuckDB's DML planner on the attached catalog, and postgres_scanner addresses rows byctid, which CockroachDB doesn't expose. Every other metadata write goes through rawpostgres_executepassthrough and works fine; routing these DELETEs the same way would fix it (and save round trips on vanilla Postgres too). Until then the catalog grows forever.restart transaction: ... WriteTooOldError, whichRetryOnError()doesn't match (it expects Postgres's "duplicate key ... unique" message). A one-line patch (match"restart transaction") was built and verified: 0/45 failed commits vs 11/45 unpatched. The same gap technically exists for vanilla PG's "could not serialize access" message.Perf: ~2× slower metadata commits than PG17 single-node (~0.5s vs ~0.27s per attach+commit cycle) — expected for a consensus engine; data-file I/O unaffected.
Notes
RetryOnErrorpatch is included as a snippet in the report if we want to follow up with a code PR.🤖 Generated with Claude Code