Skip to content

[codex] Avoid explicit metadata transactions for snapshot reads - #20

Draft
bill-ph wants to merge 1 commit into
v1.5-variegatafrom
codex/snapshot-query-no-long-metadata-tx
Draft

[codex] Avoid explicit metadata transactions for snapshot reads#20
bill-ph wants to merge 1 commit into
v1.5-variegatafrom
codex/snapshot-query-no-long-metadata-tx

Conversation

@bill-ph

@bill-ph bill-ph commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Phase 4 of the metadata snapshot-read plan: normal snapshot-safe reads now avoid DuckLake's explicit long-lived metadata transaction path.

This keeps explicit DuckLake metadata transactions for write/current/maintenance paths while routing normal SnapshotQuery reads through a no-explicit-transaction metadata connection. Postgres-backed snapshot reads still go through postgres_query, so postgres-scanner may continue creating its own implicit read transaction underneath; that remains Phase 5 scope.

Changes

  • Split DuckLake metadata connection creation so SnapshotQuery can run without BeginTransaction() while Execute, CurrentQuery, and RawQuery keep the explicit transaction path.
  • Added SnapshotQueryInTransaction for maintenance/commit paths that still require explicit metadata transaction behavior.
  • Threaded the explicit transaction intent through SQLite/default and Postgres metadata managers.
  • Extended DuckLake metadata logging with metadata_api and ducklake_explicit_metadata_transaction so tests can assert the boundary directly.
  • Added focused SQLLogic tests for normal snapshot reads, inlined snapshot reads, Postgres snapshot reads, pinned snapshot reads after newer commits, and maintenance preservation across flush/rewrite/cleanup.

Validation

  • git diff --check
  • python3 scripts/check_metadata_query_intent.py
  • cmake --build build/release --target unittest -j 8
  • cmake --build build/release --target duckdb -j 8
  • cmake --build build/release --target ducklake_loadable_extension -j 8
  • ./build/release/test/unittest --test-dir ./ test/sql/metadata/snapshot_reads_no_ducklake_tx.test
  • ./build/release/test/unittest --test-dir ./ test/sql/metadata/inlined_snapshot_reads_no_ducklake_tx.test
  • ./build/release/test/unittest --test-dir ./ test/sql/metadata/maintenance_snapshot_reads_keep_ducklake_tx.test
  • ./build/release/test/unittest --test-dir ./ test/sql/metadata/snapshot_reads_stay_pinned_after_commits.test
  • ./build/release/test/unittest --test-dir ./ test/sql/general/ducklake_metadata_logging.test
  • PGHOST=127.0.0.1 PGUSER=postgres DUCKLAKE_CI=1 ./build/release/test/unittest --test-config test/configs/postgres.json --test-dir ./ test/sql/metadata/postgres_snapshot_reads_no_ducklake_tx.test

@bill-ph
bill-ph force-pushed the codex/snapshot-query-no-long-metadata-tx branch from 8b769eb to 05ee7bf Compare May 19, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant