Merge upstream v1.5-variegata (through 542514e1) into posthog/v1.5.5 - #36
Merged
fuziontech merged 13 commits intoJul 30, 2026
Merged
Conversation
Backport 1223 partition remap to v1.5
Fix inline cleanup but on v1.5
Test uses JOIN on attached ducklake catalog, causing 'Multiple streaming scans' error
* Bump DuckDB submodule to v1.5.5 Moves the pin from v1.5.3 (14eca11b) to v1.5.5 (d8cdaa33). No DuckLake source changes were needed - the extension builds clean against v1.5.5 and the full suite passes: 16083 assertions in 451 test cases. Note this does NOT bring in VARIANT extract pushdown. That lives only on duckdb main; the v1.5-variegata branch is 20 commits past v1.5.5 and still ships the pre-rewrite reader (variant_shredded_conversion.cpp, no IsPushdownExtract), so upstream is not backporting it to the 1.5 line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mntu6YhMuqNBFD9snXqvBR * Build against v1.5.5 in CI, not just locally The submodule bump alone was not enough. MainDistributionPipeline's get-duckdb-version job reads .github/duckdb-version and passes it to _extension_distribution.yml, which checks out that DuckDB version itself rather than using the submodule. So CI - including the tagged release build that produces ducklake-linux-{amd64,arm64}.duckdb_extension - would still have compiled against v1.5.3 while the submodule said v1.5.5. Local `make release` uses the submodule and was genuinely built and tested against v1.5.5; only the CI path was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mntu6YhMuqNBFD9snXqvBR * Port upstream DuckDB-test skips for v1.5.5 Bumping to v1.5.5 pulls in DuckDB's newer test suite, and the DBMS Catalog Tests job runs that suite against a DuckLake backend. 11 of its 4087 cases failed - 10 of them tests that do not exist at v1.5.3 - so these are newly-added upstream conformance tests exposing long-standing DuckLake gaps, not regressions from the bump. Upstream ducklake already made exactly these calls when it moved to duckdb main (see "Fix and skip duckdb tests in ducklake", which added 136 lines to this file). Every entry here is ported verbatim from upstream main's attach_ducklake.json, reasons included, so we are not inventing policy: - native compression/storage-file internals (patas, rle, dict_fsst) - native db-file invalidation, checkpoint and WAL-revert semantics - DuckLake gives table entries a new object id on ALTER, so the oid stability assertion cannot hold - parquet writer does not support non-root VARIANT columns - TIMETZ second-precision offsets are quantized by parquet storage - two with upstream's own FIXME/"different error message" notes test_add_col_if_not_exists_default is covered by upstream's skip_error_messages entry rather than a path skip, so that string is added too. Note the diff is larger than the change: json round-trip normalised pre-existing inconsistent indentation. Semantically this is exactly +10 skip paths and +1 skip_error_message, nothing removed or altered - verified by comparing the parsed JSON before and after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mntu6YhMuqNBFD9snXqvBR --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rtitioning Add hive_file_pattern option to flush
Consolidates the two release lines onto posthog/v1.5.5 as the single branch for the DuckDB v1.5.5 era. Both branches already pinned the same DuckDB submodule commit (d8cdaa33); this merge brings over the .github/duckdb-version bump and attach_ducklake.json config updates that landed via #33 on the v1.5.3 branch, alongside the v1.5.5 CI and test-skip commits unique to this branch. After this merges, PRs should target posthog/v1.5.5 and the next release tag (v1.0-posthog.7) cuts from here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fork/posthog/v1.5.3: Bump DuckDB submodule to v1.5.5 (#33)
…ease line Brings in 10 upstream commits on top of the posthog/v1.5.3 -> posthog/v1.5.5 consolidation merge (2c92a42): - 84ef2d1 / 17ced71: Backport partition-id remap fix (duckdb#1288, backport of duckdb#1223). Adds DuckLakePartition::local_partition_id so commit retries remap data-file partition ids off the original transaction-local id instead of a stale id overwritten by a failed commit attempt. Also makes a no-op SET PARTITIONED BY (same fields/transforms) skip the alter entirely, and adds an InternalException guard in DuckLakeTransactionState::GetNewDataFile against committing a data file with a transaction-local partition id. - d8a1881 / c812f6e / 7e41b97: Clean up empty superseded inlined tables (duckdb#1321): drop per-table inlined-file-deletion tables when tables are dropped (metadata manager cleanup list), and drop superseded inlined tables client-side on commit when only flushes occurred. - 542514e / 664400c / d0ce7df / 16be379: Respect per-table hive_file_pattern option in ducklake_flush_inlined_data (duckdb#1348), with test and quack.json skip entry. - 22a6078: upstream-internal merge commit (no content). Conflicts: none. Git auto-merged all 7 modified files; each merged hunk was reviewed against the fork's versions of the commit path (ducklake_transaction.cpp Commit(), ducklake_transaction_state.cpp GetNewDataFile, ducklake_metadata_manager.cpp DropTables cleanup, ducklake_flush_inlined_data.cpp) and applies cleanly to fork behavior: - The client-side superseded-inlined-table drop lands in the no-changes Commit() branch only; the fork's server-side commit path (FlushChanges / ducklake_server_side_commit.cpp) is untouched. - The partition remap fix flows through the shared GetNewPartitionKey/GetNewDataFile pipeline that the fork's server-side commit also uses; where local_partition_id is unset (e.g. staged-commit replays, catalog-loaded partitions) the IsValid() fallback preserves prior behavior. - No .github/CI files were touched by upstream in this range. DuckDB submodule pin kept at d8cdaa33 (v1.5.5); upstream did not move it in this range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
benben
approved these changes
Jul 30, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the fork up to date with upstream's v1.5-stable branch: the 10 commits on
duckdb/ducklake@v1.5-variegatasince the July 10 sync. Zero textual conflicts (git auto-merged all 7 files; every hunk in fork-critical files was still hand-reviewed against the fork's passthrough/server-side commit paths). DuckDB submodule pin stays atd8cdaa33(v1.5.5); no upstream CI files touched.What comes in
DuckLakePartition::local_partition_idso retries remap data-file partition ids from the original transaction-local id instead of one overwritten by a failed attempt; no-opSET PARTITIONED BYskips the alter; newInternalExceptionguard inGetNewDataFile. Directly relevant to our retry-heavy workloads.DropEmptySupersededInlinedTablesClientSide()quack routing.ducklake_flush_inlined_datarespects per-tablehive_file_pattern(Add hive_file_pattern option to flush duckdb/ducklake#1348) + new tests.Fork-interaction review
FlushChanges()/server-side/staged-commit paths reuse the sharedGetNewDataFile/GetNewPartitionKeypipeline, so the remap fix covers them;local_partition_idfalls back to prior behavior where unset (staged replays, catalog loads). Staged commits stage no catalog alters — no gap.ducklake_transaction_state.cpponly inGetNewDataFile; Add commit-loop instrumentation and ducklake_commit_stats() #35 touchesCommit/RetryOnError— disjoint. Verified conflict-free withgit merge-tree --write-treeagainst87f29f29. The newInternalExceptionthrow inside the commit loop will be classified non-retryable by Add commit-loop instrumentation and ducklake_commit_stats() #35's counters, which is correct for an invariant violation.Test plan
GEN=ninja make releasecleanpartition_commit_retry_remap.test), data_inlining 45/45 (incl. both new cleanup tests), plus basic/general/insert/delete/update/alter/catalog smoketest/sql/quack/noop_partition_alter_data_files.testis exercised only here)🤖 Generated with Claude Code
https://claude.ai/code/session_01LUnQHp46rmLKdupmDUr7kG