Skip to content

Cherry-pick upstream #1234: fix add-files partition key lookup for repeated transforms - #26

Draft
EDsCODE wants to merge 3 commits into
mainfrom
cherry-pick-upstream-1234
Draft

Cherry-pick upstream #1234: fix add-files partition key lookup for repeated transforms#26
EDsCODE wants to merge 3 commits into
mainfrom
cherry-pick-upstream-1234

Conversation

@EDsCODE

@EDsCODE EDsCODE commented Jul 9, 2026

Copy link
Copy Markdown

Problem

ducklake_add_data_files() writes every ducklake_file_partition_value row at the last transform's partition_key_index when a partition spec applies multiple transforms to one source column — e.g. the PostHog duckling backfill's events spec year(timestamp), month(timestamp), day(timestamp). The lookup map was keyed on the bare source field index, so the three spec entries collapsed to one.

Symptoms downstream:

  • Tier-3 compaction fails with DuckLakeCompactor: Files have different hive partition path.
  • Partition pruning silently misses files.

PostHog worked around this in the Dagster backfill (PostHog/posthog#67168) with a post-register catalog fix-up, but that fix-up needs a direct catalog Postgres connection that CR-based (CNPG-shard) ducklings can't provide from Dagster, so backfills on prod-us/prod-eu currently run with the fix-up disabled (PostHog/charts#13004) and accumulate mis-stamped rows.

Changes

Cherry-picks upstream duckdb/ducklake#1234 (both commits, -x provenance preserved; zero conflicts):

  • a5a6be4d — key the partition lookup on (field index, transform) instead of bare field index; regression test with year/month/day on one timestamp column in add_file_partitioned.test.
  • fda045ba — refactor: each parsed HivePartition records its partition_key_index at map time and validation compares the full transform (handles repeated bucket transforms with different widths too); new add_file_repeated_bucket.test.

Plus a docs commit recording the cherry-pick in the "PostHog changes" list.

Testing

  • just build — clean (732/732 targets; local build needs croaring on CMAKE_PREFIX_PATH, e.g. brew install croaring).
  • ./build/release/test/unittest "test/sql/add_files/*" — 34/34 test cases, 1134 assertions, including both new regression tests.
  • just test (full suite) — all pass: 404 test cases, 14724 assertions, 18 environment-gated skips (postgres_scanner/spatial/sqlite_scanner extensions and CI-only env vars).

Release / follow-up

After merge: tag v1.0-posthog.5 (release binaries publish on tag push), bump DUCKLAKE_EXTENSION_TAG in duckgres Dockerfile + Dockerfile.worker, roll images. Then revert PostHog/charts#13004 and re-run registration on partitions registered while the fix-up was off (the fix-up's DELETE-then-INSERT is idempotent and convergent, so re-registration heals the rot).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants