Skip to content

Use planned Iceberg column stats on the dd-source Arrow 59 pin - #34

Draft
alexanderbianchi wants to merge 4 commits into
main-upstream-dd-18-11-2026from
alex.bianchi/dd-source-iceberg-stats-196245
Draft

alexanderbianchi wants to merge 4 commits into
main-upstream-dd-18-11-2026from
alex.bianchi/dd-source-iceberg-stats-196245

Conversation

@alexanderbianchi

Copy link
Copy Markdown

Based on 676f79b, the DFD revision pinned by dd-source main and by dd-source#196245. Keeps the three existing stats commits through 78f47d1, including single-pass eager scan statistics and ColumnByteStatisticsProvider.

The only change on top of that tested stack is a dependency bump: the direct dependencies, workspace patch and lockfile now use DataDog/iceberg-rust@7fda03019, matching dd-source#196245. That revision already contains apache/iceberg-rust#3424. No Iceberg cherry-pick or DataFusion 56 upgrade.

Validation: cargo tree --locked resolves one Iceberg revision, DataFusion 55.0.0 and Arrow 59.3.0. The statistics and nested_column_statistics integration suites (21 tests) and the column_byte_statistics provider tests (5 tests) pass with --locked.

(cherry picked from commit 692e362)
(cherry picked from commit 0a4193b)
(cherry picked from commit 4edf0ec)
Exports datafusion_distributed::ColumnByteStatisticsProvider so an embedder can
register it in a StatisticsRegistry and keep column-byte estimates through
grouping keys and inner joins. It is not registered automatically by Iceberg.

(cherry picked from commit 29248e7)
Request column statistics from the scan (iceberg-rust with_column_stats,
apache/iceberg-rust#3424) and summarize them from the planned FileScanTasks
instead of re-reading manifests, so a scan reads its manifests once.

Point iceberg and iceberg-storage-opendal at DataDog/iceberg-rust 9354b42
(upstream main on Arrow 59), the revision dd-source uses, and patch
datafusion-iceberg's apache/iceberg-rust dependency to the same source so
FileScanTask is one type.

The retained metrics stay on each task, so they are also serialized to workers
with the file list and counted by the planning memory reservation: iceberg-rust
has no way to drop them from a planned task.

(cherry picked from commit db8db25)
Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant