Skip to content

feat: derive eager Iceberg column statistics from planned files - #35

Draft
alexanderbianchi wants to merge 1 commit into
main-upstream-dd-18-11-2026from
alex.bianchi/iceberg-selected-file-stats-review
Draft

alexanderbianchi wants to merge 1 commit into
main-upstream-dd-18-11-2026from
alex.bianchi/iceberg-selected-file-stats-review

Conversation

@alexanderbianchi

Copy link
Copy Markdown

Eager scans use the metrics retained on their selected FileScanTasks instead of reopening manifests. Projected column sizes inform relation estimates; scan parallelism still uses whole-file lengths, capped by the number of selected files. The task-count handler now visits the selected partition groups once without allocating or sorting file indices.

This is the bottom of a two-PR stack. It does not add a statistics provider or change join selection. Schema evolution is out of scope. Downstream workspaces using the fork must align their root Iceberg patch; the dd-source pin will be handled separately.

Test plan: cargo test --locked -p datafusion-distributed-iceberg --test desired_task_count --test statistics --test nested_column_statistics.

Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant