Skip to content

feat: configure Iceberg reader concurrency and pruning - #126

Draft
alexanderbianchi wants to merge 1 commit into
apache:mainfrom
alexanderbianchi:codex/reader-config
Draft

alexanderbianchi wants to merge 1 commit into
apache:mainfrom
alexanderbianchi:codex/reader-config

Conversation

@alexanderbianchi

Copy link
Copy Markdown
Contributor

Add an interim session-to-reader mapping using the current iceberg-rust pin, based on the reader controls in DFD #595 / #720.

  • Add iceberg.execution.data_file_concurrency_limit per output partition. Preserve the current default of one and normalize zero to one, matching iceberg-rust #3514.
  • Honor datafusion.execution.parquet.pruning for reader row-group filtering.
  • Centralize these settings with the existing batch-size and page-index mapping in a private reader-construction function. Explicit with_reader(...) overrides continue to take precedence.

Once the dependency includes iceberg-rust #3514, replace the private setter mapping with ArrowReaderOptions and with_options(...). The session configuration can remain the same across that update.

Tests use the existing SQL harness: disabling row-group pruning increases data reads without changing results, and zero concurrency still completes a scan. Confirmed the tests fail when the pruning mapping and zero normalization are removed. Existing batch-size and page-index tests remain in place.

Validation:

  • cargo test --workspace --locked
  • cargo clippy --workspace --locked --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Developed with AI assistance.

Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant