Skip to content

feat: execute lazy Iceberg scans across local partitions - #127

Open
alexanderbianchi wants to merge 6 commits into
apache:mainfrom
alexanderbianchi:feat/pr84-parallel-lazy-scan
Open

alexanderbianchi wants to merge 6 commits into
apache:mainfrom
alexanderbianchi:feat/pr84-parallel-lazy-scan

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #84.

Lazy Iceberg scans now expose the session’s target_partitions without listing files during query planning. Following Gabriel’s approach in #128, the first parallel reader to be polled starts one TableScan::plan_files() stream. A producer sends the resulting file tasks to an unbounded Flume queue. Each scan partition takes a receiver: the next available reader gets the next unclaimed file, not a copy of it. Files can be read before planning finishes.

flowchart LR
    M[Iceberg manifests] --> P["TableScan::plan_files()"]
    P -->|producer sends file tasks| Q["Unbounded Flume queue · one per parallel execution"]
    Q -->|next unclaimed task| A[Scan partition 0]
    Q -->|next unclaimed task| B[Scan partition 1]
    Q -->|next unclaimed task| C[Scan partition N−1]
Loading

The queue and producer belong to DataFusion’s sibling state for one execution, not to the table provider. A late reader can take remaining files without restarting discovery; it cannot take over a partially read file. Unopened receivers can keep file discovery running after an active reader stops. Queuing a file does not read its Parquet data. Resetting the physical plan starts an independent execution. With one output partition the reader opens the planning stream directly, without a producer or queue.

Discovery uses the TableScan already built for the resolved snapshot, projected columns, and predicate. The snapshot row estimate still applies to the whole scan, not to each parallel reader. Eager file grouping and its selected-file statistics are unchanged.

Comparing Engines

In Trino’s Iceberg connector, the coordinator calls scan.planFiles() as the scheduler requests splits. A worker receives a data-file range and its applicable delete files, not a manifest to expand. Spark can read manifests on executors, but that is a planning job: the matching file metadata comes back to the driver.

This PR keeps the same distinction between planning files and assigning reads. File planning is shared by the scan, not repeated per partition. The data source uses one task provider to plan the scan’s files. Each local partition claims the next task, with its data file and applicable deletes already identified; it doesn’t scan manifests to find work of its own. What changes is how local DataFusion partitions claim those tasks—not where manifests are planned or how work is sent across machines.

Benchmark

lazy_scan measures execution including file discovery, excluding fixture creation and SQL/physical planning. Both queries project fare_amount from the same warm local fixture: 160 manifests, 4,000 Parquet files, 4 million rows, four Tokio workers, and target_partitions = 4. The baseline is main plus only the benchmark; it exposes one Iceberg scan partition.

Query Main + benchmark This PR Elapsed time
Full scan 907.92 ms 238.35 ms 73.7% less (3.8× speedup)
LIMIT 1 477.28 µs 564.04 µs 18.2% more (+87 µs)

These are Criterion point estimates from 20 samples, not cold object-store measurements. The full scan benefits from parallel readers; the one-row query is slower than the single-partition baseline. For comparison, on the previous #127 base, replacing the four-reader mutex implementation with Flume took the full scan from 242.90 ms to 227.51 ms, with no significant LIMIT 1 change in that run.

To reproduce, use the same CARGO_TARGET_DIR on both revisions:

# First commit (benchmark only)
cargo bench --locked -p iceberg-datafusion --bench lazy_scan -- --sample-size 20 --save-baseline before
# PR head
cargo bench --locked -p iceberg-datafusion --bench lazy_scan -- --sample-size 20 --baseline before
`

/// Builds the scan and returns the stream of its file tasks. The stream
/// performs no work until first polled.
fn build_discovery(&self) -> Result<FileScanTaskStream> {
let scan = match self.snapshot_id {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbianchi
alexanderbianchi marked this pull request as ready for review November 20, 2026 05:09

@gabotechs gabotechs left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are some opportunities for improvement in how this PR is structured, I think it can be made significantly simpler and with better structured shared state management.

One thing that stands out is that this PR seems to be re-inventing its own work-stealing mechanism, rather than using existing great libraries in the Rust ecosystem like https://github.com/zesterer/flume.

I've experimented with a simpler approach in #128, split in 3 different commits just to better show the diff:

  • d3fec8e, just a +102-25 LOC change for full support for partitioning, just storing the state in the LazyFileScanTaskProvider
  • aa611e6, moving statefulness fully to upstream's sibling state API, which I have to admit I don't love, but it was already agreed upstream as the standard way of sharing state across DataSource partitions.
  • Another commit with test snapshot updates not relevant here

Additionally, I think this project could really benefit from some benchmarks to qualify these kind of changes.

Comment thread crates/datafusion/src/read/scan.rs Resolved
@alexanderbianchi
alexanderbianchi force-pushed the feat/pr84-parallel-lazy-scan branch from 5ce46c2 to 26aab26 Compare November 20, 2026 21:01
@alexanderbianchi

alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@gabotechs pulled in your approach here, with a benchmark as well. I imagine we want a proper benchmark harness but I found the test harness pretty useful so I think it's worth commiting.

Review guide

  1. 7147efe — Full-scan and LIMIT 1 benchmarks
  2. e1f4448 — DataFusion errors in file-task streams
  3. aa00a75 — One Flume queue, competing scan partitions
  4. 87b3142 — Queue lifetime scoped to one execution
  5. 8180e56 — Regression tests and plan snapshots
  6. 26aab26 — AGENTS.md update

Why the diff is larger than #128

#127 is +835/−130; Gabriel’s #128 is +246/−104. Of the 589 extra added lines, 296 are benchmark code and dependency changes, 258 are tests and fixtures, and one is AGENTS.md. Only 34 are under crates/datafusion/src; both PRs use the same Flume queue design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multiple output partitions for lazy file discovery

2 participants