Repository navigation
feat: execute lazy Iceberg scans across local partitions - #127
alexanderbianchi wants to merge 6 commits into
Conversation
| /// Builds the scan and returns the stream of its file tasks. The stream | ||
| /// performs no work until first polled. | ||
| fn build_discovery(&self) -> Result<FileScanTaskStream> { | ||
| let scan = match self.snapshot_id { |
There was a problem hiding this comment.
I think there are some opportunities for improvement in how this PR is structured, I think it can be made significantly simpler and with better structured shared state management.
One thing that stands out is that this PR seems to be re-inventing its own work-stealing mechanism, rather than using existing great libraries in the Rust ecosystem like https://github.com/zesterer/flume.
I've experimented with a simpler approach in #128, split in 3 different commits just to better show the diff:
- d3fec8e, just a +102-25 LOC change for full support for partitioning, just storing the state in the
LazyFileScanTaskProvider - aa611e6, moving statefulness fully to upstream's sibling state API, which I have to admit I don't love, but it was already agreed upstream as the standard way of sharing state across DataSource partitions.
- Another commit with test snapshot updates not relevant here
Additionally, I think this project could really benefit from some benchmarks to qualify these kind of changes.
Use the generated taxi fixture to scan 4 million rows across 4,000 files. Measure collection on a fresh physical plan, leaving fixture creation and SQL planning outside the timed region.
5ce46c2 to
26aab26
Compare
|
@gabotechs pulled in your approach here, with a benchmark as well. I imagine we want a proper benchmark harness but I found the test harness pretty useful so I think it's worth commiting. Review guide
Why the diff is larger than #128#127 is +835/−130; Gabriel’s #128 is +246/−104. Of the 589 extra added lines, 296 are benchmark code and dependency changes, 258 are tests and fixtures, and one is AGENTS.md. Only 34 are under |
Closes #84.
Lazy Iceberg scans now expose the session’s
target_partitionswithout listing files during query planning. Following Gabriel’s approach in #128, the first parallel reader to be polled starts oneTableScan::plan_files()stream. A producer sends the resulting file tasks to an unbounded Flume queue. Each scan partition takes a receiver: the next available reader gets the next unclaimed file, not a copy of it. Files can be read before planning finishes.flowchart LR M[Iceberg manifests] --> P["TableScan::plan_files()"] P -->|producer sends file tasks| Q["Unbounded Flume queue · one per parallel execution"] Q -->|next unclaimed task| A[Scan partition 0] Q -->|next unclaimed task| B[Scan partition 1] Q -->|next unclaimed task| C[Scan partition N−1]The queue and producer belong to DataFusion’s sibling state for one execution, not to the table provider. A late reader can take remaining files without restarting discovery; it cannot take over a partially read file. Unopened receivers can keep file discovery running after an active reader stops. Queuing a file does not read its Parquet data. Resetting the physical plan starts an independent execution. With one output partition the reader opens the planning stream directly, without a producer or queue.
Discovery uses the
TableScanalready built for the resolved snapshot, projected columns, and predicate. The snapshot row estimate still applies to the whole scan, not to each parallel reader. Eager file grouping and its selected-file statistics are unchanged.Comparing Engines
In Trino’s Iceberg connector, the coordinator calls
scan.planFiles()as the scheduler requests splits. A worker receives a data-file range and its applicable delete files, not a manifest to expand. Spark can read manifests on executors, but that is a planning job: the matching file metadata comes back to the driver.This PR keeps the same distinction between planning files and assigning reads. File planning is shared by the scan, not repeated per partition. The data source uses one task provider to plan the scan’s files. Each local partition claims the next task, with its data file and applicable deletes already identified; it doesn’t scan manifests to find work of its own. What changes is how local DataFusion partitions claim those tasks—not where manifests are planned or how work is sent across machines.
Benchmark
lazy_scanmeasures execution including file discovery, excluding fixture creation and SQL/physical planning. Both queries projectfare_amountfrom the same warm local fixture: 160 manifests, 4,000 Parquet files, 4 million rows, four Tokio workers, andtarget_partitions = 4. The baseline is main plus only the benchmark; it exposes one Iceberg scan partition.LIMIT 1These are Criterion point estimates from 20 samples, not cold object-store measurements. The full scan benefits from parallel readers; the one-row query is slower than the single-partition baseline. For comparison, on the previous #127 base, replacing the four-reader mutex implementation with Flume took the full scan from 242.90 ms to 227.51 ms, with no significant
LIMIT 1change in that run.To reproduce, use the same
CARGO_TARGET_DIRon both revisions: