Repository navigation
Conversation
A lazy scan exposed a single output partition, so DataFusion had one scan partition to schedule no matter how many files the scan matched. It now exposes the session's target partitions. One background task drains TableScan::plan_files() into a shared channel, and every partition pulls the next file from it, so the first file can be read while the rest are still being planned. Discovery stops once every partition has dropped its feed. FileScanTaskProvider::open now yields DataFusion results, converting to iceberg errors only where the Arrow reader needs them.
The discovery state lived on the provider, so it outlived a single execution: running the same physical plan again found the fanout already drained. DataSourceExec already creates per-execution state shared by a source's partitions, and clears it on reset_state. FileScanTaskProvider gains a create_sibling_state hook that IcebergDataSource forwards to DataFusion, and the state reaches each partition's open through FileScanTaskProviderContext.
The lazy scan now produces one partition per target partition, so plans lose the round-robin repartition above the scan, and a plain LIMIT coalesces the scan's partitions instead of passing its single partition through. Tests that executed partition 0 of a single-partition scan now coalesce it first. Tests whose result depends on the partition count pin it: the fetch cap on whole-scan statistics and the EXPLAIN ANALYZE row counts under LIMIT use one partition, and the LIMIT read-count test uses two, so fewer than the seven taxi files can be open when the limit cancels the scan. A LIMIT without ORDER BY now only asserts how many rows come back, since any partition can supply them.
50703e2 to
aa44991
Compare
| ) -> Result<iceberg::scan::FileScanTaskStream> { | ||
| ) -> Result<BoxStream<'static, Result<FileScanTask>>> { |
There was a problem hiding this comment.
The issue with FileScanTaskStream is that the errors returned in the messages are iceberg::Errors, not DataFusion errors. I did not notice in our past work, but the API here should probably be full DataFusion.
alexanderbianchi
left a comment
There was a problem hiding this comment.
Some findings, not sure how big of an issue either are. I'm going to work back into #127 whatever I can.
| partition: Option<usize>, | ||
| schema: &Schema, | ||
| ) -> Result<Statistics> { | ||
| if partition.is_some_and(|partition| partition != 0) { |
There was a problem hiding this comment.
We need to keep the snapshot row estimate for the whole scan, but return unknown stats for each reader when there’s more than one partition? The readers claim files at runtime, so we don’t know either reader’s share during planning.
I tried this on the taxi fixture with target_partitions = 2:
let scan = harness.scan().await?;
assert_eq!(scan.output_partitioning().partition_count(), 2);
let stats = StatisticsContext::new();
let rows = |partition| {
stats.compute(scan.as_ref(), &StatisticsArgs::new().with_partition(partition))
.map(|s| s.num_rows)
};
// rows(None) → Inexact(175000)
// rows(Some(0)) → Inexact(175000)
// rows(Some(1)) → error: Invalid Iceberg scan partition Some(1)Partition 1 is valid, and partition 0 is getting the whole table’s estimate. Changing only the bounds check would give both readers 175,000 rows. For this fixture I’d expect Inexact(175000) globally and Absent for each reader.
| partition: usize, | ||
| context: Arc<TaskContext>, | ||
| ) -> Result<SendableRecordBatchStream> { | ||
| self.open_with_args(OpenArgs::new(partition, context)) |
There was a problem hiding this comment.
Is dropping support for direct DataSource::open on lazy scans intentional? I tried the same call on the parent and on this PR, with target_partitions = 1 so there’s no second reader to coordinate with:
let plan = harness.scan().await?;
let source = iceberg_source(&plan).unwrap();
let mut rows = source.open(0, harness.state().task_ctx())?;
assert!(rows.try_next().await?.is_some());On the parent this returns a batch. On this PR, open returns Missing sibling state for LazyFileScanTaskProvider. The normal DataSourceExec path still works because it supplies sibling state; it’s the direct call that changed.
We should preserve the single-partition case, or, if direct opens are no longer supported for lazy scans, document that callers should execute the DataSourceExec.
Closes #84.
A lazy Iceberg scan only exposes one output partition, so DataFusion can schedule just one scan partition however many files the scan matches, and has to add a round-robin repartition above it to get any parallelism. This makes the lazy scan expose the session's
target_partitionsinstead, without listing files up front.The partitions share one
TableScan::plan_files()stream. A background task drains it into a channel that every partition pulls from, so whichever partition is free takes the next file, and the first file can be read while the rest are still being planned. Once every partition has dropped its feed, for example when aLIMITis satisfied, discovery stops.The shared discovery state is created per execution through DataFusion's sibling state (
DataSource::create_sibling_state/open_with_args), the same hookFileScanConfiguses for its shared work queue.FileScanTaskProvidergets acreate_sibling_statehook thatIcebergDataSourceforwards, so re-running a plan afterreset_statestarts a fresh discovery instead of finding the previous one drained. The commits are split so the first one shows the idea with the state kept on the provider, and the second moves it into sibling state.FileScanTaskProvider::opennow yields DataFusion results; errors only get converted to iceberg errors where the Arrow reader needs them.One trade-off: a
LIMITwithoutORDER BYcan now open up to one file per partition before the limit stops the scan, rather than stopping after the first file.