Skip to content

feat: read lazy Iceberg scans across target partitions - #128

Draft
gabotechs wants to merge 3 commits into
apache:mainfrom
gabotechs:feat/lazy-scan-target-partitions
Draft

gabotechs wants to merge 3 commits into
apache:mainfrom
gabotechs:feat/lazy-scan-target-partitions

Conversation

@gabotechs

Copy link
Copy Markdown
Contributor

Closes #84.

A lazy Iceberg scan only exposes one output partition, so DataFusion can schedule just one scan partition however many files the scan matches, and has to add a round-robin repartition above it to get any parallelism. This makes the lazy scan expose the session's target_partitions instead, without listing files up front.

The partitions share one TableScan::plan_files() stream. A background task drains it into a channel that every partition pulls from, so whichever partition is free takes the next file, and the first file can be read while the rest are still being planned. Once every partition has dropped its feed, for example when a LIMIT is satisfied, discovery stops.

before                                    after

DataSourceExec (1 partition)              plan_files() ──► shared channel
  plan_files() ──► p0 reads every file                       ├──► p0 reads next file
RepartitionExec: RoundRobinBatch(N)                          ├──► p1 reads next file
  spreads batches over N partitions                          └──► pN-1 reads next file

The shared discovery state is created per execution through DataFusion's sibling state (DataSource::create_sibling_state / open_with_args), the same hook FileScanConfig uses for its shared work queue. FileScanTaskProvider gets a create_sibling_state hook that IcebergDataSource forwards, so re-running a plan after reset_state starts a fresh discovery instead of finding the previous one drained. The commits are split so the first one shows the idea with the state kept on the provider, and the second moves it into sibling state.

FileScanTaskProvider::open now yields DataFusion results; errors only get converted to iceberg errors where the Arrow reader needs them.

One trade-off: a LIMIT without ORDER BY can now open up to one file per partition before the limit stops the scan, rather than stopping after the first file.

A lazy scan exposed a single output partition, so DataFusion had one scan
partition to schedule no matter how many files the scan matched. It now
exposes the session's target partitions. One background task drains
TableScan::plan_files() into a shared channel, and every partition pulls the
next file from it, so the first file can be read while the rest are still
being planned. Discovery stops once every partition has dropped its feed.

FileScanTaskProvider::open now yields DataFusion results, converting to
iceberg errors only where the Arrow reader needs them.
The discovery state lived on the provider, so it outlived a single execution:
running the same physical plan again found the fanout already drained.
DataSourceExec already creates per-execution state shared by a source's
partitions, and clears it on reset_state. FileScanTaskProvider gains a
create_sibling_state hook that IcebergDataSource forwards to DataFusion, and
the state reaches each partition's open through FileScanTaskProviderContext.
@gabotechs
gabotechs marked this pull request as draft November 20, 2026 10:49
The lazy scan now produces one partition per target partition, so plans lose
the round-robin repartition above the scan, and a plain LIMIT coalesces the
scan's partitions instead of passing its single partition through. Tests that
executed partition 0 of a single-partition scan now coalesce it first.

Tests whose result depends on the partition count pin it: the fetch cap on
whole-scan statistics and the EXPLAIN ANALYZE row counts under LIMIT use one
partition, and the LIMIT read-count test uses two, so fewer than the seven
taxi files can be open when the limit cancels the scan. A LIMIT without ORDER
BY now only asserts how many rows come back, since any partition can supply
them.
Comment on lines -73 to +74
) -> Result<iceberg::scan::FileScanTaskStream> {
) -> Result<BoxStream<'static, Result<FileScanTask>>> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with FileScanTaskStream is that the errors returned in the messages are iceberg::Errors, not DataFusion errors. I did not notice in our past work, but the API here should probably be full DataFusion.

@alexanderbianchi alexanderbianchi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some findings, not sure how big of an issue either are. I'm going to work back into #127 whatever I can.

partition: Option<usize>,
schema: &Schema,
) -> Result<Statistics> {
if partition.is_some_and(|partition| partition != 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to keep the snapshot row estimate for the whole scan, but return unknown stats for each reader when there’s more than one partition? The readers claim files at runtime, so we don’t know either reader’s share during planning.

I tried this on the taxi fixture with target_partitions = 2:

let scan = harness.scan().await?;
assert_eq!(scan.output_partitioning().partition_count(), 2);

let stats = StatisticsContext::new();
let rows = |partition| {
    stats.compute(scan.as_ref(), &StatisticsArgs::new().with_partition(partition))
        .map(|s| s.num_rows)
};

// rows(None)    → Inexact(175000)
// rows(Some(0)) → Inexact(175000)
// rows(Some(1)) → error: Invalid Iceberg scan partition Some(1)

Partition 1 is valid, and partition 0 is getting the whole table’s estimate. Changing only the bounds check would give both readers 175,000 rows. For this fixture I’d expect Inexact(175000) globally and Absent for each reader.

partition: usize,
context: Arc<TaskContext>,
) -> Result<SendableRecordBatchStream> {
self.open_with_args(OpenArgs::new(partition, context))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is dropping support for direct DataSource::open on lazy scans intentional? I tried the same call on the parent and on this PR, with target_partitions = 1 so there’s no second reader to coordinate with:

let plan = harness.scan().await?;
let source = iceberg_source(&plan).unwrap();
let mut rows = source.open(0, harness.state().task_ctx())?;
assert!(rows.try_next().await?.is_some());

On the parent this returns a batch. On this PR, open returns Missing sibling state for LazyFileScanTaskProvider. The normal DataSourceExec path still works because it supplies sibling state; it’s the direct call that changed.

We should preserve the single-partition case, or, if direct opens are no longer supported for lazy scans, document that callers should execute the DataSourceExec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multiple output partitions for lazy file discovery

2 participants