Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The playground config holds `[[catalogs]]` entries with `name`, `type` (`rest` o

### Read path and filter pushdown

`IcebergDataSource` converts the projection to column names and the filters to a single Iceberg `Predicate` (`read/predicate.rs`). It applies `limit` in-stream, has one output partition, and delegates the actual read to iceberg's `TableScan::to_arrow()`.
`FileScanTaskProviderBuilder` converts the projection to column names and the filters to a single Iceberg `Predicate` (`read/predicate.rs`). Lazy discovery advertises the session's target partition count and shares one discovery stream per execution. In a parallel lazy scan, a producer sends planned file tasks to one unbounded Flume queue. Each partition takes a receiver on first poll; a late reader can claim remaining tasks without restarting discovery. A single-partition scan reads directly without a queue. Use DataFusion's `reset_plan_states` to run the same parallel scan again; partition-open order is not an execution boundary. `IcebergDataSource` reads tasks with a shared `ArrowReader` and applies fetch per partition. Eager planning still collects tasks into round-robin groups.

Both providers report every filter as `Inexact`, so DataFusion re-applies the original filters after the scan. The pushed-down predicate therefore only has to be *implied by* the original filter. It may match extra rows but must never drop a row that matches. Unconvertible parts are dropped: an `AND` keeps whichever side converted, while an `OR` needs both sides. Preserve this soundness rule when extending `predicate.rs`; for example, it is why casts and some NaN arithmetic are deliberately not pushed down. Removing a cast is left to DataFusion's simplifier (`unwrap_cast_in_comparison`), which rewrites a comparison only when the result doesn't change.

Expand Down
215 changes: 215 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions crates/datafusion/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,20 @@ autotests = false
async-trait = { workspace = true }
bytes = "1"
dashmap = "6"
flume = "0.12"
datafusion = { workspace = true }
futures = "0.3"
iceberg = { workspace = true }
parquet = "59.2"
tokio = { workspace = true }
uuid = { version = "1.18", features = ["v7"] }

[[bench]]
name = "lazy_scan"
harness = false

[dev-dependencies]
criterion = "0.8"
iceberg-datafusion-test-tools = { workspace = true }
insta = "1"
parquet = "59.2"
Expand Down
Loading