Skip to content

bench: measure large string IN lists on the DataFusion Iceberg scan - #130

Draft
alexanderbianchi wants to merge 2 commits into
apache:mainfrom
alexanderbianchi:codex/taxi-in-list-bench
Draft

alexanderbianchi wants to merge 2 commits into
apache:mainfrom
alexanderbianchi:codex/taxi-in-list-bench

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Benchmark

This Criterion benchmark measures how long DataFusion takes to plan and collect a query of the form SELECT resource_key FROM taxi WHERE resource_key IN (...). It builds a seven-file Iceberg taxi table once, generates a distinct 63–68 byte string key for each trip, and checks the exact matching keys before timing each list size. The companion Iceberg reader change starts using a hash set at 32 distinct string literals. apache/iceberg-rust#3739

Results

At 400 keys, mean query time fell from 155.08 ms to 14.91 ms (10.40× faster). Both readers returned the same 400 keys out of 175,000 trips.

Distinct keys Baseline wall time (ms) Patched wall time (ms) Speedup
1 5.68 5.19 1.10×
16 10.97 11.11 0.99×
31 17.84 16.68 1.07×
32 17.01 11.27 1.51×
100 43.04 14.04 3.07×
369 139.31 13.33 10.45×
400 155.08 14.91 10.40×
1,000 372.62 14.01 26.59×

These are Criterion’s mean wall-clock estimates from 20 samples per case on an Apple M5 Pro, using the same seven January 2024 NYC taxi source files for both builds. Each run used one local query at a time with four Tokio workers and four DataFusion target partitions. The 95% confidence intervals at 400 keys were 149.88–163.15 ms before and 14.66–15.23 ms after. Taxi fixture

Run it

cargo bench -p iceberg-datafusion --bench taxi_in_list

The default uses the repository’s deterministic taxi fixture. To reproduce the table above, set TAXI_DATA_DIR to the archived fixture’s data/ directory with its Git LFS files materialized. Both runs used DataFusion 55.2.0, Arrow 59.3.0 and reader baseline 9354b42. The second build added only the hash-set commit. Baseline reader · Reader change

With the baseline reader selected, save a Criterion baseline. After selecting the patched reader, compare in the same target directory.

cargo bench -p iceberg-datafusion --bench taxi_in_list -- --save-baseline scalar_arrow59 --warm-up-time 1 --measurement-time 3
cargo bench -p iceberg-datafusion --bench taxi_in_list -- --baseline scalar_arrow59 --warm-up-time 1 --measurement-time 3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant