Skip to content

bench(reader): measure large string IN predicates with the ArrowReader harness - #3741

Draft
alexanderbianchi wants to merge 6 commits into
apache:mainfrom
alexanderbianchi:codex/reader-in-list-bench
Draft

alexanderbianchi wants to merge 6 commits into
apache:mainfrom
alexanderbianchi:codex/reader-in-list-bench

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #2557 and #3739. This draft carries Liang-Chi Hsieh’s three Criterion harness commits from #2558, rebased onto 986153cb5, and adds string, Int32, and Int64 IN cases to the with_predicate group.

What changes are included in this PR?

The reader scans 262,144 rows from warm local Snappy-compressed Parquet files. For each key type, it evaluates lists of 2, 4, 8, 16, 32, 400, and 5,000 distinct literals against the same data in one file or 64 files. Each scan projects the row ID and key. Before timing, each case verifies all 131,072 expected row IDs. Every timed scan checks the row count.

Criterion binds a fresh predicate outside each timed iteration. The timer includes building file tasks, cloning the predicate per file, building any index on first use, and reading the batches. The reader uses 8,192-row batches and one file at a time. The draft also ports the harness to the current FileScanTask and ArrowReaderOptions builders.

Are these changes tested?

Tables report Criterion mean scan time in milliseconds on an Apple M5 Pro, with 20 samples per case. Each reader ran separately with the same harness.

Large string lists

The original reader is 986153cb5. The indexed reader is 63ebab2e4 from #3739. The FNV control changes only the hasher used for membership lookups.

Distinct keys Files Original (ms) FNV (ms) foldhash (ms) Original / foldhash FNV / foldhash
400 1 106.62 4.15 3.54 30.11× 1.17×
400 64 113.86 7.49 6.92 16.46× 1.08×
5,000 1 1,093.15 5.80 4.14 264.16× 1.40×
5,000 64 1,102.81 9.25 10.65 103.56× 0.87×

At 5,000 string keys across 64 files, the first scan comparison favored FNV. Repeating it in reverse order measured 9.34 ms with FNV and 9.85 ms with foldhash.

Where does the index pay off?

Each cell shows scalar → foldhash at the same list size. Bold marks the first tested size where indexing won in both file layouts. These controls used a 1-second warm-up and a 2-second measurement target.

Keys String, 1 file String, 64 files Int32, 1 file Int32, 64 files Int64, 1 file Int64, 64 files
2 3.35 → 3.23 6.47 → 6.69 1.60 → 1.79 4.55 → 4.98 1.62 → 1.87 4.64 → 6.26
4 3.96 → 3.61 7.17 → 6.60 1.74 → 1.88 4.81 → 4.90 1.65 → 1.87 4.69 → 4.83
8 5.44 → 3.60 8.43 → 6.67 1.88 → 1.77 5.01 → 4.74 1.78 → 1.88 4.84 → 4.82
16 8.29 → 3.51 11.39 → 7.24 2.10 → 1.82 6.19 → 4.83 1.98 → 1.86 5.14 → 4.74
32 16.07 → 3.67 19.96 → 6.67 2.84 → 1.90 6.16 → 4.90 2.46 → 1.88 5.63 → 4.95

Two strings already favored hashing in one file, but the scalar path was faster across 64 files. Eight Int64 keys were close across 64 files, and the scalar path was faster in one file. The reader therefore starts indexing at 4 strings, 8 Int32s, or 16 Int64s for this workload.

Integer hashers

Both probes used an index at each cardinality below. Values include index construction and scan time.

Type Distinct keys Files FNV (ms) foldhash (ms) FNV / foldhash
Int32 400 1 1.88 1.82 1.04×
Int32 400 64 4.95 4.82 1.03×
Int32 5,000 1 2.09 1.93 1.08×
Int32 5,000 64 5.04 5.33 0.95×
Int64 400 1 2.10 1.90 1.11×
Int64 400 64 5.83 5.01 1.17×
Int64 5,000 1 2.25 1.95 1.15×
Int64 5,000 64 6.01 5.46 1.10×

Int64 was faster with foldhash in both file layouts at 400 and 5,000 keys. At 5,000 Int32 keys across 64 files the first run favored FNV, with overlapping 95% confidence intervals. Reversing the run order gave 5.02 ms with FNV and 4.97 ms with foldhash. The repeat’s 95% intervals were 4.99–5.04 ms and 4.94–5.01 ms, respectively.

Reproduce

Start from this draft’s branch in a disposable checkout. This runs the large-list comparison on the original reader and on the reviewed reader with the same benchmark source.

git fetch https://github.com/apache/iceberg-rust.git pull/3739/head
bench_large() {
  cargo bench --locked -p iceberg --bench arrow_reader -- \
    'with_predicate/(string|int32|int64)_in_(1_file|64_files)/(400|5000)$' \
    --warm-up-time 1 --measurement-time 3 --sample-size 20 --noplot --save-baseline "$1"
}
bench_large original
git cherry-pick 986153cb5..63ebab2e4
bench_large foldhash

To repeat the FNV comparison, replace use foldhash::fast::FixedState; in crates/iceberg/src/expr/predicate.rs with use fnv::FnvBuildHasher as FixedState; and run bench_large fnv.

For the crossover measurements, the harness was paired with reader 0c658b41e. In SetExpression::membership_index, set MIN_INDEX_SIZE to usize::MAX for the scalar control and to 2 for the indexed control. Apply the integer-hasher change from 6c9ea0658 to use foldhash for all three types. Run the same Criterion command with the filter with_predicate/(string|int32|int64)_in_(1_file|64_files)/(2|4|8|16|32)$ and --measurement-time 2.

AI Disclosure

Codex adapted the harness, wrote the cases, and ran the comparisons. The fixture uses short user-key-* strings and a 50% match rate. The companion DataFusion benchmark measures longer resource keys and a much lower match rate. End-to-end taxi benchmark

viirya and others added 4 commits November 20, 2026 15:30
There were no in-repo benchmarks for the ArrowReader, so every performance
claim in the perf epic (apache#2172) had to be validated against external
workloads. This adds a criterion harness that writes Parquet files to a
local temp dir and reads them back through the normal FileIO path, measuring
the per-FileScanTask overhead that dominates scans of tables with many small
files.

Benchmark groups:
- many_small_files: scans 16/64/256 small files, reporting files/sec so
  per-file overhead is directly visible.
- concurrency: a fixed corpus read at concurrency 1/4/16, exercising both
  the single-concurrency fast path and the buffered/flattened multi-task
  path.
- migrated_table: files without embedded field IDs read via name mapping,
  isolating the migrated-table schema-resolution cost.
- same_file_splits: one multi-row-group file read as 1/8/32 byte-range
  tasks, surfacing the redundant per-split metadata fetch that metadata
  caching (item #5) targets.
- with_predicate: scans carrying a bound predicate with row-group filtering
  and row selection enabled, exercising the per-task row-filter setup.

This gives a reproducible baseline for evaluating reader optimizations such
as operator caching (apache#2177) and metadata reuse.

(cherry picked from commit 562f4c7)
Wall-clock time on the local FS cannot show the benefit of round-trip-saving
optimizations, but it also undersells the same-file-split problem, whose cost
is redundant *bytes fetched* rather than latency. Report ScanMetrics::bytes_read
as the same_file_splits throughput basis (a deterministic count, not a sample),
and print the read amplification per split count.

Reading one ~1.2MB file as 32 byte-range splits fetches ~15.7x the file size,
because each split re-fetches the Parquet metadata independently. This is the
machine-independent evidence for the cost that same-file metadata caching
(proposed in apache#2172, attempted in apache#2100) targets; a caching implementation
would drive the amplification back toward 1x.

(cherry picked from commit 736466f)
Address review feedback from mbutrovich:

- Every group now asserts the expected total row count, so a change that
  reads fewer or duplicated rows fails loudly instead of reporting a bogus
  win. The same_file_splits assertion doubles as a live regression guard
  for the apache#2614 sub-row-group-split duplication.
- The module doc now states the measurement regime explicitly: warm-cache
  CPU/decode path on local FS; operator caching (apache#2177) and the latency
  items (apache#2173, apache#2181) are not measurable here and a flat local result
  must not be read as "no benefit"; only Int64/Utf8 decode is exercised.

(cherry picked from commit 44b6354)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants