Repository navigation
bench(reader): measure large string IN predicates with the ArrowReader harness - #3741
Draft
alexanderbianchi wants to merge 6 commits into
Draft
alexanderbianchi wants to merge 6 commits into
alexanderbianchi wants to merge 6 commits into
Conversation
There were no in-repo benchmarks for the ArrowReader, so every performance claim in the perf epic (apache#2172) had to be validated against external workloads. This adds a criterion harness that writes Parquet files to a local temp dir and reads them back through the normal FileIO path, measuring the per-FileScanTask overhead that dominates scans of tables with many small files. Benchmark groups: - many_small_files: scans 16/64/256 small files, reporting files/sec so per-file overhead is directly visible. - concurrency: a fixed corpus read at concurrency 1/4/16, exercising both the single-concurrency fast path and the buffered/flattened multi-task path. - migrated_table: files without embedded field IDs read via name mapping, isolating the migrated-table schema-resolution cost. - same_file_splits: one multi-row-group file read as 1/8/32 byte-range tasks, surfacing the redundant per-split metadata fetch that metadata caching (item #5) targets. - with_predicate: scans carrying a bound predicate with row-group filtering and row selection enabled, exercising the per-task row-filter setup. This gives a reproducible baseline for evaluating reader optimizations such as operator caching (apache#2177) and metadata reuse. (cherry picked from commit 562f4c7)
Wall-clock time on the local FS cannot show the benefit of round-trip-saving optimizations, but it also undersells the same-file-split problem, whose cost is redundant *bytes fetched* rather than latency. Report ScanMetrics::bytes_read as the same_file_splits throughput basis (a deterministic count, not a sample), and print the read amplification per split count. Reading one ~1.2MB file as 32 byte-range splits fetches ~15.7x the file size, because each split re-fetches the Parquet metadata independently. This is the machine-independent evidence for the cost that same-file metadata caching (proposed in apache#2172, attempted in apache#2100) targets; a caching implementation would drive the amplification back toward 1x. (cherry picked from commit 736466f)
Address review feedback from mbutrovich: - Every group now asserts the expected total row count, so a change that reads fewer or duplicated rows fails loudly instead of reporting a bogus win. The same_file_splits assertion doubles as a live regression guard for the apache#2614 sub-row-group-split duplication. - The module doc now states the measurement regime explicitly: warm-cache CPU/decode path on local FS; operator caching (apache#2177) and the latency items (apache#2173, apache#2181) are not measurable here and a flat local result must not be read as "no benefit"; only Int64/Utf8 decode is exercised. (cherry picked from commit 44b6354)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #2557 and #3739. This draft carries Liang-Chi Hsieh’s three Criterion harness commits from #2558, rebased onto
986153cb5, and adds string, Int32, and Int64INcases to thewith_predicategroup.What changes are included in this PR?
The reader scans 262,144 rows from warm local Snappy-compressed Parquet files. For each key type, it evaluates lists of 2, 4, 8, 16, 32, 400, and 5,000 distinct literals against the same data in one file or 64 files. Each scan projects the row ID and key. Before timing, each case verifies all 131,072 expected row IDs. Every timed scan checks the row count.
Criterion binds a fresh predicate outside each timed iteration. The timer includes building file tasks, cloning the predicate per file, building any index on first use, and reading the batches. The reader uses 8,192-row batches and one file at a time. The draft also ports the harness to the current
FileScanTaskandArrowReaderOptionsbuilders.Are these changes tested?
Tables report Criterion mean scan time in milliseconds on an Apple M5 Pro, with 20 samples per case. Each reader ran separately with the same harness.
Large string lists
The original reader is
986153cb5. The indexed reader is63ebab2e4from #3739. The FNV control changes only the hasher used for membership lookups.At 5,000 string keys across 64 files, the first scan comparison favored FNV. Repeating it in reverse order measured 9.34 ms with FNV and 9.85 ms with foldhash.
Where does the index pay off?
Each cell shows scalar → foldhash at the same list size. Bold marks the first tested size where indexing won in both file layouts. These controls used a 1-second warm-up and a 2-second measurement target.
Two strings already favored hashing in one file, but the scalar path was faster across 64 files. Eight Int64 keys were close across 64 files, and the scalar path was faster in one file. The reader therefore starts indexing at 4 strings, 8 Int32s, or 16 Int64s for this workload.
Integer hashers
Both probes used an index at each cardinality below. Values include index construction and scan time.
Int64 was faster with foldhash in both file layouts at 400 and 5,000 keys. At 5,000 Int32 keys across 64 files the first run favored FNV, with overlapping 95% confidence intervals. Reversing the run order gave 5.02 ms with FNV and 4.97 ms with foldhash. The repeat’s 95% intervals were 4.99–5.04 ms and 4.94–5.01 ms, respectively.
Reproduce
Start from this draft’s branch in a disposable checkout. This runs the large-list comparison on the original reader and on the reviewed reader with the same benchmark source.
To repeat the FNV comparison, replace
use foldhash::fast::FixedState;incrates/iceberg/src/expr/predicate.rswithuse fnv::FnvBuildHasher as FixedState;and runbench_large fnv.For the crossover measurements, the harness was paired with reader
0c658b41e. InSetExpression::membership_index, setMIN_INDEX_SIZEtousize::MAXfor the scalar control and to2for the indexed control. Apply the integer-hasher change from6c9ea0658to use foldhash for all three types. Run the same Criterion command with the filterwith_predicate/(string|int32|int64)_in_(1_file|64_files)/(2|4|8|16|32)$and--measurement-time 2.AI Disclosure
Codex adapted the harness, wrote the cases, and ran the comparisons. The fixture uses short
user-key-*strings and a 50% match rate. The companion DataFusion benchmark measures longer resource keys and a much lower match rate. End-to-end taxi benchmark