Repository navigation
perf(arrow): use shared membership sets for large IN predicates - #3739
alexanderbianchi wants to merge 9 commits into
Conversation
| fastnum = { workspace = true } | ||
| flate2 = { workspace = true } | ||
| fnv = { workspace = true } | ||
| foldhash = { workspace = true } |
There was a problem hiding this comment.
I can revert the last commit if we don't want this new dep. But this is what DF uses as well, and did benchmark better.
Iceberg rust bench
| Workload | 400 keys, FNV → foldhash | 5,000 keys, FNV → foldhash |
|---|---|---|
| Evaluate an 8,192-row UTF-8 batch | 47.3 → 25.9 µs (−45%) | 46.0 → 26.5 µs (−42%) |
| Scan 262,144 rows in one Parquet file | 3.05 → 2.31 ms (−24%) | 3.28 → 2.56 ms (−22%) |
| Scan the same rows across 64 files | 5.93 → 5.13 ms (−13%) | 6.96 → 6.20 ms (−11%) |
Datafusion Bench
| Distinct keys | FNV | foldhash | Time saved |
|---|---|---|---|
| 32 | 11.27 ms | 5.65 ms | 50% |
| 400 | 15.09 ms | 7.87 ms | 48% |
| 1,000 | 14.25 ms | 8.56 ms | 40% |
There was a problem hiding this comment.
I can revert the last commit if we don't want this new dep.
I'd keep it. DataFusion already depends on foldhash = "0.2", so DataFusion-based consumers already build it, and foldhash has no dependencies of its own.
Should Int and Long use FixedState too? FNV hashes one byte at a time, so each i64 row costs 8 dependent multiplies before the probe starts. This is the Int64 arm of probe_set at a16ffe7, release build on aarch64:
ldr x0, [x9, x0, lsl #3] ; load the i64
and x1, x0, #0xff
eor x1, x1, x10 ; FNV offset basis
mul x1, x1, x11 ; x11 = FNV prime, round 1 of 8
... ; rounds 2-7
eor x1, x1, x0, lsr #56
mul x2, x1, x11 ; round 8foldhash hashes a single integer with one widening multiply in finish. With HashSet<i64, FixedState> the same arm becomes:
ldr x1, [x9, x1, lsl #3] ; load the i64
eor x2, x1, x10
mul x3, x2, x11
umulh x2, x2, x11 ; independent of the mul above
eor x2, x2, x3Could you add an Int64 case to the reader benchmark and compare the two hashers? If it holds up, SetIndex would also use one hasher for every variant.
| (SetIndex::String(keys), DataType::Utf8) => { | ||
| BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated) | ||
| } | ||
| (SetIndex::String(keys), DataType::LargeUtf8) => { |
There was a problem hiding this comment.
We could be storing the minimum and the maximum length of each value to rule our keys before we even hash them.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @alexanderbianchi, this is a big win for long IN lists, and sharing the index through the Arc keeps the per-file setup cost off the scan path. The extra copy of each string in SetIndex::String looks fine to me, because it is bounded by the literal list the predicate already holds and every file task shares it.
The rewritten description no longer has the template's AI Disclosure section, and "Are these changes tested?" now lists only the benchmarks. Could you restore the disclosure and name the unit tests there? The description becomes the squashed commit message.
| fn probe_set(index: &SetIndex, column: &ArrayRef, negated: bool) -> Option<BooleanArray> { | ||
| Some(match (index, column.data_type()) { | ||
| (SetIndex::Int(keys), DataType::Int32) => { | ||
| BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| { | ||
| keys.contains(&x) ^ negated | ||
| }) | ||
| } | ||
| (SetIndex::Long(keys), DataType::Int64) => { | ||
| BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| { | ||
| keys.contains(&x) ^ negated | ||
| }) | ||
| } | ||
| (SetIndex::String(keys), DataType::Utf8) => { | ||
| BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated) | ||
| } | ||
| (SetIndex::String(keys), DataType::LargeUtf8) => { | ||
| BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated) | ||
| } | ||
| (SetIndex::String(keys), DataType::Utf8View) => { | ||
| BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated) | ||
| } | ||
| _ => return None, | ||
| }) | ||
| } |
There was a problem hiding this comment.
Should probe_set handle dictionary-encoded columns? A Parquet file whose embedded ARROW:schema declares a dictionary type comes back from the decoder as a dictionary array. Files migrated from non-Iceberg writers often carry Arrow types like this. I wrote a Dictionary(Int32, Utf8) column with ArrowWriter and read it through ArrowReader with a 64-string IN. set_predicate saw Dictionary(Int32, Utf8), the index was built, and probe_set returned None, so every batch took the per-literal loop.
DataFusion's InListExpr handles this case with DictionaryFilter. It probes the dictionary values once and then uses take with the keys, which carries key nulls through. The same shape fits here:
| fn probe_set(index: &SetIndex, column: &ArrayRef, negated: bool) -> Option<BooleanArray> { | |
| Some(match (index, column.data_type()) { | |
| (SetIndex::Int(keys), DataType::Int32) => { | |
| BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| { | |
| keys.contains(&x) ^ negated | |
| }) | |
| } | |
| (SetIndex::Long(keys), DataType::Int64) => { | |
| BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| { | |
| keys.contains(&x) ^ negated | |
| }) | |
| } | |
| (SetIndex::String(keys), DataType::Utf8) => { | |
| BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated) | |
| } | |
| (SetIndex::String(keys), DataType::LargeUtf8) => { | |
| BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated) | |
| } | |
| (SetIndex::String(keys), DataType::Utf8View) => { | |
| BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated) | |
| } | |
| _ => return None, | |
| }) | |
| } | |
| fn probe_set( | |
| index: &SetIndex, | |
| column: &dyn Array, | |
| negated: bool, | |
| ) -> std::result::Result<Option<BooleanArray>, ArrowError> { | |
| if let Some(dictionary) = column.as_any_dictionary_opt() { | |
| let Some(values) = probe_set(index, dictionary.values().as_ref(), negated)? else { | |
| return Ok(None); | |
| }; | |
| return Ok(Some( | |
| take(&values, dictionary.keys(), None)?.as_boolean().clone(), | |
| )); | |
| } | |
| Ok(Some(match (index, column.data_type()) { | |
| (SetIndex::Int(keys), DataType::Int32) => { | |
| BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| { | |
| keys.contains(&x) ^ negated | |
| }) | |
| } | |
| (SetIndex::Long(keys), DataType::Int64) => { | |
| BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| { | |
| keys.contains(&x) ^ negated | |
| }) | |
| } | |
| (SetIndex::String(keys), DataType::Utf8) => { | |
| BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated) | |
| } | |
| (SetIndex::String(keys), DataType::LargeUtf8) => { | |
| BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated) | |
| } | |
| (SetIndex::String(keys), DataType::Utf8View) => { | |
| BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated) | |
| } | |
| _ => return Ok(None), | |
| })) | |
| } |
This needs use arrow_select::take::take;, and the call in set_predicate becomes:
if let Some(index) = &index
&& let Some(result) = probe_set(index, column.as_ref(), negated)?
{
return Ok(result);
}Adding DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8)) to the encodings in string_set_filters_match_across_arrow_encodings covers it. This probes every dictionary value in each batch, including values no key references. That is still one pass over the dictionary, where the current loop makes one pass per literal.
| predicate_fn(batch).unwrap() | ||
| } | ||
|
|
||
| fn check_set_filter(column: ArrayRef, ty: PrimitiveType, literals: Vec<Datum>) { |
There was a problem hiding this comment.
check_set_filter compares results, and the hash probe and the scalar loop return the same results. So these tests pass whether or not the index is used. I deleted the Int64 and Utf8View arms from probe_set, and both integer_set_filters_preserve_nulls_and_physical_casts and string_set_filters_match_across_arrow_encodings still passed. Could we add a test that calls probe_set directly and checks that each supported column type returns Some?
#[test]
fn set_index_probes_supported_column_types() {
use arrow_array::{Int32Array, Int64Array, StringArray};
use arrow_cast::cast::cast;
use super::probe_set;
use crate::expr::BoundPredicate;
let index = |ty, literals: Vec<Datum>| {
let schema = Arc::new(
Schema::builder()
.with_fields([NestedField::optional(1, "key", Type::Primitive(ty)).into()])
.build()
.unwrap(),
);
let BoundPredicate::Set(set) = Reference::new("key")
.is_in(literals)
.bind(schema, true)
.unwrap()
else {
panic!("expected a set predicate");
};
set.membership_index().unwrap()
};
let ints = index(PrimitiveType::Int, (0..64).map(Datum::int).collect());
let longs = index(PrimitiveType::Long, (0..64).map(Datum::long).collect());
let strings = index(
PrimitiveType::String,
(0..64).map(|i| Datum::string(i.to_string())).collect(),
);
let int_column: ArrayRef = Arc::new(Int32Array::from(vec![1, 100]));
let long_column: ArrayRef = Arc::new(Int64Array::from(vec![1, 100]));
let string_column: ArrayRef = Arc::new(StringArray::from(vec!["1", "100"]));
let expected = Some(BooleanArray::from(vec![true, false]));
assert_eq!(probe_set(&ints, &int_column, false), expected);
assert_eq!(probe_set(&longs, &long_column, false), expected);
for ty in [DataType::Utf8, DataType::LargeUtf8, DataType::Utf8View] {
let column = cast(&string_column, &ty).unwrap();
assert_eq!(probe_set(&strings, &column, false), expected);
}
// An Int32 file column under a promoted long field takes the scalar path.
assert_eq!(probe_set(&longs, &int_column, false), None);
}With the dictionary change, probe_set returns a Result, so each assertion needs an .unwrap(), and the encodings loop can include the dictionary type.
| pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> { | ||
| // Short lists benefit from Arrow's vectorized scalar comparisons. | ||
| const MIN_INDEX_SIZE: usize = 32; | ||
| if self.literals().len() < MIN_INDEX_SIZE { | ||
| return None; | ||
| } | ||
| self.literals | ||
| .index | ||
| .get_or_init(|| SetIndex::new(self.literals()).map(Arc::new)) | ||
| .clone() |
There was a problem hiding this comment.
Could the 32-literal cutoff live in the reader instead of in SetExpression? The comment explains it in terms of Arrow's scalar comparison kernels, which is a reader concern. predicate.rs doesn't otherwise know about Arrow, and the Arrow evaluation lives in PredicateConverter. membership_index() can build the index whenever it's asked, and set_predicate can decide whether to ask:
| pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> { | |
| // Short lists benefit from Arrow's vectorized scalar comparisons. | |
| const MIN_INDEX_SIZE: usize = 32; | |
| if self.literals().len() < MIN_INDEX_SIZE { | |
| return None; | |
| } | |
| self.literals | |
| .index | |
| .get_or_init(|| SetIndex::new(self.literals()).map(Arc::new)) | |
| .clone() | |
| pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> { | |
| self.literals | |
| .index | |
| .get_or_init(|| SetIndex::new(self.literals()).map(Arc::new)) | |
| .clone() | |
| } |
and in predicate_visitor.rs:
/// Shorter `IN` lists are faster with Arrow's vectorized scalar comparisons than with a hash probe.
const MIN_INDEX_SIZE: usize = 32;
// in set_predicate
let index = if set.literals().len() >= MIN_INDEX_SIZE {
set.membership_index()
} else {
None
};On the value itself, the DataFusion table in the description suggests the hash set wins well below 32. At 31 keys this PR still runs the per-literal loop and takes 16.11 ms. At 32 keys it switches to the hash set and takes 5.40 ms. At 16 keys the loop takes 10.76 ms, twice what the hash set takes at 32.
DataFusion's InListExpr never runs one comparison pass per literal for a constant list. It uses a branchless compare for up to 32 four-byte or 16 eight-byte values and a hash set above that. I'm not suggesting we port the branchless filter. Could you add cases at 4, 8, 16, and 32 literals, for strings and Int64, to the harness in #3741, and set the cutoff from those numbers?
| #[test] | ||
| fn cloned_set_predicates_share_the_index_and_serialize_as_literals() { | ||
| let predicate = Reference::new("key").is_in((0..64).map(Datum::int)); | ||
| let encoded = serde_json::to_value(&predicate).unwrap(); | ||
| assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64); | ||
| let restored: Predicate = serde_json::from_value(encoded.clone()).unwrap(); | ||
| let clone = predicate.clone().not().rewrite_not(); | ||
| let (Predicate::Set(original), Predicate::Set(negated)) = (&predicate, &clone) else { | ||
| panic!("expected set predicates"); | ||
| }; | ||
| assert!(Arc::ptr_eq( | ||
| &original.membership_index().unwrap(), | ||
| &negated.membership_index().unwrap() | ||
| )); | ||
| assert_eq!(predicate, restored); | ||
| assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded); | ||
| } | ||
|
|
There was a problem hiding this comment.
This test checks sharing on an unbound Predicate, but the reader shares the index through a BoundPredicate. Each file task clones the bound predicate, and the reader calls rewrite_not on it, which goes through BoundPredicate::negate for a NOT. If BoundPredicate::negate went back to SetExpression::new(...), this test would still pass and every file would build its own index. Could the test bind first?
| #[test] | |
| fn cloned_set_predicates_share_the_index_and_serialize_as_literals() { | |
| let predicate = Reference::new("key").is_in((0..64).map(Datum::int)); | |
| let encoded = serde_json::to_value(&predicate).unwrap(); | |
| assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64); | |
| let restored: Predicate = serde_json::from_value(encoded.clone()).unwrap(); | |
| let clone = predicate.clone().not().rewrite_not(); | |
| let (Predicate::Set(original), Predicate::Set(negated)) = (&predicate, &clone) else { | |
| panic!("expected set predicates"); | |
| }; | |
| assert!(Arc::ptr_eq( | |
| &original.membership_index().unwrap(), | |
| &negated.membership_index().unwrap() | |
| )); | |
| assert_eq!(predicate, restored); | |
| assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded); | |
| } | |
| #[test] | |
| fn cloned_set_predicates_share_the_index_and_serialize_as_literals() { | |
| let predicate = Reference::new("bar") | |
| .is_in((0..64).map(Datum::int)) | |
| .bind(table_schema_simple(), true) | |
| .unwrap(); | |
| let encoded = serde_json::to_value(&predicate).unwrap(); | |
| assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64); | |
| let restored: BoundPredicate = serde_json::from_value(encoded.clone()).unwrap(); | |
| let clone = predicate.clone().negate().rewrite_not(); | |
| let (BoundPredicate::Set(original), BoundPredicate::Set(negated)) = (&predicate, &clone) | |
| else { | |
| panic!("expected set predicates"); | |
| }; | |
| assert!(Arc::ptr_eq( | |
| &original.membership_index().unwrap(), | |
| &negated.membership_index().unwrap() | |
| )); | |
| assert_eq!(predicate, restored); | |
| assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded); | |
| } |
The Predicate import in the test module becomes unused with this change.
| let scalars = scalars | ||
| .get_or_try_init(&arrow_literals) | ||
| .map_err(|err| ArrowError::ExternalError(Box::new(err)))?; | ||
| let mut acc = constant_bool_array(negated, batch.num_rows()); |
There was a problem hiding this comment.
This also happens on main. - I tried a file containing 0, 32, and null, with the table column promoted to Int64. For a list containing 0 through 31 plus 2147483648, IN should return 0 and NOT IN should return 32. Both returned no rows. The large value becomes null when we cast it down.
Use the first tested list size where the foldhash index improves the scan over both one and 64 files: four string keys, eight Int32 keys, or sixteen Int64 keys. Review: apache#3739 (comment)
|
@mbutrovich responded to each comment with the corresponding commit. Each comment has it's own commit, and the benchmark requests are pushed to that branch. Re-included AI disclosure + list of unit test in the description. |
Exercise index construction for each literal type before probing the Arrow column, including dictionary-encoded strings. Review: apache#3739 (comment)
File tasks can clone and rewrite a predicate before its first lookup. Require both cold clones to initialize the same index. Review: apache#3739 (comment)
Which issue does this PR close?
Related to #3350. Large
INlists can also come from ordinary SQL scan predicates.What changes are included in this PR?
Today, the Arrow reader evaluates
key IN (10, 20, 30)by comparing the entire decodedkeycolumn against10, then20, then30, and OR-ing the resulting bitmaps. With 5,000 literals and an 8,192-row batch, that’s about 41 million comparisons per batch.This PR builds a hash set once and shares it across file tasks that clone the bound predicate. The same 8,192-row batch takes 8,192 lookups. For an Arrow dictionary column, the reader probes each dictionary value once and uses the keys to select the result for each row.
The reader uses the index starting at 4 distinct strings, 8 Int32s, or 16 Int64s. Shorter lists, unsupported literal types, and file columns that require a cast use the existing comparison kernels.
NOT INinverts the membership result. Both operators preserve nulls, including null dictionary keys and values.AI Disclosure
OpenAI Codex assisted with the implementation, unit tests, and benchmark runs. The reader cutoffs come from warm local scans with a 50% match rate. The crossover can shift with file layout and selectivity.
Are these changes tested?
integer_set_filters_preserve_nulls_and_physical_castsandstring_set_filters_match_across_arrow_encodingsexerciseINandNOT INon sliced, nullable arrays, including a promoted integer field and the three Arrow string encodings.set_index_probes_supported_column_typesbindsINpredicates and checks directly which physical types their indexes can probe.dictionary_set_probes_preserve_null_keys_and_valueschecks both operators on a dictionary with repeated keys, null keys, and a null value.cloned_set_predicates_share_the_index_and_serialize_as_literalsclones and rewrites a bound predicate before the first lookup, then verifies that both predicates share one index and deserialization creates a fresh one.Both benchmarks check the exact matching rows before timing. Tables report Criterion mean wall time in milliseconds from 20 samples per case on an Apple M5 Pro.
End-to-end DataFusion
DataFusion 55.2 planned and collected
SELECT resource_key FROM taxi WHERE resource_key IN (...)over 175,000 taxi rows in seven Parquet files. Each requested key matched one row. Both builds used the same Arrow 59 reader baseline (9354b42). Taxi benchmarkIceberg Arrow reader
The reader scanned 262,144 rows in 8,192-row batches and returned 131,072 matching row IDs. Both file layouts held the same rows, with one file reader active at a time. The companion PR also compares scalar and indexed evaluation at the same list sizes to show how the cutoffs were chosen. Reader benchmark and reproduction steps
FNV vs foldhash
All three typed indexes use
foldhash::fast::FixedState. After hashing, the set checks full-value equality, including the complete string for string keys. Each comparison uses the same index strategy and changes only the hasher.At 5,000 string keys across 64 files, the first scan comparison favored FNV. Repeating it in reverse order measured 9.34 ms with FNV and 9.85 ms with foldhash.
Int64 was faster with foldhash in both file layouts at 400 and 5,000 keys. At 5,000 Int32 keys across 64 files the first run favored FNV, with overlapping 95% confidence intervals. Reversing the run order gave 5.02 ms with FNV and 4.97 ms with foldhash.