Skip to content

perf(arrow): use shared membership sets for large IN predicates - #3739

Open
alexanderbianchi wants to merge 9 commits into
apache:mainfrom
alexanderbianchi:codex/arrow-in-list
Open

alexanderbianchi wants to merge 9 commits into
apache:mainfrom
alexanderbianchi:codex/arrow-in-list

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #3350. Large IN lists can also come from ordinary SQL scan predicates.

What changes are included in this PR?

Today, the Arrow reader evaluates key IN (10, 20, 30) by comparing the entire decoded key column against 10, then 20, then 30, and OR-ing the resulting bitmaps. With 5,000 literals and an 8,192-row batch, that’s about 41 million comparisons per batch.

This PR builds a hash set once and shares it across file tasks that clone the bound predicate. The same 8,192-row batch takes 8,192 lookups. For an Arrow dictionary column, the reader probes each dictionary value once and uses the keys to select the result for each row.

The reader uses the index starting at 4 distinct strings, 8 Int32s, or 16 Int64s. Shorter lists, unsupported literal types, and file columns that require a cast use the existing comparison kernels. NOT IN inverts the membership result. Both operators preserve nulls, including null dictionary keys and values.

AI Disclosure

OpenAI Codex assisted with the implementation, unit tests, and benchmark runs. The reader cutoffs come from warm local scans with a 50% match rate. The crossover can shift with file layout and selectivity.

Are these changes tested?

  • integer_set_filters_preserve_nulls_and_physical_casts and string_set_filters_match_across_arrow_encodings exercise IN and NOT IN on sliced, nullable arrays, including a promoted integer field and the three Arrow string encodings.
  • set_index_probes_supported_column_types binds IN predicates and checks directly which physical types their indexes can probe. dictionary_set_probes_preserve_null_keys_and_values checks both operators on a dictionary with repeated keys, null keys, and a null value.
  • cloned_set_predicates_share_the_index_and_serialize_as_literals clones and rewrites a bound predicate before the first lookup, then verifies that both predicates share one index and deserialization creates a fresh one.

Both benchmarks check the exact matching rows before timing. Tables report Criterion mean wall time in milliseconds from 20 samples per case on an Apple M5 Pro.

End-to-end DataFusion

DataFusion 55.2 planned and collected SELECT resource_key FROM taxi WHERE resource_key IN (...) over 175,000 taxi rows in seven Parquet files. Each requested key matched one row. Both builds used the same Arrow 59 reader baseline (9354b42). Taxi benchmark

Distinct keys Original reader (ms) This PR (ms) Speedup
1 4.95 4.88 1.02×
16 11.22 5.14 2.19×
31 16.40 5.27 3.11×
32 16.47 5.61 2.93×
100 41.89 6.90 6.07×
369 136.99 8.28 16.54×
400 152.35 7.94 19.19×
1,000 365.16 8.47 43.13×

Iceberg Arrow reader

The reader scanned 262,144 rows in 8,192-row batches and returned 131,072 matching row IDs. Both file layouts held the same rows, with one file reader active at a time. The companion PR also compares scalar and indexed evaluation at the same list sizes to show how the cutoffs were chosen. Reader benchmark and reproduction steps

Distinct string keys Files Original reader (ms) This PR (ms) Speedup
400 1 106.62 3.54 30.11×
400 64 113.86 6.92 16.46×
5,000 1 1,093.15 4.14 264.16×
5,000 64 1,102.81 10.65 103.56×

FNV vs foldhash

All three typed indexes use foldhash::fast::FixedState. After hashing, the set checks full-value equality, including the complete string for string keys. Each comparison uses the same index strategy and changes only the hasher.

Workload Distinct keys FNV (ms) foldhash (ms) FNV / foldhash
DataFusion taxi, string 16 10.01 5.14 1.95×
DataFusion taxi, string 31 10.68 5.27 2.03×
DataFusion taxi, string 32 10.53 5.61 1.88×
DataFusion taxi, string 100 13.38 6.90 1.94×
DataFusion taxi, string 369 12.73 8.28 1.54×
DataFusion taxi, string 400 14.01 7.94 1.76×
DataFusion taxi, string 1,000 13.58 8.47 1.60×
Reader, string, 1 file 400 4.15 3.54 1.17×
Reader, string, 64 files 400 7.49 6.92 1.08×
Reader, string, 1 file 5,000 5.80 4.14 1.40×
Reader, string, 64 files 5,000 9.25 10.65 0.87×
Reader, Int32, 1 file 400 1.88 1.82 1.04×
Reader, Int32, 64 files 400 4.95 4.82 1.03×
Reader, Int32, 1 file 5,000 2.09 1.93 1.08×
Reader, Int32, 64 files 5,000 5.04 5.33 0.95×
Reader, Int64, 1 file 400 2.10 1.90 1.11×
Reader, Int64, 64 files 400 5.83 5.01 1.17×
Reader, Int64, 1 file 5,000 2.25 1.95 1.15×
Reader, Int64, 64 files 5,000 6.01 5.46 1.10×

At 5,000 string keys across 64 files, the first scan comparison favored FNV. Repeating it in reverse order measured 9.34 ms with FNV and 9.85 ms with foldhash.

Int64 was faster with foldhash in both file layouts at 400 and 5,000 keys. At 5,000 Int32 keys across 64 files the first run favored FNV, with overlapping 95% confidence intervals. Reversing the run order gave 5.02 ms with FNV and 4.97 ms with foldhash.

Comment thread crates/iceberg/Cargo.toml
fastnum = { workspace = true }
flate2 = { workspace = true }
fnv = { workspace = true }
foldhash = { workspace = true }

@alexanderbianchi alexanderbianchi Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert the last commit if we don't want this new dep. But this is what DF uses as well, and did benchmark better.

Iceberg rust bench

Workload 400 keys, FNV → foldhash 5,000 keys, FNV → foldhash
Evaluate an 8,192-row UTF-8 batch 47.3 → 25.9 µs (−45%) 46.0 → 26.5 µs (−42%)
Scan 262,144 rows in one Parquet file 3.05 → 2.31 ms (−24%) 3.28 → 2.56 ms (−22%)
Scan the same rows across 64 files 5.93 → 5.13 ms (−13%) 6.96 → 6.20 ms (−11%)

Datafusion Bench

apache/datafusion-iceberg#130

Distinct keys FNV foldhash Time saved
32 11.27 ms 5.65 ms 50%
400 15.09 ms 7.87 ms 48%
1,000 14.25 ms 8.56 ms 40%

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert the last commit if we don't want this new dep.

I'd keep it. DataFusion already depends on foldhash = "0.2", so DataFusion-based consumers already build it, and foldhash has no dependencies of its own.

Should Int and Long use FixedState too? FNV hashes one byte at a time, so each i64 row costs 8 dependent multiplies before the probe starts. This is the Int64 arm of probe_set at a16ffe7, release build on aarch64:

ldr  x0, [x9, x0, lsl #3]   ; load the i64
and  x1, x0, #0xff
eor  x1, x1, x10            ; FNV offset basis
mul  x1, x1, x11            ; x11 = FNV prime, round 1 of 8
...                         ; rounds 2-7
eor  x1, x1, x0, lsr #56
mul  x2, x1, x11            ; round 8

foldhash hashes a single integer with one widening multiply in finish. With HashSet<i64, FixedState> the same arm becomes:

ldr   x1, [x9, x1, lsl #3]  ; load the i64
eor   x2, x1, x10
mul   x3, x2, x11
umulh x2, x2, x11           ; independent of the mul above
eor   x2, x2, x3

Could you add an Int64 case to the reader benchmark and compare the two hashers? If it holds up, SetIndex would also use one hasher for every variant.

@alexanderbianchi alexanderbianchi Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adressed in 6c9ea06
benchmark in 2d55fc4
Description includes integer cases

(SetIndex::String(keys), DataType::Utf8) => {
BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::LargeUtf8) => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could be storing the minimum and the maximum length of each value to rule our keys before we even hash them.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alexanderbianchi, this is a big win for long IN lists, and sharing the index through the Arc keeps the per-file setup cost off the scan path. The extra copy of each string in SetIndex::String looks fine to me, because it is bounded by the literal list the predicate already holds and every file task shares it.

The rewritten description no longer has the template's AI Disclosure section, and "Are these changes tested?" now lists only the benchmarks. Could you restore the disclosure and name the unit tests there? The description becomes the squashed commit message.

Comment on lines +334 to +357
fn probe_set(index: &SetIndex, column: &ArrayRef, negated: bool) -> Option<BooleanArray> {
Some(match (index, column.data_type()) {
(SetIndex::Int(keys), DataType::Int32) => {
BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::Long(keys), DataType::Int64) => {
BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::String(keys), DataType::Utf8) => {
BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::LargeUtf8) => {
BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::Utf8View) => {
BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated)
}
_ => return None,
})
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probe_set handle dictionary-encoded columns? A Parquet file whose embedded ARROW:schema declares a dictionary type comes back from the decoder as a dictionary array. Files migrated from non-Iceberg writers often carry Arrow types like this. I wrote a Dictionary(Int32, Utf8) column with ArrowWriter and read it through ArrowReader with a 64-string IN. set_predicate saw Dictionary(Int32, Utf8), the index was built, and probe_set returned None, so every batch took the per-literal loop.

DataFusion's InListExpr handles this case with DictionaryFilter. It probes the dictionary values once and then uses take with the keys, which carries key nulls through. The same shape fits here:

Suggested change
fn probe_set(index: &SetIndex, column: &ArrayRef, negated: bool) -> Option<BooleanArray> {
Some(match (index, column.data_type()) {
(SetIndex::Int(keys), DataType::Int32) => {
BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::Long(keys), DataType::Int64) => {
BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::String(keys), DataType::Utf8) => {
BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::LargeUtf8) => {
BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::Utf8View) => {
BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated)
}
_ => return None,
})
}
fn probe_set(
index: &SetIndex,
column: &dyn Array,
negated: bool,
) -> std::result::Result<Option<BooleanArray>, ArrowError> {
if let Some(dictionary) = column.as_any_dictionary_opt() {
let Some(values) = probe_set(index, dictionary.values().as_ref(), negated)? else {
return Ok(None);
};
return Ok(Some(
take(&values, dictionary.keys(), None)?.as_boolean().clone(),
));
}
Ok(Some(match (index, column.data_type()) {
(SetIndex::Int(keys), DataType::Int32) => {
BooleanArray::from_unary(column.as_primitive::<Int32Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::Long(keys), DataType::Int64) => {
BooleanArray::from_unary(column.as_primitive::<Int64Type>(), |x| {
keys.contains(&x) ^ negated
})
}
(SetIndex::String(keys), DataType::Utf8) => {
BooleanArray::from_unary(column.as_string::<i32>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::LargeUtf8) => {
BooleanArray::from_unary(column.as_string::<i64>(), |s| keys.contains(s) ^ negated)
}
(SetIndex::String(keys), DataType::Utf8View) => {
BooleanArray::from_unary(column.as_string_view(), |s| keys.contains(s) ^ negated)
}
_ => return Ok(None),
}))
}

This needs use arrow_select::take::take;, and the call in set_predicate becomes:

            if let Some(index) = &index
                && let Some(result) = probe_set(index, column.as_ref(), negated)?
            {
                return Ok(result);
            }

Adding DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8)) to the encodings in string_set_filters_match_across_arrow_encodings covers it. This probes every dictionary value in each batch, including values no key references. That is still one pass over the dictionary, where the current loop makes one pass per literal.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 1ca9b4b

predicate_fn(batch).unwrap()
}

fn check_set_filter(column: ArrayRef, ty: PrimitiveType, literals: Vec<Datum>) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_set_filter compares results, and the hash probe and the scalar loop return the same results. So these tests pass whether or not the index is used. I deleted the Int64 and Utf8View arms from probe_set, and both integer_set_filters_preserve_nulls_and_physical_casts and string_set_filters_match_across_arrow_encodings still passed. Could we add a test that calls probe_set directly and checks that each supported column type returns Some?

    #[test]
    fn set_index_probes_supported_column_types() {
        use arrow_array::{Int32Array, Int64Array, StringArray};
        use arrow_cast::cast::cast;

        use super::probe_set;
        use crate::expr::BoundPredicate;

        let index = |ty, literals: Vec<Datum>| {
            let schema = Arc::new(
                Schema::builder()
                    .with_fields([NestedField::optional(1, "key", Type::Primitive(ty)).into()])
                    .build()
                    .unwrap(),
            );
            let BoundPredicate::Set(set) = Reference::new("key")
                .is_in(literals)
                .bind(schema, true)
                .unwrap()
            else {
                panic!("expected a set predicate");
            };
            set.membership_index().unwrap()
        };
        let ints = index(PrimitiveType::Int, (0..64).map(Datum::int).collect());
        let longs = index(PrimitiveType::Long, (0..64).map(Datum::long).collect());
        let strings = index(
            PrimitiveType::String,
            (0..64).map(|i| Datum::string(i.to_string())).collect(),
        );

        let int_column: ArrayRef = Arc::new(Int32Array::from(vec![1, 100]));
        let long_column: ArrayRef = Arc::new(Int64Array::from(vec![1, 100]));
        let string_column: ArrayRef = Arc::new(StringArray::from(vec!["1", "100"]));
        let expected = Some(BooleanArray::from(vec![true, false]));
        assert_eq!(probe_set(&ints, &int_column, false), expected);
        assert_eq!(probe_set(&longs, &long_column, false), expected);
        for ty in [DataType::Utf8, DataType::LargeUtf8, DataType::Utf8View] {
            let column = cast(&string_column, &ty).unwrap();
            assert_eq!(probe_set(&strings, &column, false), expected);
        }
        // An Int32 file column under a promoted long field takes the scalar path.
        assert_eq!(probe_set(&longs, &int_column, false), None);
    }

With the dictionary change, probe_set returns a Result, so each assertion needs an .unwrap(), and the encodings loop can include the dictionary type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 3c20cdf

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followed up cleanup in 25daa58

Comment on lines +350 to +359
pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> {
// Short lists benefit from Arrow's vectorized scalar comparisons.
const MIN_INDEX_SIZE: usize = 32;
if self.literals().len() < MIN_INDEX_SIZE {
return None;
}
self.literals
.index
.get_or_init(|| SetIndex::new(self.literals()).map(Arc::new))
.clone()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the 32-literal cutoff live in the reader instead of in SetExpression? The comment explains it in terms of Arrow's scalar comparison kernels, which is a reader concern. predicate.rs doesn't otherwise know about Arrow, and the Arrow evaluation lives in PredicateConverter. membership_index() can build the index whenever it's asked, and set_predicate can decide whether to ask:

Suggested change
pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> {
// Short lists benefit from Arrow's vectorized scalar comparisons.
const MIN_INDEX_SIZE: usize = 32;
if self.literals().len() < MIN_INDEX_SIZE {
return None;
}
self.literals
.index
.get_or_init(|| SetIndex::new(self.literals()).map(Arc::new))
.clone()
pub(crate) fn membership_index(&self) -> Option<Arc<SetIndex>> {
self.literals
.index
.get_or_init(|| SetIndex::new(self.literals()).map(Arc::new))
.clone()
}

and in predicate_visitor.rs:

/// Shorter `IN` lists are faster with Arrow's vectorized scalar comparisons than with a hash probe.
const MIN_INDEX_SIZE: usize = 32;

// in set_predicate
let index = if set.literals().len() >= MIN_INDEX_SIZE {
    set.membership_index()
} else {
    None
};

On the value itself, the DataFusion table in the description suggests the hash set wins well below 32. At 31 keys this PR still runs the per-literal loop and takes 16.11 ms. At 32 keys it switches to the hash set and takes 5.40 ms. At 16 keys the loop takes 10.76 ms, twice what the hash set takes at 32.

DataFusion's InListExpr never runs one comparison pass per literal for a constant list. It uses a branchless compare for up to 32 four-byte or 16 eight-byte values and a hash set above that. I'm not suggesting we port the branchless filter. Could you add cases at 4, 8, 16, and 32 literals, for strings and Int64, to the harness in #3741, and set the cutoff from those numbers?

@alexanderbianchi alexanderbianchi Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 63ebab2
benchmark in 89beb26

Comment on lines +895 to +912
#[test]
fn cloned_set_predicates_share_the_index_and_serialize_as_literals() {
let predicate = Reference::new("key").is_in((0..64).map(Datum::int));
let encoded = serde_json::to_value(&predicate).unwrap();
assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64);
let restored: Predicate = serde_json::from_value(encoded.clone()).unwrap();
let clone = predicate.clone().not().rewrite_not();
let (Predicate::Set(original), Predicate::Set(negated)) = (&predicate, &clone) else {
panic!("expected set predicates");
};
assert!(Arc::ptr_eq(
&original.membership_index().unwrap(),
&negated.membership_index().unwrap()
));
assert_eq!(predicate, restored);
assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test checks sharing on an unbound Predicate, but the reader shares the index through a BoundPredicate. Each file task clones the bound predicate, and the reader calls rewrite_not on it, which goes through BoundPredicate::negate for a NOT. If BoundPredicate::negate went back to SetExpression::new(...), this test would still pass and every file would build its own index. Could the test bind first?

Suggested change
#[test]
fn cloned_set_predicates_share_the_index_and_serialize_as_literals() {
let predicate = Reference::new("key").is_in((0..64).map(Datum::int));
let encoded = serde_json::to_value(&predicate).unwrap();
assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64);
let restored: Predicate = serde_json::from_value(encoded.clone()).unwrap();
let clone = predicate.clone().not().rewrite_not();
let (Predicate::Set(original), Predicate::Set(negated)) = (&predicate, &clone) else {
panic!("expected set predicates");
};
assert!(Arc::ptr_eq(
&original.membership_index().unwrap(),
&negated.membership_index().unwrap()
));
assert_eq!(predicate, restored);
assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded);
}
#[test]
fn cloned_set_predicates_share_the_index_and_serialize_as_literals() {
let predicate = Reference::new("bar")
.is_in((0..64).map(Datum::int))
.bind(table_schema_simple(), true)
.unwrap();
let encoded = serde_json::to_value(&predicate).unwrap();
assert_eq!(encoded["Set"]["literals"].as_array().unwrap().len(), 64);
let restored: BoundPredicate = serde_json::from_value(encoded.clone()).unwrap();
let clone = predicate.clone().negate().rewrite_not();
let (BoundPredicate::Set(original), BoundPredicate::Set(negated)) = (&predicate, &clone)
else {
panic!("expected set predicates");
};
assert!(Arc::ptr_eq(
&original.membership_index().unwrap(),
&negated.membership_index().unwrap()
));
assert_eq!(predicate, restored);
assert_eq!(serde_json::to_value(&predicate).unwrap(), encoded);
}

The Predicate import in the test module becomes unused with this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in eb8026f

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followed up cleanup in 5c7d3f6

let scalars = scalars
.get_or_try_init(&arrow_literals)
.map_err(|err| ArrowError::ExternalError(Box::new(err)))?;
let mut acc = constant_bool_array(negated, batch.num_rows());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also happens on main. - I tried a file containing 0, 32, and null, with the table column promoted to Int64. For a list containing 0 through 31 plus 2147483648, IN should return 0 and NOT IN should return 32. Both returned no rows. The large value becomes null when we cast it down.

alexanderbianchi added a commit to alexanderbianchi/iceberg-rust that referenced this pull request Nov 20, 2026
alexanderbianchi added a commit to alexanderbianchi/iceberg-rust that referenced this pull request Nov 20, 2026
@alexanderbianchi

Copy link
Copy Markdown
Contributor Author

@mbutrovich responded to each comment with the corresponding commit. Each comment has it's own commit, and the benchmark requests are pushed to that branch. Re-included AI disclosure + list of unit test in the description.

Exercise index construction for each literal type before probing the Arrow column, including dictionary-encoded strings.

Review: apache#3739 (comment)
File tasks can clone and rewrite a predicate before its first lookup. Require both cold clones to initialize the same index.

Review: apache#3739 (comment)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants