Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ fastnum = { version = "0.7", default-features = false, features = [
faststr = "0.2.31"
flate2 = "1.1.5"
fnv = "1.0.7"
foldhash = "0.2"
form_urlencoded = "1.2.2"
futures = "0.3"
hive_metastore = "0.2.0"
Expand Down
1 change: 1 addition & 0 deletions crates/iceberg/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ derive_builder = { workspace = true }
fastnum = { workspace = true }
flate2 = { workspace = true }
fnv = { workspace = true }
foldhash = { workspace = true }

@alexanderbianchi alexanderbianchi Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert the last commit if we don't want this new dep. But this is what DF uses as well, and did benchmark better.

Iceberg rust bench

Workload 400 keys, FNV → foldhash 5,000 keys, FNV → foldhash
Evaluate an 8,192-row UTF-8 batch 47.3 → 25.9 µs (−45%) 46.0 → 26.5 µs (−42%)
Scan 262,144 rows in one Parquet file 3.05 → 2.31 ms (−24%) 3.28 → 2.56 ms (−22%)
Scan the same rows across 64 files 5.93 → 5.13 ms (−13%) 6.96 → 6.20 ms (−11%)

Datafusion Bench

apache/datafusion-iceberg#130

Distinct keys FNV foldhash Time saved
32 11.27 ms 5.65 ms 50%
400 15.09 ms 7.87 ms 48%
1,000 14.25 ms 8.56 ms 40%

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert the last commit if we don't want this new dep.

I'd keep it. DataFusion already depends on foldhash = "0.2", so DataFusion-based consumers already build it, and foldhash has no dependencies of its own.

Should Int and Long use FixedState too? FNV hashes one byte at a time, so each i64 row costs 8 dependent multiplies before the probe starts. This is the Int64 arm of probe_set at a16ffe7, release build on aarch64:

ldr  x0, [x9, x0, lsl #3]   ; load the i64
and  x1, x0, #0xff
eor  x1, x1, x10            ; FNV offset basis
mul  x1, x1, x11            ; x11 = FNV prime, round 1 of 8
...                         ; rounds 2-7
eor  x1, x1, x0, lsr #56
mul  x2, x1, x11            ; round 8

foldhash hashes a single integer with one widening multiply in finish. With HashSet<i64, FixedState> the same arm becomes:

ldr   x1, [x9, x1, lsl #3]  ; load the i64
eor   x2, x1, x10
mul   x3, x2, x11
umulh x2, x2, x11           ; independent of the mul above
eor   x2, x2, x3

Could you add an Int64 case to the reader benchmark and compare the two hashers? If it holds up, SetIndex would also use one hasher for every variant.

@alexanderbianchi alexanderbianchi Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adressed in 6c9ea06
benchmark in 2d55fc4
Description includes integer cases

form_urlencoded = { workspace = true }
futures = { workspace = true }
iceberg-property-macro = { workspace = true }
Expand Down
Loading
Loading