feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds#6630
Conversation
github-actions
Bot
added
enhancement
a3df856 to
9193496
Compare
The default lancedb wheel targets target-cpu=haswell (AVX2 + FMA + F16C) and SIGILLs at import on pre-Haswell silicon (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD). Per westonpace's review on lancedb#3324 (fast by default, extra steps to work on legacy), the default wheel baseline stays fast for modern users; pre-Haswell users get a separately-published 'lancedb-compat' wheel. Adds a 'lancedb-compat' matrix entry to pypi-publish.yml that builds with RUSTFLAGS=-C target-cpu=x86-64-v2 (Nehalem-class baseline). The compat wheel relies on runtime SIMD dispatch in the embedded lance crate (landing via lance-format/lance#6630) to pick the appropriate kernel tier at load time, so it still goes fast on modern hardware. Generalizes build_linux_wheel and upload_wheel composites with optional package-name and rustflags inputs (defaults preserve current behavior for the existing 4 lancedb matrix entries). Documents the choice in python/README.md: pip install lancedb-compat for pre-Haswell hosts; same import lancedb API. Maintainer setup required before this can ship: register lancedb-compat on PyPI and configure trusted publishing. Blocked on lance-format/lance#6630.
7db5171 to
26aa7f4
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
8691248 to
448a275
Compare
The default lancedb wheel targets target-cpu=haswell (AVX2 + FMA + F16C) and SIGILLs at import on pre-Haswell silicon (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD). Per westonpace's review on lancedb#3324 (fast by default, extra steps to work on legacy), the default wheel baseline stays fast for modern users; pre-Haswell users get a separately-published 'lancedb-compat' wheel. Adds a 'lancedb-compat' matrix entry to pypi-publish.yml that builds with RUSTFLAGS=-C target-cpu=x86-64-v2 (Nehalem-class baseline). The compat wheel relies on runtime SIMD dispatch in the embedded lance crate (landing via lance-format/lance#6630) to pick the appropriate kernel tier at load time, so it still goes fast on modern hardware. Generalizes build_linux_wheel and upload_wheel composites with optional package-name and rustflags inputs (defaults preserve current behavior for the existing 4 lancedb matrix entries). Documents the choice in python/README.md: pip install lancedb-compat for pre-Haswell hosts; same import lancedb API. Maintainer setup required before this can ship: register lancedb-compat on PyPI and configure trusted publishing. Blocked on lance-format/lance#6630.
448a275 to
126a6a5
Compare
github-actions
Bot
added
A-index
126a6a5 to
26a520b
Compare
|
Benchmarked this PR on two modern x86_64 boxes (the "no slowdown on newer hardware" question). As-is it regresses the f32 cosine batch path on AVX2-only CPUs; a small refinement fixes that while keeping the AVX-512 win.
PR as-is vs base:
Cause: Refinement: gate the dispatch on After (fix vs base):
Fast by default, AVX-512 still wins where it helps. Notes: the +6% at dim 1024 on AVX2 is a codegen-unit artifact (the kernel there is byte-identical to pre-PR), not the dispatch path. I only reworked f32 cosine — (Downstream context: paperless-ngx/paperless-ngx#12970.) |
|
I'd love the PR to my branch! Thank you!!!
…On Wed, Jun 10, 2026, 4:01 PM Trenton H ***@***.***> wrote:
*stumpylog* left a comment (lance-format/lance#6630)
<#6630 (comment)>
Benchmarked this PR on two modern x86_64 boxes (the "no slowdown on newer
hardware" question). As-is it regresses the f32 cosine *batch* path on
AVX2-only CPUs; a small refinement fixes that while keeping the AVX-512 win.
cargo bench -p lance-linalg --bench cosine, criterion, same-session
base→PR→fix (Cosine(f32, scalar) is a build-independent control ≈ 0% on
clean runs). auto-vectorized = cosine_distance_batch dim 1024; simd,f32x8
= dim 8.
*PR as-is vs base:*
Broadwell (AVX2) Zen 4 (AVX-512)
dim 1024 *+24%* −6%
dim 8 *+36%* *+78%*
Cause: f32::cosine_batch does the match *SIMD_SUPPORT + a
#[target_feature] call *per vector*. On a wheel already built with
target-cpu=haswell, that's pure overhead vs the pre-PR inlined f32x16/
f32x8 kernel.
*Refinement:* gate the dispatch on #[cfg(target_feature = "avx2")] —
AVX2-baseline builds use the inlined (byte-identical to pre-PR) kernels
with no per-vector dispatch, and dispatch to AVX-512 *once per batch*,
large dims only (a masked 512-bit load loses to a plain AVX2 load at 8
lanes). Sub-AVX2 builds keep your runtime path.
*After (fix vs base):*
Broadwell (AVX2) Zen 4 (AVX-512)
dim 1024 +6% *−7%*
dim 8 flat −3%
Fast by default, AVX-512 still wins where it helps.
Notes: the +6% at dim 1024 on AVX2 is a codegen-unit artifact (the kernel
there is byte-identical to pre-PR), not the dispatch path. I only reworked
f32 cosine — dot/l2/norm_l2 have the same per-vector shape and likely the
same regression. Branch:
https://github.com/stumpylog/lance/tree/cosine-batch-dispatch-fix — happy
to PR it against your branch.
(Downstream context: paperless-ngx/paperless-ngx#12970
<paperless-ngx/paperless-ngx#12970>.)
—
Reply to this email directly, view it on GitHub
<#6630?email_source=notifications&email_token=ABKN6LEKEHRXNJUUKNCC2WD47G5AJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXGM4TKMRQGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4673952033>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKN6LDI2FDF6RKPOSM7AR347G5AJAVCNFSNUABFKJSXA33TNF2G64TZHM2TCMJWHEYTGOBQHNEXG43VMU5TIMZUGA2TQMBRHA4KC5QC>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/ABKN6LBY7GQWJQTQAFWJUM347G5AJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXGM4TKMRQGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG>
and Android
<https://github.com/notifications/mobile/android/ABKN6LAYQNGZLTE6W6ENDGL47G5AJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRXGM4TKMRQGMZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>.
Download it today!
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Just a friendly bump now that there are other folks with similar needs. I'm wondering if you could nudge someone on this team our way. Thank you very much. |
a877569 to
782877e
Compare
…-source builds Adds 5-tier runtime SIMD dispatch (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) to the 10 hot f32/f64 distance kernels in lance-linalg, matching the per-tier dispatch shape already used by dot_u8.rs / cosine_u8.rs / l2_u8.rs and the f16/bf16 paths in norm_l2.rs. Today, `import lancedb` SIGILLs on AVX-without-AVX2 CPUs (Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller) because the wheel bakes AVX2 into every compiled function with no runtime guard. numpy and pyarrow handle the same hardware via runtime dispatch; this PR brings lance to parity for the from-source legacy build path. Per westonpace's review feedback on lancedb/lancedb#3324 — fast by default, extra steps to work on legacy — the workspace .cargo/config.toml baseline stays at target-cpu=haswell. The qemu-pre-haswell CI job sets RUSTFLAGS=-C target-cpu=x86-64-v2 only in its own env block so the runtime-dispatch path gets exercised under qemu Nehalem without affecting any other build. CONTRIBUTING.md documents the from-source legacy build for users on pre-Haswell hardware. Changes: - lance-core::utils::cpu: extend SimdSupport with Avx and AvxFma tiers, add #[non_exhaustive] for forward compatibility, add SimdInfo + simd_info() introspection API mirroring pyarrow.runtime_info(). - lance-linalg::distance::{cosine, dot, l2, norm_l2}: per-tier dispatch via match *SIMD_SUPPORT for all f32/f64 hot kernels. AVX-512 inner kernels use _mm512_* intrinsics directly; AVX+FMA kernels use the existing f32x8 / f64x8 SIMD types; AVX-only kernels use raw _mm256_mul/_mm256_add intrinsics because f32x8::multiply_add lowers to an FMA instruction. AVX2-host dispatch routes to the AvxFma kernel where the body uses no AVX2-specific intrinsics (Avx2 | AvxFma => x86::FOO_avx_fma) — eliminates ~480 lines of byte-identical AVX2/AvxFma per-tier function pairs across all 10 hot kernels. Documents AvxFma/Avx fall-through to scalar in the u8 and dist_table dispatchers (their AVX2 inners use integer ops not available below AVX2). - lance-linalg::simd::{f32, f64}: drop the dead AVX-512 specializations of f32x16 / f64x8 (gated on target_feature="avx512f" which no project CI configuration enables). Per-tier dispatch in distance/* uses raw _mm512_* intrinsics directly. f32x8::gather adds a runtime AVX2 guard with a scalar fallback for x86_64 hosts without AVX2. - python: expose lance.simd_info() (native binding + re-export in lance/__init__.py with a pytest) so users can verify which SIMD tier the runtime dispatched to without rebuilding. - .github/workflows/rust.yml: new qemu-pre-haswell job that builds with RUSTFLAGS=-C target-cpu=x86-64-v2 and runs lance-linalg lib tests under qemu-x86_64 -cpu Nehalem, catching SIGILL leaks in the legacy-build path before they ship. - CONTRIBUTING.md: documents the legacy build (RUSTFLAGS=-C target-cpu=x86-64-v2 cargo build --release). Zero new external dependencies; Cargo.lock unchanged. lance-linalg lib tests all green (83/83). Verified end-to-end on Sandy Bridge Xeon E5-2609 via the companion lancedb wheel build: pre-PR `pip install lancedb` SIGILLs; post-PR a from-source build with the documented RUSTFLAGS override produces a wheel where import + vector search work at the AVX tier. Closes lance-format#6618.
…=avx2) On AVX2-baseline builds (the default haswell wheel), cosine_batch uses inlined non-target_feature kernels (base-equivalent) plus a single per-batch AVX-512 check, avoiding the per-vector runtime-dispatch + target_feature tax that regressed the modern path. Sub-AVX2 builds keep the runtime dispatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
782877e to
6141fed
Compare
|
addressed all CI issues that I saw @wjones127 thank you so much |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
The runtime-dispatch codecov run flagged the f32 cosine batch kernels and the scalar x86 gather fallback as uncovered. Both are only reached at runtime on sub-AVX2 hosts, so on the AVX2+ CI runner they were never exercised even though the surrounding per-vector kernels have full parity tests. Add direct parity tests that call cosine_batch_avx, cosine_batch_avx_fma, and cosine_batch_avx512 across the 8/16/general dimension arms, comparing each batch entry against the scalar cosine_fast reference and gating each tier on the matching is_x86_feature_detected! check. Add a direct test for gather_scalar_x86 so the non-AVX2 gather path is covered on hosts that route through the AVX2 gather at runtime.
|
the review request was accidental @wjones127 my apologies. This still needs benchmarks and also it needs to address what is discussed here #6630 (comment) |
Every kernel the Avx2 tier dispatches to is annotated
`#[target_feature(enable = "avx,fma")]`, but the tier was selected on
`is_x86_feature_detected!("avx2")` alone. AVX2 does not imply FMA in the ISA.
No shipping AVX2 part lacks FMA, so this was latent rather than live, but it is
the same class of fault the rest of this work exists to prevent.
Check FMA explicitly, document the tier's contract, and add a test that fails
if a tier is ever selected on a host that cannot run its kernels.
Runtime dispatch was happening once per vector inside the batch loops. On a build whose baseline already implies AVX2 — the default `haswell` wheel — the old code inlined the kernel directly, so the `match *SIMD_SUPPORT` plus the non-inlinable `#[target_feature]` call bought nothing and cost a measured +24% at dim 1024 and +36% at dim 8 on Broadwell. Cosine was fixed this way already; dot and l2 were not. Dispatch once, for the whole batch: - On AVX2-baseline builds, call the same inlined auto-vectorized kernel the code used before dispatch existed. Only an AVX-512 host with dimension > 16 takes a wide kernel, entered once. - On sub-AVX2 builds, still dispatch, but into a kernel that loops internally. `Dot` had no batch hook at all, so `dot_distance_batch` mapped `dot_distance` per vector; add `Dot::dot_batch` with a per-vector default, so every other impl keeps its current behaviour. Both batch methods return `impl Iterator` rather than a trait object: the k-means assignment loop drives them one element at a time, and a `Box<dyn Iterator>` would trade the per-vector dispatch for a per-element virtual call. `BatchIter` keeps the lazy and eager shapes in one statically dispatched type. `do_dot_distance_arrow_batch` carried its own copy of the per-vector loop and would not have benefited; route it through `dot_distance_batch`. Public signatures are unchanged from before the dispatch work: `l2_distance_batch` still returns `impl Iterator`, `dot_distance_batch` still returns a boxed one.
Benchmarking the previous commit on two hosts turned up a regression it had introduced: L2(simd,f32x8) was +154% on an AVX2-only EPYC 7B13 and +78.7% on an AVX-512 Xeon, while the dim-1024 benchmarks were flat. The cause was the `BatchIter` enum. Wrapping the batch iterator loses two things the bare `Map<ChunksExact, _>` has, and both matter only when the per-vector work is small: - `Map` overrides `Iterator::fold` to drive `ChunksExact` in one inlined, auto-vectorized loop. Through the enum, `fold` fell back to calling `next()` per element. - `Map<ChunksExact, _>` is `TrustedLen`, so `.collect()` preallocates exactly. No enum can be, and `TrustedLen` cannot be implemented outside std. `fold` can be delegated; `TrustedLen` cannot, and the hot consumers do not all use `fold` — `argmin_value_float` drives the iterator through `enumerate()`, and the arrow batch paths `.collect()`. So on an AVX2-baseline build, return the bare `Map` over the inlined scalar kernel: byte-for-byte the iterator the code produced before runtime dispatch existed. That removes the per-vector dispatch without adding anything, which is the only shape that is non-regressing by construction rather than by measurement. `BatchIter` now exists only for sub-AVX2 builds, where the eager `#[target_feature]` kernels need a common return type and there is no prior behaviour to regress. It delegates `fold`/`for_each` and implements `ExactSizeIterator` so that path stays reasonable too.
Benchmarks found a regression — holding this PR@wjones127 I ran the AVX2 and AVX-512 benchmarks you asked for. They turned up a real regression in this PR, so it isn't merge-ready. Details below; I'm working the root cause now. @stumpylog you called this in June — The finding
The control is flat on both hosts and dim-1024 is flat, so this is a real effect localized to small dimensions — exactly where the per-vector dispatch overhead dominates the per-vector work. Dim 8 is not academic: it's the sub-vector width PQ uses. Two attempted fixes have not moved it, which means my causal model is wrong rather than my patch. Bisecting now. Methodcriterion, one machine, one session, base → PR, on two GCE VMs:
Both verified with No base I'll follow up with the full numbers once the regression is fixed and the run is clean. |
Hoisting the tier choice out of the per-vector loop is only half the job. The
first attempt also replaced the kernel with the auto-vectorized scalar one,
on the theory that an AVX2-baseline build makes the SIMD kernel redundant.
Benchmarks say otherwise: at dim 8, `L2(simd,f32x8)` went from -41.6% (before
this work) to +156% against upstream/main. Removing a cheap branch is worthless
if it costs the kernel behind it.
The baseline already guarantees avx2+fma, so call the AVX+FMA kernel directly:
no runtime check, and the `#[target_feature]` contract is satisfied statically,
so it inlines into the loop.
Measured on an AMD EPYC 7B13 (avx2, fma, no avx512f), against upstream/main:
before this work with this commit
L2(simd,f32x8) dim 8 -41.6% -45.4%
L2(f32, simd) dim 1024 +1.5% +0.4%
Dot(f32, SIMD) -0.1% -0.5%
The iterator stays a bare `Map`: `Map<ChunksExact, _>` is `TrustedLen`, so
`.collect()` preallocates, and `Map::fold` drives `ChunksExact` in one inlined
loop. Any wrapper — trait object or enum — loses both.
`cfg(target_feature = "avx2")` does not imply `fma` — `-C target-feature=+avx2` sets one and not the other. The batch path guarded on it calls a kernel declared `#[target_feature(enable = "avx,fma")]`, so such a build emitted `vfmadd` while claiming a baseline without FMA. Gate the direct-kernel arm on both features, and make the runtime-dispatch arm its exact complement so the two remain total. Verified by building the crate at `+avx2`, `+avx2,+fma`, `+avx`, `target-cpu=haswell`, and the default baseline.
`dot_scalar`/`l2_scalar` chunk by 16 lanes. At or below that width the chunking
degenerates to the scalar remainder loop and vectorizes nothing, which is why
the explicit AVX kernel is worth ~40% at dim 8 — the width PQ sub-vectors use.
Above 16 lanes the autovectorizer is already good, and on Zen 3 the 8-wide
kernel loses to it.
Measured on an AMD EPYC 7B13 (avx2, fma, no avx512f), two repetitions each,
against upstream/main:
kernel everywhere this commit
L2(simd,f32x8) -42.2% / -41.4% (unchanged, kernel)
Dot(f32, SIMD) +1.9% / +3.6% (scalar, == base)
So above the threshold keep exactly the kernel the pre-dispatch code used. The
wide path is then byte-identical to base and cannot regress; the narrow path
keeps the win. The branch is loop-invariant.
Benchmark results — no regressions on either CPU@wjones127 here are the AVX2 and AVX-512 numbers. @stumpylog, no need to rerun anything. Same method as your June run: criterion, one machine, one session, base → PR. No base
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector width — improves 25–46%. On the two >2% rows
The head deltas sit inside the base-vs-base spread. That is what "identical code path" looks like on a shared machine — and above dim 16 it is the same code path, see below.
What changed@stumpylog called it in June: Both now choose the SIMD tier once per batch instead of once per vector. The subtlety is which kernel the batch path should then call.
The benchmark caught one bug. Fixing it led me to two more.What the benchmark caught — a real slowdown. An earlier version of this change moved the dispatch out of the per-vector loop, which was the point. But it also swapped the SIMD kernel out for the plain auto-vectorized one, on the assumption that an AVX2 baseline makes the hand-written SIMD kernel redundant. It doesn't. On FMA hardware that swap cost +156% on What the benchmark couldn't catch — two FMA bugs I found while fixing the first one. Fixing that meant actually reading the dispatch table, and that's where the other two turned up. Neither of these can ever show up in a benchmark, because no real CPU triggers them — you only get there with hand-picked build flags:
Both now check for FMA explicitly. The compile-time gate and the runtime check are each the exact complement of the other, so there's no third path left open. Verified by building at |
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
This is looking better. Thanks for the benchmark results, they look good. Once clippy lints are fixed, I'm happy to approve and merge.
|
thank you! |
The per-batch `dot_batch_f32_avx*` / `l2_batch_f32_avx*` wrappers are only called from the runtime-dispatch path, which is compiled out when the build baseline already guarantees avx2+fma (the repo's `target-cpu=haswell` default). Under that baseline the wrappers, their test helper, and their tests were unused, so `cargo clippy -D warnings` failed on dead_code. Gate the definitions and their tests with the same `not(all(target_feature = "avx2", target_feature = "fma"))` cfg as their caller, so definition, dispatch, and test compile together. No behavior change: at the avx2+fma baseline `dot_batch`/`l2_batch` already inline the kernel directly; below it the runtime dispatch and these wrappers remain.
|
@wjones127 clippy lints are fixed in b1670de |
) Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled function. Per [westonpace's review](#3324 (comment)), the default `lancedb` wheel stays fast; pre-Haswell users get a separately-published `lancedb-compat` wheel. ## Summary - Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline). Same Python API (`import lancedb` works) — files install to the same namespace, so the two wheels conflict at install time and users pick one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` / `tensorflow-cpu`. - Generalizes `build_linux_wheel` and `upload_wheel` composites with optional `package-name` and `rustflags` inputs (defaults preserve the existing 4 `lancedb` matrix entries verbatim). - Documents the choice in `python/README.md`: `pip install lancedb-compat` for pre-Haswell hosts. The default `.cargo/config.toml` baseline is unchanged. ## Sequencing 1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in lance.~~ **Done — merged.** 2. lancedb's lance dep is bumped to a release that includes it (separate PR / normal cadence). 3. This PR's `lancedb-compat` wheel build path starts producing a wheel that runs on pre-Haswell hardware. **Maintainer setup**: register `lancedb-compat` on PyPI and configure trusted publishing. ## Verified end-to-end on Sandy Bridge Xeon E5-2609 Verification was done locally against a fork-pinned lance dep that includes the runtime dispatch implementation, using the same `RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI: ``` $ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release $ pip install ./target/wheels/lancedb-*.whl $ python verify.py PASS: import + simd dispatch + table create + vector search all work. ``` Pre-fix on the same CPU (default `pip install lancedb`): `Illegal instruction (core dumped)`. Full reproducer (deps + clone + build + verification): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df. Fork-internal verification PR with the dep bump and full logs: [`tobocop2#2`](tobocop2#2). ## Benchmarks — no regressions on modern CPUs from the lance-side change These are the numbers I ran for the lance PR, confirming the runtime dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that existing users install. Criterion, one machine, one session, base → PR, no `RUSTFLAGS` override. Full methodology, null experiments, and logs: [lance-format/lance#6630 benchmark comment](lance-format/lance#6630 (comment)) and the [logs gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2). | benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade Lake (`avx512f`) | |---|---|---| | `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% | | `Cosine(f64, scalar)` | −0.34% | −1.94% | | `Cosine(u8, SIMD)` | +2.30% | +3.63% | | `Dot(f16, SIMD)` | −0.58% | +0.61% | | `Dot(f32, SIMD)` | +0.34% | **−6.08%** | | `Dot(f32, arrow_arity)` | +0.02% | −0.00% | | `L2(f32, scalar)` | −0.10% | −0.02% | | `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% | | **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** | | `L2(u8, SIMD)` | +0.42% | −3.11% | | `NormL2(f32, SIMD)` | −1.02% | −4.17% | | `NormL2(f64, SIMD)` | +3.51% | −0.58% | Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector width — improves 25–46%. --- To be transparent: this isn't my domain of expertise and the lance-side implementation is AI-generated. I verified it works end-to-end on the failing hardware. Happy to roll in feedback.
3 participants
Tracks #6618. On x86_64 CPUs without AVX2 — Sandy Bridge / Ivy Bridge / Westmere on Intel, Bulldozer / Piledriver / Steamroller on AMD —
import lancedbSIGILLs because the wheel bakes AVX2 + FMA into every compiled function with no runtime guard. numpy and pyarrow handle the same hardware via runtime CPU dispatch.Summary
lance-linalg::distance::{cosine, dot, l2, norm_l2}. Samematch *SIMD_SUPPORT+mod x86 { #[target_feature] pub unsafe fn ... }shape asdot_u8.rs/cosine_u8.rs/l2_u8.rs. Where the AVX2 and AVX+FMA kernel bodies use no AVX2-specific intrinsics, the dispatch matchesAvx2 | AvxFmato a shared kernel.lance.simd_info()Python introspection mirroringpyarrow.runtime_info()so users can verify which tier the runtime selected.qemu-pre-haswellCI job that builds withRUSTFLAGS="-C target-cpu=x86-64-v2"(env-var-scoped to that one job — workspace.cargo/config.tomlis unchanged) and runslance-linalglib tests underqemu-x86_64 -cpu Nehalem.CONTRIBUTING.md:RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release.Per westonpace's review on lancedb/lancedb#3324, the workspace baseline stays at
target-cpu=haswell. Modern wheels are unchanged; legacy users opt into the lower baseline at build time.Benchmark
The AVX2 path on modern hardware is preserved as one of the per-tier kernels and the workspace baseline still bakes AVX2 into surrounding code, so by construction the modern compile is unchanged. Numbers still pending — Codespace's 30-min idle timeout killed my last full
cargo bench -p lance-linalg --bench {cosine,dot,l2,norm_l2}run mid-suite (even withnohup— the VM itself sleeps). If anyone can recommend a free resource that holds a benchmark for ~1 hour, or a maintainer-preferred narrower bench shape, I'd appreciate the pointer.Pre-Haswell verification on Sandy Bridge Xeon E5-2609 (the hardware the published wheel SIGILLs on) via the companion lancedb wheel build: pre-PR
pip install lancedbSIGILLs at import; post-PR a from-source build with the documentedRUSTFLAGSoverride produces a wheel where import + table-create + vector-search all work at the AVX tier. PASS output intobocop2/lancedb#2.Incidental fix: FMA not verified before the AVX2 tier
While auditing the dispatch table for this PR I found a latent soundness bug that predates it. On
mainthe tier is chosen withis_x86_feature_detected!("avx2")alone, but every kernel the AVX2 tier dispatches to is#[target_feature(enable = "avx,fma")]. AVX2 does not imply FMA in the x86 ISA, so a host with AVX2 but no FMA would select that tier and executevfmadd— an illegal instruction.No shipping AVX2 part lacks FMA, so this is latent rather than live, but it is exactly the class of fault this PR exists to remove. The fix checks FMA explicitly, documents the tier's contract, and adds a test that fails if any tier is ever selected on a host that cannot run its kernels. Tracked separately as #7732 so it stays on the record independent of the perf work.
Test plan
cargo test -p lance-linalg --lib— 83/83 on aarch64 dev boxcargo clippy --all-targets -- -D warningsclean;cargo fmt --checkclean;Cargo.lockunchangedis_x86_feature_detected!()so each runs on hosts that can execute its tierRUSTFLAGSoverrideTo be transparent: this isn't my domain of expertise and the implementation is AI-generated — I stuck to the existing
mod x86 { #[target_feature] }precedent and verified end-to-end on the failing hardware, but wanted to be upfront. Happy to roll in feedback.Summary by CodeRabbit
lance.simd_info()(Python) to report the runtime SIMD tier, target architecture, and detected host features.simd_info()tests and expanded dispatched-vs-scalar parity coverage, plus a new regression seed.