bug(python): import lancedb crashes with SIGILL on pre-Haswell x86_64 (Sandy Bridge / Ivy Bridge / FX-7500)

[tobocop2]

LanceDB version

0.31.0b9 (every published wheel since the target-cpu=haswell baseline was set is affected)

What happened?

On x86_64 CPUs without AVX2 — Intel Sandy Bridge / Ivy Bridge / Westmere, and AMD Bulldozer / Piledriver / Steamroller (e.g. FX-7500) — import lancedb crashes immediately:

$ python -c "import lancedb"
[1]  Illegal instruction (core dumped)

The Rust extension is compiled with target-cpu=haswell + target-feature=+avx2,+fma,+f16c (this repo's .cargo/config.toml), so AVX2 and FMA instructions are baked into every compiled function. The binary traps on the first AVX2 instruction at load time, before any Python-level lancedb code runs.

Numpy and pyarrow run cleanly on the same hardware via runtime CPU dispatch. On my Sandy Bridge Xeon E5-2609:

Library	What it ships	Behavior
pyarrow	runtime_info.simd_level == 'avx'	Imports cleanly
numpy	baseline=X86_V2	Imports cleanly
lancedb	no runtime guard, heavy AVX2	SIGILL

The root cause lives in lance (which lancedb embeds) — tracked at lance-format/lance#6618. This issue is the lancedb-side companion: even after lance lands runtime dispatch, the lancedb wheel-build config (.cargo/config.toml) needs its baseline lowered to actually ship a wheel that runs on the affected hardware.

Are there known steps to reproduce?

On any pre-Haswell x86_64 Linux box, or in a qemu-x86_64 -cpu Nehalem container:

pip install lancedb
python -c "import lancedb"

Reliably reproduces the SIGILL.

The candidate lancedb-side fix lives at #2 — small and config-only: bump the lance dep to a runtime-dispatch-aware version, lower this repo's .cargo/config.toml baseline to x86-64-v2 (matches numpy), and add a wheel-import smoke test in CI.

End-to-end reproduction + the candidate fix's wheel build + verification is one script (handles Manjaro / Arch / Ubuntu / Debian, with BENCH=1 and REPRO_BUG=1 modes): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df

Verified on the failing hardware: built the wheel from tobocop2/lancedb#2 and installed on my Intel Xeon E5-2609 (Sandy Bridge):

$ grep -m1 'model name' /proc/cpuinfo
model name      : Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz

$ grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(sse4_2|avx|avx2|fma|avx512f)$' | sort -u
avx
sse4_2

$ python verify.py
Python: 3.12.7
lancedb version: 0.31.0b9
query result: [{'vec': [1.0, 2.0, 3.0], '_distance': 0.0}]
PASS: import + simd dispatch + table create + vector search all work.

Pre-fix on the same CPU: Illegal instruction (core dumped) at import lancedb.

---

I would really appreciate this. I'm working on https://github.com/tobocop2/lilbee and I'm really invested in this project — it would be incredible if it could target older architectures. It would be incredible to make my project accessible to older hardware and I would really love if this could happen.

To be transparent: the PRs are on my fork (not opened upstream) because this isn't my domain of expertise and the implementation is AI-generated. I verified it works end-to-end on the failing hardware (the Sandy Bridge box above), but wanted to be upfront about that.

• lance fork PR: feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds lance#2
• lancedb fork PR: build: bump lance to runtime-dispatch fix; document RUSTFLAGS override for pre-Haswell #2

I'd be happy to target my PRs upstream if it makes sense and I'd be happy to roll in any feedback.