From a052c04c9c8e0d2639ad188cce1f7af68d1fd6f4 Mon Sep 17 00:00:00 2001 From: Heng Qian Date: Tue, 15 Sep 2026 15:20:05 +0800 Subject: [PATCH 1/3] Drop the numpy<2.0 runtime pin from the Python bindings The bindings' metadata had NumPy's compatibility rule inverted. requirements.txt asks for numpy>=2.0 headers at build time, but pyproject.toml pinned the runtime to numpy<2.0. NumPy's C API is backward compatible the other way round: a module compiled against 2.x headers loads on any runtime from its target level up, 1.x and 2.x alike, while one compiled against 1.x refuses to import under 2.x. The pin bit hardest on Python 3.13+, where numpy<2.0 has no wheel and pip built numpy 1.26 from source against an interpreter it never supported, corrupting arrays silently. The benchmark skill, the seismic_mmap demo and the CI job all carried workarounds for this (install numpy>=2.1 over the package; stay on Python 3.12). Fix the direction instead of working around it: - pyproject.toml: require numpy>=1.26 with no upper bound. 1.26 is the floor because loader.py imports numpy._core, which older 1.x lacks. - CMakeLists.txt: refuse to configure against numpy<2.0 headers, so the open-ended runtime requirement cannot be undermined by a 1.x build. - swignsparse.swig: set NPY_TARGET_VERSION to NPY_1_25_API_VERSION, the C-API level numpy 1.26 ships, so the runtime floor is explicit rather than whatever the numpy 2.x providing the headers defaults to. - conftest.py: drop the guard that rejected numpy>=2; numpy's own import_array() already fails loudly on a real mismatch. - CI: build against numpy 2.x on Python 3.12 and 3.13, then on the 3.12 leg downgrade to numpy<2.0 and run the tests again to cover the floor. - Docs: remove the workaround text from DEVELOPER_GUIDE.md, the benchmark skill and the seismic_mmap docstring. Verified locally on Amazon Linux 2023 / Python 3.12: built against numpy 2.5.1, python_tests pass on numpy 2.5.1 and on numpy 1.26.4 (140/140 each), and configuring against a numpy 1.26 environment fails with the new CMake error. Signed-off-by: Heng Qian --- .claude/skills/benchmark-seismic/SKILL.md | 6 ++-- .github/workflows/CI.yml | 34 +++++++++++++++++------ DEVELOPER_GUIDE.md | 7 ++++- demos/seismic_mmap.py | 8 ++---- nsparse/python/CMakeLists.txt | 14 ++++++++++ nsparse/python/pyproject.toml | 8 ++++-- nsparse/python/requirements.txt | 4 +++ nsparse/python/swignsparse.swig | 7 +++++ python_tests/conftest.py | 20 ++++++------- 9 files changed, 75 insertions(+), 33 deletions(-) diff --git a/.claude/skills/benchmark-seismic/SKILL.md b/.claude/skills/benchmark-seismic/SKILL.md index f5d78c4..227f980 100644 --- a/.claude/skills/benchmark-seismic/SKILL.md +++ b/.claude/skills/benchmark-seismic/SKILL.md @@ -157,9 +157,9 @@ OMP_NUM_THREADS=1 python demos/seismic_mmap.py \ --reuse-index --keep ``` -Needs `numpy>=2.1` installed *over* the nsparse package: `pyproject.toml` pins `numpy<2.0`, which on -Python 3.13+ resolves to a numpy older than the interpreter and silently corrupts arrays (`a - b` -overwrites `a`) while every check still passes. Pin Python 3.12 or force the newer numpy. +Install the bindings with `pip install build/nsparse/python`. They are built against numpy 2.x +headers (`nsparse/python/CMakeLists.txt` enforces it) and run on numpy 1.26+ and 2.x alike, so no +numpy override or interpreter pin is needed. ### Peak build memory — `disk_build_mem_bench` diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 00e8942..da33cae 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -164,6 +164,12 @@ jobs: needs: check-files if: needs.check-files.outputs.RUN_BUILD_AND_TEST == 'true' runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + # 3.13 has no numpy 1.x wheel, which is what used to make the numpy<2.0 + # pin resolve to a source build older than the interpreter. + python-version: ['3.12', '3.13'] steps: - name: Checkout uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 @@ -173,18 +179,18 @@ jobs: sudo apt-get update sudo apt-get install -y swig - # Pinned to 3.12 because nsparse/python/pyproject.toml requires numpy<2.0, - # and numpy 1.26 is the last release supporting that constraint. On 3.13+ - # the pin resolves to a numpy older than the interpreter, which corrupts - # arrays at runtime rather than failing to install. - name: Set up Python id: python uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: - python-version: '3.12' + python-version: ${{ matrix.python-version }} + # The extension is compiled against the numpy installed here, which must be + # 2.x (nsparse/python/requirements.txt; CMake refuses otherwise): a module + # built against 2.x headers loads on numpy 1.26+ and 2.x alike, whereas one + # built against 1.x cannot import under numpy 2. - name: Install build dependencies - run: python -m pip install --upgrade pip "numpy<2.0" packaging setuptools wheel pytest + run: python -m pip install --upgrade pip -r nsparse/python/requirements.txt packaging setuptools wheel pytest - name: Configure run: | @@ -197,9 +203,9 @@ jobs: - name: Build run: cmake --build build -j$(nproc) - # --no-deps keeps the numpy installed above. Letting pip re-resolve here can - # swap numpy after the extension was compiled against its headers, and a - # mismatched pair corrupts search results instead of erroring. + # --no-deps keeps the numpy installed above, so the first test run below + # uses the very numpy the extension was compiled against; the 1.x run then + # swaps it deliberately. - name: Install the built package run: | cd build/nsparse/python @@ -215,6 +221,16 @@ jobs: - name: Run Python tests run: python -m pytest python_tests -v -rX + # A module built against numpy 2.x headers must keep loading and passing on + # the newest 1.x, the floor pyproject.toml declares. Only the 3.12 leg can + # exercise this: 3.13 has no numpy 1.x wheel. + - name: Run Python tests on numpy 1.x + if: matrix.python-version == '3.12' + run: | + python -m pip install "numpy<2.0" + python -c "import numpy; print('numpy', numpy.__version__)" + python -m pytest python_tests -v -rX + Build-nsparse-MacOS: name: Build and Test nsparse on MacOS needs: check-files diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 933e545..a099e29 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -186,13 +186,18 @@ cover the internals SWIG deliberately hides (`MmapCursor`, borrowing They need the bindings built and installed: ```bash +pip install -r nsparse/python/requirements.txt pytest # numpy 2.x headers for the build cmake -S . -B build -DNSPARSE_ENABLE_PYTHON=ON cmake --build build -j -pip install "numpy<2.0" pytest pip install --no-deps build/nsparse/python pytest python_tests -v ``` +The extension is compiled against the numpy present at configure time, which +must be 2.x (CMake refuses otherwise). A module built that way loads on numpy +1.26 and later as well as on 2.x, so the installed package places no upper +bound on numpy; one built against 1.x headers would not import under numpy 2. + One file per index type, named after the use case being exercised (`test_happy_case`, `test_with_id_map`, `test_exact_match`, ...). Accuracy is checked against an independent numpy brute-force oracle in diff --git a/demos/seismic_mmap.py b/demos/seismic_mmap.py index 0f7f4dc..4a2e472 100644 --- a/demos/seismic_mmap.py +++ b/demos/seismic_mmap.py @@ -35,12 +35,8 @@ Usage: python demos/seismic_mmap.py [options] -Needs a numpy that supports the running interpreter. Installing the nsparse -package pulls numpy<2.0 (see pyproject.toml), and on Python 3.13+ that resolves -to a numpy released before the interpreter existed, which silently corrupts live -arrays: `a - b` overwrites `a`, so scores turn to zeros midway through a run -while every check still passes. Install numpy>=2.1 over it (pip complains about -the pin; the complaint is the bug, not the fix). +Runs on whatever numpy the nsparse package is installed with, 1.26+ or 2.x: the +extension is built against numpy 2.x headers and loads on either. Exits non-zero if any check fails. """ diff --git a/nsparse/python/CMakeLists.txt b/nsparse/python/CMakeLists.txt index 5b22b82..a619b62 100644 --- a/nsparse/python/CMakeLists.txt +++ b/nsparse/python/CMakeLists.txt @@ -17,6 +17,20 @@ find_package(Python REQUIRED COMPONENTS Interpreter Development.Module NumPy ) +# The extension must be compiled against numpy 2.x headers. NumPy's C API is +# backward compatible in that direction only: a module built against 2.x loads +# on 1.26+ and 2.x (see NPY_TARGET_VERSION in swignsparse.swig), whereas one +# built against 1.x refuses to import under 2.x. pyproject.toml's open-ended +# numpy requirement relies on this, so enforce it here rather than trusting +# requirements.txt to have been read. +if(Python_NumPy_VERSION VERSION_LESS 2.0) + message(FATAL_ERROR + "nsparse Python bindings must be built against numpy>=2.0 headers; found " + "numpy ${Python_NumPy_VERSION} at ${Python_NumPy_INCLUDE_DIRS}. Install a " + "newer numpy for ${Python_EXECUTABLE} (pip install -r " + "nsparse/python/requirements.txt) and reconfigure.") +endif() + find_package(SWIG REQUIRED COMPONENTS python) include(${SWIG_USE_FILE}) diff --git a/nsparse/python/pyproject.toml b/nsparse/python/pyproject.toml index 45b33d5..91cc336 100644 --- a/nsparse/python/pyproject.toml +++ b/nsparse/python/pyproject.toml @@ -6,8 +6,12 @@ build-backend = "setuptools.build_meta" name = "nsparse" version = "0.1.0" description = "Python bindings for NSPARSE sparse vector search library" -requires-python = ">=3.8" -dependencies = ["numpy>=1.20.0,<2.0", "packaging"] +requires-python = ">=3.9" +# No upper bound on numpy: the extension is compiled against numpy 2.x headers +# (requirements.txt; enforced in CMakeLists.txt), and a module built that way +# loads on numpy 1.26+ and 2.x alike. 1.26 is the floor because loader.py +# imports numpy._core, which older 1.x releases do not provide. +dependencies = ["numpy>=1.26", "packaging"] license = "MIT" [tool.setuptools] diff --git a/nsparse/python/requirements.txt b/nsparse/python/requirements.txt index 88dd02a..915f926 100644 --- a/nsparse/python/requirements.txt +++ b/nsparse/python/requirements.txt @@ -1 +1,5 @@ +# Build-time requirement for the SWIG extension: compile against numpy 2.x +# headers so the built module loads on numpy 1.26+ and 2.x alike (a module +# built against 1.x headers cannot import under numpy 2). The runtime bound +# lives in pyproject.toml. numpy>=2.0 diff --git a/nsparse/python/swignsparse.swig b/nsparse/python/swignsparse.swig index 2fa3015..9669787 100644 --- a/nsparse/python/swignsparse.swig +++ b/nsparse/python/swignsparse.swig @@ -11,6 +11,13 @@ %{ #define SWIG_FILE_WITH_INIT +// Oldest numpy the built module loads on. Compiled against numpy 2.x headers +// (CMakeLists.txt enforces this), the module runs on any numpy from this C-API +// level up, 1.x and 2.x alike; numpy's import_array() rejects an older runtime +// with a clear error rather than misbehaving. 1.25 is the C-API level numpy +// 1.26 ships, matching the floor in pyproject.toml. Without this the floor +// would silently follow whichever numpy 2.x happened to provide the headers. +#define NPY_TARGET_VERSION NPY_1_25_API_VERSION #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION #include #include diff --git a/python_tests/conftest.py b/python_tests/conftest.py index f9b2613..02666b9 100644 --- a/python_tests/conftest.py +++ b/python_tests/conftest.py @@ -8,9 +8,13 @@ """Fixtures for the black-box index tests. Everything here goes through the installed extension module and the SWIG -surface only. Two harness guards run before any test, because both failure -modes are silent: a shadowed import gives an incomplete module, and a -mismatched numpy corrupts arrays rather than erroring. +surface only. One harness guard runs before any test, because its failure mode +is silent: a shadowed import gives an incomplete module rather than an error. + +No numpy guard is needed: the extension is built against numpy 2.x headers +(nsparse/python/CMakeLists.txt enforces it) and loads on numpy 1.26+ and 2.x +alike, while numpy's own import_array() refuses an incompatible runtime at +import time instead of misbehaving. """ import os @@ -39,7 +43,7 @@ def pytest_configure(config): - """Fail loudly on the two silent harness failure modes.""" + """Fail loudly on the silent harness failure mode.""" # Importing from the repo root resolves the source tree `nsparse/` as a # namespace package (__file__ is None) when the extension is not installed. # The result is a module without index_factory, which reads as a test bug. @@ -51,14 +55,6 @@ def pytest_configure(config): "Install the built extension: pip install --no-deps " "build/nsparse/python" ) - # nsparse/python/pyproject.toml pins numpy<2.0. A newer numpy than the one - # the extension was compiled against corrupts arrays at runtime instead of - # failing to import, which would look like a search regression. - if int(np.__version__.split(".")[0]) >= 2: - raise pytest.UsageError( - f"numpy {np.__version__} is incompatible with these bindings " - "(pyproject pins numpy<2.0); results would be silently corrupt." - ) @pytest.fixture(scope="session") From 192ee366f34d7d4da9779871d67debcf4aae1508 Mon Sep 17 00:00:00 2001 From: Heng Qian Date: Tue, 15 Sep 2026 15:20:06 +0800 Subject: [PATCH 2/3] docs: the index header has carried a format version since #36 The benchmark skill and DISK_SEISMIC_BENCH.md still said the serialized header has no version and that an old binary silently misreads a new file. IndexHeader has carried a per-type kFormatVersion since #36, and read_index rejects an unknown version. Restate the gotcha as what it now is: a layout change shipped without a version bump is still misread silently, so cached .dat files must be regenerated on layout changes. Signed-off-by: Heng Qian --- .claude/skills/benchmark-seismic/SKILL.md | 13 +++++++------ benchmarks/DISK_SEISMIC_BENCH.md | 5 +++-- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/.claude/skills/benchmark-seismic/SKILL.md b/.claude/skills/benchmark-seismic/SKILL.md index 227f980..fe3d44d 100644 --- a/.claude/skills/benchmark-seismic/SKILL.md +++ b/.claude/skills/benchmark-seismic/SKILL.md @@ -182,12 +182,13 @@ recall = np.mean([len(set(map(int, results[i])) & set(map(int, correct[i]))) ## Gotchas that have invalidated real runs -- **The serialized `.dat` header carries no format version** — `write_header` writes only fourcc and - dimension. A new binary reading an old file throws `std::length_error` (loud), but an **old binary - reading a new file silently loads a garbage index** with the correct `num_vectors` and wrong - results. Any change to on-disk layout (e.g. the alignment padding in `nsparse/io/align.h`) - invalidates every cached `.dat`. Regenerate them whenever the tree changes, and distrust a - suspiciously large speedup — one apparent 100× win was a binary misparsing a stale file. +- **A cached `.dat` is only as safe as its format version.** The header carries a per-index-type + version (`kFormatVersion`, checked by `read_index`), so a file from a layout this binary does not + know fails loudly — but only if the layout change also bumped the version. A layout change shipped + without a bump (e.g. to the alignment padding in `nsparse/io/align.h`) still **silently loads a + garbage index** with the correct `num_vectors` and wrong results. Regenerate cached `.dat` files + whenever the on-disk layout changes, and distrust a suspiciously large speedup — one apparent 100× + win was a binary misparsing a stale file. - **mmap requires the unquantized write path.** `seismic_sq` is not mmap-able, so enabling `kUseMmap` means writing plain `seismic` and losing 8-bit quantization — the index file roughly doubles. Do not attribute a latency change to residency when quantization moved with it. diff --git a/benchmarks/DISK_SEISMIC_BENCH.md b/benchmarks/DISK_SEISMIC_BENCH.md index c70a50d..faaef79 100644 --- a/benchmarks/DISK_SEISMIC_BENCH.md +++ b/benchmarks/DISK_SEISMIC_BENCH.md @@ -70,7 +70,8 @@ larger on disk but only its summaries stay resident) to expose the disk benefit. - **`base_small` is a smoke test, not a benchmark** — it fits in cache, so the memory-bound behavior that is the whole point disappears. Only report `base_full`. -- Regenerate the `.dat` files whenever the source tree changes; the serialized format - carries no version and an old binary silently misreads a new file. +- Regenerate the `.dat` files whenever the on-disk layout changes. The header carries a + per-type format version, so a mismatch is caught only if the change bumped + `kFormatVersion`; one that did not is misread silently. - Query threads are pinned to 1 (`OMP_NUM_THREADS=1`) — per-query cost is the metric and the workload is bandwidth-bound. From 3e681ee4212933081054cc79ab21ebb14094808c Mon Sep 17 00:00:00 2001 From: Heng Qian Date: Tue, 15 Sep 2026 18:02:31 +0800 Subject: [PATCH 3/3] Make the declared numpy floor real, and trim the comments Review follow-ups: - loader.py: detect SVE through getauxval(AT_HWCAP) alone. The numpy.distutils path returned False on numpy >= 2, the very numpy this change steers to, so SVE silently stopped being selected on aarch64. - loader.py: fall back to numpy.core when numpy._core is absent. The _core shim only appeared in 1.26.1, so the declared floor of 1.26 admitted a release (1.26.0) on which the package failed to import. - CI: install the package without --no-deps so pip resolves pyproject's requirement for real, and run the 1.x leg on numpy==1.26.0, the exact floor, instead of whatever numpy<2.0 happens to resolve to. - Cut the comments down to what the code does: one or two lines in CMakeLists.txt and swignsparse.swig, none in pyproject.toml or the CI steps, no paragraphs explaining removed checks or absent flags in conftest.py, the demo docstring or the benchmark skill. State only what NPY_TARGET_VERSION pins (the C-API level) and leave the package floor to pyproject.toml. Signed-off-by: Heng Qian --- .claude/skills/benchmark-seismic/SKILL.md | 4 +--- .github/workflows/CI.yml | 20 ++++------------- DEVELOPER_GUIDE.md | 8 +++---- demos/seismic_mmap.py | 3 --- nsparse/python/CMakeLists.txt | 16 +++++--------- nsparse/python/loader.py | 27 ++++++++++------------- nsparse/python/pyproject.toml | 4 ---- nsparse/python/requirements.txt | 5 +---- nsparse/python/swignsparse.swig | 8 ++----- python_tests/conftest.py | 7 +----- 10 files changed, 29 insertions(+), 73 deletions(-) diff --git a/.claude/skills/benchmark-seismic/SKILL.md b/.claude/skills/benchmark-seismic/SKILL.md index fe3d44d..1bab589 100644 --- a/.claude/skills/benchmark-seismic/SKILL.md +++ b/.claude/skills/benchmark-seismic/SKILL.md @@ -157,9 +157,7 @@ OMP_NUM_THREADS=1 python demos/seismic_mmap.py \ --reuse-index --keep ``` -Install the bindings with `pip install build/nsparse/python`. They are built against numpy 2.x -headers (`nsparse/python/CMakeLists.txt` enforces it) and run on numpy 1.26+ and 2.x alike, so no -numpy override or interpreter pin is needed. +Install the bindings with `pip install build/nsparse/python`; any numpy 1.26+ or 2.x works. ### Peak build memory — `disk_build_mem_bench` diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index da33cae..9b8d286 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -167,8 +167,6 @@ jobs: strategy: fail-fast: false matrix: - # 3.13 has no numpy 1.x wheel, which is what used to make the numpy<2.0 - # pin resolve to a source build older than the interpreter. python-version: ['3.12', '3.13'] steps: - name: Checkout @@ -185,10 +183,6 @@ jobs: with: python-version: ${{ matrix.python-version }} - # The extension is compiled against the numpy installed here, which must be - # 2.x (nsparse/python/requirements.txt; CMake refuses otherwise): a module - # built against 2.x headers loads on numpy 1.26+ and 2.x alike, whereas one - # built against 1.x cannot import under numpy 2. - name: Install build dependencies run: python -m pip install --upgrade pip -r nsparse/python/requirements.txt packaging setuptools wheel pytest @@ -203,13 +197,10 @@ jobs: - name: Build run: cmake --build build -j$(nproc) - # --no-deps keeps the numpy installed above, so the first test run below - # uses the very numpy the extension was compiled against; the 1.x run then - # swaps it deliberately. - name: Install the built package run: | cd build/nsparse/python - python -m pip install --no-deps . + python -m pip install . - name: Import check run: python -c "import nsparse; print(nsparse.index_factory)" @@ -221,14 +212,11 @@ jobs: - name: Run Python tests run: python -m pytest python_tests -v -rX - # A module built against numpy 2.x headers must keep loading and passing on - # the newest 1.x, the floor pyproject.toml declares. Only the 3.12 leg can - # exercise this: 3.13 has no numpy 1.x wheel. - - name: Run Python tests on numpy 1.x + # The floor pyproject.toml declares, exactly. 3.13 has no numpy 1.x wheel. + - name: Run Python tests on the numpy floor if: matrix.python-version == '3.12' run: | - python -m pip install "numpy<2.0" - python -c "import numpy; print('numpy', numpy.__version__)" + python -m pip install "numpy==1.26.0" python -m pytest python_tests -v -rX Build-nsparse-MacOS: diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index a099e29..6f0d2a7 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -189,14 +189,12 @@ They need the bindings built and installed: pip install -r nsparse/python/requirements.txt pytest # numpy 2.x headers for the build cmake -S . -B build -DNSPARSE_ENABLE_PYTHON=ON cmake --build build -j -pip install --no-deps build/nsparse/python +pip install build/nsparse/python pytest python_tests -v ``` -The extension is compiled against the numpy present at configure time, which -must be 2.x (CMake refuses otherwise). A module built that way loads on numpy -1.26 and later as well as on 2.x, so the installed package places no upper -bound on numpy; one built against 1.x headers would not import under numpy 2. +The bindings are compiled against the numpy present at configure time, which +must be 2.x; a module built that way runs on numpy 1.26+ and 2.x. One file per index type, named after the use case being exercised (`test_happy_case`, `test_with_id_map`, `test_exact_match`, ...). Accuracy is diff --git a/demos/seismic_mmap.py b/demos/seismic_mmap.py index 4a2e472..b657385 100644 --- a/demos/seismic_mmap.py +++ b/demos/seismic_mmap.py @@ -35,9 +35,6 @@ Usage: python demos/seismic_mmap.py [options] -Runs on whatever numpy the nsparse package is installed with, 1.26+ or 2.x: the -extension is built against numpy 2.x headers and loads on either. - Exits non-zero if any check fails. """ diff --git a/nsparse/python/CMakeLists.txt b/nsparse/python/CMakeLists.txt index a619b62..04f6ebf 100644 --- a/nsparse/python/CMakeLists.txt +++ b/nsparse/python/CMakeLists.txt @@ -17,18 +17,12 @@ find_package(Python REQUIRED COMPONENTS Interpreter Development.Module NumPy ) -# The extension must be compiled against numpy 2.x headers. NumPy's C API is -# backward compatible in that direction only: a module built against 2.x loads -# on 1.26+ and 2.x (see NPY_TARGET_VERSION in swignsparse.swig), whereas one -# built against 1.x refuses to import under 2.x. pyproject.toml's open-ended -# numpy requirement relies on this, so enforce it here rather than trusting -# requirements.txt to have been read. +# A module built against numpy 2.x headers loads on numpy 1.x and 2.x; one built +# against 1.x cannot import under 2.x, which pyproject.toml permits. if(Python_NumPy_VERSION VERSION_LESS 2.0) - message(FATAL_ERROR - "nsparse Python bindings must be built against numpy>=2.0 headers; found " - "numpy ${Python_NumPy_VERSION} at ${Python_NumPy_INCLUDE_DIRS}. Install a " - "newer numpy for ${Python_EXECUTABLE} (pip install -r " - "nsparse/python/requirements.txt) and reconfigure.") + message(FATAL_ERROR "numpy>=2.0 headers are required to build the bindings; " + "found ${Python_NumPy_VERSION} for ${Python_EXECUTABLE} " + "(pip install -r nsparse/python/requirements.txt).") endif() find_package(SWIG REQUIRED COMPONENTS python) diff --git a/nsparse/python/loader.py b/nsparse/python/loader.py index 121c68f..01eddac 100644 --- a/nsparse/python/loader.py +++ b/nsparse/python/loader.py @@ -24,27 +24,24 @@ def supported_instruction_sets(): """ def is_sve_supported(): - if platform.machine() != "aarch64": + # AT_HWCAP (16) from the kernel; HWCAP_SVE is bit 22 on aarch64. + if platform.machine() != "aarch64" or platform.system() != "Linux": return False - if platform.system() != "Linux": - return False - import numpy + import ctypes - if Version(numpy.__version__) >= Version("2.0"): - return False - try: - import numpy.distutils.cpuinfo - - return ( - "sve" in numpy.distutils.cpuinfo.cpu.info[0].get("Features", "").split() - ) - except ImportError: - return bool(__import__("ctypes").CDLL(None).getauxval(16) & (1 << 22)) + getauxval = ctypes.CDLL(None).getauxval + getauxval.restype = ctypes.c_ulong + getauxval.argtypes = [ctypes.c_ulong] + return bool(getauxval(16) & (1 << 22)) import numpy if Version(numpy.__version__) >= Version("1.19"): - from numpy._core._multiarray_umath import __cpu_features__ + try: + # numpy 2.x, and the shim numpy 1.26.1+ ships + from numpy._core._multiarray_umath import __cpu_features__ + except ImportError: # numpy 1.x up to 1.26.0 + from numpy.core._multiarray_umath import __cpu_features__ supported = {k for k, v in __cpu_features__.items() if v} if is_sve_supported(): diff --git a/nsparse/python/pyproject.toml b/nsparse/python/pyproject.toml index 91cc336..62bc1bf 100644 --- a/nsparse/python/pyproject.toml +++ b/nsparse/python/pyproject.toml @@ -7,10 +7,6 @@ name = "nsparse" version = "0.1.0" description = "Python bindings for NSPARSE sparse vector search library" requires-python = ">=3.9" -# No upper bound on numpy: the extension is compiled against numpy 2.x headers -# (requirements.txt; enforced in CMakeLists.txt), and a module built that way -# loads on numpy 1.26+ and 2.x alike. 1.26 is the floor because loader.py -# imports numpy._core, which older 1.x releases do not provide. dependencies = ["numpy>=1.26", "packaging"] license = "MIT" diff --git a/nsparse/python/requirements.txt b/nsparse/python/requirements.txt index 915f926..846e74e 100644 --- a/nsparse/python/requirements.txt +++ b/nsparse/python/requirements.txt @@ -1,5 +1,2 @@ -# Build-time requirement for the SWIG extension: compile against numpy 2.x -# headers so the built module loads on numpy 1.26+ and 2.x alike (a module -# built against 1.x headers cannot import under numpy 2). The runtime bound -# lives in pyproject.toml. +# Build-time headers only; the runtime bound is in pyproject.toml. numpy>=2.0 diff --git a/nsparse/python/swignsparse.swig b/nsparse/python/swignsparse.swig index 9669787..717ebc5 100644 --- a/nsparse/python/swignsparse.swig +++ b/nsparse/python/swignsparse.swig @@ -11,12 +11,8 @@ %{ #define SWIG_FILE_WITH_INIT -// Oldest numpy the built module loads on. Compiled against numpy 2.x headers -// (CMakeLists.txt enforces this), the module runs on any numpy from this C-API -// level up, 1.x and 2.x alike; numpy's import_array() rejects an older runtime -// with a clear error rather than misbehaving. 1.25 is the C-API level numpy -// 1.26 ships, matching the floor in pyproject.toml. Without this the floor -// would silently follow whichever numpy 2.x happened to provide the headers. +// Pin the C-API floor (the 1.25/1.26 level) rather than inherit whatever the +// numpy 2.x providing the headers defaults to. The package floor is pyproject's. #define NPY_TARGET_VERSION NPY_1_25_API_VERSION #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION #include diff --git a/python_tests/conftest.py b/python_tests/conftest.py index 02666b9..7264672 100644 --- a/python_tests/conftest.py +++ b/python_tests/conftest.py @@ -10,11 +10,6 @@ Everything here goes through the installed extension module and the SWIG surface only. One harness guard runs before any test, because its failure mode is silent: a shadowed import gives an incomplete module rather than an error. - -No numpy guard is needed: the extension is built against numpy 2.x headers -(nsparse/python/CMakeLists.txt enforces it) and loads on numpy 1.26+ and 2.x -alike, while numpy's own import_array() refuses an incompatible runtime at -import time instead of misbehaving. """ import os @@ -52,7 +47,7 @@ def pytest_configure(config): ): raise pytest.UsageError( f"nsparse resolved to an incomplete module ({nsparse.__file__!r}). " - "Install the built extension: pip install --no-deps " + "Install the built extension: pip install " "build/nsparse/python" )