diff --git a/.claude/skills/benchmark-seismic/SKILL.md b/.claude/skills/benchmark-seismic/SKILL.md index f5d78c4..1bab589 100644 --- a/.claude/skills/benchmark-seismic/SKILL.md +++ b/.claude/skills/benchmark-seismic/SKILL.md @@ -157,9 +157,7 @@ OMP_NUM_THREADS=1 python demos/seismic_mmap.py \ --reuse-index --keep ``` -Needs `numpy>=2.1` installed *over* the nsparse package: `pyproject.toml` pins `numpy<2.0`, which on -Python 3.13+ resolves to a numpy older than the interpreter and silently corrupts arrays (`a - b` -overwrites `a`) while every check still passes. Pin Python 3.12 or force the newer numpy. +Install the bindings with `pip install build/nsparse/python`; any numpy 1.26+ or 2.x works. ### Peak build memory — `disk_build_mem_bench` @@ -182,12 +180,13 @@ recall = np.mean([len(set(map(int, results[i])) & set(map(int, correct[i]))) ## Gotchas that have invalidated real runs -- **The serialized `.dat` header carries no format version** — `write_header` writes only fourcc and - dimension. A new binary reading an old file throws `std::length_error` (loud), but an **old binary - reading a new file silently loads a garbage index** with the correct `num_vectors` and wrong - results. Any change to on-disk layout (e.g. the alignment padding in `nsparse/io/align.h`) - invalidates every cached `.dat`. Regenerate them whenever the tree changes, and distrust a - suspiciously large speedup — one apparent 100× win was a binary misparsing a stale file. +- **A cached `.dat` is only as safe as its format version.** The header carries a per-index-type + version (`kFormatVersion`, checked by `read_index`), so a file from a layout this binary does not + know fails loudly — but only if the layout change also bumped the version. A layout change shipped + without a bump (e.g. to the alignment padding in `nsparse/io/align.h`) still **silently loads a + garbage index** with the correct `num_vectors` and wrong results. Regenerate cached `.dat` files + whenever the on-disk layout changes, and distrust a suspiciously large speedup — one apparent 100× + win was a binary misparsing a stale file. - **mmap requires the unquantized write path.** `seismic_sq` is not mmap-able, so enabling `kUseMmap` means writing plain `seismic` and losing 8-bit quantization — the index file roughly doubles. Do not attribute a latency change to residency when quantization moved with it. diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 00e8942..9b8d286 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -164,6 +164,10 @@ jobs: needs: check-files if: needs.check-files.outputs.RUN_BUILD_AND_TEST == 'true' runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ['3.12', '3.13'] steps: - name: Checkout uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 @@ -173,18 +177,14 @@ jobs: sudo apt-get update sudo apt-get install -y swig - # Pinned to 3.12 because nsparse/python/pyproject.toml requires numpy<2.0, - # and numpy 1.26 is the last release supporting that constraint. On 3.13+ - # the pin resolves to a numpy older than the interpreter, which corrupts - # arrays at runtime rather than failing to install. - name: Set up Python id: python uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: - python-version: '3.12' + python-version: ${{ matrix.python-version }} - name: Install build dependencies - run: python -m pip install --upgrade pip "numpy<2.0" packaging setuptools wheel pytest + run: python -m pip install --upgrade pip -r nsparse/python/requirements.txt packaging setuptools wheel pytest - name: Configure run: | @@ -197,13 +197,10 @@ jobs: - name: Build run: cmake --build build -j$(nproc) - # --no-deps keeps the numpy installed above. Letting pip re-resolve here can - # swap numpy after the extension was compiled against its headers, and a - # mismatched pair corrupts search results instead of erroring. - name: Install the built package run: | cd build/nsparse/python - python -m pip install --no-deps . + python -m pip install . - name: Import check run: python -c "import nsparse; print(nsparse.index_factory)" @@ -215,6 +212,13 @@ jobs: - name: Run Python tests run: python -m pytest python_tests -v -rX + # The floor pyproject.toml declares, exactly. 3.13 has no numpy 1.x wheel. + - name: Run Python tests on the numpy floor + if: matrix.python-version == '3.12' + run: | + python -m pip install "numpy==1.26.0" + python -m pytest python_tests -v -rX + Build-nsparse-MacOS: name: Build and Test nsparse on MacOS needs: check-files diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 933e545..6f0d2a7 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -186,13 +186,16 @@ cover the internals SWIG deliberately hides (`MmapCursor`, borrowing They need the bindings built and installed: ```bash +pip install -r nsparse/python/requirements.txt pytest # numpy 2.x headers for the build cmake -S . -B build -DNSPARSE_ENABLE_PYTHON=ON cmake --build build -j -pip install "numpy<2.0" pytest -pip install --no-deps build/nsparse/python +pip install build/nsparse/python pytest python_tests -v ``` +The bindings are compiled against the numpy present at configure time, which +must be 2.x; a module built that way runs on numpy 1.26+ and 2.x. + One file per index type, named after the use case being exercised (`test_happy_case`, `test_with_id_map`, `test_exact_match`, ...). Accuracy is checked against an independent numpy brute-force oracle in diff --git a/benchmarks/DISK_SEISMIC_BENCH.md b/benchmarks/DISK_SEISMIC_BENCH.md index c70a50d..faaef79 100644 --- a/benchmarks/DISK_SEISMIC_BENCH.md +++ b/benchmarks/DISK_SEISMIC_BENCH.md @@ -70,7 +70,8 @@ larger on disk but only its summaries stay resident) to expose the disk benefit. - **`base_small` is a smoke test, not a benchmark** — it fits in cache, so the memory-bound behavior that is the whole point disappears. Only report `base_full`. -- Regenerate the `.dat` files whenever the source tree changes; the serialized format - carries no version and an old binary silently misreads a new file. +- Regenerate the `.dat` files whenever the on-disk layout changes. The header carries a + per-type format version, so a mismatch is caught only if the change bumped + `kFormatVersion`; one that did not is misread silently. - Query threads are pinned to 1 (`OMP_NUM_THREADS=1`) — per-query cost is the metric and the workload is bandwidth-bound. diff --git a/demos/seismic_mmap.py b/demos/seismic_mmap.py index 0f7f4dc..b657385 100644 --- a/demos/seismic_mmap.py +++ b/demos/seismic_mmap.py @@ -35,13 +35,6 @@ Usage: python demos/seismic_mmap.py [options] -Needs a numpy that supports the running interpreter. Installing the nsparse -package pulls numpy<2.0 (see pyproject.toml), and on Python 3.13+ that resolves -to a numpy released before the interpreter existed, which silently corrupts live -arrays: `a - b` overwrites `a`, so scores turn to zeros midway through a run -while every check still passes. Install numpy>=2.1 over it (pip complains about -the pin; the complaint is the bug, not the fix). - Exits non-zero if any check fails. """ diff --git a/nsparse/python/CMakeLists.txt b/nsparse/python/CMakeLists.txt index 5b22b82..04f6ebf 100644 --- a/nsparse/python/CMakeLists.txt +++ b/nsparse/python/CMakeLists.txt @@ -17,6 +17,14 @@ find_package(Python REQUIRED COMPONENTS Interpreter Development.Module NumPy ) +# A module built against numpy 2.x headers loads on numpy 1.x and 2.x; one built +# against 1.x cannot import under 2.x, which pyproject.toml permits. +if(Python_NumPy_VERSION VERSION_LESS 2.0) + message(FATAL_ERROR "numpy>=2.0 headers are required to build the bindings; " + "found ${Python_NumPy_VERSION} for ${Python_EXECUTABLE} " + "(pip install -r nsparse/python/requirements.txt).") +endif() + find_package(SWIG REQUIRED COMPONENTS python) include(${SWIG_USE_FILE}) diff --git a/nsparse/python/loader.py b/nsparse/python/loader.py index 121c68f..01eddac 100644 --- a/nsparse/python/loader.py +++ b/nsparse/python/loader.py @@ -24,27 +24,24 @@ def supported_instruction_sets(): """ def is_sve_supported(): - if platform.machine() != "aarch64": + # AT_HWCAP (16) from the kernel; HWCAP_SVE is bit 22 on aarch64. + if platform.machine() != "aarch64" or platform.system() != "Linux": return False - if platform.system() != "Linux": - return False - import numpy + import ctypes - if Version(numpy.__version__) >= Version("2.0"): - return False - try: - import numpy.distutils.cpuinfo - - return ( - "sve" in numpy.distutils.cpuinfo.cpu.info[0].get("Features", "").split() - ) - except ImportError: - return bool(__import__("ctypes").CDLL(None).getauxval(16) & (1 << 22)) + getauxval = ctypes.CDLL(None).getauxval + getauxval.restype = ctypes.c_ulong + getauxval.argtypes = [ctypes.c_ulong] + return bool(getauxval(16) & (1 << 22)) import numpy if Version(numpy.__version__) >= Version("1.19"): - from numpy._core._multiarray_umath import __cpu_features__ + try: + # numpy 2.x, and the shim numpy 1.26.1+ ships + from numpy._core._multiarray_umath import __cpu_features__ + except ImportError: # numpy 1.x up to 1.26.0 + from numpy.core._multiarray_umath import __cpu_features__ supported = {k for k, v in __cpu_features__.items() if v} if is_sve_supported(): diff --git a/nsparse/python/pyproject.toml b/nsparse/python/pyproject.toml index 45b33d5..62bc1bf 100644 --- a/nsparse/python/pyproject.toml +++ b/nsparse/python/pyproject.toml @@ -6,8 +6,8 @@ build-backend = "setuptools.build_meta" name = "nsparse" version = "0.1.0" description = "Python bindings for NSPARSE sparse vector search library" -requires-python = ">=3.8" -dependencies = ["numpy>=1.20.0,<2.0", "packaging"] +requires-python = ">=3.9" +dependencies = ["numpy>=1.26", "packaging"] license = "MIT" [tool.setuptools] diff --git a/nsparse/python/requirements.txt b/nsparse/python/requirements.txt index 88dd02a..846e74e 100644 --- a/nsparse/python/requirements.txt +++ b/nsparse/python/requirements.txt @@ -1 +1,2 @@ +# Build-time headers only; the runtime bound is in pyproject.toml. numpy>=2.0 diff --git a/nsparse/python/swignsparse.swig b/nsparse/python/swignsparse.swig index 2fa3015..717ebc5 100644 --- a/nsparse/python/swignsparse.swig +++ b/nsparse/python/swignsparse.swig @@ -11,6 +11,9 @@ %{ #define SWIG_FILE_WITH_INIT +// Pin the C-API floor (the 1.25/1.26 level) rather than inherit whatever the +// numpy 2.x providing the headers defaults to. The package floor is pyproject's. +#define NPY_TARGET_VERSION NPY_1_25_API_VERSION #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION #include #include diff --git a/python_tests/conftest.py b/python_tests/conftest.py index f9b2613..7264672 100644 --- a/python_tests/conftest.py +++ b/python_tests/conftest.py @@ -8,9 +8,8 @@ """Fixtures for the black-box index tests. Everything here goes through the installed extension module and the SWIG -surface only. Two harness guards run before any test, because both failure -modes are silent: a shadowed import gives an incomplete module, and a -mismatched numpy corrupts arrays rather than erroring. +surface only. One harness guard runs before any test, because its failure mode +is silent: a shadowed import gives an incomplete module rather than an error. """ import os @@ -39,7 +38,7 @@ def pytest_configure(config): - """Fail loudly on the two silent harness failure modes.""" + """Fail loudly on the silent harness failure mode.""" # Importing from the repo root resolves the source tree `nsparse/` as a # namespace package (__file__ is None) when the extension is not installed. # The result is a module without index_factory, which reads as a test bug. @@ -48,17 +47,9 @@ def pytest_configure(config): ): raise pytest.UsageError( f"nsparse resolved to an incomplete module ({nsparse.__file__!r}). " - "Install the built extension: pip install --no-deps " + "Install the built extension: pip install " "build/nsparse/python" ) - # nsparse/python/pyproject.toml pins numpy<2.0. A newer numpy than the one - # the extension was compiled against corrupts arrays at runtime instead of - # failing to import, which would look like a search regression. - if int(np.__version__.split(".")[0]) >= 2: - raise pytest.UsageError( - f"numpy {np.__version__} is incompatible with these bindings " - "(pyproject pins numpy<2.0); results would be silently corrupt." - ) @pytest.fixture(scope="session")