-
Notifications
You must be signed in to change notification settings - Fork 20
Fix PyLucene sidecar packaging and add smoke tests #174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nvzm123
wants to merge
15
commits into
NVIDIA:main
Choose a base branch
from
nvzm123:pr-147
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
1c0f648
Add PyLucene integration support
cjnolet e27a53b
Fix benchmark version marker
cjnolet 3cab384
Fix PyLucene sidecar packaging and update cuVS version
nvzm123 6e3cf20
Merge remote-tracking branch 'upstream/main' into pr-147
nvzm123 7ec17fe
Expand PyLucene smoke coverage
nvzm123 5ade7af
Merge branch 'main' into pr-147
nvzm123 692c3db
Make writer telemetry on-demand
nvzm123 0e6bb43
Avoid duplicate binary format initialization
nvzm123 eeff876
Cache binary quantized vector formats
nvzm123 b76e2f5
Expose writer diagnostics through format descriptions
nvzm123 bf23e24
Use Lucene 102 binary vector formats
nvzm123 f7a8c66
Use the standard jar for PyLucene
nvzm123 7084157
Add a public PyLucene test entrypoint
nvzm123 d13184d
Move PyLucene tests under examples Python
nvzm123 df836cc
Clarify binary format version handling
nvzm123 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,6 +20,100 @@ mvn clean compile package | |
|
|
||
| The artifacts would be built and available in the target / folder. | ||
|
|
||
| ### Using with PyLucene | ||
|
|
||
| PyLucene embeds a JVM and starts it with the classpath passed to `lucene.initVM(...)`. | ||
| Because PyLucene's generated Python module only exposes the Java classes it was built | ||
| to wrap, use Lucene's service provider lookup to load `cuvs-lucene` codecs from | ||
| Python instead of importing `com.nvidia.cuvs.lucene` classes directly. | ||
|
|
||
| Build the standard cuvs-lucene jar: | ||
|
|
||
| ```sh | ||
| mvn clean package -DskipTests | ||
| ``` | ||
|
|
||
| Then start PyLucene with the base `cuvs-java` jar, the standard `cuvs-lucene` | ||
| jar, and PyLucene's own Lucene classpath: | ||
|
|
||
| ```python | ||
| import os | ||
| from pathlib import Path | ||
|
|
||
| import lucene | ||
|
|
||
| cuvs_java_jar = Path(os.environ["CUVS_LUCENE_CUVS_JAVA_JAR"]) | ||
| cuvs_lucene_jar = next( | ||
| jar | ||
| for jar in Path("target").glob("cuvs-lucene-*.jar") | ||
| if "-jar-with-" not in jar.name | ||
| and not jar.name.endswith(("-sources.jar", "-javadoc.jar")) | ||
| ) | ||
| lucene.initVM( | ||
| classpath=os.pathsep.join( | ||
| [str(cuvs_java_jar), str(cuvs_lucene_jar), lucene.CLASSPATH] | ||
| ), | ||
| vmargs=[ | ||
| "--enable-native-access=ALL-UNNAMED", | ||
| "--add-modules=jdk.incubator.vector", | ||
| ], | ||
| ) | ||
|
|
||
| from org.apache.lucene.codecs import Codec | ||
|
|
||
| codec = Codec.forName("Lucene101AcceleratedHNSWCodec") | ||
| ``` | ||
|
|
||
| Use the returned `codec` with `IndexWriterConfig.setCodec(codec)`. The standard | ||
| artifact includes `cuvs-lucene` classes and service descriptors. | ||
| PyLucene must provide Lucene classes, and the base multi-release `cuvs-java` jar | ||
| must be present separately on the JVM classpath. Do not use a native classifier | ||
| `cuvs-java` jar here unless you also want to rely on its embedded native | ||
| libraries; the base jar uses native libraries from | ||
| `LD_LIBRARY_PATH`/`java.library.path`. | ||
|
|
||
| To run the PyLucene pytest smoke suite against a local PyLucene environment: | ||
|
|
||
| ```sh | ||
| ./test_pylucene.sh | ||
| ``` | ||
|
|
||
| The script builds and validates the jar before invoking pytest. To invoke pytest | ||
| directly against existing artifacts instead: | ||
|
|
||
| ```sh | ||
| CUVS_LUCENE_JAR=/path/to/cuvs-lucene.jar \ | ||
| CUVS_LUCENE_CUVS_JAVA_JAR=/path/to/cuvs-java.jar \ | ||
| python3 -m pytest -q -s examples/Python/test_pylucene_smoke.py | ||
| ``` | ||
|
|
||
| To run an expanded GPU end-to-end pytest suite through CPU HNSW, | ||
| CAGRA-to-HNSW, and CAGRA search paths: | ||
|
|
||
| ```sh | ||
| ./test_pylucene.sh --gpu-e2e | ||
| ``` | ||
|
|
||
| The expanded suite runs the `gpu-basic`, `gpu-segments`, `cpu-hnsw`, and | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great description here. How to build and run tests should really be in a separate build and install guide. I think this is okay for now, especially since we are moving cuVS-Lucene to cuVS, but it's something to consider. |
||
| `cagra-hnsw` case groups. The basic cases cover `hnsw`, `cagra`, `hnsw-single`, | ||
| and `cagra-single`. The segment cases cover 1-segment indexes, 10-segment | ||
| indexes, 10 segments force-merged to 1, and 100 segments force-merged to 10 for | ||
| both HNSW and CAGRA. The CPU HNSW cases force the accelerated HNSW codec through | ||
| its Lucene CPU fallback path in the same run, including 10 segments force-merged | ||
| to 1 and 100 segments force-merged to 10. The CAGRA-to-HNSW cases explicitly | ||
| cover one-layer and three-layer HNSW graphs built from CAGRA with NN_DESCENT, | ||
| `graphDegree=32`, and `intermediateGraphDegree=64`. The base matrix uses 2,000 | ||
| documents and 32 dimensions; high-segment cases use at least 257 rows per | ||
| segment to avoid expected cuVS graph-degree clamps on tiny per-segment datasets. | ||
| The suite checks Lucene SPI discovery, jar packaging, index file suffixes | ||
| (`.vex`/`.vem` for HNSW and `.vcag`/`.vemc` for CAGRA), indexed vector metadata, | ||
| unfiltered KNN, filtered KNN, missing-vector documents, deletions, and force | ||
| merge behavior. To run a subset or resize the test: | ||
|
|
||
| ```sh | ||
| ./test_pylucene.sh --gpu-e2e --cases=gpu-segments --rows=5000 --dims=64 --topk=20 | ||
| ``` | ||
|
|
||
| ### Running Tests | ||
|
|
||
| ```sh | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| #!/bin/bash | ||
|
|
||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | ||
| exec "${REPO_ROOT}/test_pylucene.sh" "$@" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like this adds yet another jar, which is going to add up quickly. Can we please just add the pylucene dependencies directly to the main jar?
cc @imotov for thoughts here. Assuming it doesn't bloat the dependency chain, it would be good to have a minimal set of rich build artifacts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked a bit closer into it. I don't see any significant differences between
cuvs-lucene-26.10.0.jarandcuvs-lucene-26.10.0-jar-with-pylucene-dependencies.jar. Thecuvs-lucene-26.10.0.jarfile is built in a slightly different way so it contains a bit more metadata, but class-wise the files are identical:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for looking into this @imotov. Do you think we should just use the existing package? @nvzm123 do you foresee any issues removing the special pylucene jar and just using the existing package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there was a legitimate reason for @nvzm123 to create that package in the first place, I’d love to understand the rationale. If it was created by mistake and/or is no longer needed, I don’t see a compelling reason to keep it. As far as I can tell, it doesn’t currently do anything useful that
cuvs-lucene-26.10.0.jarcannot do.