Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,100 @@ mvn clean compile package

The artifacts would be built and available in the target / folder.

### Using with PyLucene

PyLucene embeds a JVM and starts it with the classpath passed to `lucene.initVM(...)`.
Because PyLucene's generated Python module only exposes the Java classes it was built
to wrap, use Lucene's service provider lookup to load `cuvs-lucene` codecs from
Python instead of importing `com.nvidia.cuvs.lucene` classes directly.

Build the standard cuvs-lucene jar:

```sh
mvn clean package -DskipTests
```

Then start PyLucene with the base `cuvs-java` jar, the standard `cuvs-lucene`
jar, and PyLucene's own Lucene classpath:

```python
import os
from pathlib import Path

import lucene

cuvs_java_jar = Path(os.environ["CUVS_LUCENE_CUVS_JAVA_JAR"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this adds yet another jar, which is going to add up quickly. Can we please just add the pylucene dependencies directly to the main jar?

cc @imotov for thoughts here. Assuming it doesn't bloat the dependency chain, it would be good to have a minimal set of rich build artifacts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked a bit closer into it. I don't see any significant differences between cuvs-lucene-26.10.0.jar and cuvs-lucene-26.10.0-jar-with-pylucene-dependencies.jar. The cuvs-lucene-26.10.0.jar file is built in a slightly different way so it contains a bit more metadata, but class-wise the files are identical:

(cuvs-shared) *[main][/raid/imotov/Projects/NVIDIA/cuvs-lucene/target]$ diff <(jar tvf cuvs-lucene-26.10.0.jar) <(jar tvf cuvs-lucene-26.10.0-jar-with-pylucene-dependencies.jar) 
1,2c1,2
<      0 Fri Jul 17 22:26:54 UTC 2026 META-INF/
<     99 Fri Jul 17 22:26:54 UTC 2026 META-INF/MANIFEST.MF
---
>      0 Fri Jul 17 22:27:08 UTC 2026 META-INF/
>     25 Fri Jul 17 22:27:08 UTC 2026 META-INF/MANIFEST.MF
8,10d7
<      0 Fri Jul 17 22:26:54 UTC 2026 META-INF/maven/
<      0 Fri Jul 17 22:26:54 UTC 2026 META-INF/maven/com.nvidia.cuvs.lucene/
<      0 Fri Jul 17 22:26:54 UTC 2026 META-INF/maven/com.nvidia.cuvs.lucene/cuvs-lucene/
57,58d53
<   8679 Thu Jul 16 23:45:28 UTC 2026 META-INF/maven/com.nvidia.cuvs.lucene/cuvs-lucene/pom.xml
<     70 Fri Jul 17 22:26:54 UTC 2026 META-INF/maven/com.nvidia.cuvs.lucene/cuvs-lucene/pom.properties

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this @imotov. Do you think we should just use the existing package? @nvzm123 do you foresee any issues removing the special pylucene jar and just using the existing package?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there was a legitimate reason for @nvzm123 to create that package in the first place, I’d love to understand the rationale. If it was created by mistake and/or is no longer needed, I don’t see a compelling reason to keep it. As far as I can tell, it doesn’t currently do anything useful that cuvs-lucene-26.10.0.jar cannot do.

cuvs_lucene_jar = next(
jar
for jar in Path("target").glob("cuvs-lucene-*.jar")
if "-jar-with-" not in jar.name
and not jar.name.endswith(("-sources.jar", "-javadoc.jar"))
)
lucene.initVM(
classpath=os.pathsep.join(
[str(cuvs_java_jar), str(cuvs_lucene_jar), lucene.CLASSPATH]
),
vmargs=[
"--enable-native-access=ALL-UNNAMED",
"--add-modules=jdk.incubator.vector",
],
)

from org.apache.lucene.codecs import Codec

codec = Codec.forName("Lucene101AcceleratedHNSWCodec")
```

Use the returned `codec` with `IndexWriterConfig.setCodec(codec)`. The standard
artifact includes `cuvs-lucene` classes and service descriptors.
PyLucene must provide Lucene classes, and the base multi-release `cuvs-java` jar
must be present separately on the JVM classpath. Do not use a native classifier
`cuvs-java` jar here unless you also want to rely on its embedded native
libraries; the base jar uses native libraries from
`LD_LIBRARY_PATH`/`java.library.path`.

To run the PyLucene pytest smoke suite against a local PyLucene environment:

```sh
./test_pylucene.sh
```

The script builds and validates the jar before invoking pytest. To invoke pytest
directly against existing artifacts instead:

```sh
CUVS_LUCENE_JAR=/path/to/cuvs-lucene.jar \
CUVS_LUCENE_CUVS_JAVA_JAR=/path/to/cuvs-java.jar \
python3 -m pytest -q -s examples/Python/test_pylucene_smoke.py
```

To run an expanded GPU end-to-end pytest suite through CPU HNSW,
CAGRA-to-HNSW, and CAGRA search paths:

```sh
./test_pylucene.sh --gpu-e2e
```

The expanded suite runs the `gpu-basic`, `gpu-segments`, `cpu-hnsw`, and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great description here. How to build and run tests should really be in a separate build and install guide. I think this is okay for now, especially since we are moving cuVS-Lucene to cuVS, but it's something to consider.

`cagra-hnsw` case groups. The basic cases cover `hnsw`, `cagra`, `hnsw-single`,
and `cagra-single`. The segment cases cover 1-segment indexes, 10-segment
indexes, 10 segments force-merged to 1, and 100 segments force-merged to 10 for
both HNSW and CAGRA. The CPU HNSW cases force the accelerated HNSW codec through
its Lucene CPU fallback path in the same run, including 10 segments force-merged
to 1 and 100 segments force-merged to 10. The CAGRA-to-HNSW cases explicitly
cover one-layer and three-layer HNSW graphs built from CAGRA with NN_DESCENT,
`graphDegree=32`, and `intermediateGraphDegree=64`. The base matrix uses 2,000
documents and 32 dimensions; high-segment cases use at least 257 rows per
segment to avoid expected cuVS graph-degree clamps on tiny per-segment datasets.
The suite checks Lucene SPI discovery, jar packaging, index file suffixes
(`.vex`/`.vem` for HNSW and `.vcag`/`.vemc` for CAGRA), indexed vector metadata,
unfiltered KNN, filtered KNN, missing-vector documents, deletions, and force
merge behavior. To run a subset or resize the test:

```sh
./test_pylucene.sh --gpu-e2e --cases=gpu-segments --rows=5000 --dims=64 --topk=20
```

### Running Tests

```sh
Expand Down
9 changes: 9 additions & 0 deletions ci/test_pylucene_smoke.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
exec "${REPO_ROOT}/test_pylucene.sh" "$@"
Loading