Neptune

Normally, you'd bring your lakehouse data to qdrant. With neptune, you bring qdrant to your lakehouse.

Neptune is a lakehouse-native vector search system that builds HNSW indexes over quantized vectors using Qdrant's battle-tested segment internals. Parquet files are serialized into an Apache Puffin sidecar stored right next to the data. No dedicated cluster, and no ETL.

s3://my-bucket/my-data/
├── data_001.parquet          full-precision f32 vectors + payload columns
├── data_002.parquet
└── neptune_index.puffin      HNSW graph + TurboQuant-4bit vectors + address map

Traversal runs on compressed vectors mmapped from the Puffin blobs; the top candidates can be exactly rescored by range-reading the full-precision vectors from the source Parquet row groups.

See DESIGN_DOC.md for the full architecture.

Usage

# Build an index next to the parquet files (local dir, s3://, gs://, az://)
neptune index path/to/parquet/
neptune index s3://my-bucket/my-data/ --quantization turboquant-4bit
neptune index s3://my-bucket/my-data/ --vector-column embedding --distance cosine

# Inline lightweight payload fields into the index so --with-payload never
# touches Parquet at query time ('*' = all non-vector columns; keep heavy
# text/blob columns out).
neptune index s3://my-bucket/my-data/ --payload-fields "id,title,url"

# Query
neptune query s3://my-bucket/my-data/ --query-vector "0.1,0.2,0.3" --top-k 10
neptune query s3://my-bucket/my-data/ --query-vector emb.npy --top-k 10 --rescore --with-payload
# Force full payloads (all columns) from Parquet, bypassing the inlined store:
neptune query s3://my-bucket/my-data/ --query-vector emb.npy --with-payload --full-payload

# Inspect (footer-only read; does not download the index)
neptune info s3://my-bucket/my-data/

# Pre-download the index blobs into the local cache so the first query
# starts warm (idempotent)
neptune warm s3://my-bucket/my-data/

Quantization options: turboquant-4bit (default), turboquant-2bit, turboquant-1.5bit, turboquant-1bit, int8, binary, binary-1.5bit, binary-2bit, pq-x4 … pq-x64. Distances: cosine (default), dot, euclid, manhattan.

S3/GCS/Azure credentials come from the standard environment variables (AWS_ACCESS_KEY_ID, AWS_REGION, GOOGLE_SERVICE_ACCOUNT, …).

Logging & profiling

Every pipeline stage logs its wall-clock time at debug level — file scanning, quantization, HNSW linking, per-blob writes/downloads, cache extraction, mmap load, traversal, and per-row-group rescore fetches:

LOG_LEVEL=debug neptune index path/to/parquet/   # or: neptune -v index …
neptune -v query … 2>&1 | grep 'stage:'          # just the timings

# Full tracing syntax (overrides LOG_LEVEL/-v), e.g. include object_store:
RUST_LOG=neptune_core=debug,object_store=debug neptune index s3://…

Download tuning

Index blobs are pulled from object storage as concurrent ranged GETs. A single connection to S3 tops out around 30 MB/s no matter how fast the local NIC is, so throughput scales with the number of requests in flight rather than with read speed. Defaults: 16 concurrent requests; chunk size auto-scales (length/16, clamped to 2–16 MiB) so small blobs still fan out fully. This reaches roughly 180 MiB/s from in-region S3.

Blobs are fetched concurrently (and each is itself range-split), so extraction time is bounded by the largest blob's transfer. Alternatively, NEPTUNE_COALESCE_MIB=N downloads containers up to N MiB whole in one ranged pass and splits them locally — fewer requests, but measured slower on in-region EC2 (the parallel per-blob round trips hide under the big blob's transfer while coalescing pays a full local copy); useful mainly on high-latency links where round trips dominate.

NEPTUNE_DOWNLOAD_CONCURRENCY=32 neptune query s3://…   # in-flight requests
NEPTUNE_DOWNLOAD_CHUNK_MIB=8    neptune query s3://…   # fixed chunk override
NEPTUNE_COALESCE_MIB=256        neptune query s3://…   # whole-file mode ≤256 MiB

Peak memory during extraction is bounded by concurrency × chunk size.

Workspace

• crates/neptune-core — all logic: Puffin container, Parquet IO, address map, build & query pipelines. Qdrant crates are imported as git dependencies pinned to the v1.19.0 tag.
• crates/neptune-cli — thin neptune binary (clap) over the core crate.

Query-time blobs are extracted once into a digest-keyed cache ($NEPTUNE_CACHE_DIR, default ~/.cache/neptune) and mmapped from there.

Benchmarking

bench/README.md covers launching the EC2 bench box with SkyPilot, running the CLI stage-timing benchmarks against in-region S3, and driving the cold/warm/hot ladder against neptune-serve.

Building

Requires Rust ≥ 1.97 (edition 2024, matching qdrant's MSRV), a C compiler, and CMake (for aws-lc-sys). The first build compiles the full qdrant dependency tree and takes a while.

cargo build --release
cargo test -p neptune-core        # includes an end-to-end recall test