Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,18 @@
`paimon-vindex-core` contains the Rust implementations and seek-based readers
for IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN.

The Rust reader supports distance range search for IVF-FLAT and IVF-RQ with
squared L2, using `DistanceBand`, `VectorRangeSearchParams`, and CSR
`RangeSearchResult` buffers. Both families support single and batch queries,
with or without a serialized Roaring allow-list, and a fixed positive `nprobe`.
IVF-FLAT tests exact distances; IVF-RQ tests its one-bit or full multi-bit
estimated distances. Results are uncapped and unordered. Probing every list
removes the IVF coverage gap, but not IVF-RQ's quantization error. The range
path does not change top-K search or the v1 storage format.

See the [range search guide](../docs/range-search.html) for membership,
validation, filtering, and statistics. C/JNI range bindings are not included.

The DiskANN and Vamana code is an independent Apache-licensed implementation
based on the published algorithms and this project's existing storage
abstractions. It does not incorporate source code from Microsoft's
Expand Down
2 changes: 1 addition & 1 deletion core/src/collect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ pub(crate) trait Collector {
fn cutoff(&self) -> f32;

/// Delivers one row, with the value the family's scan computed for it. For
/// IVF-Flat that value is an exact distance.
/// IVF-Flat that value is an exact distance; for IVF-RQ it is an estimate.
///
/// Fallible because a collector may own a resource the scan cannot see: the
/// oversized-list path streams chunks through a callback, and without a
Expand Down
24 changes: 15 additions & 9 deletions core/src/index.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1846,7 +1846,7 @@ impl<R: SeekRead> VectorIndexReader<R> {
/// Without this a family that cannot serve range search would report
/// `Unsupported` for an invalid width or a mismatched metric, and a caller
/// treating `Unsupported` as "fall back to a scan" would silently paper over
/// its own bug. The IVF-Flat reader repeats the metric check because it is a
/// its own bug. Each supporting reader repeats the metric check because it is a
/// public entry point in its own right; the comparison is two enum reads, so
/// the duplication costs nothing measurable.
fn validate_range_request(&self, params: &VectorRangeSearchParams) -> io::Result<()> {
Expand All @@ -1866,7 +1866,9 @@ impl<R: SeekRead> VectorIndexReader<R> {
}

/// Distance range search. For the contract see
/// [`IVFFlatIndexReader::range_search`].
/// [`IVFFlatIndexReader::range_search`] and
/// [`IVFRQIndexReader::range_search`]. IVF-RQ membership uses estimated
/// distances rather than distances to the original vectors.
///
/// The empty-band short-circuit lives **inside each family's reader**, so a
/// family that cannot do range search at all still fails loud for every
Expand All @@ -1880,15 +1882,16 @@ impl<R: SeekRead> VectorIndexReader<R> {
self.validate_range_request(&params)?;
match self {
Self::IvfFlat(reader) => reader.range_search(query, params),
Self::IvfRq(_) => Err(range_unsupported("ivf_rq")),
Self::IvfRq(reader) => reader.range_search(query, params),
Self::IvfSq(_) => Err(range_unsupported("ivf_sq")),
Self::IvfPq(_) => Err(range_unsupported("ivf_pq")),
Self::DiskAnn(_) => Err(range_unsupported("diskann")),
}
}

/// Range search restricted to a serialized Roaring allow-list. For the
/// contract see [`IVFFlatIndexReader::range_search_with_roaring_filter`].
/// contract see [`IVFFlatIndexReader::range_search_with_roaring_filter`]
/// and [`IVFRQIndexReader::range_search`].
pub fn range_search_with_roaring_filter(
&mut self,
query: &[f32],
Expand All @@ -1903,15 +1906,16 @@ impl<R: SeekRead> VectorIndexReader<R> {
let filter = decode_roaring_filter(roaring_filter_bytes)?;
match self {
Self::IvfFlat(reader) => reader.range_search_with_filter(query, params, Some(&filter)),
Self::IvfRq(_) => Err(range_unsupported("ivf_rq")),
Self::IvfRq(reader) => reader.range_search_with_filter(query, params, Some(&filter)),
Self::IvfSq(_) => Err(range_unsupported("ivf_sq")),
Self::IvfPq(_) => Err(range_unsupported("ivf_pq")),
Self::DiskAnn(_) => Err(range_unsupported("diskann")),
}
}

/// Batched distance range search. For the contract see
/// [`IVFFlatIndexReader::range_search`].
/// [`IVFFlatIndexReader::range_search`] and
/// [`IVFRQIndexReader::range_search`].
pub fn range_search_batch(
&mut self,
queries: &[f32],
Expand All @@ -1922,7 +1926,7 @@ impl<R: SeekRead> VectorIndexReader<R> {
self.validate_range_request(&params)?;
match self {
Self::IvfFlat(reader) => reader.range_search_batch(queries, query_count, params),
Self::IvfRq(_) => Err(range_unsupported("ivf_rq")),
Self::IvfRq(reader) => reader.range_search_batch(queries, query_count, params),
Self::IvfSq(_) => Err(range_unsupported("ivf_sq")),
Self::IvfPq(_) => Err(range_unsupported("ivf_pq")),
Self::DiskAnn(_) => Err(range_unsupported("diskann")),
Expand All @@ -1944,7 +1948,9 @@ impl<R: SeekRead> VectorIndexReader<R> {
Self::IvfFlat(reader) => {
reader.range_search_batch_with_filter(queries, query_count, params, Some(&filter))
}
Self::IvfRq(_) => Err(range_unsupported("ivf_rq")),
Self::IvfRq(reader) => {
reader.range_search_batch_with_filter(queries, query_count, params, Some(&filter))
}
Self::IvfSq(_) => Err(range_unsupported("ivf_sq")),
Self::IvfPq(_) => Err(range_unsupported("ivf_pq")),
Self::DiskAnn(_) => Err(range_unsupported("diskann")),
Expand Down Expand Up @@ -2669,7 +2675,7 @@ fn validate_query(query: &[f32], dimension: usize) -> io::Result<()> {
validate_finite_values(query, dimension, "query")
}

/// Only IVF-Flat implements range search so far. The other families return
/// IVF-Flat and IVF-RQ implement range search. The other families return
/// `Unsupported`, meaning "we cannot serve this request, please fall back",
/// rather than "the call has a bug". For DiskANN the reason is a lasting one:
/// graph traversal is inherently k-oriented and has no natural radius
Expand Down
Loading
Loading