DiskSeismic: Add disk_seismic_index by zirui-song-18 · Pull Request #32 · opensearch-project/neural-sparse-cpp

zirui-song-18 · 2026-08-21T05:51:46Z

Description

Added the main classes for DiskSeismicIndex

Issues Resolved

RFC: [RFC] Disk-resident forward index for SEISMIC, with an inline on-disk layout #21

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Zirui Song <zrsong@amazon.com>

chishui · 2026-08-26T04:42:18Z

Could you post the benchmark of this index comparing with SeismicIndex?
Do you plan to make it support quantization? I assume the quantized index could be using smaller disk side and faster query time.
Do we support options to either store inline or as forward index

chishui · 2026-08-26T04:45:16Z

+                 const IDSelector* id_selector, size_t element_size,
+                 detail::TopKHolder<idx_t>& heap,
+                 absl::flat_hash_set<idx_t>& visited) {
+    if (fwd != nullptr) {


what about throw exceptions if either fwd or vectors is null as I think you can do nothing without them

It will throw, but in search() before entering OpenMP. If we throw in score_block(), it will go out of OMP parallel.

chishui · 2026-08-26T04:52:42Z

+        const auto data = vectors->get_all_data();
+        for (const idx_t doc_id : clusters[pl].get_docs(cid)) {
+            const idx_t start = data.indptr_data[doc_id];
+            const size_t len = data.indptr_data[doc_id + 1] - start;


let's assign variables outside of for loop

const idx_t* const indptr = data.indptr_data; const idx_t* const indices = data.indices_data; const float* const values = data.values_data;

Good suggestions. Ack

chishui · 2026-08-26T05:04:28Z

+    throw_if_any_null(indptr, indices, values);
+    const size_t indptr_size = n + 1;
+    const size_t nnz = indptr[n];
+    if (vectors_ == nullptr) {


reset num_vectors_ to 0 here?

chishui · 2026-08-26T05:09:45Z

+                              SearchParameters* search_parameters)
+    -> pair_of_score_id_vectors_t {
+    if (num_vectors_ == 0 || n == 0) {
+        return {std::vector<std::vector<float>>(n),


we use INVALID_IDX as not found doc, please check other indices

chishui · 2026-08-26T05:11:31Z

+                   dynamic_cast<const SeismicSearchParameters*>(
+                       search_parameters)) {
+        cut = seismic_parameters->cut;
+    }


what about else situation?

cut/k_prime has been initialized with default. else should be fine with nothing.

chishui · 2026-08-26T05:11:56Z

+                       search_parameters)) {
+        cut = seismic_parameters->cut;
+    }
+    if (k_prime <= 0) {


what if k_prime is smaller than k?

k_prime here refers to the number of blocks to be fully examined. It has nothing to do with k.

chishui · 2026-08-26T05:13:02Z

+    }
+
+    std::vector<std::vector<float>> result_distances(n);
+    std::vector<std::vector<idx_t>> result_labels(n);


here set default labels to 0

Use INVALID_IDX now.

chishui · 2026-08-26T05:14:12Z

+    // score. score_summaries_transposed dots the full query with each cluster
+    // summary, so scores are comparable across posting lists for the global
+    // ranking.
+    std::vector<BlockCandidate> candidates;


are sizes of both variables known already? can you call .reserve()

Now add a pre-scan:

size_t total_clusters = 0; size_t max_clusters = 0; for (const term_t term : cuts) { if (term < clustered_inverted_lists.size()) { const size_t nc = clustered_inverted_lists[term].cluster_size(); total_clusters += nc; max_clusters = std::max(max_clusters, nc); } } candidates.reserve(total_clusters); score_scratch.reserve(max_clusters);

no, no, I mean if you already know the size, if not, we don't do this just to calculate the size and reserve which could be even heavier.

Got it. That makes sense.

chishui · 2026-08-26T05:20:57Z

+    }
+
+    for (size_t i = 0; i < q_len; ++i) {
+        dense[q_indices[i]] = 0.0F;


this for loop may have multiple cache misses, is the performance of this for loop fine?

We only reset the number of q_len position in query, which is quite small. This is much cheaper than memset the whole dim table.

zirui-song-18 · 2026-08-26T07:00:17Z

Thanks for @chishui 's suggestions, a benchmark result can be seen here:
Host: g5.12xlarge (AMD EPYC 7R32 Zen2, 48 vCPU, 186.7 GiB, avx2+fma, noavx512f)
Data: 8.8M docs with 6,980 queries (MS MARCO); k 10

1. Iso-recall readout along the two Pareto frontiers

anchor	recall	paired cell	recall	Δrecall	p50 ratio	p99 ratio
Seis `c3/hf=1.15`	0.914011	Disk `c3/k'=25`	0.915014	+0.0010	1.44×	3.14×
Seis `c3/hf=1.2`	0.925817	Disk `c3/k'=32`	0.926963	+0.0011	1.44×	3.24×
Seis `c5/hf=1.1`	0.929943	Disk `c5/k'=25`	0.930960	+0.0010	1.39×	2.92×
Seis `c3/hf=1.25`	0.934212	Disk `c3/k'=40`	0.934943	+0.0007	1.48×	3.30×
Seis `c5/hf=1.2`	0.952521	Disk `c5/k'=40`	0.952765	+0.0002	1.44×	3.15×
Seis `c8/hf=1.2`	0.963639	Disk `c8/k'=50`	0.965143	+0.0015	1.33×	2.88×
Seis `c5/hf=1.3`	0.964269	Disk `c8/k'=50`	0.965143	+0.0009	1.36×	3.00×
Seis `c8/hf=1.5`	0.981232	Disk `c8/k'=200`	0.982106	+0.0009	1.13×	2.53×
Disk `c3/k'=15`	0.875917	Seis `c3/hf=1.05`	0.876218	+0.0003	1.43×	3.02×
Disk `c3/k'=30`	0.924298	Seis `c3/hf=1.2`	0.925817	+0.0015	1.50×	3.32×
Disk `c5/k'=30`	0.940745	Seis `c8/hf=1.1`	0.942507	+0.0018	1.58×	3.37×
Disk `c8/k'=400`	0.983625	Seis `c8/hf=2`	0.984112	+0.0005	1.57×	2.93×

Ratio > 1 = DiskSeismic faster. The p50 margin is 1.2–1.5× everywhere and narrows above recall 0.97;
the p99 margin is ~2.8–3.1× and does not narrow.

2. Footprint

variant	file on disk	RssAnon	RssFile (pages this process touched)	VmHWM
SeismicIndex in-memory `c3/hf=1.0`	13.35 GiB	13.371 GiB	0.004 GiB	13.376 GiB
SeismicIndex mmap `c3/hf=1.0`	13.35 GiB	0.011 GiB	13.359 GiB	13.369 GiB
DiskSeismic mmap `c3/k'=50`	114.65 GiB	0.332 GiB	35.828 GiB	36.160 GiB

Exact sizes: base_full.seismic.dat 14,339,730,216 B; base_full.disk_seismic.dat
123,106,695,136 B — 8.59× larger on disk.

3. Two secondary results

	recall@10	p50	p99	RssAnon
SeismicIndex in-memory (documented default)	0.843367	0.1905	0.6325	13.371 GiB
SeismicIndex mmap, identical params	0.843367	0.1547 (1.23×)	0.4760 (1.33×)	0.011 GiB
DiskSeismic `cut=3, k'=11`	0.840358 (−0.0030)	0.1054 (1.81×)	0.1640 (3.86×)	0.332 GiB
DiskSeismic `cut=3, k'=12`	0.851619 (+0.0083)	0.1105 (1.72×)	0.1680 (3.76×)	0.332 GiB

zirui-song-18 · 2026-08-26T07:05:31Z

Could you post the benchmark of this index comparing with SeismicIndex?

Do you plan to make it support quantization? I assume the quantized index could be using smaller disk side and faster query time.

Do we support options to either store inline or as forward index

The benchmark results can be seen in my above comment;
Not yet — DiskSeismicIndex currently stores float32 values. But quantization is a natural fit and most of the plumbing already exists. The inline forward-index format (writer/reader/validator) already supports element_size ∈ {1,2,4}, and SeismicScalarQuantizedIndex (seismic_sq) is a complete reference for 8/16-bit scalar quantization: encode-on-add, element_size-dispatched scoring kernels (compute_similarity → uint8/uint16 dot products), query re-quantization, score decoding, and mmap of codes. The remaining work is contained within DiskSeismicIndex: drop the hardcoded float element size, quantize in add(), make score_block/score_doc element_size-aware (reuse the existing uint8 kernels instead of the float-only dot_product_float_dense), persist a quantizer header, and decode scores.
Both layouts are available, but as two index types selected via the factory descriptor, not as a flag on a single index:

seismic (SeismicIndex) — the classic forward index: each document's vector is stored once in a global CSR (SparseVectors) and gathered by indptr[doc_id]. Loadable either in-memory or mmap (IndexIoFlag::kUseMmap).
disk_seismic (DiskSeismicIndex) — the inline layout: each document's vector is duplicated into every block it belongs to, laid out block-contiguously (InlineForwardIndex). mmap-only.

So you pick the storage by building "seismic,…" vs "disk_seismic,…" — there's no store=inline|forward parameter on one index.

chishui · 2026-08-26T07:40:36Z

+void DiskSeismicIndex::write_index(IOWriter* io_writer) {
+    const uint64_t nv = num_vectors_;
+    io_writer->write(const_cast<uint64_t*>(&nv), sizeof(uint64_t), 1);
+    SeismicInvertedListsWriter inv_list_writer(clustered_inverted_lists);


check if clustered list are written twice

Good catch, partially. The summaries are written once (only in the invlists section) and the per-doc vectors are written once (only in the inline forward section — DiskSeismic drops plain seismic's full-CSR forward write). What is written twice is the per-cluster doc-id membership: docs_/offsets_ in the invlists section and again as per-block doc_id[] in the inline forward. And on the mmap search path the invlists docs_/offsets_ are never read (membership comes from the inline doc_id[]; only the summaries + cluster counts are used) — so that copy is dead weight on disk. It's ~M×4 bytes (M = total doc-cluster memberships, ~0.4% of the file here), so it's a cleanliness/correctness fix rather than a size lever (quantization is the size lever). I can add a summaries-only serialize/deserialize path for the DiskSeismic invlists so docs_/offsets_ aren't persisted — docs_ is still needed in RAM at build time to lay out the blocks, just not on disk.

But the cost of that change is not small. Maybe we can leave it a separate commit later.

chishui · 2026-08-26T08:42:48Z

+    const detail::InlineForwardIndex* fwd =
+        fwd_.num_blocks() > 0 ? &fwd_ : nullptr;
+    const SparseVectors* vectors = fwd == nullptr ? vectors_.get() : nullptr;
+    const size_t element_size = fwd != nullptr


so here, can you throw when fwd and vectors are null?

oh, never mind

but can you quit early though?

Done — switched from throwing to quitting early. The no-source case (fwd_.num_blocks()==0 && vectors_==nullptr) is now folded into the early-return at the top of search(), alongside num_vectors_==0 || n==0, returning the k-length padded empty result. A plain return is safe even though the scoring runs under OpenMP (only an escaping throw would be a problem), and it's checked once before the parallel region rather than per block.

Signed-off-by: Zirui Song <zrsong@amazon.com>

chishui

Please update the duplicate serialization in next PR and also include a python integration tests.

zirui-song-18 requested review from chishui, model-collapse and yuye-aws as code owners August 21, 2026 05:51

DiskSeismic: Add disk_seismic_index

cf5ca78

Signed-off-by: Zirui Song <zrsong@amazon.com>

zirui-song-18 force-pushed the disk-seismic-index branch from 74fbcb6 to cf5ca78 Compare August 24, 2026 02:42

chishui reviewed Aug 26, 2026

View reviewed changes

zirui-song-18 force-pushed the disk-seismic-index branch from b84974e to b3183f2 Compare August 26, 2026 08:32

zirui-song-18 requested a review from chishui August 26, 2026 08:34

chishui reviewed Aug 26, 2026

View reviewed changes

Address Liyun's comments

e76c7cd

Signed-off-by: Zirui Song <zrsong@amazon.com>

zirui-song-18 force-pushed the disk-seismic-index branch from b3183f2 to e76c7cd Compare August 26, 2026 08:56

zirui-song-18 requested a review from chishui August 26, 2026 08:56

chishui approved these changes Aug 26, 2026

View reviewed changes

chishui merged commit 439b992 into opensearch-project:main Aug 26, 2026
9 checks passed

zirui-song-18 mentioned this pull request Aug 26, 2026

DiskSeismic: add python tests & de-dup serialization #33

Merged

chishui mentioned this pull request Aug 27, 2026

DiskSeismicIndex does not override add_with_ids and cannot be used with IDMapIndex #34

Closed

Conversation

zirui-song-18 commented Aug 21, 2026

Description

Issues Resolved

Uh oh!

chishui commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zirui-song-18 Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zirui-song-18 commented Aug 26, 2026

1. Iso-recall readout along the two Pareto frontiers

2. Footprint

3. Two secondary results

Uh oh!

zirui-song-18 commented Aug 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chishui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

chishui commented Aug 26, 2026 •

edited

Loading

zirui-song-18 Aug 26, 2026 •

edited

Loading