Pack the inline forward index's blocks instead of padding each to a page - #60
Merged
chishui merged 1 commit intoSep 24, 2026
Merged
Conversation
A block of the inline forward index was written page-aligned: each one starts on a 4096 boundary, so the tail of its last page is padding. At the standard lambda/beta a block holds about ten documents, which makes that padding a large share of the file -- and none of it is ever read, since a query touches a block's own payload. On MS MARCO base_full (lambda=6000 beta=400 alpha=0.4, 8-bit disk_seismic_sq) it was 24.3 GiB of a 74.2 GiB index. Make InlineLayout::kPacked the default, which the format already supports: the header carries the effective alignment and a reader honours whatever the file declares. So this is not a format change -- an index written before this still loads, it just has to be rebuilt to shrink -- and no kFormatVersion bump is needed. base_full comes down to 50.0 GiB (-32.7%). Nothing regresses. Padding cost storage and page cache rather than page faults, so removing it leaves query work alone: on base_full, first (cold) query pass -9.6%, warm p50 0.221 -> 0.217 ms, peak RSS 6.57 -> 6.04 GiB. Blocks now share pages, which is where the resident-set gain comes from. Also adds benchmarks/index_size_stats, which re-parses a serialized index and prints where its bytes are plus what a narrower encoding of each array would save -- how the padding was found, and what sizes the remaining levers (delta-coded component ids, narrower off[], 4-bit values) before any of them is built. And a labels_out.txt argument to sq_residency_bench, so a layout change can be shown to be exact: dump one build's top-k, score the other against it. With a fixed seed the two layouts' label files are byte-identical. Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
chishui
requested review from
model-collapse,
yuye-aws and
zirui-song-18
as code owners
September 23, 2026 07:06
zirui-song-18
approved these changes
Sep 23, 2026
Comment on lines
+183
to
+186
| if (layout.end <= kU16Max) { | ||
| stats->blocks_off_fits_u16 += 1; | ||
| stats->c_off_u16 += (n_docs + 1) * sizeof(uint16_t); | ||
| } else { |
Collaborator
There was a problem hiding this comment.
layout.end <= kU16Max tests the block's total byte length, but an off[] entry holds an element offset in [0, total_nnz] (off[n_docs] == total_nnz) — so a block's off[] fits a u16 table iff total_nnz <= 0xFFFF, not layout.end <= 0xFFFF. layout.end is always larger (it also counts doc_id[]/comps[]/vals[]), so this undercounts both blocks_off_fits_u16 and the u16 saving. For a ~4.5 KB block the two agree, but consider gating on total_nnz.
| const std::vector<InvertedListClusters>* lists_ = nullptr; | ||
| const SparseVectors* vectors_ = nullptr; | ||
| uint64_t write_page_size_ = kDefaultPageSize; | ||
| InlineLayout write_layout_ = InlineLayout::kPageAligned; |
Collaborator
There was a problem hiding this comment.
Nit: Consider setting this to kPacked?
Collaborator
|
I also tested on 138M documents spreading on 3 data nodes: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of #59.
Inline-forward blocks were page-aligned, so each block's tail page is padding no query reads:
24.3 of 74.2 GiB on
base_fulldisk_seismic_sq, where blocks hold ~ten documents. Defaults toInlineLayout::kPacked.No format change — the header carries the alignment and readers honour either, so older files
still load and the A/B ran one binary over two files.
base_full,lambda=6000 beta=400 alpha=0.4, mmap,cut=3 k_prime=50 k=10, 6,980 queries,1 thread, cold caches, aligned → packed: file 74.205 → 49.967 GiB (−32.7%), cold pass
982,994 → 888,829 ms (−9.6%), warm p50 0.221 → 0.217 ms,
VmHWM6.567 → 6.042 GiB (−8.0%),recall@10 0.9252 → 0.9261 (seed noise). Padding cost storage and cache, not faults.
Adds
index_size_stats; at one seed both layouts' labels match exactly.--signoff