Problem
A disk_seismic(_sq) .dat runs several times the size of its corpus, much of it carrying no
information. The inline forward index dominates; duplicating each document's vector per block is
the design, not the issue. On top of that:
- Every block is padded to a page (
kPageAligned, kDefaultPageSize = 4096 —
io/inline_forward_index_io.h:82,92) yet holds roughly ten documents at standard
lambda/beta, and a query never reads the padding. InlineLayout::kPacked already exists
(inline_forward_index.h:46); the header records the alignment.
[u16 comps[total_nnz]] (:29) is flat, though a document's ids ascend and so compress.
doc_id[], off[] (:113-114) are uint32_t, beyond what a block needs.
Proposal
- Default to
kPacked: no new encoding, no format bump.
- Add byte accounting, so later steps are sized rather than guessed.
- Then evaluate separately, each kept only if latency holds: delta-coded ids, narrower
off[],
4-bit values.
Acceptance: smaller .dat, recall and latency unchanged.
Problem
A
disk_seismic(_sq).datruns several times the size of its corpus, much of it carrying noinformation. The inline forward index dominates; duplicating each document's vector per block is
the design, not the issue. On top of that:
kPageAligned,kDefaultPageSize = 4096—io/inline_forward_index_io.h:82,92) yet holds roughly ten documents at standardlambda/beta, and a query never reads the padding.InlineLayout::kPackedalready exists(
inline_forward_index.h:46); the header records the alignment.[u16 comps[total_nnz]](:29) is flat, though a document's ids ascend and so compress.doc_id[],off[](:113-114) areuint32_t, beyond what a block needs.Proposal
kPacked: no new encoding, no format bump.off[],4-bit values.
Acceptance: smaller
.dat, recall and latency unchanged.