Skip to content

[FEATURE] Shrink the on-disk size of the DiskSeismic indexes #59

Description

@chishui

Problem

A disk_seismic(_sq) .dat runs several times the size of its corpus, much of it carrying no
information. The inline forward index dominates; duplicating each document's vector per block is
the design, not the issue. On top of that:

  • Every block is padded to a page (kPageAligned, kDefaultPageSize = 4096 —
    io/inline_forward_index_io.h:82,92) yet holds roughly ten documents at standard
    lambda/beta, and a query never reads the padding. InlineLayout::kPacked already exists
    (inline_forward_index.h:46); the header records the alignment.
  • [u16 comps[total_nnz]] (:29) is flat, though a document's ids ascend and so compress.
  • doc_id[], off[] (:113-114) are uint32_t, beyond what a block needs.

Proposal

  1. Default to kPacked: no new encoding, no format bump.
  2. Add byte accounting, so later steps are sized rather than guessed.
  3. Then evaluate separately, each kept only if latency holds: delta-coded ids, narrower off[],
    4-bit values.

Acceptance: smaller .dat, recall and latency unchanged.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions