Repository navigation

Bound the seismic build's memory by batching the term space - #45

Merged

zirui-song-18 merged 15 commits into

opensearch-project:mainfrom

chishui:seismic-batched-build

Sep 3, 2026

Commits on Sep 3, 2026
- io: expose write_header so a streamed payload can lay out its own header
  chishui
  committed
  6affd30
- Bound the seismic build's memory by batching the term space
  chishui
  committed
  0bfb160
- benchmarks: add a peak-RSS driver for the batched build
  chishui
  committed
  955ebcf
- docs: document building an index larger than memory
  chishui
  committed
  fdc207b
- benchmarks: report peak RssAnon, not just peak RSS
  chishui
  committed
  aa44b64
- Cut term windows to equal clustering load, not equal width
  chishui
  committed
  4e61d95
- Weight term windows by both build phases, not just one
  chishui
  committed
  70a0a39
- docs: correct the batched-build memory table
  chishui
  committed
  3e1f5dc
- Leave a batched build holding the index it wrote
  chishui
  committed
  72d16b4
- Bound the disk indexes' build memory too, by spilling their lists
  chishui
  committed
  5af526f
- docs: keep the reasoning, drop the corpus it was measured on
  chishui
  committed
  cb6a5ce
- Spill windows to scratch instead of writing an index from build()
  chishui
  committed
  e69abbe
- Let the spill own its own file, and take the build out of MmapIndex
  chishui
  committed
  51a0d18
- tests: assert when the spill is gone, not how it went
  chishui
  committed
  922c10d
- Own the spill from creation, so a failed build removes it
  chishui
  committed
  3586141