Repository navigation
Conversation
MTPLX_SSD_INCREMENTAL_ENCODE only reached put_entry. Entries at or above the writer backlog budget, live-ref sessions and the write-out before an idle release all stream through spill_entry, which built its codec without the session's block memo: every KV block, recurrent state and checkpoint was evaluated, copied to the host and hashed again, and incremental_reused_blobs stayed 0 on a server that writes only large sessions. spill_entry now passes the same per-session memo and replaces it with the write's content keys once its row is installed, so queued and streamed writes of one session feed each other. A memo hit is referenced only after known_blob claims the digest and finds the blob on disk; a blob gone since then is captured and written in the same spill (incremental_missing_blobs) instead of failing the write. Checkpoints a turn keeps from the previous one are referenced like KV blocks. The store stays byte-identical to a full encode (tests for both writers).
The fingerprint pass reduced each blocked tensor's whole full-block prefix at once: an unbounded unit on the model-owner thread (issue youssofal#505) whose uint32 temporaries are several times the tensor's size. With the memo on the streaming spill, which also runs when the memory guard writes an entry out before releasing it, that peak lands where memory is short. Blocks are now fingerprinted per chunk of at most unit_bytes, the first time a block of the chunk needs one. A block's fingerprint depends only on its own bytes, so the content keys are unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
MTPLX_SSD_INCREMENTAL_ENCODE=1should stop each turn from copying and hashing the whole session again when only the new tail is new on disk. It was only wired intoput_entry, the queued writer. Long sessions are written byspill_entryinstead: entries above the writer backlog budget, live-reference sessions, and the write-out before an idle session is released. On a server with long agent sessions, 81 of 83 writes in one day took that path, andincremental_reused_blobsstayed at 0.How it works, in plain terms
The incremental encode keeps a small table per session: for each block, a content fingerprint computed on the GPU, and the blob on disk that holds it. The streaming writer now uses the same table. A matching block is only referenced after the writer has claimed the blob and seen it on disk. A blob that has disappeared is simply written again in the same spill. Recurrent checkpoints carried over unchanged from the previous turn are referenced the same way. Fingerprints are now taken in runs of at most 32 MiB, so the pass no longer builds large temporary arrays when the memory guard writes an entry out.
What changes
spill_entrypasses the per-session memo to its codec, and updates it after the manifest row is installed.TreeCodec.known_blob: an optional hook that confirms a memo hit before it is referenced.incremental_*counters now include spills.Results
Qwen 3.8 27B, a 49K-token conversation with three follow-up turns, written via
spill_entry(3 runs without, 4 with):Tests
tests/test_cold_tier_incremental_encode.pycovers:Cold-tier, SSD and session-bank tests: 326 passed, with the switch off and on.
Notes
Incremental encode stays opt-in: it is as safe as its content key. #596 closes a known fingerprint gap and is recommended first.