Repository navigation
Conversation
added 3 commits
October 13, 2026 15:53
…ken lists in prefix lookups lookup_prefix_boundary read and json-parsed the token list of every manifest row on each request, then dropped the rows under min_useful_matched_tokens. Rows with prefix_len below that floor cannot rank (matched <= prefix_len), so the query now leaves them out; they are only inspected when nothing else ranks, so the miss reason stays the same. Parsed token tuples are memoized per entry (keyed on entry_id and token_hash, bounded by total tokens, dropped on delete and rewrite) and shared with _restore_row.
The prompt was rebuilt with tuple(int(t) for t in ids) 8-12 times per request (session bank lookups, engine session, token_prefix in generation), about 2.5 ms each at 85K tokens. as_token_tuple returns a TokenTuple, a tuple subclass that marks "all plain ints", and returns it unchanged when it is handed one again. The prompt is normalized where restore_or_prefill_prompt_state and the admission probe receive it; lists and other sequences still work everywhere.
dominant-prefix estimate, exact lookup, near-prefix candidates with their diagnostics and the reread receipt each compared the prompt with every RAM entry (~0.45 ms per entry at 80K tokens). shared_prefix_len remembers the answer per (prompt, entry) pair when both are TokenTuple objects, so the later passes reuse it. Plain lists and tuples take the old compare.
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 14, 2026
…incl. single-rounding fix), 134 (youssofal#629), new 135 (fused lse prefill rounds twice)
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 14, 2026
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 21, 2026
…ed time, fix proposal
The prefix lookup walks the candidate rows in the same order on every turn.
With more tokens in those rows than the memo cap (2.5M), evicting the least
recently used row dropped each row just before its next use, so the memo
never hit and every turn parsed every row again. On a store of 79 rows and
5.7M tokens a lookup for a 56K prompt took 343 ms with 0 of 306 rows from the
memo, against 8.6 ms when the rows stay parsed.
Rows are now held as array('I') (4 bytes per id instead of about 40 for a
tuple of ints), the cap is 32M tokens (about 128 MB), and a full memo keeps
its residents and returns a newcomer unmemoized instead of evicting. The
exact-prefix check in _restore_row compares through _starts_with, which
accepts both the array and the tuple fallback for ids that do not fit.
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 21, 2026
jvmenen
pushed a commit
to jvmenen/MTPLX
that referenced
this pull request
Oct 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small host-side changes that remove work repeated on every request. Behaviour is unchanged; only timing differs.
Why: the per-turn cost of the prefix lookups grows with the size of the SSD manifest and the prompt, not with the work the turn needs. With 24 saved conversations of 60-85K tokens, a turn spent roughly 100-200 ms re-reading and comparing every saved token list.
lookup_prefix_boundary): rows shorter thanmin_useful_matched_tokenscannot beat the caller's RAM match, so the query leaves them out (prefix_len >= floor). They are only looked at when nothing else ranks, so the miss reason stays the same. Parsed token tuples are cached per entry (keyed on entry id and token hash, bounded by total tokens, dropped on delete and rewrite), and_restore_rowuses the same cache.mtplx/token_ids.py,TokenTuple) and later layers skip the re-conversion. Lists and other sequences still work everywhere.Micro-benchmark (pure Python, 24 rows of 60-85K tokens, RAM match 80K tokens, median of 5): cold scan 208 ms -> 13 ms with a warm cache (the first turn after start parses each row once); 10 prompt conversions 28 ms -> 2 ms; 4 passes over 10 RAM entries 176 ms -> 42 ms.
Verified: new tests compare the lookup against the old unfiltered scan on a synthetic manifest (various
prefix_len, floors including the equal-to-floor boundary, deletes, rewrites, rewrites by another process), check the cache bound, and check that list andTokenTupleprompts give identical bank results and diagnostics. Existing cold tier and session bank tests pass.