serve: single-slot MTP speculation in the mux path (KV_SLOTS=1) — #492/#358 - #581
Merged
Merged
Conversation
5 tasks
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the single-slot half of #492 and #358:
coli serveatKV_SLOTS=1now keeps MTP speculation instead of disabling it.Why it's safe at one slot
MTP/n-gram speculation is disabled in the mux (
g_draft=0) because the draft/verify batch isn't ragged-safe when slots sit at different positions. But atKV_SLOTS=1there is no ragged batch — decode is a single contiguous sequence, the exact regimespec_decodealready serves incoli chat,coli run, andrun_serve. So the gate becomesif(nctx>1) g_draft=0;— multi-slot unchanged, single-slot keeps the engine's resolved draft setting.How it's wired
spec_decodegains alogit_outparam (the 4 existing callers passNULL, unchanged behavior) so the mux scheduler can own the continuation logits across calls.mux_submit: a single-slot request with drafts active and no per-request grammar takes the new spec path — the prefill logits are stashed on the request, no pending-token cycle.spec_decodecall (same contract asrun_serve), streaming tokens via amux_dataemit callback, thenmux_done. No chunk-splicing — that would re-enterspec_decodemid-turn and duplicate the boundary token;Ctrl-Cstill interrupts throughg_intrinsidespec_decodelike every serve path.[MTP] disabled in multiplexed servebanner now fires only forKV_SLOTS>1; single-slot prints[MTP] single-slot serve: speculation active.Measured (6x RTX 5090, GLM-5.2 int4, frozen usage snapshot)
Correctness gate — the maintainer's bar, "same tokens as
coli chat": on a high-confidence prompt (low near-tie margin), single-slot serve spec (DRAFT=3) is token-exact with greedy (DRAFT=0):Determinism: two runs of the spec path on the same prompt produce byte-identical output.
On near-tie divergence: on a low-confidence prompt, spec and greedy do diverge — but so do
DRAFT=3andDRAFT=0in plain text mode (" What about black and white"vs" What about for printing"on the same prompt). That's the known #163/#510 kernel-family effect: greedy's S=1 forward and speculation's S=1+g verify forward batch the MoE differently, flipping near-ties. It is a property of speculation on this container, present identically incoli chat, not introduced by this PR — and it's sharpest on the per-row int4 container (another reason for the gs64 default in #579). Under SPEC_PIN (default on) and a group-scaled container it's minimized.Scope
Non-spec paths untouched (verified:
KV_SLOTS=1 DRAFT=0still runs the shared-batch greedy path normally). Speculation on a single-slot server is pure win where verify forwards are bandwidth-bound — the exact ask in #492/#358.