Skip to content

serve: single-slot MTP speculation in the mux path (KV_SLOTS=1) — #492/#358 - #581

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:feat/serve-single-slot-mtp
Jul 24, 2026
Merged

serve: single-slot MTP speculation in the mux path (KV_SLOTS=1) — #492/#358#581
JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:feat/serve-single-slot-mtp

Conversation

@ZacharyZcR

Copy link
Copy Markdown
Contributor

Closes the single-slot half of #492 and #358: coli serve at KV_SLOTS=1 now keeps MTP speculation instead of disabling it.

Why it's safe at one slot

MTP/n-gram speculation is disabled in the mux (g_draft=0) because the draft/verify batch isn't ragged-safe when slots sit at different positions. But at KV_SLOTS=1 there is no ragged batch — decode is a single contiguous sequence, the exact regime spec_decode already serves in coli chat, coli run, and run_serve. So the gate becomes if(nctx>1) g_draft=0; — multi-slot unchanged, single-slot keeps the engine's resolved draft setting.

How it's wired

  • spec_decode gains a logit_out param (the 4 existing callers pass NULL, unchanged behavior) so the mux scheduler can own the continuation logits across calls.
  • mux_submit: a single-slot request with drafts active and no per-request grammar takes the new spec path — the prefill logits are stashed on the request, no pending-token cycle.
  • Scheduler: the spec slot runs the whole turn through one spec_decode call (same contract as run_serve), streaming tokens via a mux_data emit callback, then mux_done. No chunk-splicing — that would re-enter spec_decode mid-turn and duplicate the boundary token; Ctrl-C still interrupts through g_intr inside spec_decode like every serve path.
  • Grammar requests keep the forced-draft path (its acceptance is ~1 where it fires; the two draft sources shouldn't fight over one forward). Multi-slot and the non-spec pending cycle are byte-for-byte unchanged.
  • The load-time [MTP] disabled in multiplexed serve banner now fires only for KV_SLOTS>1; single-slot prints [MTP] single-slot serve: speculation active.

Measured (6x RTX 5090, GLM-5.2 int4, frozen usage snapshot)

Correctness gate — the maintainer's bar, "same tokens as coli chat": on a high-confidence prompt (low near-tie margin), single-slot serve spec (DRAFT=3) is token-exact with greedy (DRAFT=0):

prompt "Q: What is 2+2? Answer with one number only. A:"
  DRAFT=3 (spec):   " 4\nWhat is "
  DRAFT=0 (greedy): " 4\nWhat is "   → identical

Determinism: two runs of the spec path on the same prompt produce byte-identical output.

On near-tie divergence: on a low-confidence prompt, spec and greedy do diverge — but so do DRAFT=3 and DRAFT=0 in plain text mode (" What about black and white" vs " What about for printing" on the same prompt). That's the known #163/#510 kernel-family effect: greedy's S=1 forward and speculation's S=1+g verify forward batch the MoE differently, flipping near-ties. It is a property of speculation on this container, present identically in coli chat, not introduced by this PR — and it's sharpest on the per-row int4 container (another reason for the gs64 default in #579). Under SPEC_PIN (default on) and a group-scaled container it's minimized.

Scope

Non-spec paths untouched (verified: KV_SLOTS=1 DRAFT=0 still runs the shared-batch greedy path normally). Speculation on a single-slot server is pure win where verify forwards are bandwidth-bound — the exact ask in #492/#358.

@JustVugg
JustVugg merged commit de62a18 into JustVugg:dev Jul 24, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants