server: preempt a slot instead of ending every conversation when the KV pool fills by danielhanchen · Pull Request #184 · unslothai/llama.cpp

danielhanchen · 2026-09-05T01:09:43Z

Summary

With --kv-unified --parallel N, every slot is told it has the whole context while all of them share one pool of cells. Four chats that each fit on their own are admitted together, grow into the pool, and when the next decode does not fit update_slots() enters the KV-full retry ladder and ends with send_error on every processing slot. Four users lose four conversations at once, none of them anywhere near their own limit.

This change makes the server park a slot instead. When the cells the next decode will need do not fit, one running slot is chosen, its sequence is copied to host RAM with llama_state_seq_get_data_ext, its cells are released, and its task stays alive in a new SLOT_STATE_PREEMPTED state. When the pool has room again the state is copied back with llama_state_seq_set_data_ext and decoding continues from the same token, with the same sampler state. The client sees a pause in its stream and nothing else. The pause is invisible over the wire, so the OpenAI-compatible API, the built-in web UI and any third-party client all get it without changes.

All of it is in tools/server/server-context.cpp, inside server_slot and update_slots(), tagged [TAG_PREEMPT].

Policy

update_preemption() runs before pre_decode() on every iteration of update_slots(), only when kv_unified is set and there is more than one slot.
Idle slots holding a finished conversation's cached prompt are purged first, through the existing try_clear_idle_slots(). A running conversation is never asked to wait while a finished one is holding cells.
Victim choice: the slot with the most tokens is the leader and is never preempted, so one chat always makes progress and the pool cannot thrash. Among the rest, a slot already preempted PREEMPT_N_STARVED (3) times is passed over while any other candidate exists. The smallest remaining slot is parked, which frees the least work per pause. A slot just given a task is measured by the prefix it keeps, for the leader as for the victims.
Resume order: the most-preempted parked slot first, then the one that has waited longest, but a slot that does not fit yet does not hold up a smaller one that does. The smaller one is the first to be parked again if the pool fills, so the head of the line loses nothing.
A slot still processing its prompt is a victim too: between two chunks of a prompt is as clean a boundary as between two sampled tokens, and a slot that has not started yet holds at most a cached prefix. So two prompts that do not fit together do not fail together, and a large prompt arriving beside a running chat waits for it instead of ending it. A slot just given a task is first trimmed to the prefix it shares with the previous request's prompt, which is what the batch builder does anyway, so it is sized, charged and parked by what the new request will use. Its next-step need counts from that prefix as well; the trim is a partial removal, so a memory that cannot remove part of a sequence has the whole stale sequence cleared instead. A child of an n_cmpl > 1 request still waiting for its parent's prompt holds a previous request's cells until the copy, and is charged them on its own. A parked slot is left alone by every failure sweep, the decode error and the exception paths alike.
Parent/child (n_cmpl > 1) slots are not preempted. They share cells through seq_cp, so a per-sequence save and restore would free less than it costs to put back.
/metrics gains the counters n_preempt_total and n_resume_total and the gauges requests_preempted and preempt_ram_bytes; each /slots entry gains is_preempted and n_preempt. A client can tell a parked request from a slow one, and an operator can see the parked host RAM.
--preempt-ram N (env LLAMA_ARG_PREEMPT_RAM) bounds the host RAM parked sequences may hold, default 8192 MiB like --cache-ram. A slot that would not fit under the budget is not parked; when nothing can be parked the KV-full path runs as before. --preempt-ram 0 disables preemption.
A slot released while parked (cancelled or failed) frees its host mirror and clears its prompt so the next task on that slot cannot prefix-match against an empty cache.

Why the smallest slot

A discrete-step simulation of the pool (scripts/preempt_policy_sim.py in the Unsloth workspace, costs taken from the measured runs) compares victim choices with everything else held equal: keep the leader, restore most-preempted first, fit-first, the same anti-starvation rule. With four chats the choice barely matters (within 1 percent of makespan). With eight chats on 8192 or 16384 cells parking the smallest slot gives the shortest makespan, the shortest mean completion, the least waiting, the fewest preemptions and the fewest cells copied; parking the largest is the worst on every count (5 to 6 percent longer, 40 to 50 percent more cells copied); parking the newest arrival, which is what vLLM does, is within 1 percent of smallest. Restoring from host RAM beats recomputing the sequence by 3 to 7 percent of makespan at these sizes, which is the argument for a server-side save over a client-side resume.

The same question put to the live server, with everything else in place: LLAMA_SERVER_PREEMPT_POLICY (a test knob, off unless set) selects the non-leader the planner parks: smallest, largest, youngest (the most recent task, what vLLM's scheduler preempts) or oldest, with the leader kept and the starvation guard applied under all four. Qwen3.5-4B with two MTP drafts, exact mode on, a 2048-cell pool for four slots, four chats of 900 tokens (about 4400 cells wanted), three rounds per cell, P0 byte compared with its solo run every round:

policy	four at once: wall	slowest chat	aggregate tok/s	parks per round	staggered 3 s apart: wall
smallest (shipped)	11.0 s	11.0 s	326	14	12.9 s
youngest (what vLLM preempts)	11.1 s	11.1 s	324	14	12.9 s
oldest	11.1 s	11.1 s	324	17	12.9 s
largest	11.6 s	11.6 s	310	12	13.0 s

Every round of every policy is byte identical to the solo run, with no last resort and no context error. Smallest is best on makespan and aggregate throughput, by a little; largest is worst, by about 5 percent, since it parks and restores the most tokens. Four chats of 900 tokens took 11.0 s where four solo runs in a row would take 13.4 s at the measured 268 tok/s. Chats arriving 3 s apart never reach pressure and are the control.

Results

Qwen3.5-4B UD-Q4_K_XL with the embedded MTP head, --parallel 4 --kv-unified --spec-type draft-mtp --spec-draft-n-max 2 --flash-attn on, four concurrent streaming chats with roughly 1000-token prompts, no max_tokens, temperature 0, seed 1234.

build	`-c`	completed	errors	gen tokens	wall	preemptions
master (`e9e0d99`)	8192	1 of 4	3	6861	31.9 s	n/a
master + #182	8192	0 of 4	4	0	6.7 s	n/a
this branch, first cut	8192	4 of 4	0	22944	66.9 s	7
this branch	8192	4 of 4	0	17803	50.5 s	8
master + #182	16384	0 of 4	4		20.8 s	n/a
this branch	16384	4 of 4	0		45.8 s	2

On master the retry ladder halves n_batch and three chats die on the speculative sub-batch index assertion (ggml-org#24840, fixed by #182). With #182 all four die together on Context size has been exceeded. With this branch failed to find free space in the KV cache is logged zero times: the ladder is never entered.

The watermark fires at 8185 to 8192 wanted cells out of 8192, so it is not preempting early. Releasing 2000 to 4000 cells takes 105 to 420 ms; restoring takes 28 to 85 ms into an empty pool and 300 to 420 ms into a nearly full one. Aggregate throughput across the four chats was 353 tok/s against 279 tok/s for running the same four chats one after another, because the server still batches whenever the pool allows. The first cut restored strictly in priority order; letting a smaller parked slot through when the head does not fit took the run from 66.9 s to 50.5 s on the same load.

Exactness

Four concurrent greedy streams do not reproduce four solo greedy streams even without this change, because the batch shape differs and the matmul reductions are not shape-invariant. A chat that was never preempted diverged from its solo run after 50 characters. So a concurrent-versus-solo comparison cannot measure the pause.

The env var LLAMA_SERVER_PREEMPT_EVERY=N preempts a slot every N generated tokens regardless of pressure. With one request at a time the batch shape is identical with and without it, so the pause is the only difference:

prompt	forced preemptions	chars	identical to the unforced run
0	17	13449	yes
1	20	19454	yes
2	34	29889	yes
3	22	15545	yes

91 preemptions, byte-identical output, identical token counts, MTP drafting on throughout. Two further runs of the same check on prompts 0 and 2, one on the first cut (68 preemptions) and one on the final build (50 preemptions), also matched byte for byte. The save and restore round trip is exact and the sampler survives it.

A second model: Qwen3.6-35B-A3B UD-Q4_K_XL with its MTP head

Same flags and load, a mixture-of-experts model with the draft head active:

`-c`	completed	errors	gen tokens	wall	preemptions	aggregate tok/s
8192	4 of 4	0	23191	82.6 s	9	281
16384	4 of 4	0	42146	128.3 s	6	328

Forced preemption every 200 tokens on prompts 0 and 2, 69 park and restore cycles: byte-identical to the unforced run (7059 and 6915 tokens). After the run /metrics reported n_preempt_total 9, n_resume_total 9, requests_preempted 0, preempt_ram_bytes 0.

Cost when it does not fire

update_preemption() is a loop over the slots summing prompt.n_tokens() and returns immediately when the pool has room. One chat at a time on the same server flags, two prompts, two runs each, master against this branch:

prompt	gen tokens	master tok/s	this branch tok/s	output
0	3387	260.7, 263.6	267.1, 269.2	byte-identical
2	6917	317.0, 317.9	317.4, 314.4	byte-identical

Tests

tools/server/tests/unit/test_preempt.py, thirteen tests on the two-slot unified pool with the stories260K model:

one request with LLAMA_SERVER_PREEMPT_EVERY=8 produces the same tokens as the same request without the knob, with at least six park and resume cycles in the log;
two requests that each fit alone (8 prompt plus 160 generated in 256 cells) but not together both finish with 160 tokens, no truncation and no context error;
two 150-token prompts that do not fit together both finish, so a slot still processing its prompt is parked and resumed;
a slot generating 230 tokens beside a 150-token prompt generating 90 both finish;
--preempt-ram 0 parks nothing and the requests fail the old way.
after a pressure run /metrics shows the preemptions and resumes, no request still parked and no parked RAM, and /slots shows no slot parked;
two 250-token prompts in a 256-cell pool both complete: a parked prompt too close to the pool size to leave the scheduling margin is still restored once the pool is empty (before the fix it was parked for ever);
with LLAMA_SERVER_PREEMPT_PLANNER=off, two generations that fit alone and not together reach the retry ladder and both finish through its last resort, with nothing parked ahead of the decode;
the same with a prompt in flight when the pool runs out: the chunk that was in the batch given up is processed once after the rewind, so the prompt token count is exact.
--preempt-ram -1 (no limit) enables the last resort as well: the same pair finishes through it;
two 12000-token generations in a 256-cell pool with context shift on both finish: a resident cycling through context shifts is parked for a head that has waited two seconds, and the two take turns.
three unending generations in a 384-cell pool with context shift on all finish: the rotation parks the resident whose cells let the head in, the smallest such, and failing one the largest;
one request for two completions, a parent and a child sharing the prompt, that do not fit the pool together: a family alone in the pool has no victim, so the request gets the context error it would get alone and the next request is served normally.

All pass on this branch with and without GPU offload, three runs each. The first two fail on master, the second with Context size has been exceeded.

Last resort

The planner parks ahead of the decode, so the retry ladder is only reached when its estimate was wrong. When it was, one token finding no cell used to end every processing slot with the context error, parked slots included. Now, with more than one conversation resident and a budget to park into, every resident slot is rewound to the token boundary the cache is at (a batch is applied one chunk at a time and the chunk that failed left nothing behind: tokens that were never decoded come off, the sampled token stays in the slot and goes into the next batch the way it went into this one, a prompt whose last chunk was in the batch is marked not done again), the smallest slots are parked until the planner's own bound holds, the batch is given up and the next pass rebuilds it from the survivors. One conversation that does not fit alone, and a multimodal prompt, keep the old path. Parked slots are out of the error sweep either way. With speculation on, a slot's sampled token and its draft have to stay in one view, so a batch holding a draft is not narrowed: after purging an idle slot the ladder goes to the last resort straight away.

Exact mode (#194) with the planner off, -c 2048, four chats of 512 tokens against a solo run, greedy: 4B and 35B, with and without MTP drafts, identical in every round, six last resorts per cell, no context error.

Limitations

These are known and are the reason this is a first cut rather than the whole feature.

Above --preempt-ram nothing more is parked; there is no fall back to recompute yet. The parked state is about 36 KiB per token for this model.
A parked slot cannot evict anyone to make room for itself. It waits for the leader, which in the four-chat run meant waits of up to 24 s. A slot preempted three times is passed over while another candidate exists, but when it is the only one it is parked again rather than letting the pool fail.
The pool estimate sums each slot's tokens. For SWA models that overstates what the pool holds, so preemption fires early on those rather than late. A pure recurrent model (Mamba, RWKV) holds one state per sequence whatever its length, so the count says nothing there: preemption is off for those, said at load, and hybrids keep their attention cache and stay on. Parent/child requests are not preempted at all and a group that outgrows the pool still fails the old way.
Two models were tested, a dense 4B and a 35B-A3B mixture of experts, both with MTP drafting. SWA and hybrid memory take different branches inside the state save and restore, and the interaction with context checkpoints on those models is unverified.
A single chat that outgrows the whole pool on its own still gets Context size has been exceeded, as does a parked prompt that cannot fit the empty pool. Those are now the only ways this server ends a conversation on pool pressure.

Relation to #182 and #183

#182 fixes the speculative sub-batch index once the retry ladder narrows the batch. #183 reduces the damage on a full pool from every slot to one slot. This change stops the pool from filling in the first place, and when the ladder is reached anyway it parks instead of ending anyone. With a park budget a batch holding a draft is no longer narrowed, so #182 matters on this branch only under --preempt-ram 0; it still merges cleanly.

Follow-up commits after review

The rotation credited the parked head's bytes as leaving, since the head is restored on the pass that parks the resident, but the resident is parked before the head is restored and freed, so both states are held at once and --preempt-ram was not a cap on what is held. The rotation now asks the plain budget check: a budget that holds one sequence but not two does not rotate, and the head says so once per park and waits for a resident to finish, or to shrink after a context shift. Preemption is off for a pure recurrent cache, as above. Tests: mamba-130m under --kv-unified with the forced-park knob set, two completions finish and nothing is parked; three generations in a pool one of them fills under a budget that holds the parked heads but not a head and the resident together, the refusal logged, every stream finishes 6000 tokens, no context error. Harness 15 of 15.

One more: the reservation for a slot just given a request measured its next prompt chunk from the prompt it still mirrored, while the used count and the resume check already counted from the retained prefix, so a request shorter than the previous one on that slot reserved one cell for a chunk of hundreds. The three figures now share one helper, the common prefix for a started slot and nothing when the request does not cache its prompt. Harness 15 of 15.

And the context shift and the planner now run inside the guarded part of the step, with pre_decode() and batch.render(): ahead of it, a shift that failed to rebuild a slot's tokens or a park that failed to allocate ended the loop on an uncaught exception instead of telling the slots.

What a started slot keeps is now decided by one rule shared by the retained figure and the normalisation, the batch builder's: nothing when the request does not cache its prompt, otherwise the shared prefix cut short of an aLoRA invocation.

The resume order is read from the environment on every load: the flag is process-global and a reload with the variable unset, or another context in the same process, kept the previous load's order.

The order belongs to the context now rather than to the process, and the recurrent flag is assigned from the model on every load, so a reload with an attention model after a recurrent one gets preemption back.

The pool is measured by the cells each slot physically holds. A slot just given a request keeps the previous request's prompt until the batch builder trims it to the shared prefix, and with continuous batching off that can be well behind a running generation; counted by the prefix, a resume was found to fit and attempted against occupied cells, and the parked request failed its wait. When nothing fits, the restore pass now trims every started slot to the prefix its request keeps before idle slots are cleared, so the cells it will not use are released ahead of the batch builder.

…KV pool fills With --parallel N --kv-unified there is one pool of cells and every slot believes it owns all of them. When the pool fills, llama_decode returns 1, the retry ladder in decode() halves n_batch down to 1, and the server calls send_error on EVERY processing slot: "Context size has been exceeded". Four chats sharing a 8192-cell pool on Qwen3.5-4B-MTP die together after six seconds, none of them anywhere near its own 8192 limit. The code already says what should happen instead: "TODO: try to terminate only the largest active slot/sequence and continue with the rest". Terminate nothing. Once per update_slots(), before the batch is built, compare what the pool holds against what the next decode will ask for. If it does not fit, take the cells back from one slot: copy its sequence out with llama_state_seq_get_data_ext, release the cells, and park the slot in a new SLOT_STATE_PREEMPTED. When the pool has room the copy goes back with llama_state_seq_set_data_ext and the slot carries on. The task, the sampler, the generated text and the position the stream has reached never left the slot, so the continuation is the one the slot would have produced without the pause, and a streaming client sees a gap and nothing else. The check sits before the batch is built on purpose: at that point every slot is at a token boundary, prompt.tokens is exactly what the cache holds for it, and no draft is in flight, so a slot can be removed without unpicking a half-decoded batch. The speculative draft is dropped with the cells, which costs the step its speedup and nothing else. Victim policy: keep the slot that is furthest along, since it is the closest to finishing and to giving its cells back, and among the rest prefer one that has not been preempted three times already, then the smallest. A prompt cached on an idle slot is cheaper than a conversation waiting to continue, so try_clear_idle_slots() is asked first, both before preempting anyone and before deciding a resume does not fit. Measured on Qwen3.5-4B-UD-Q4_K_XL with an embedded MTP head, --parallel 4 --kv-unified -c 8192, four streaming chats with 1000-token prompts at temperature 0: base 4 of 4 chats killed by "Context size has been exceeded" after 6.7 s with this 4 of 4 chats completed, 0 errors, 7 preemptions, 7 resumes, 22944 tokens in 66.9 s (343 tok/s aggregate) and the retry ladder never fires at all. At -c 16384 the same load still kills all four on the base and still completes all four here. LLAMA_SERVER_PREEMPT_EVERY=N preempts every generating slot every N generated tokens regardless of pressure. With one request on an idle server the batch has the same shape at every step, so it isolates the resume from batch nondeterminism: over 91 forced preemptions across four prompts, every continuation is byte-identical to the same prompt run without any.

Two tests on the two-slot unified pool. The first runs one request with LLAMA_SERVER_PREEMPT_EVERY=8 and asserts the tokens match the same request without the knob. The second runs two requests that each fit alone but not together and asserts both finish with no context error. Both fail on master: the knob is unknown there, and the second request dies with Context size has been exceeded.

…, and bound the parked state with --preempt-ram A slot still processing its prompt is between two chunks of it, which is as clean a boundary as between two sampled tokens, so it is a victim too: two prompts that do not fit together no longer fail together, and a large prompt arriving beside a running chat waits for it instead of ending it. A slot that has not started yet holds at most a cached prefix and is parked the same way, which is how it waits. Restoring takes the most-preempted parked slot first, but one that does not fit yet no longer holds up a smaller one that does: the smaller one is the first to be parked again if the pool fills, so the head of the line loses nothing. --preempt-ram N (LLAMA_ARG_PREEMPT_RAM) bounds the host RAM parked sequences may hold, default 8192 MiB like --cache-ram. A slot that would not fit under the budget is not parked, and when nothing can be parked the KV-full path runs as before. --preempt-ram 0 disables preemption. The prompt batching pass skips parked slots explicitly. Speculation is only restarted on restore for a slot that was generating; one parked mid-prompt starts it when its prompt is done, as it always did. Tests: two prompts that overflow the pool together, a generating slot beside a large prompt, and --preempt-ram 0 restoring the old behaviour.

chatgpt-codex-connector · 2026-09-05T01:09:53Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T03:38:35.547113Z	`3c99faf`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Counters n_preempt_total and n_resume_total, gauges requests_preempted and preempt_ram_bytes, and is_preempted plus n_preempt on each /slots entry, so a client can tell a parked request from a slow one and an operator can see the parked host RAM. A parked slot no longer counts as busy in n_busy_slots_per_decode, since it took no part in the decode.

danielhanchen · 2026-09-05T17:18:27Z

One more commit, 75944db80 Batch fragmented sequence restores through bounded host staging.

A parked sequence whose cells were scattered across the pool restores as thousands of small synchronous device copies per layer, each of which has to win the GPU. Under a foreign job at full utilisation single restores took 65 to 175 s and froze the server. The restore loop now groups the fragments per tensor, stages the tensor once on the host, patches every fragment into it and writes it back in one transfer, for tensors up to 64 MiB with 64 or more fragments; contiguous restores keep their original path and a failed staging allocation falls back to the individual copies.

Measured on the 4B with two MTP drafts at -c 8192, four slots, four concurrent chats generating 8192 tokens per run, on a shared GPU, before and after interleaved:

forced parks every 64 tokens, 124 restores each	min ms	median ms	max ms	over 1 s
before, `63cdac4d7`	144	296	36625	10
after, `75944db80`	82	90	104	0

Plain runs: before 320, 83228 and 83381 ms; after 105, 107 and 109 ms. Park time is unchanged at 217 to 248 ms on both sides. All eight runs completed 4 of 4 with zero errors on both binaries. The distributions do not overlap: the whole after range sits below the before minimum, and the claim is the shape, restore cost proportional to the data and insensitive to contention, rather than a single ratio.

Exactness: 1000 tokens, seed 0, temperature 0, forced parks (15 restores) against unforced, byte-identical. test-state-restore-fragmented passes on CPU and on CUDA with the 4B (55 MB sequence state, all three snapshots byte-identical after the batched restore); unit/test_preempt.py 6 passed.

Parked slots came back most-preempted first, then longest parked, and a smaller slot could pass a head that did not fit yet, on the reasoning that the smaller slot is the first to be parked again and so costs the head nothing. It does cost the head: the small slot squeezes in, grows, and is parked again with one more preemption to its name, which puts it ahead of the head next time as well. In a four-chat Studio run on an 8192-cell pool one chat parked at 4160 tokens waited 7 minutes 22 seconds while two smaller chats were restored and parked three times each. The order is now the line itself: the slot parked longest comes back first, and nobody else while it does not fit, which is what a first come first served queue of preempted requests amounts to. Simulated with the smallest-slot victim policy over 60 seeds, this cuts the longest single wait at eight chats by 2.5 to 3x (60 to 78 s down to 24 to 28 s) for 0 to 3 percent of makespan at 8192 cells and 3 to 6 percent at 16384, and parks less often and copies fewer cells at 8192. At four chats every order is within 2 percent on everything. LLAMA_SERVER_PREEMPT_RESUME=pass keeps the previous order. The victim side is unchanged, including the starvation guard.

danielhanchen · 2026-09-06T12:42:13Z

Pushed 50b617a: parked slots come back head of line by park time, and nobody passes a head that does not fit yet. LLAMA_SERVER_PREEMPT_RESUME=pass keeps the previous order (most-preempted first, then longest parked, and a smaller slot may pass the head). The victim side and its starvation guard are unchanged.

Why: in a four-chat Studio run on an 8192-cell pool, one chat parked at 4160 tokens waited 7 minutes 22 seconds while two smaller chats were restored and parked three times each. The previous order let the small slot squeeze in, grow, and be parked again with one more preemption to its name, which put it ahead of the head next time as well.

Simulated with the smallest-slot victim policy over 60 seeds (scripts/preempt_policy_sim.py --resume-orders), longest single wait in seconds, makespan in parentheses as a fraction of the previous order:

workload	previous order	head of line
measured, 8 chats, 8192 cells	60.4 (1.00)	25.8 (0.99)
uniform, 8 chats, 8192 cells	33.8 (1.00)	14.8 (1.01)
heavy, 8 chats, 8192 cells	72.6 (1.00)	23.6 (1.00)
measured, 8 chats, 16384 cells	24.4 (1.00)	16.7 (1.06)
heavy, 8 chats, 16384 cells	78.2 (1.00)	27.8 (1.03)
four chats, any workload	within 2 percent on every figure

At 8192 cells it also parks less often and copies fewer cells. Live on the #197 binary, eight chats of about 1000 prompt plus 2048 generated tokens (ignore_eos) through four slots at -c 8192 with two MTP drafts, three interleaved pairs on one GPU: head of line 128.6 / 129.7 / 129.8 s wall, longest wait 13.5 / 13.5 / 14.0 s; previous order 128.6 / 130.7 / 124.1 s, longest wait 12.9 / 15.1 / 14.2 s; 8 of 8 completions and zero context overflows in every run. Equal-length chats are the shape where order cannot matter, and it does not; the change costs nothing there and the fairness gain is on uneven chats like the Studio run above.

On the 35B in Studio (exact concurrency on, MTP drafts), two four-chat GUI drives on the new order finished 8 of 8 with 9 parks each and no errors; the largest parked chat was restored after 28 s and the slowest chat finished in 174 s.

Server tests: 6 passed here, and 11, 7 and 16 on the async, exact and integration branches after merging.

…e that is neither head nor pass LLAMA_SERVER_PREEMPT_RESUME was read lazily on the first resume and never announced, unlike the other LLAMA_SERVER_PREEMPT_* knobs, and any value other than pass silently meant head of line. It is now read in load_model() next to LLAMA_SERVER_PREEMPT_EVERY: head is the default, pass is logged as a warning, and anything else fails the load with a message.

danielhanchen · 2026-09-06T13:01:28Z

Pushed 8057a74: LLAMA_SERVER_PREEMPT_RESUME is now read once in load_model next to the other preemption knobs, head is the default, pass is logged, and any other value fails the load with a message. Tests 6 of 6.

… of parking it for ever A parked slot was restored only when its sequence, its next step and the scheduling margin all fit, even with the pool empty. A prompt within n_ctx that was parked before it took any cells, but too close to n_ctx to leave room for the margin, therefore never fit, and since a restore was never attempted it never reached the restore-failure path either: it stayed parked for ever. With parked slots restored head of line, such a head would have held every slot behind it as well. Two changes. The margin is headroom for the other residents, so with nothing resident there is nobody to keep it for and a sequence that fits the pool exactly is let back in. A parked sequence whose next step would not fit an empty pool at all is the single-conversation overflow the KV-full path reports, so it is reported the same way, "Context size has been exceeded", and released, and the line is rescanned without it. New test: two prompts of 240 tokens on a 256-cell pool, sent together, both complete. Server preemption tests 7 of 7.

danielhanchen · 2026-09-06T15:50:16Z

25c8466c2 fixes a case the resume loop could park for ever.

The loop admits a parked slot when occupied + need + margin <= n_cells. A prompt that fits the pool but not the pool minus the margin (250 tokens in a 256-cell pool, say) was never admitted, and since the request itself passed the n_ctx check it was never failed either. It sat parked with nothing ever tried. Under head of line it would have held the line for every slot behind it.

Two changes:

The margin is waived when nothing is resident. There is nothing for a resident slot to grow into, so the margin has no job.
A parked slot whose need exceeds the pool alone is failed with the context error and released before the restore pass, so it cannot block anyone.

Regression test test_two_prompts_near_the_context_size_both_complete: two 250-token prompts at n_ctx 256, n_batch 256, four tokens each. On the build before this commit the second prompt parks with no cells, the first completes and releases at 253 tokens, and no restore is ever attempted; the test times out at 240 s. On this commit both complete with four tokens each. Full suite 7 passed.

…h states at once and the RAM cap says so A recurrent cache holds one state per sequence whatever its length, so the token count the planner measures says nothing about it: two long conversations were parked in turn on a cache with room to spare. Preemption is off for a pure recurrent model (llama_model_is_recurrent), said so at load, planner and last resort both; a hybrid keeps its attention cache and stays on. The rotation credited the parked head's bytes as leaving, since the head is restored on the same pass, but the resident is parked before the head is restored and freed, so both states are held at once and --preempt-ram was not a cap on what is held. The rotation now asks the plain budget check: a budget that holds one sequence but not two does not rotate, and the head says so once per park and waits for a resident to finish, or to shrink after a shift. Tests: mamba-130m under --kv-unified with the forced-park knob set, two completions finish and nothing is parked; three generations in a pool one of them fills under a budget that holds the parked heads but not a head and the resident together, the refusal logged, every stream finishes, no context error.

danielhanchen · 2026-09-07T00:47:52Z

@codex review

chatgpt-codex-connector

💡 Codex Review

llama.cpp/tools/server/server-context.cpp

Line 3529 in 3800dde

server_task task(SERVER_TASK_TYPE_NEXT_RESPONSE);

Keep context shifting inside the exception boundary

When a context shift throws, for example because rebuilding new_tokens or reinserting them runs out of memory, pre_decode_shift() now executes before the existing try block. The exception therefore escapes update_slots() and the uncaught server_queue::start_loop() callback, terminating the inference loop instead of reporting the failure to the active slots as the previous in-pre_decode() implementation did; include the shift and preemption calls in the guarded section.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… as its used count already did preempt_kv_reserve() measured a started slot's next prompt chunk from the prompt it still mirrors (task tokens minus prompt tokens), so a request shorter than the previous one on that slot reserved one cell for a chunk of hundreds, while preempt_kv_used() and preempt_n_need() already counted from the retained prefix. The three now share preempt_n_retained(): the common prefix for a started slot, nothing when the request does not cache its prompt, the prompt otherwise. The new chunk always fits where the mirrored prompt was, so the undercount could only bite a restore decided in the same pass, and the retry ladder hid it for ordinary requests; the figures agree now either way.

… of the step Both ran ahead of the try block around pre_decode() and batch.render(), so a shift that failed to rebuild a slot's tokens, or a park that failed to allocate, left update_slots() on an uncaught exception and ended the loop instead of telling the slots. They run inside the same guard now, and the existing handler ends the affected slots with the message; a parked slot took no part in what failed and is spared, as before.

danielhanchen · 2026-09-07T01:18:33Z

Right. The shift and the planner ran ahead of the guard, so a shift that failed to rebuild a slot's tokens, or a park that failed to allocate, left update_slots() on an uncaught exception instead of telling the slots. Both now run inside the same try as pre_decode() and batch.render(), and the existing handler ends the affected slots with the message. Fixed in af56090.

…uilder's The retained-prefix figure and the started-slot normalisation each took the common prefix whole, while the batch builder keeps none of it when the request does not cache its prompt and cuts it short of an aLoRA invocation. Both now ask preempt_n_keep(), which applies the same rule, so the planner counts, reserves and parks a started slot by the state the request will actually keep.

The flag is process-global and was set only when LLAMA_SERVER_PREEMPT_RESUME was pass, so a reload with the variable unset, or another context loaded in the same process, kept the previous load's order. It is reset to the default and read again on every load.

…otation delay At 6000 tokens the resident finished in under three seconds on a fast host, before the parked heads had waited the two seconds a rotation needs, so nothing was ever asked for or refused and the test failed on timing.

…s assigned on every load The resume order was a file-static flag, so two contexts in one process shared it and the later load decided for both. It is a member now, read from the environment at load. And preempt_recurrent was set and never cleared, so a context reloaded with an attention model after a recurrent one kept preemption off; it is assigned from the model on every load.

danielhanchen · 2026-09-07T03:05:28Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 301caa31da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-07T03:18:32Z

+        if (slot.state == SLOT_STATE_STARTED && slot.task) {
+            return (int32_t) preempt_n_keep(slot);


Normalize started slots before counting their retained prefix

When a newly assigned SLOT_STATE_STARTED slot still physically holds a previous request's larger cached prompt, this returns only the new request's common prefix even though the stale suffix is not removed until that slot enters the prompt-building path. If another request is already parked, the resume check can therefore repeatedly decide that it fits and attempt restoration against cells that are still occupied. With --no-cont-batching (or incompatible LoRA batches), the started slot may remain unprocessed behind a long generation, and after 60 seconds the parked request is incorrectly failed with failed to restore the preempted sequence. Normalize started slots before the resume pass, or continue counting their actual cached tokens until normalization has occurred.

Useful? React with 👍 / 👎.

Fixed in 3c99faf. The pool is now measured by the cells each slot physically holds, so a started slot carrying its previous prompt is counted in full until it is trimmed. When nothing fits, the restore pass trims every started slot to the prefix its request keeps (preempt_normalize_started_all) before clearing idle slots, so the released cells let a parked slot that fits come back without waiting on the batch builder. 15 of 15 server tests pass.

…e a resume A slot just given a task keeps the previous request's prompt in the pool until the batch builder trims it to the shared prefix, which with continuous batching off can be well behind a running generation. The planner counted such a slot by that prefix, so a resume could be found to fit and attempted against cells still occupied, and the parked request then failed its 60 s wait. The pool is now measured by what each slot physically holds. When nothing fits, every started slot is trimmed to the prefix its request keeps before idle slots are cleared, so the cells it will not use are released ahead of the batch builder and a parked slot that fits without them comes back.

danielhanchen · 2026-09-07T03:31:27Z

@codex review

chatgpt-codex-connector · 2026-09-07T03:38:34Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 3c99fafdf0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Daniel Han and others added 3 commits September 5, 2026 01:04