server: --pipeline-groups, run the slots over several contexts of one model by danielhanchen · Pull Request #187 · unslothai/llama.cpp

danielhanchen · 2026-09-05T10:46:23Z

Adds --pipeline-groups N to llama-server: the slots are run over N independent
llama_context objects created from the same model, each with its own batch, its own sampling
and its own decode thread. The model weights, the task queue, the results queue and the HTTP
layer are shared. With N = 1 nothing changes.

The point is a layer split across two machines. With one context the split is a two-stage
pipeline fed one batch at a time, so each stage is idle while the other computes. With two
groups there are two batches in flight.

Result

Two DGX Sparks (GB10, 111.9 Gb/s per rail), Qwen3.8-27B-UD-Q4_K_XL, layer split over one
ggml-rpc-server on the peer, -c 16384 --parallel 32 --cache-ram 0 -fa on -ngl 99 -t 6,
32 concurrent clients, 64 requests, npp 128 / ntg 256, best of two repeats, all cells inside one
clock state (2386 to 2402 MHz local, 2433 to 2463 MHz peer, no thermal cap markers).

device order	groups	tok/s (run a / run b)	TPOT ms	TTFT median s	GPU busy local	GPU busy peer
`CUDA0,RPC0`	1	94.9 / 92.9	310 / 311	7.4 / 7.6	44 / 47 pc	43 / 41 pc
`CUDA0,RPC0`	2	75.5 / 76.3	395 / 398	6.3 / 5.8	43 / 42 pc	44 / 47 pc
`CUDA0,RPC0`	2, sampling pool off	73.9 / 75.3	408 / 404	6.5 / 5.8	44 / 43 pc	45 / 46 pc
`RPC0,CUDA0`	1	99.7 / 98.1	295 / 301	7.5 / 6.6	46 / 43 pc	47 / 46 pc
`RPC0,CUDA0`	2	130.4 / 131.3	223 / 220	5.7 / 6.1	76 / 76 pc	79 / 76 pc
`RPC0,CUDA0`	1, `--backend-sampling`	94.8 / 94.9	309 / 311	7.5 / 7.4	42 / 42 pc	44 / 43 pc
`RPC0,CUDA0`	2, `--backend-sampling`	120.4 / 123.7	233 / 230	8.0 / 6.4	71 / 74 pc	73 / 75 pc

1.32x over one context on the same split, at 76 to 79 percent GPU busy per node against 43 to 47
percent. Two things had to be true at once:

The device order matters more than the feature. With -sm layer the devices are filled in
the order they are listed, so the last device holds the output layer. List the RPC device
first and the local one last (--device RPC0,CUDA0): the logits are then produced locally,
which removes a n_vocab * n_rows * 4 byte return from every decode step (31.8 MB per step at
32 rows on this 248320-token vocabulary) and lets the sampler read them out of local memory.
Sampling had to come off the critical path. See below.

Slot count sweep, RPC0,CUDA0, --cache-ram 0, -c sized so every slot keeps 512 tokens:

slots	groups 1	groups 2	ratio
32	99.7 tok/s	130.4 tok/s	1.31x
64	117.2	157.2	1.34x
128	124.2	170.1	1.37x
256	124.1 (1 request in error)	130.5 (126 of 512 in error)	not comparable, see limits

What the cost was, and how it was found

LLAMA_SERVER_PIPE_PROF=1 (added here) prints, per group and every five seconds, how long an
iteration spends waiting for the engine, in pre_decode, in llama_decode, in
llama_synchronize and in post_decode, and splits post_decode per token into sampling,
detokenization, stop-string handling and the result queue. On the pair, per group iteration:

	groups 1, 32 rows	groups 2, 16 rows per group
wait for the engine lock	0.00 ms	0.00 ms
`pre_decode`	0.01 ms	0.01 ms
`llama_decode`	145 ms	200 to 262 ms
`llama_synchronize`	148 ms	95 to 126 ms
re-take the engine lock	0.00 ms	0.00 ms
`post_decode`	21 ms	54 to 76 ms
whole iteration	316 ms	401 ms
per token: `common_sampler_sample`	0.60 ms	3.6 to 4.6 ms
per token: `common_token_to_piece`	0.002 ms	0.002 ms
per token: `process_token` (stop strings, streaming)	0.065 ms	0.18 ms
per token: `queue_results.send`	0.061 ms	0.18 ms

Sampling one row builds a candidate array over a 248320-token vocabulary, about 4 MB of memory
traffic. With one context that pass runs in the gap where both GPUs are idle and gets the full
memory bandwidth of the node. With two groups it runs against the other group's GPU work on the
same LPDDR5X and costs six to seven times more per row, while sitting on each group's critical
path between the synchronize and the next submit. Every other candidate was measured and
falsified: the results queue and the task queue are three orders of magnitude too small, and the
engine lock was never contended (0.00 ms in every window; forcing the old single shared host lock
back on with an A/B switch gave 74.7 against 73.5 to 75.3 tok/s).

Changes

--pipeline-groups N, parsed in tools/server because it only means anything for the server.
Slots are partitioned contiguously; each context gets n_ctx / N and n_seq_max = P / N, so
the per-slot context and the total KV over all groups are what the user asked for. Slot
selection still runs over all slots, so prompt-cache similarity and the slot save / restore
endpoints behave exactly as before. N > 1 is refused together with speculative decoding,
multimodal, control vectors and --sleep-idle.
Each group samples its rows over a small worker pool. The rows are independent (own sampler,
own row of the logits), so the tokens are the ones the serial pass would have produced. The
thread budget is divided by the number of groups, so a pipeline-groups run is not simply given
more CPU than the single-context run. LLAMA_SERVER_SAMPLE_THREADS=1 turns it off.
The engine lock is per group, so the host path of one group does not exclude the other's. This
is worth nothing on its own on this hardware and is reported as such, but it is what the
feature is supposed to guarantee.
Task processing pauses only the group that owns the slot the task touches, instead of every
group. This is what fixed TTFT (29 s to 7 s at 32 concurrent).
get_available_slot() called prompt_save / prompt_load, which read and write the slot's
sequence KV, before the guard waited for the owning group's decode to finish. With
--cache-ram 0 the cache is null so it never fired; with the cache on it is a live race
against a running decode and it matches the 29 s TTFT and the abort seen while developing this.
The cache update now happens after the wait.
server_metrics is no longer written from several group threads at once, and each group counts
its own slots instead of every slot of the server (which double counted n_busy_slots).
ggml-rpc: a whole message is now atomic on the wire. The socket is cached per endpoint and
shared by every backend of that endpoint, and a message was three unlocked writes, so two
contexts interleaved their command streams and --pipeline-groups 2 aborted with "Remote RPC
server crashed or returned malformed response" within seconds. Responses are handed out in
request order by a ticket, so a waiter does not hold the send lock. last_graph_uid moved to
the connection and is checked under that lock, which closes a hazard where GRAPH_RECOMPUTE
could re-run the other context's graph.

Correctness

Per-group speculative decoding (the section below), on top of the proofs already listed here:

Single GPU, five prompts, 48 tokens, temperature 0: greedy output byte identical between the PR
base, --pipeline-groups 1 and --pipeline-groups 2 on this branch, md5
e154ffeace8e6d57298e1963f16529b5 for all three.
Qwen3.5-4B-MTP on one GPU, --parallel 8: groups 1 and groups 2 give the same sequential greedy
md5 with MTP (4dae418e8227bddb0fc0e93ab11a9e67) and the same without it
(43e68a9c6ffafce623dd8089192cdfc3); all eight slots of both groups serve requests and
/metrics reports drafted and accepted tokens in both MTP arms.
tools/server/tests/unit/test_speculative.py (--model-draft sidecar, context shift,
context-not-exceeded, parallel requests): 6 passed with one group, 6 passed with two.
test_basic.py, test_completion.py, test_ctx_shift.py, test_slot_save.py: 59 passed,
1 skipped, 1 failed in both arms, the failure being a preset whose model is not in the offline
cache. Slot save / restore and context shift therefore also run over two groups.
Greedy output byte-identical at N = 1, N = 2 and N = 4 over a CPU-only two-RPC split,
five prompts, md5 177dc61e0703eba3bdaf7bf1131f0458, same as the unmodified binary.
tools/server/tests run serially against this build and against the unmodified branch head on
the same machine: 325 passed, 233 failed, 6 skipped, 9 errors on both, with byte-identical
failure name sets. The failures are pre-existing in that environment (the cached test models do
not match), not introduced here.
--pipeline-groups 2 with the sampling pool on and off gives the same test results.

Speculative decoding per group

--pipeline-groups N now combines with speculative decoding (--spec-type draft-mtp with the MTP
head inside the GGUF, and --model-draft with a sidecar draft model). Each group owns its own
speculative state:

one draft or MTP llama_context, created by common_speculative_init_from_params against the
group's target context, so ctx_other and the next-token embedding hooks point at that context;
one common_speculative, sized for the group's P/N sequences and addressed by the slot's
sequence id inside the group (slot.seq_id, which is slot.id with one group);
the draft batches go through the group's draft context on the group's decode thread, like
llama_decode already does; the two task-queue yields around the drafter are only taken on the
single-group main-thread path.

Slot save / restore, context checkpoints (the drafter state is stored and restored with the
checkpoint) and the prompt cache behave as before, per slot. With N = 1 the code path is the same
as before, spelled through groups[0]. With --model-draft the sidecar model is loaded once per
group. validate_pipeline_groups no longer refuses drafters; --mmproj, --control-vector and
--sleep-idle-seconds are still refused with N > 1.

Two DGX Sparks, Qwen3.8-27B-UD-Q4_K_XL, layer split over one ggml-rpc-server on the peer
(--rpc peer:50055 --device RPC0,CUDA0 -sm layer), -c 16384 --parallel 32 --cache-ram 0 -fa on -ngl 99 -t 6, real-text prompts, npp 128 / ntg 256, 2 requests per client. MTP is
--spec-type draft-mtp --spec-draft-n-max 3 off the head inside the GGUF. Two passes with the arm
order reversed, because the SM clock decays through a window; the cells below all ran with both
GPUs at 2390 to 2400 MHz and no cap marker (the three cells that were capped or straddled a cap
stage are listed after the table, and each of them has a clean repeat).

groups	MTP	8 users tok/s	TPOT ms	32 users tok/s	TPOT ms	accepted/drafted 8 / 32
1	no	55.3	136	96.9	311	-
1	yes	95.5 / 93.6	74 / 75	112.9 / 114.2	233 / 232	0.80 / 0.64
2	no	54.0	139	115.7	219	-
2	yes	77.1	84	133.8	200	0.72 / 0.68

At 32 users the combination is the best cell measured: 1.38x over one context without speculation,
1.17x over one context with MTP and 1.16x over two groups without it. At 8 users it is not: one
context with MTP is faster (95.5 against 77.1), because two groups halve the rows per group and the
draft head is at its most valuable when the batch is small and the step is memory bound. The
crossover is between 8 and 32 rows, which is the same crossover the --pipeline-groups result
itself has.

Cells with a clock caveat, each superseded by the clean repeat in the table: one group without MTP
in the first pass ran at 1828 MHz (53.6 / 95.5), two groups without MTP in the reversed pass at
1690 MHz (51.5 / 108.7), two groups with MTP in the reversed pass straddled a cap stage
(92.0 / 128.3). Every 32-user cell in every arm, including the two without speculation, had one
request of 64 return no tokens; it is present in the arms this PR does not touch, so it is not a
property of per-group speculation.

Known limits

Use --cache-ram 0 with a layer split. The RAM prompt cache moves a whole slot state on
every slot handover, and on a split most of that state lives on the remote node, so it crosses
the wire. At 32 concurrent clients on the pair, with the cache at its default: one group 75.9
tok/s and 33 s median TTFT (against 99.7 and 7.5 with --cache-ram 0); two groups 6.9 tok/s
with 14 of 64 requests timing out, because the handover runs on the single task thread and the
other group starves while it does. This is a property of the cache on a split, and one group is
already badly hurt by it, but two groups make it much worse and it is not fixed here.
At 256 slots the two-group cell finished 386 of 512 requests, the rest ending with a truncated
HTTP response; the server log records no error, so the truncation is on the HTTP path at 256
concurrent streams, and the node crossed 80 C during that cell. That row is reported with its
error count and is not a clean measurement. 32, 64 and 128 slots are clean.
The gain is specific to a layer split. On a single node there is nothing to pipeline.

A layer split over two nodes is a two-stage pipeline that a single llama_context feeds one batch at a time, so each stage sits idle while the other one computes. With --pipeline-groups N the server creates N llama_contexts from the one model, partitions its slots between them and gives each group its own batch and its own decode thread, so there are N batches in flight and both stages have work. Each context is created with n_seq_max = n_parallel / N and n_ctx = n_ctx / N, so the per-slot context and the total KV memory are unchanged. Slots are partitioned contiguously and carry the sequence id they use inside their own context. Slot selection for a new task still runs over all slots, so prompt cache similarity, the slot endpoints and the KV prefix reuse behave exactly as before. The model weights, the task queue, the results queue and the HTTP layer are shared. Task processing pauses the decode loops for the moment it looks at the slots. Speculative decoding, multimodal and idle sleeping are refused with N > 1 rather than half supported. With the default N = 1 there is one context, one batch and one update loop on the main thread, no locks and no extra threads.

- the unlock around llama_decode is now RAII, so a throwing decode cannot leave a group marked busy (which would wedge every later task) nor return to the error handler without the engine lock - n_cmpl is rejected when it exceeds the slots of one group, instead of being deferred forever: the child slots take their KV from the parent, so they have to live in the parent's context - refuse --control-vector with more than one group, common_init_from_params only applies it to the context it creates - the queued prompt stats and the empty batch kill switch move into the group, they were shared counters flushed per group - post_decode uses the group's context, and the detokenize calls in the result path use the slot's own context - free the contexts already created if a later one fails, and do not index groups[0] when no model is loaded

…onnection One socket is cached per endpoint and is therefore shared by every backend of that endpoint, including the backends of different llama_contexts. A message is written as three unlocked send_data calls, so two contexts interleave their command streams and the server sees a malformed request within seconds. Make a whole message atomic on the wire, and hand the responses out in request order with a ticket, so a thread waiting for its response does not hold the send lock and the other contexts can keep submitting. last_graph_uid was kept per endpoint device while the graph it refers to is stored by the server per connection, and it was read and written without a lock, so two contexts on one connection could make RPC_CMD_GRAPH_RECOMPUTE re-run the other one's graph. Track it per connection and device and check it under the send lock. server: pause only the group that owns the slot a task touches process_single_task stopped every pipeline group for every task and waited for all the in-flight decodes. Holding the engine is already enough to keep the slot state stable, so only wait for the group whose context the task touches: the owning group for completions, cancel, control and slot save / restore / erase, every group for --cache-idle-slots and SET_LORA, none for metrics, /slots and get-lora.

…s own engine lock At 32 concurrent clients on a two-node layer split, post_decode cost 4.3 ms per slot with --pipeline-groups 2 against 0.85 ms per slot with one context, and it sits on each group's critical path between llama_synchronize and the next submit. Per-group timings show it is common_sampler_sample: 0.60 ms per row with one context, 3.6 to 4.6 ms per row with two, because the candidate array of a 248320-token vocabulary is about 4 MB of memory traffic per row and with a second group the pass runs against the other group's GPU work instead of in the gap when both GPUs are idle. - each group now samples its rows over a small worker pool. The rows are independent, so the tokens are the ones the serial pass would have produced; greedy output is byte-identical at N = 1, 2 and 4. The thread budget is divided by the number of groups, so a pipeline-groups run is not simply given more CPU. LLAMA_SERVER_SAMPLE_THREADS=1 turns it off. - the engine lock is now per group, so the host path of one group no longer excludes the other's. Measured on the pair this is worth nothing on its own (74.7 against 73.5 to 75.3 tok/s), which is reported as a falsification, but it is what the feature is supposed to guarantee and it is needed before the sampling pool can overlap anything. - get_available_slot() called prompt_save / prompt_load, which read and write the slot's sequence KV, before the guard waited for the owning group's decode. With --cache-ram 0 the cache is null so it never fired; with the cache on it is a live race against a running decode. The cache update now happens after the wait. - server_metrics is no longer written from several group threads at once, and each group only counts its own slots instead of every slot of the server. - LLAMA_SERVER_PIPE_PROF=1 prints the per-group host path every five seconds. That is how the cost above was found. README: document that a layer split must list the RPC device first and the local device last, so the output layer and its logits stay local.

chatgpt-codex-connector · 2026-09-05T23:18:43Z

You have reached your Codex usage limits for security reviews. Please try again later.

chatgpt-codex-connector · 2026-09-05T23:18:51Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-06T11:09:16.085094Z	`a1dd7c5`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8de4320163

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-05T23:22:41Z


+    // strip the server-only --pipeline-groups before the common parser sees it


Preserve pipeline groups when spawning router children

When the server starts in router mode, stripping this option before server_models receives argc/argv removes it from the base_preset built in server-models.cpp:407-415; that preset is what gets merged into every child configuration at server-models.cpp:548-552. Consequently, llama-server --pipeline-groups 2 accepts the flag on the router, but each spawned model process starts without it and uses the default of 1, so the new feature silently has no effect in this supported mode. Preserve the setting in the child arguments or otherwise propagate it across the process boundary.

Useful? React with 👍 / 👎.

A common_speculative and its draft (or MTP) context are bound to one target context, so with --pipeline-groups > 1 the server refused every drafter. The speculative state now lives in struct server_group: every group creates its own draft / MTP context against its own target context, sizes its own common_speculative for the group's sequences and addresses it by the slot's sequence id inside the group. The draft batches go through the group's draft context on the group's decode thread; the task-queue yields around the drafter are only taken on the single-group main-thread path, like llama_decode already does. Slot save / restore, checkpoints and the prompt cache carry the draft state per slot as before. With one group seq_id == id and the path is the same as before, spelled through groups[0]. With --model-draft the sidecar model is loaded once per group. validate_pipeline_groups no longer refuses --model-draft / --spec-type.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1dd7c5e8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-06T11:09:14Z

            if (ret == 0 && has_output) {
+                prof_timer ts(&grp.prof.t_sync, prof_on);
                llama_synchronize(ctx_tgt);


Keep no-output decodes busy until synchronization

When a prompt is split into batches, intermediate batches commonly have has_output == false, so this skips llama_synchronize() even though llama_decode() is asynchronous. The decode_window destructor then marks the group non-busy, allowing the task thread to acquire its guard and modify that context—for example by launching another slot, updating the prompt cache, or handling a slot operation—while the batch is still executing. Keep the group busy until the asynchronous work is synchronized before exposing the context to task processing.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-06T11:09:14Z

                    if (params_base.cache_idle_slots) {
+                        // this walks every slot of every group
+                        guard.wait_for_all();


Reacquire released group locks before waiting for all

With --pipeline-groups > 1 --cache-idle-slots, the earlier guard.wait_for(slot->id_group) releases every other group's lock, but this subsequent wait_for_all() assumes all entries in lks are still owned and never reacquires them. If another group is busy, it passes an unlocked unique_lock to condition_variable::wait; even when the predicate skips that call, the following cache loop reads and clears the other groups' slots and contexts without excluding their worker threads. This can race or fail as soon as a new task starts while another group is decoding.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-06T11:09:14Z

+                    per_tk(pr.t_sampl), per_tk(pr.t_sampl_par), per_tk(pr.t_piece), per_tk(pr.t_proc),
+                    per_tk(pr.t_send), per_tk(pr.t_post));
+
+            pr = server_group_prof();


Synchronize profiler snapshots across groups

When LLAMA_SERVER_PIPE_PROF=1 and multiple groups are active, the reporting group reads and resets every group's plain server_group_prof fields here while the other workers concurrently update those fields, including during their unlocked decode windows. The atomic report timestamp only selects a reporter; it does not protect these counters, so profiling introduces C++ data races and can emit corrupted measurements. Snapshot or reset each group's counters under appropriate synchronization.

Useful? React with 👍 / 👎.

danielhanchen added 4 commits September 5, 2026 03:28

danielhanchen marked this pull request as ready for review September 5, 2026 23:18

chatgpt-codex-connector Bot reviewed Sep 5, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 6, 2026

View reviewed changes

danielhanchen mentioned this pull request Sep 6, 2026

Studio: two-Spark serving orchestrator with an async replica router unslothai/unsloth#10323

Open

Navigation Menu

Uh oh!

server: --pipeline-groups, run the slots over several contexts of one model - #187

server: --pipeline-groups, run the slots over several contexts of one model#187
danielhanchen wants to merge 5 commits into
masterfrom
feature/pipeline-groups

danielhanchen commented Sep 5, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		// strip the server-only --pipeline-groups before the common parser sees it

Uh oh!

Conversation

danielhanchen commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Result

What the cost was, and how it was found

Changes

Correctness

Speculative decoding per group

Known limits

Uh oh!

chatgpt-codex-connector Bot commented Sep 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

danielhanchen commented Sep 5, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 5, 2026 •

edited

Loading