spec : add adaptive MTP draft depth (draft-mtp-adaptive)

[stew675]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

Suggested configuration to use is: --spec-type draft-mtp-adaptive --spec-draft-n-max 12

Note: It is not recommended to set spec-draft-n-max to values lower than 7 with the Adaptive MTP algorithm. In this instance the algorithm isn't able to reach depths high enough to provide the benefits it aims to achieve. If you find yourself in a situation where insufficient VRAM prevents you from setting a depth of at least 7, or more preferably 8+, then it's generally going to be best to just stick with regular MTP.

The algorithm is a counting based state machine with a climb counter and a weighted drop-pressure accumulator. The depth climbs one step after a number consecutive verifies that accepted every drafted token; any miss adds (N - acceptance) to the drop pressure and the depth drops one step once it exceeds a certain amount. High depths fall quickly (a total miss adds N), low depths hold, and at the floor no pressure accumulates at all. The specific climb difficulty and drop pressure values were empirically determined over a wide range of tests.

--spec-draft-n-max bounds the upper adaptive range and both the floor and the cold-start depth default to 3. The default floor of 3 may be adjusted with --spec-draft-n-min-adaptive, although in testing 3 seems to be the best value to choose here.

Overview of the climb-cost algorithm

It was experimentally determined that a draft MTP depth of 2 or 3 is close to optimal for reasoning and prose.
More typically an MTP depth of 3 was found to be optimal, and so this was chosen as the baseline default.

When the depth is less than three the algorithm allows for an easy transition from a depth of 1 to 2.
A slightly higher cost from 2 to 3 is used to minimise the depth oscillating between two and three.

To climb past a depth of three requires passing a hardened barrier. The hardened barrier is in place to
minimise easily reaching a depth of four which is known to be harmful to performance for reasoning and prose.

Beyond a depth of four the barrier to climb higher is gradually reduced.
This allows for predictive runs to quickly ascend as the admission rates indicate that this is useful.

Latest Testing and Performance Results

(Updated 23 Aug 2026)

Latest performance results can be seen here: #27210 (comment)

Extended discussion of results can be seen here: #27210 (comment)

Test Setup

• Model: Qwen3.8-27B Q8_0 (Qwen3.8-27B-Q8_0.gguf
• Hardware: 2x AMD Radeon AI PRO R9700 (32.6 GiB each), tensor split, Ryzen 9
  9950X3D2, ROCm, HIP_VISIBLE_DEVICES=0,2, GGML_CUDA_DISABLE_GRAPHS=0
• Server: llama-server from branch adaptive-mtp (build 10459, 0c6426b,
  freshly rebuilt)
• Decode: greedy (temp 0.0, seed 675), top-k 20, top-p 0.95, min-p 0.001,
  ctx 262144, f16 KV, --parallel 1, mlock, reasoning on
  (--reasoning-budget 65536 --reasoning-preserve)
• 2 repeats per cell; tok/s = mean of timings.predicted_per_second
  (generation only, >= 4000 tokens per request)

Configurations Tested

ID	spec type	args
C0	none	-
C1	draft-mtp, fixed 3	--spec-draft-n-max 3
C2	draft-mtp, fixed 12, p-min	--spec-draft-n-max 12 --spec-draft-p-min 0.75
C3	draft-mtp-adaptive 3..12	--spec-draft-n-min-adaptive 3 --spec-draft-n-max 12
C4	adaptive 3..12 + p-min	C3 + --spec-draft-p-min 0.75
C5	ngram-mod	--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
C6	adaptive + ngram-mod	C3 args + C5 args
C7	fixed 3 + ngram-mod	C1 args + C5 args
C8	fixed 12 + p-min + ngram-mod	C2 args + C5 args

Results

config	reasoning	prose	code	recall
C0 baseline	30.03	30.08	30.03	29.93
C1 fixed 3	51.68	55.89	69.28	81.94
C2 fixed 12 + p-min 0.75	38.59	43.46	71.46	151.25
C3 adaptive 3..12	51.42	56.64	78.03	147.46
C4 adaptive + p-min 0.75	40.67	43.35	70.92	147.43
C5 ngram-mod	30.03	30.10	29.98	292.88
C6 adaptive + ngram-mod	51.48	56.61	73.06	321.25
C7 fixed 3 + ngram-mod	51.65	55.90	68.36	324.10
C8 fixed 12 + p-min + ngram-mod	41.34	43.73	64.39	325.68

Additional information

The total diff size is +519/-35

This PR touches on 15 files which seems large at first, but this mostly arises from integrating the new command line options. The bulk of the functional changes take place within speculative.cpp After the recent code-review (Ref: #27210 (comment) ), I rewrote the MTP vs Adaptive MTP invocations to keep the original MTP behaviour exactly the same as before, even if the code-path to get there has now changed. I also believe that I successfully argued for why the GDN changes should remain: Ref: #27210 (comment)

218 lines are for the unit test file. 92 lines are for the header file that contains the algorithm itself.

The rest of the line changes are the wiring of the algorithm into the llama.cpp code base, and the command line option handling.

I tried to keep the size of the functional changes to llama.cpp itself as small as possible, and I tried to keep this diff as independent as I could so that unless it's specifically activated, it won't interfere with normal llama.cpp operations.

"Horses for Courses"

While Adaptive MTP generally works well for most hardware and models, it can make performance worse on older hardware that cannot run the deep drafting and acceptance steps fast enough to realise the potential gains it offers. In these instances it is generally better to not enable Adaptive MTP and just stick with regular MTP with a low fixed depth.

Dense Models generally receive the most benefit, but gains on MoE models are generally weaker. For MoE models it's generally preferable to lower the maximum depth to no more than 8 and then assess. If the performance is no better than using a fixed MTP depth, then that's fine. Just use a fixed MTP depth instead.

Related Issues/PRs

I searched known issues and open PRs and did not find anything precisely mentioning this sort of feature at first.

@Us5rName pointed out in the comments below that they have a similar feature PR here: https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes. I designed the exact algorithm. AI chose the best place for it, and added it. I then went over it all, reviewed it, cleaned up and changed half of it to better reflect my intent, and then tested and refined the algorithm manually over the day.

[Us5rName]

Hi!

I've also created a fork with adaptive length for mtp and dflash with a rolling window based heuristic.

https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

and this PR #25726 with the groundwork.

Would you like to colab?

[theIvanR]

Excellent job! I would propose to merge this with my fix for a more robust datatype selection in MTP implementation for nvidia cards.

(bug)
#25713
(and solution)
#25680

[stew675]

Excellent job! I would propose to merge this with my fix for a more robust datatype selection in MTP implementation for nvidia cards.

(bug) #25713 (and solution) #25680

I corrected a different MTP issue in this commit to my personal fork's working branch here: stew675@b2655d3

I really should file a PR for that, but its exposure range is pretty narrow. My fix targets which custom kernel runs for F32-activation small batches. It fixes a corner case for the handling of BF16 KV caches though (most of the work in my branch there targets adding BF16 KV to the ROCm backend), and my code exposed that latent bug.

It appears there is a handful of inconsistencies in the mainline MTP code for certain corner cases and you have found another. I do see that your PR got closed. Did you ever work on a more targetted fix as per the reviewer's suggestion?

[stew675]

Hi!

I've also created a fork with adaptive length for mtp and dflash with a rolling window based heuristic.

https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window

and this PR #25726 with the groundwork.

Would you like to colab?

Thank you for bringing your PR up. I searched for similar PR's and I don't know why yours didn't show up.

I've looked over your branch and groundwork PR. There's a number of conceptual similarities, both being counting based heuristics. Both are sliding window approach, with yours being explicit, and mine being implicit. Both employ a form of "depth stickiness". Yours has a lot more knobs where I made a conscious effort to keep the knobs to a minimum. I do like your Bias knob though. That could be a genuinely useful knob for users to tweak for certain models to adjust the level of depth stickiness. The falling back approaches do differ significantly in concept. I spent quite a bit of time trying difference approaches, and then dialing in the one I settled on.

Your personal branch seems to target DFlash too, right?

I'm fine with us combining efforts, but we'd need to settle on what the UX approach should be. I'm a big fan of KISS. Years ago I would've had knobs for everything, but nowadays I'm a firm believer of use as few knobs as possible.

Do you have any performance comparisons of your implementation and how it affects both prose and code? I tried pretty hard to ensure that prose performance was affected as little as possible, so I'm curious to see results on how you solved that issue.

At the end of the day though, we both need one of the maintainers to offer some guidance on what they'd like to see. That will ultimately set the direction on what the next steps will be here.

Edited to correct my earlier statement of both being implicit sliding windows. That was my mistake earlier.

[simongonzalezdc]

Numbers from our side supporting adaptive draft depth, measured on a Strix Halo APU (gfx1151, Ryzen AI Max+ 395) serving Qwen3.8-27B with draft-mtp + ngram-mod speculative decoding (datapost: #27154):

At --spec-draft-n-max 12, draft acceptance splits sharply by workload class:

• code / low-reasoning traffic: ~0.96 acceptance. Deep drafts pay off; going from n-max 3 to n-max 12 was the difference between ~40 and ~60 tok/s class generation on this chip.
• heavy thinking/reasoning traffic: ~0.5 acceptance. Deep drafts mostly miss, so most drafted tokens become wasted verification work that adds per-token cost.

The second lane replicates independently on a different Strix Halo system: acceptance "way below 50%" under xhigh-reasoning coding traffic, with n-max 12 regressing generation (peak ~26 t/s, dips into the ~5-6 t/s range) versus lower n-max values — that user is now moving down to n-max 2 (comment: #27154 (comment)).

So a fixed n-max serves one lane and starves the other: deep enough for the ~0.96-acceptance lane, or shallow enough to cap wasted verification in the ~0.5-acceptance lane, but not both. Acceptance-driven draft depth is the knob that spans both, which is exactly what this PR adds.

[stew675]

I went over all the check failures and it appears that they are all pre-existing/known flaky tests unrelated to my code changes.

Now, I do have another commit that's almost ready which actually solves the 3% performance regression on hard prose, and actually boosts normal prose. The question I have is shall I wait for this PR to go ahead and file a followup, or shall I attach that change to this same PR. I'm fine either way.

[mndodd]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

[stew675]

I went over all the check failures and it appears that they are all pre-existing/known flaky tests unrelated to my code changes.
Now, I do have another commit that's almost ready which actually solves the 3% performance regression on hard prose, and actually boosts normal prose. The question I have is shall I wait for this PR to go ahead and file a followup, or shall I attach that change to this same PR. I'm fine either way.

If this is working, then you don’t really have many options.

Yeah, I'll cover it in a followup PR. That will give me more time to refine it further.

[nomandormosh]

Getting this error with plain draft-mtp:

 invalid adaptive draft range: n_min_adaptive=3, n_max=2 (n_min_adaptive must be in [1, n_max])                                                                                                

It occurs whenever the effective n_max < 3 (e.g. --spec-draft-n-max 2). The range check added in
speculative.cpp:1401 runs unconditionally in the ctor, but the ctor is shared by both draft-mtp and draft-mtp-adaptive, and n_min_adaptive (default 3) is only used in adaptive mode. To fix it, I moved the check inside the if (adaptive) block.

[stew675]

Getting this error with plain draft-mtp:

 invalid adaptive draft range: n_min_adaptive=3, n_max=2 (n_min_adaptive must be in [1, n_max])                                                                                                

It occurs whenever the effective n_max < 3 (e.g. --spec-draft-n-max 2). The range check added in speculative.cpp:1401 runs unconditionally in the ctor, but the ctor is shared by both draft-mtp and draft-mtp-adaptive, and n_min_adaptive (default 3) is only used in adaptive mode. To fix it, I moved the check inside the if (adaptive) block.

Thank you for catching this. I'll be pushing a fix shortly.

[stew675]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

You've provided both a broad range (8-16) and an ambiguous range of p-min > 0.5

I'm guessing you meant --spec-draft-p-min, and not --min-p?

Just to be clear, what's your ask here? For me to test all depths from 8-16 and vaguely a number of --spec-draft-p-min values as a matrix? Wouldn't that be highly dependent on what exactly it is you're doing? It's kind of the point of this PR that you don't need to be fiddling about trying to dial in the perfect depth and p-value each time.

It's also been my experience that raising --spec-draft-p-min, whether it be normal or adaptive MTP, absolutely destroys prose performance. With the rise of deep thinking models (like our good friend Qwen3.8 here) there's likely to be significant amounts of time spent generating prose before code gets written.

This is why I've been focusing on preventing/minimising harm to prose performance. We don't want trash that thinking performance just to get some more t/s on code generation and end up slower overall.

[mndodd]

Overview

Implements adaptive mtp with a new option --spec-type draft-mtp-adaptive

How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results.

You've provided both a broad range (8-16) and an ambiguous range of p-min > 0.5

I'm guessing you meant --spec-draft-p-min, and not --min-p?

Just to be clear, what's your ask here? For me to test all depths from 8-16 and vaguely a number of --spec-draft-p-min values as a matrix? Wouldn't that be highly dependent on what exactly it is you're doing? It's kind of the point of this PR that you don't need to be fiddling about trying to dial in the perfect depth and p-value each time.

No ask, just curious because I had your exact though this weekend "man, we need an adaptive n-max or something." and then found spec-draft-p-min and it gave me the sort of results I was looking for, with Qwen3.8.

It's also been my experience that raising --spec-draft-p-min, whether it be normal or adaptive MTP, absolutely destroys prose performance. With the rise of deep thinking models (like our good friend Qwen3.8 here) there's likely to be significant amounts of time spent generating prose before code gets written.

This is why I've been focusing on preventing/minimising harm to prose performance. We don't want trash that thinking performance just to get some more t/s on code generation and end up slower overall.

Your PR is on my list to test.

[bucknova]

Tested the fixed-depth equivalent of this on the target hardware/backend this PR doesn't cover yet — Intel Arc Pro B70 (BMG G31, 32GB), llama.cpp SYCL via LocalAI intel-sycl-f16-llama-cpp gallery build 2026-08-03, Qwen3.8-27B UD-Q4_K_XL, 256k ctx, q8_0 KV. Fixed draft-mtp depth sweep, 700-token decode, single stream:

depth	t/s
2	~29
3	32
4	31
6	24.2

Draft acceptance 75–90% (mean accepted length 2.5–4.7) across runs. Two observations that support the adaptive approach directly:

1. The optimum is sharp and shallow. Peak at 3, and 6 is a 25% loss — deeper drafting actively hurts on this backend. Your default floor of 3 is exactly where I landed by hand.

2. The penalty is dispatch-bound, so it's worse on SYCL than CUDA/ROCm. Each extra draft step pays the SYCL per-kernel dispatch cost (100–500µs vs ~5µs CUDA), so the "keep it at the floor unless acceptance earns it deeper" logic should pay off proportionally more here than on the platforms already benchmarked.

Happy to run the adaptive build against the same 700-token workload on this B70 once it's testable and post the side-by-side.

[Us5rName]

@stew675

Do you have any performance comparisons of your implementation and how it affects both prose and code? I tried pretty hard to ensure that prose performance was affected as little as possible, so I'm curious to see results on how you solved that issue.

I ran my heuristic against static length in speedbench with qwen 3.6 35B and mtp

roleplay category was used for prose
coding category was used for coding

*note: on my hardware (rtx 4080 super + rx 7900 xtx, both using vulkan backend), I noticed the maximum draft length that doesn't hurt my performance was 5, and the minimum draft length that didn't hurt it was 2.

draft length 2 was used for prose, and draft length 5 was used for coding.

got very similar results in both the non adaptive case and the adaptive case.

Speedbench, qwen 3.6 35B UD-Q4_K_XL, using mtp:

coding, draft length 5: 157.70 tps
coding, adaptive draft length 2-5: 163.19 tps

roleplay, draft length 2: 147.71 tps
roleplay, adaptive draft length 2-5: 149.08 tps

In my personal experience with my fork, the draft length of my heuristic usually stabilizes at 2 tokens for prose,
3-4 tokens in math,
and 4-5 tokens in code.

Tell me if there are additional files/logs you would like to see.

Your personal branch seems to target DFlash too, right?

Yes. By targeting Dflash I meant I copy pasted the code from my mtp implementation into Dflash - the heuristic works exactly the same in both cases.

In the code, Because the draft length update is checked on every accept(), which every draft spec implementation has, it can absolutely be ported to draft-simple and draft-eagle3, I don't use them so I didn't apply the heuristic to them.

I now also deduplicated code in my fork by using your approach of putting the heuristic in a header, and added a short explanation on the algorithm I used, so it should be easier to read and trivial to port to other draft implementations.

I looked at your code, and it seems like adding the same heuristic you implemented in mtp to Dflash (and other spec types) should be simple, and I think adding your heuristic to Dflash as well is a good experiment because even with Dflash generating all tokens at once, the target model can still try to verify "useless tokens" (In my experiments, my heuristic worked well on Dflash)

I'm fine with us combining efforts, but we'd need to settle on what the UX approach should be. I'm a big fan of KISS. Years ago I would've had knobs for everything, but nowadays I'm a firm believer of use as few knobs as possible.

I agree I that have too many knobs, and when I use my fork I always use --spec-adaptive-length-default (sets my knobs to params that worked well for me)

However, I still think that 1-3 knobs of configuration is still good to have, to let people experiment.

I think a good approach would be like with n-gram speculative decoding - configurable but includes a parameter that sets sane defaults (--spec-default).
What do you think?

[stew675]

@stew675 , small enhancement from my side:

diff --git a/common/speculative.cpp b/common/speculative.cpp
index 0c7fb38d9..459eea494 100644
--- a/common/speculative.cpp
+++ b/common/speculative.cpp
@@ -1457,6 +1457,12 @@ struct common_speculative_impl_draft_mtp : public common_speculative_impl {
             return;
         }

+        // new generation: the depth learned for the previous content is stale,
+        // so the controller starts from the floor again
+        if (adaptive) {
+            adaptive_ctrl[seq_id].reset(params.n_max, params.n_min_adaptive);
+        }
+
         auto * ctx_dft = this->params.ctx_dft;
         const llama_pos pos_max = llama_memory_seq_pos_max(llama_get_memory(ctx_dft), seq_id);

Its a fix for issue with adaptive which has depth from previous state... If you want just pick it up.... however, if you intend to use previous state and let adaptive decrease/increase N this will create unnecessary work

Ooh, good catch! Thank you for that. I see that another work-flow hasn't been approved yet, so now's a good time to merge that in.

[stew675]

Sorry, my mistake — I had --fit on when I tested earlier. With --fit off, the model loads without any issues. Thanks for the help!

All good. I'm just glad we got you sorted out in the end.

[simongonzalezdc]

Follow-up from our August 18 datapost — same Strix Halo box, but this time at the PR head (45e3d26) with the embedded-MTP draft and the adaptive controller, not ngram-mod. Qwen3.8-27B UD-Q4_K_XL from unsloth, -fit off, 16K context, q4_0 KV, temp 0.2, 300 max tokens. Five code and five prose prompts, two reps each — n=10 per cell. Throughput is the median wall-clock tok/s over those ten; acceptance is the share of drafted tokens the target model accepted.

• adaptive [3..12] — code 18.61 tok/s (61.7% accepted), prose 14.88 (41.9%)
• fixed n=4 — code 17.65 (53.5%), prose 13.98 (35.4%)
• fixed n=12 — code 17.72 (24.5%), prose 11.79 (13.2%)

Code is a wash on this run — the edge adaptive shows there is smaller than the run-to-run spread we see with this setup, so I'm not claiming anything from it.

Prose is where it shows: adaptive is ~6% faster than fixed-4 and accepts 6.5 points more of what it drafts.

The fixed-12 row is the one I keep looking at. 13.2% acceptance means most of those deep drafts were verification work that went nowhere, and on this chip drafts share memory bandwidth with the target so that work isn't free. This is the lane I called out in the August post — deep drafting into low-acceptance traffic — showing up in one run.

We ran -fit off throughout on Linux with this quant — zero load failures.

Happy to share the raw per-request jsonl, or run whatever config anyone wants on this box.

[Hudendudel]

Been using --spec-type draft-mtp-adaptive --spec-draft-n-max 12 for about 2 weeks now with Qwen3.8-27B-UD-Q4_K_XL on the R9700 on Linux/ROCm. It's one of the few methods that speed up both coding and prose in a single setting without sacrificing either. Getting roughly 34 tok/s on prose, 72 tok/s on code. As a comparison, before that i used a static n-max of 3 as a middle ground which gave about also 34 tok/s for prose yet 53 tok/s for code. Would be great to see this land in main. @CISC @ggerganov

[simongonzalezdc]

Follow-up from our August 18 datapost — same Strix Halo box, but this time at the PR head (45e3d26) with the embedded-MTP draft and the adaptive controller, not ngram-mod. Qwen3.8-27B UD-Q4_K_XL from unsloth, -fit off, 16K context, q4_0 KV, temp 0.2, 300 max tokens. Five code and five prose prompts, two reps each — n=10 per cell. Throughput is the median wall-clock tok/s over those ten; acceptance is the share of drafted tokens the target model accepted.

• adaptive [3..12] — code 18.61 tok/s (61.7% accepted), prose 14.88 (41.9%)
• fixed n=4 — code 17.65 (53.5%), prose 13.98 (35.4%)
• fixed n=12 — code 17.72 (24.5%), prose 11.79 (13.2%)

Code is a wash on this run — the edge adaptive shows there is smaller than the run-to-run spread we see with this setup, so I'm not claiming anything from it.

Prose is where it shows: adaptive is ~6% faster than fixed-4 and accepts 6.5 points more of what it drafts.

The fixed-12 row is the one I keep looking at. 13.2% acceptance means most of those deep drafts were verification work that went nowhere, and on this chip drafts share memory bandwidth with the target so that work isn't free. This is the lane I called out in the August post — deep drafting into low-acceptance traffic — showing up in one run.

We ran -fit off throughout on Linux with this quant — zero load failures.

Happy to share the raw per-request jsonl, or run whatever config anyone wants on this box.

Small precision note on my post above. Throughput is the median wall-clock tok/s across the n=10 runs; acceptance is the aggregate share of drafted tokens the target model accepted across those runs. The adaptive-vs-fixed-4 prose throughput delta is +6.4%. The code comparison is descriptive of this single run only; I should not have characterized it beyond that. Finally, the hardware explanation in that post was not measured and should not be treated as a result.

[JCraigWasTaken]

I tried this on two AMD MI50 cards (gfx906) running Qwen3.8-27B at 8-bit, split across both cards, on the mxxm gfx906 fork of llama.cpp (https://github.com/mxxm-t/mx-llama.cpp) at its b10811 build. Sampling at temperature 0.8, reasoning off, compared against the fixed 3-token draft.

Letting the draft length move between 2 and 6: a file-rewrite prompt went from 75.7 to 79.8 tok/s, fresh code went from 68-71 down to 66.3, and fresh prose from 42.4 down to 41.3, with the length changing 36 times over the run. Between 2 and 4: 79.4 / 69.8 / 41.5 on the same three prompts.

I think the loss on code and prose comes from how this backend caches its GPU work: it keeps a separate recorded version per draft length, every new length costs a few slow rounds to set up, and lengths that are only used now and then get their recorded version dropped as unstable. Both showed up in the logs during these runs.

The other change in this PR, starting the recurrent-state copy at the first slot a rollback can reach, helped on its own: with the fixed 3-token draft, prose went from 41.3 to 42.4 tok/s with it. It might be worth splitting that part out so it can go in separately.

Happy to rerun with other settings if that helps.

[stew675]

@JCraigWasTaken

Limiting the depth range from 2-6 isn't really expected to make a huge difference from, say, setting a fixed MTP depth of 3. There are always short bursts, even for predictable content, where the acceptance rates collapse for short periods, and it's in those periods that the algorithm needs enough of a buffer to ride over those short collapses.

When the algorithm is prevented from scaling to the higher depths where the biggest speedups are found for predictable content, then those short periods of collapsed acceptance means that it will fall back to assuming that it's working with prose again far too quickly, and so it's not being given the chance to ride those sweet deep predictive highs for long enough for it to provide a real speedup.

It should be noted that for llama.cpp, adjusting the depth does come with a small temporary performance hit, and the algorithm aims to amortise that cost through sustaining the high depths long enough to hide it. When the peak depth is set too low, it cannot do that. In my belief therefore that the algorithm working given the constraints you've set for it to work under.

If, for memory reasons, that you'd like to reduce the upper range, then 8 would be about the lower end of where that should be. As the max-gain is dropped much below 8, then the algorithm essentially collapses back to not being a whole lot different to simply running a fixed depth of 2, 3, or 4.

In a car analogy sense, you've basically put a speed governor on a Ferrari that limits it from going faster than the regular speed limit, and are then wondering why the car can't reach its destination any faster than a Toyota.

I would guess, if anything, what probably needs to change here is to have the adaptive MTP code emit a warning at startup that if the maximum depth has been set to a value of 6 or lower that it's probably best to just stick with normal fixed MTP instead. Even a max depth of 7 is going to be kind of borderline. A maximum depth of 8 is the lowest I would set the maximum at to be sure of starting to see what the algorithm aims to deliver.

I'll add a note to the PR description to point this out.

[JCraigWasTaken]

Thanks, that is a fair point about the cap. I reran with the ranges you suggested, same setup as before: two AMD MI50 cards, Qwen3.8-27B at 8-bit split across both, the mxxm gfx906 fork at b10811, temperature 0.8, reasoning off. Three prompts (a file rewrite, fresh code, fresh prose), code and prose twice each. Decode speed in tokens per second; the fixed 3-token draft was run twice as a control.

	file rewrite	fresh code	fresh prose
fixed 3 (first run)	76.1	70.4	43.3
adaptive 2..8	64.3	50.3	42.7
adaptive 3..12	77.6	48.1	42.8
fixed 3 (second run)	75.8	70.3	41.7

The controller did what you describe. On code it climbed to depth 9 to 12 and stayed there for most of the run (58 depth changes, 41 of them between 9 and 12). Acceptance per position held up out to the deep positions, for example 0.93, 0.87, 0.79, 0.64, 0.51, 0.41, 0.36, 0.29, 0.21, 0.14, 0.09, 0.05 on one code prompt, and the mean accepted length per round went from 3.7 tokens to about 6.3.

It still lost 30 percent on code because each round got much more expensive. From the server's own timings, one round at fixed depth 3 took about 53 ms, of which 9 ms was drafting. At depth 12 a round took about 131 ms, of which 24 ms was drafting. So verifying 13 tokens cost about 2.5 times as much as verifying 4 on this hardware, while the round produced 1.7 times as many tokens. I since measured the forward-pass cost against batch size directly on this build: 31 ms at 1 token, 42 at 4, 69 at 8, then a jump to 107 at 9 and flat to 32, which is the backend's kernel switching from a narrow mat-vec to a 32-column tile. Details in mxxm-t#10. On a card where a 13-token batch is still bandwidth-bound that cost would not show, which would explain why your numbers and ours differ.

Prose was a wash at every setting. Acceptance at position 1 is under 50 percent on prose here, so the controller sat at the floor. The file rewrite gained 2 percent at 3..12 and lost 15 percent at 2..8, where the depth moved 45 times over the run.

I agree a startup warning for low caps makes sense. It may also be worth capping on measured round time rather than on depth alone, since on compute-limited cards the best depth is low even when acceptance is high. Raw logs are available if useful.

[stew675]

@JCraigWasTaken

At this stage it sounds like the algorithm just isn't a good fit for your hardware. The cost of the drafting at high depth is high enough that it offers no gains for you. In your scenario, it may just be best to stick with the normal MTP use case. It's interesting because yours is the first scenario reported where it hasn't provided the proposed gains. Can't win 'em all I guess, and this is EXACTLY why I made this feature an explicitly optional configuration.

[Jaxx7594]

Hello, is it normal that Qwen 3.8 27b reasoning speed has tanked by ~5-10tk/s, whilst code creation has been boosted by 10-20? Using the recommended settings; spec-draft-n-max=12. Using a 7800XT with a custom llama.cpp build optimised for the GPU. Vulkan inference.

[stew675]

Hello, is it normal that Qwen 3.8 27b reasoning speed has tanked by ~5-10tk/s, whilst code creation has been boosted by 10-20? Using the recommended settings; spec-draft-n-max=12. Using a 7800XT with a custom llama.cpp build optimised for the GPU. Vulkan inference.

On that GPU with that model, that is not even remotely normal. Something is wrong, but there's not enough information to say more than that.

[Jaxx7594]

Hello, is it normal that Qwen 3.8 27b reasoning speed has tanked by ~5-10tk/s, whilst code creation has been boosted by 10-20? Using the recommended settings; spec-draft-n-max=12. Using a 7800XT with a custom llama.cpp build optimised for the GPU. Vulkan inference.

On that GPU with that model, that is not even remotely normal. Something is wrong, but there's not enough information to say more than that.

Here is my preset.ini, if that helps:
[*]
device = Vulkan0
flash-attn = on
no-mmproj-offload = true
cache-type-k = q5_1
cache-type-v = q5_1
parallel = 1
load-on-startup = false
sleep-idle-seconds = 300
ngl = 999
batch-size = 2048
ubatch-size = 1024
reasoning-preserve = true
cache-prompt = true
kv-unified = 1
context-shift = true
load-mode = mlock

[qwen3.8-27B-Q2-MTP]
fit = off
model = ./Qwen3.8-MTP/Q2_K_XL/Qwen3.8-27B-UD-Q2_K_XL.gguf
model-draft = ./Qwen3.8-MTP/MTP/mtp-Qwen3.8-27B-Q4_0.gguf
mmproj = ./Qwen3.8-MTP/mmproj/mmproj-BF16.gguf
ctx-size = 120000
spec-type = draft-mtp-adaptive
spec-draft-n-max = 12
temperature = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0

and the command:
llama-server --models-preset ./preset.ini --models-max 1 --port 9080 --host 0.0.0.0

[Jaxx7594]

Apologies for providing so little information initially.. rereading it now, it was a rather useless message in terms of debugging value 😅

[Jaxx7594]

Generally, with 2 heads and standard mtp, on long tasks the tk/s will average at about 50 in low context lengths (sub 30k), and degrade from there until about 30tk/s at ~100k tokens. With adaptive, it sometimes hits 70 with coding tasks, but in thinking processes will be pinned at ~30. Drops by ~10 in both types as it approaches ~100k context length.

[Jaxx7594]

My build is slightly unique due to it cherry picking the commits from this PR automatically onto the main branch, so I can get the latest features as well as adaptive MTP. Theres potential that may be the issue. I will test on a build of this PR tomorrow.

[satoyon]

Jaxx7594

Since Qwen3.8-27B tends to generate longer reasoning chains, the default spec-draft-n-min-adaptive value (3) might be too restrictive, causing draft acceptance penalties when transitioning to reasoning mode.
You could try lowering spec-draft-n-min-adaptive to 2 or even 1 to see if it helps mitigate this issue.

[pwilkin]

/bot review

[ggml-gh-bot]

Automated code review

Review of PR #27210: adaptive MTP draft depth

I reviewed the diff plus surrounding code in common/speculative.cpp, tools/server/server-context.cpp, src/llama-memory-recurrent.cpp, and src/models/delta-net-base.cpp. Overall the feature is well integrated (priority list, type maps, static_assert, need_n_rs_seq, server gating all updated), the new state machine is simple and well unit-tested, and concrete perf data is provided. Findings below.

Blocking

(point 1) Undisclosed change in src/models/delta-net-base.cpp:501 - the build_conv_state loop skip is not mentioned anywhere in the PR description ("the main functional changes take place within a single file, speculative.cpp"), yet it changes behavior for every delta-net user with n_rs_seq > 0, not just adaptive MTP. It also relies on a nontrivial invariant: a rollback never rewinds past the start of the seq's last ubatch (rollback <= ubatch.n_seq_tokens - 1). Note the old code wrote the pre-ubatch state into the skipped slots (via the s_idx = max(0, ...) clamp), while the new code leaves whatever stale data was there from a previous round. llama_memory_recurrent::seq_rm only bounds the rollback by n_rs_seq, not by the last ubatch size, so the invariant holds only by convention among callers. The fused GDN op already makes the same assumption ("writes the last min(n_seq_tokens, K) snapshots"), which supports the change, but this needs to be either split into its own PR with its own justification and testing, or explicitly documented in the description together with an argument for why the invariant holds for every seq_rm caller.

(point 2) Stale feedback to the controller on server replay rounds - accept() (common/speculative.cpp:1811) feeds n_last[seq_id], which is only updated inside draft(). On the server checkpoint path (server-context.cpp:3914, use_ckpt_tgt for COMMON_CONTEXT_SEQ_RM_TYPE_FULL memories - i.e. exactly the recurrent models), a partially-rejected round returns before common_speculative_accept, sets spec_draft = accepted and spec_is_replay = true; the next round reuses that draft without calling impl->draft(), yet common_speculative_accept is still called (server-context.cpp:3940) with is_other == false for this impl. The controller then gets update(stale n_last, replay n_accepted), accumulating bogus drop pressure (the replayed draft is shorter than the original, so it registers as a miss). This corrupts the depth for recurrent models, which appear to be untested here. Fix: track whether this impl actually produced the draft for the round being accepted (e.g. clear n_last or set a flag when the round reuses a cached draft) and skip the update otherwise.

Will slow the review

(point 3) Missing docs - docs/speculative.md (spec-type list at ~line 227, options block at ~line 244, type table at ~line 369) and tools/server/README.md (spec-type values at ~line 273, options table at ~line 263) are not updated for draft-mtp-adaptive and --spec-draft-n-min-adaptive.

(point 4) Config validation is late and fatal - --spec-draft-n-min-adaptive (common/arg.cpp:4190) accepts any int, including negative values; the failure only surfaces as GGML_ABORT deep in the impl constructor (common/speculative.cpp:1452/1455). Reject value < 1 in the arg handler like --spec-draft-n-max does, and prefer throw std::runtime_error over GGML_ABORT for user-facing config, consistent with the sibling draft impls.

(point 5) New file under tests/ - per project rules, adding a new test file requires maintainer approval. The test itself is small, model-free, and reasonable; just flag it for sign-off.

(point 6) Stale comment in tests/test-arg-parser.cpp:278 - "the adaptive floor defaults to 2" but the assert checks 3 (the actual default). Fix the comment.

(point 7) Copy-pasted type check - the "is MTP or MTP_ADAPTIVE in types" std::find expression is now duplicated in five places (common/arg.cpp:366, common/common.cpp:1714, common/speculative.cpp:2590, tools/server/server-context.cpp:1022, plus need_n_rs_seq). Add a small helper on common_params_speculative (e.g. has_mtp(), next to the existing has_dft()) instead.

Nits

(point 8) common/arg.cpp:1305 - unrelated added blank line; drop it.

(point 9) common/speculative.cpp:1500 - in begin() the adaptive reset sits after if (N <= 0) return;, so an empty prompt would skip the reset. Move the reset above the early return.

(point 10) --spec-draft-n-min is silently ignored in adaptive mode (common/speculative.cpp:1794); log a warning if the user set it.

(point 11) common/speculative-adaptive.h:6-29 - the 30-line header comment is long, partially duplicates the PR description, and embeds CLI flag names in a library header. Trim to the core mechanism; keep the constant table rationale in climb_threshold/drop_pressure where it already is.

(point 12) The new option is registered between --spec-synth-rates and --spec-draft-p-split; consider placing it next to --spec-draft-n-min (~common/arg.cpp:4150) and matching the example set of its neighbors (LLAMA_EXAMPLE_LOOKUP is missing).

(point 13) --spec-type draft-mtp,draft-mtp-adaptive would construct two MTP impls sharing one ctx_dft (double process() decode). The hazard is pre-existing for other dft-based pairs, but the new sibling type makes the combo more likely - consider rejecting it explicitly.

The controller itself (common/speculative-adaptive.h) looks correct: the climb/drop arithmetic matches the unit tests, the is_other guard correctly excludes rounds drafted by other speculators, and the dp.n_max clamp plus the n_draft <= 0 early-out handle the truncated-draft and empty-draft cases. The main open items are (point 1) and (point 2).

This review was generated automatically by pi coding agent using zai-org/GLM-5.3. It may contain mistakes. Maintainers make the final call.

[Jaxx7594]

Jaxx7594

Since Qwen3.8-27B tends to generate longer reasoning chains, the default spec-draft-n-min-adaptive value (3) might be too restrictive, causing draft acceptance penalties when transitioning to reasoning mode. You could try lowering spec-draft-n-min-adaptive to 2 or even 1 to see if it helps mitigate this issue.

Thank you, that largely fixed the issue. Reasoning perf is approx the same as it was before using adaptive, whilst coding perf is far higher than baseline. For anyone else with the same issue/same environment, a minimum of 1 is ideal, at least for my specific environment.

[stew675]

@pwilkin

Thank you for kicking off the PR review. I believe I've addressed all the points raised. I'll copy/paste my commit message here which is ~60% my own words (some AI assist). Please let me know if anything else needs addressing.

---

Original PR Review here -> #27210 (comment)

Addressed all nits and items marked as review slowers, except the new test file (tests/test-speculative-adaptive.cpp), which needs maintainer sign-off.

For point 1 (delta-net): the change only skips conv-state snapshot writes that no legal rewind can reach.
Short verify batches are already routine upstream - for example --spec-draft-p-min truncation, a small --spec-draft-n-max, and the server's context clamp can all produce them.

The fused GDN path has written only the last min(n_seq_tokens, K) recurrent-state snapshots (leaving the rest stale) since before this PR, and that path is the default for the qwen35 family - it already depends on rewinds never leaving the seq's last ubatch. Verify batches always begin with the seq's last committed token, so a rewind can only remove drafted tokens that sit inside that batch.

This change makes the conv-state writer consistent with that existing convention. It is therefore my position that this PR builds upon established behaviour, and there is no known code path or scenario where the invariant is broken, even on the delta-net (GDN) models this change affects.

I implemented a mechanism to prevent stale accept depth feedback reaching the adaptive controller (point 2): a draft round whose partial acceptance cannot be applied to the context is reported once with the true accept count via common_speculative_accept_partial, and the checkpoint-replay round that follows no longer feeds the controller with the previous round's draft count. The mechanism is scoped so the non-adaptive MTP accept path behaves exactly
as before.

The controller was re-tested on delta-net hardware after the change (C3/C6 on Qwen3.8-27B, one repeat): throughput and acceptance match the previous measurements within run-to-run noise.