Misc. bug: When KV is in RAM the MTP draft KV cache is also in RAM

[jkSeria]

Name and Version

version: 0.3.0-dev (build 10675, commit 90c26fc)
built with MSVC 19.44.35228.0 for Windows AMD64

I've been able to reproduce on a7cc83b

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m Qwen3.8-27B-Q4_K_M.gguf -ngl 99 -c 131072 --no-kv-offload -ctk q8_0 -ctv q8_0 -fa on -ub 512 -b 2048 --spec-type draft-mtp --spec-draft-n-max 8

Problem description & steps to reproduce

When KV is in RAM the MTP draft KV cache is also kept in RAM, and copied to VRAM for every draft step. This causes a lot of redundant transfers causing a significant speed penalty. The MTP draft KV cache should just stay in VRAM instead of continuosly being copied. The same applies to a seperat draft model-s KV cache (--spec-draft-model).

Se also issue #24492

These are the measured differences when testing with Qwen 3.8 27b MTP n=8 with KV offloaded to RAM, before and after the change
(greedy, 200 tokens, fixed prompt. Numbers from the server timings object)

prompt depth	master	with fix	accept len (identical)
65,536	19.2 t/s ~ 399 ms/step	27.2 t/s ~ 281 ms/step	7.69
122,880	13.3 t/s ~ 621 ms/step	21.1 t/s ~ 393 ms/step	8.33

Tested on a system with an RTX 5090 and 5200 MT/s DDR5 RAM. Issue found when trying to optimize throughput for KV offladed (to RAM) with YARN for past native context for the model.

First Bad Commit

No response

Relevant log output

Logs

llama_kv_cache:        CPU KV buffer size = 17408.00 MiB   # target, intended
llama_kv_cache:        CPU KV buffer size =  2048.00 MiB   # MTP draft (the bug)

[jkSeria]

There might be an argument for adding a flag for this, and I have a draft of one. But from my point of view this should have been the default all along: the draft KV stays where the draft is computed. A flag could then move the draft KV to RAM for the people who need that.

The one case I found where #28117 has a cost is a separate draft model (--spec-draft-model). In my test (Gemma-4-E4B as a drafter for itself, 64k context) net VRAM went up by a few hundred MB, and that number grows with context and draft size. With the default -fit on this does not overflow, fit just quietly gives the target fewer GPU layers or less context. A flag would make that controllable for users on tight VRAM, while everyone else (MTP and separate draft models alike) gets the speed-up by default.

The params_dft = params_base copy dates to PR #10455 (Nov 2024), the MTP path reused it in PR #22673, and as far as I can tell offload was never discussed in either.

---

I should also mention that I have only tested this on my own setup.
I have no access to a unified-memory machine like a DGX Spark or a Mac.

---

A separate question. I have been working on a performance improvement that allocates the non-offloaded KV cache in the pinned (page-locked) host buffer instead of pageable memory. It roughly doubles decode speed at depth (340-480k) and adds 12-27% to prefill.

I was going to open an issue for it, but because of the one-open-PR rule for new contributors I cannot open the PR yet. What is best practice here? Should I close this PR, open the other issue, and submit one PR that fixes both? The two fixes stack, but they are separate issues in separate files.