llama: GPU-resident LRU cache for host-offloaded MoE expert weights

[csantiago78]

Summary

GPU-resident LRU cache for MoE expert weights that live in host memory (via -ot ...exps=CPU, -ncmoe, etc.). Decode on a host-offloaded MoE layer is bound by host RAM bandwidth: every token streams the routed experts' weights from system RAM. This PR serves the recently used experts from VRAM instead.

Opt-in via --moe-expert-cache N (slots per host-resident expert layer; --moe-expert-cache-inserts caps uploads per layer per decode step). Fully inert when disabled.

Motivation / measurements

Measured on Qwen3.8-Flash-Next UD-Q4_K_XL (512 experts, 10 routed/token, 28 expert layers pinned to host by -ot), 2x RTX 3090 + dual-Xeon host (single populated RAM channel per socket):

• Routing was instrumented over a 54k-record mixed workload (code / math / prose / multilingual):
  • No exploitable static skew: a top-32 "hot expert" list learned on half the workload covers only ~10% of the other half (uniform = 6.2%). Static pinning of experts is a dead end.
  • Strong temporal locality: a per-layer LRU-64 would hit ~67%, LRU-128 ~81%. This is what the cache exploits.
• End-to-end decode: 18.4 -> 24.2 tok/s (+31%) with 48 slots/layer (~4.1 GiB VRAM) at 2 uploads/layer/step.
• Overhead when enabled but cold is ~zero (measured with uploads disabled).

Mechanism (no new CUDA kernels)

• Per cached layer: companion tensors [ne0, ne1, K+1] for up/gate/down in the device buffer of that layer's router. Slot K is permanently zero (the "dummy" slot).
• An I32 expert id -> slot table per layer, two copies:
  • device copy: ggml_get_rows remaps selected_experts into slot ids for a second mul_mat_id chain over the cache tensors. Uncached ids map to the zero slot and contribute exactly 0.
  • host copy: passed as src[3] to the CPU mul_mat_id, which skips cached ids and zeroes their dst rows.
• The two down-projection outputs are summed - the split is exact by construction (each expert is computed on exactly one side).
• Decode-only (n_tokens == 1); batches/prefill build the stock graph, so batch offload is untouched.
• Updates are throttled and asynchronous: evictions are published at a decode-boundary sync point, slices are copied by a worker thread via ggml_backend_tensor_set, and the new mapping is only published at a later sync point after the copy completed - a running graph can never read a torn slot. (Synchronous uploads were measured to eat the entire win.)

Known discussion points (why this is a draft)

• The cache is a per-process singleton keyed by the up_exps tensor pointer, because build_moe_ffn has no model access. Happy to rework the ownership (e.g. hang it off llama_model) per your preference.
• The CPU mul_mat_id observation callback (ggml_set_moe_obs_callback) is a ggml -> llama upcall; suggestions for a cleaner layering welcome.
• Only the separate gate/up + LLM_FFN_SILU path is wired; other MoE variants fall back to the stock graph.
• Multi-token decode (speculative/MTP) currently bypasses the cache (n_tokens == 1 guard); extending the remap to small batches is straightforward if the approach is acceptable.
• The routing-locality result should transfer to other MoE models (uniform aggregate usage but high temporal locality is what load-balancing losses + real text produce), but only Qwen3.8-Flash-Next has been measured.

Testing

• A/B benchmarks above; outputs coherent over thousands of tokens at 48-81% measured hit rates.
• test-arg-parser argument section passes (the URL/404 section fails in my sandbox for network reasons, unrelated).
• Will add a runtime smoke of the final CLI plumbing before marking ready for review.

[ggml-gh-bot]

Hi @csantiago78, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[csantiago78]

Runtime verification of the CLI plumbing (previously compile-checked only): built this branch with CUDA and ran llama-server with --moe-expert-cache 48 --moe-expert-cache-inserts 2 on Qwen3.8-Flash-Next UD-Q4_K_XL (2x RTX 3090, 28 expert layers host-resident via -ot). The cache correctly skips the memory-estimation pass, initializes on the real model, and decode runs at 22.7 tok/s vs ~18-19 tok/s with the cache disabled on the same layout — same speedup as the env-var version of the patch. Output coherent, clean finish_reason: stop.

[sdroege]

On Qwen3.8-Flash-Next UD-Q4_K_XL on a single R9700 with DDR5 (at 3600MT/s), this increases tg from 12.8 to 14.8 (+15%) in my testcase with --moe-expert-cache 48 --moe-expert-cache-inserts 2. Doesn't seem to have any negative impact on pp.

End-to-end decode: 18.4 -> 24.2 tok/s (+31%) with 48 slots/layer (~4.1 GiB VRAM) at 2 uploads/layer/step.

Is this additional VRAM usage considered by --fit? In my case, --fit leaves ~8GB of VRAM unused on this model currently (for whatever reason!), so it happens to fit just fine right now.

edit It is not considered during --fit

[sdroege]

Gave it some more testing.

• -c 131072 -ngl 999 -cmoe --load-mode mmap --tensor-read-lazy auto -ub 4096 -b 8192 --reasoning-effort medium --moe-expert-cache 0 --moe-expert-cache-inserts 0: 12.8t/s (baseline)
• --fit on --fit-ctx 131072 --fit-target 512 -ngl 999 -cmoe --load-mode mmap --tensor-read-lazy auto -ub 4096 -b 8192 --reasoning-effort medium --moe-expert-cache 0 --moe-expert-cache-inserts 0: 13.0t/s (auto-fitting random experts)
• -c 131072 -ngl 999 -cmoe --load-mode mmap --tensor-read-lazy auto -ub 4096 -b 8192 --reasoning-effort medium --moe-expert-cache 84 --moe-expert-cache-inserts 4: 18.0t/s (this uses exactly the available VRAM)

That's 40% faster than the baseline and --fit is basically useless.

[sdroege]

There's an interesting suboptimal behaviour though. When processing a bigger prompt, a lot of VRAM gets freed (32GB -> 22GB). Can be reproduced with my configuration above and llama-cli. When processing small prompts it's all fast, when e.g. pasting a lot of text or using /read on a bigger file, VRAM becomes a lot more empty and tg becomes slow.

I can't see anything relevant in the logs with --log-verbosity 6 that would explain this.

[sdroege]

I think the n_tokens == 1 here means that this will basically never trigger if there's a draft model. You'd (usually) validate N tokens per round and generate 1 in parallel.

[sdroege]

You actually mentioned that in the initial comment, nevermind

[sdroege]

In prefill/batch you could still update the LRU and once prefill is done, sort the prefill experts by frequency and upload the top N or the most recent ones. That improves cache-hit ratio a bit for me after bigger prefills (where otherwise the cache is mostly cold).

0001-moecache-warm-cache-from-prefill-expert-frequency.patch implements something like this if you want to take that as reference. It was mostly an experiment to see if it makes a bigger difference (it only makes a small difference but there's probably more fine-tuning possible).

[sdroege]

There's an interesting suboptimal behaviour though. When processing a bigger prompt, a lot of VRAM gets freed (32GB -> 22GB). Can be reproduced with my configuration above and llama-cli. When processing small prompts it's all fast, when e.g. pasting a lot of text or using /read on a bigger file, VRAM becomes a lot more empty and tg becomes slow.

That doesn't cause actual performance problems but the problem here is that with the MoE cache the expert weights are changing backends. Next time the graph needs to be re-allocated (big prompt), it might need less memory because of that, and then reallocation shrinks memory usage. And there's then repeated re-allocations happening regularly.

This can be avoided by only re-allocating if the new size doesn't fit in the previously allocated but that all doesn't seem ideal.

[sdroege]

AFAIU if uploading is constantly slower than decoding, at some point all slots are in flight and old experts are still uploaded while newer ones rarely end up in the cache, which should then reduce the hit rate considerably. Maybe this requires some other approach?

[sdroege]

Also related to this, by doing this you have different graphs (and number of nodes) between decode and prefill, and that can cause reallocs all the time. When doing the first decode with n_tokens==1 with a different topology, a realloc happens that uses the new small n_kv. Then over time the KV cache fills up, QSA inputs that are depending on n_kv grow above their reserved limits and then there's a realloc every 250-300 decode steps (which wouldn't happen if the prefill allocation was kept, for example).

[sirfyyn]

Some measurements on this, on Blackwell workstation cards with a 180B qwen4_exp model. I ran everything on the same file and the same card so the numbers can be compared to each other.

Setup

GPU	RTX PRO 4500 Blackwell, 32.6 GiB, sm_120
CPU / RAM	Ryzen 9950X3D, 128 GiB DDR5-6000
Model	unsloth/Qwen3.8-Flash-Next-GGUF, UD-Q4_K_XL, 104 GB, 176.94 B params
Context	262144

Throughput is token generation only (eval time … / 200 tokens), median of six sequential requests through llama-server. Mixing prompt processing into this inflates the number by ~75%, which cost me a day before I noticed.

Throughput

Configuration	t/s	n	range
stock 62acc89c, default settings	16.83	6	16.50–16.86
my build, default settings	21.95	6	21.79–22.11
my build, tuned settings	41.31	6	37.44–42.70

Tuned is --moe-expert-cache 96 --fit-target 18500 -ctk q8_0 -ctv q8_0 --cpu-strict 1.

The stock number sits inside the range already reported here (12.8 → 18.0 on a single R9700), which I think is the useful part: on default settings this machine is not faster than anyone else's. My build carries changes beyond this PR, so the middle row exists to keep those separate from the tuning.

Measured hit rate

I captured a routing trace during normal decode (GGML_MOE_LOG, 87,889 lines, 42 layers, top-k 10, N≈512) and simulated LRU over it:

Slots	Coverage	Hit rate	assumed in this thread
32	6%	69.3%	—
64	12%	81.5%	67%
96	19%	86.9%	—
128	25%	90.2%	81%
216	42%	95.6%	—
288	56%	97.9%	—
384	75%	98.5%	—
512	100%	98.5%	—

The cache does better than the LRU-64 ≈ 67% / LRU-128 ≈ 81% figures used here so far, and it saturates around 384 slots — the residual 1.5% are cold-start misses.

Batching does not reduce loads

Miss experts per token:

Slots	b=1	b=2	b=4	b=8	b=16
128	0.98	0.98	0.98	0.98	0.98
216	0.44	0.44	0.44	0.44	0.44
384	0.15	0.15	0.15	0.15	0.15

Consecutive tokens share 44.2% of their experts, so batching looks like it should help. It does not, because the cache already covers that reuse over a much longer window: reach_in_tokens = slots / (k · (1 − overlap)), i.e. 38.7 tokens at 216 slots against a batch of 4. Batching only helps while the batch exceeds the reach.

Two practical notes

Slot counts are a memory budget, not a model property. 144 slots fit my own 119 GB conversion of this model; only 96 fit the 104 GB UD-Q4_K_XL. My first tuned run died on exactly this and I spent a while blaming the wrong thing.

--spec-type draft-mtp cannot run on UD-Q4_K_XL:

llama_init_from_model: context type MTP requested but model doesn't contain MTP layers

That conversion does not carry the MTP head. My own conversion does, and MTP speculation is worth about +14% there. So the 41.31 above is the best I get on this file, not the best I get.

Caveat: q8_0 KV on this architecture needs LLAMA_ATTN_ROT_DISABLE=1, which disables the Hadamard rotation that makes quantised KV accurate. I have not measured the quality cost of that, so treat the KV quantisation part of the tuning accordingly.

[sissyhistorian-a11y]

Tested this PR at commit bccbacd on Windows with an AMD Radeon RX 7600 8GB using the Vulkan backend.

Hardware:

Ryzen 9 5900X
Radeon RX 7600 8GB
32GB DDR4-3200
Windows
Vulkan backend

Build:

GCC 15.2.0
Vulkan SDK 1.4.313.2

I used --cpu-moe with -ngl 99, and compared decode throughput with and without --moe-expert-cache.

Results:

GigaChat-20B-A3B Q4_K_M
cache OFF: 17.1 tok/s
26 slots: 20.4–20.6 tok/s
hit rate: 60.4%
cache allocation: 4622 MiB
improvement: ~20%
Qwen3-30B-A3B Q4_K_M derivative
cache OFF: 14.4 tok/s
32 slots: 16.5 tok/s
hit rate: 74.7%
cache allocation: 4316 MiB
improvement: ~14.6%

I also tested Qwen3-30B-A3B with a 16K context, q8_0 KV cache and Flash Attention:

16K / q8 KV / 32 expert-cache slots: 16.5 tok/s
KV cache: 816 MiB
expert-cache hit rate: 74.2%
~721 MiB VRAM remained free

So on this RX 7600 system, increasing the context from 4K to 16K did not reduce decode throughput in this configuration.

I also confirmed that the same expert-cache options work through the built llama-server, with /v1/chat/completions producing ~16.2 tok/s on the Qwen3 30B setup.

One build note: I needed two local Windows/toolchain workarounds:

guard flockfile / funlockfile in the diagnostic MoE logging code on MinGW
force the Vulkan shader generator to use one glslc worker because concurrent shader compilation was crashing on my system

Neither change touches the MoE expert-cache logic itself.

Overall, the cache appears to be working reliably on AMD RDNA3/Vulkan/Windows, and the speedup is significant on larger MoE models. Thanks for working on this. This did a great job.

[sissyhistorian-a11y]

I tested the n_tokens == 1 limitation mentioned above with an embedded-MTP Qwen3.6 MoE model and hit the expected issue: speculative validation builds small multi-token decode graphs, so the expert cache is bypassed.

The GGUF I tested uses the existing separate gate_exps / up_exps / down_exps path supported by this PR. This change does not add support for fused gate_up layouts or otherwise broaden the existing MoE-layout compatibility.

I tried extending the existing cache remap to small decode batches (n_tokens <= 4) by making selected_experts contiguous, flattening it to n_expert_used * n_tokens for ggml_get_rows(), then reshaping the resulting slot ids back to [n_expert_used, n_tokens].

The core change is:

ggml_tensor * mc_selected_experts = ggml_cont(ctx0, selected_experts);
mc_selected_experts = ggml_reshape_2d(
ctx0, mc_selected_experts, n_expert_used * n_tokens, 1);
mc_slot_ids = ggml_get_rows(
ctx0, mcache->dev_table, mc_selected_experts);
mc_slot_ids = ggml_reshape_2d(
ctx0, mc_slot_ids, n_expert_used, n_tokens);

together with changing the existing cache guard from n_tokens == 1 to n_tokens > 0 && n_tokens <= 4.

I built and runtime-tested this with Qwen3.6-35B-A3B using embedded MTP (--spec-type draft-mtp --spec-draft-n-max 2), and the cache path now runs successfully during MTP decode.

If useful, I can provide the complete small patch.

One additional data point: this wasn't only a runtime smoke test — combining MTP with the expert cache produced a substantial end-to-end speedup on this model.

On the same Qwen3.6-35B-A3B setup, I measured roughly:

plain decode, no MTP / no expert cache: ~17.1 tok/s
MTP (--spec-draft-n-max 2) without expert cache: ~23–24 tok/s
MTP2 + expert cache: ~30–32 tok/s

So being able to use the cache during the small validation batches is material here; the combined configuration is roughly 1.8–1.9x the plain baseline.

[Interpause]

anyone following this closer than me might want to check if there are similar ideas in #24528 or #28248

[Inovello]

Duplicate slot ids per token break the CUDA batched mul_mat_id path (n_tokens > 1)

It looks like every expert that hasn't been cached to VRAM has been mapped to the same dummy slot (n_slots), which means that a token's slot-id row can contain the same id multiple times. The CUDA batched kernels behind mul_mat_id assume that the ids of a token are going to be distinct, so the duplicates collapse into one entry. At batch 1 with quantized experts, this isn't shown because it uses a different kernel, mmvq, which simply asks for each output row which expert it needs and fetches it. Duplicates are harmless here

Reproduction without a model: with the help of an agent, I added dup_ids to test_mul_mat_id in tests/test-backend-ops.cpp (llama.cpp's built-in test program) that writes the highest expert index into every other id slot of each token. In short, since the test normally gives each token ten different expert numbers, the goal was to overwrite every second one with the highest expert index, which mimics what the cache does when it points every uncached expert of a token at the same dummy slot. On this tree (2x RTX 3090, CUDA 12.0, driver 595.84):

test-backend-ops -o MUL_MAT_ID -b CUDA0
  MUL_MAT_ID(type_a=f16,...,n=32,dup_ids=1): CUDA error: an illegal memory access was encountered

F16 cases fail already at n_tokens 1 to 8 (see the safe window section below: mmvq is not used on F16 for NVIDIA). Q4_K and Q6_K pass at n_tokens 1 to 8, and the run aborts at the first case that reaches mmq, the batched quantized kernel that takes over above 8 tokens.

For reference, here is where the assumption lives:

1. mmid.cu mm_ids_helper: lanes matching the same expert write the same store[] entry and the compact count advances once per token, so duplicates collapse to one compact row. In the dedup mode (ne11 == 1) the inverse map ids_src1 is left uninitialized for the losing lanes and the scatter quantizer writes to garbage rows. In both modes expert_bounds counts duplicates while the compact list does not, so ids_dst ranges contain uninitialized entries and write_back writes out of bounds.

2. mmf.cuh mul_mat_f<has_ids>: slot_map keeps one slot per (token, expert) and breaks at the first match, so the dst rows of duplicate slots are never written.

3. mmf.cuh mul_mat_f_switch_ids and mmq.cuh launch_mul_mat_q: grids are sized from n_tokens, but with duplicates one expert can hold up to n_tokens * n_expert_used compact rows.

4. The host sorting fallback in ggml-cuda.cu breaks after the first matching slot as well.

The safe window, measured: quantized experts take mmvq iff n_tokens <= min(MMVQ_MAX_BATCH_SIZE, get_mmvq_mmid_max_batch()), which is 8 for Q4_K, Q5_1 and Q6_K on cc 860 and on cc 1200. With the helper fixed, dup-id cases pass 6/6 at n_tokens 8 (three types, broadcast and not) and fail 18/18 at 9, 12 and 16. F16 and BF16 experts never take mmvq on NVIDIA.

The diff (gist link below) fixes the helper by:

1. Changing the way the sorter counts, so that if several lanes of one token match the same expert, each one gets its own entry in the compact list instead of the same slot being overwritten.

2. Changing the size of the sorter's shared memory store. It used to hold one entry per token which is enough when the ids of a token are distinct. The diff sizes it as n_tokens multiplied bt n_expert_used when that fits in smpbo (the maximum shared memory a block may use). If it doesn't fit, the code keeps the old size, with a comment stating that it then assumes distinct ids as before.

The diff also guards flockfile/funlockfile in ggml-cpu.c with #ifndef _WIN32 so the GGML_MOE_LOG debug logging builds on Windows. After the diff there is no CUDA error and every pre-existing MUL_MAT_ID case passes, but 24 dup-id cases still fail because of 2 (the F16 small-batch kernel) and 3 (the block-count sizing).

The duplicate cases that pass through those paths still produce wrong numbers; they just no longer crash and instead fail the CPU comparison. So the diff isn't a solution, but it is a step.

There are a few options I see right now that could be solutions:

1. You can keep the cache at one token. It is safe, but it rules out speculative decoding with the cache.

2. You could use the safe window as a gate for the cache: basically let it handle batches up to 8 tokens, only when the experts are quantized. This is what I run now; it makes n-gram drafts work on top of the cache (measured), and MTP verify batches of up to 8 should too. As verification, it's worth noting that a server at -ub 8 -b 8 is byte-identical to batch 1 on a 256-token greedy reference.

3. You can also stop producing duplicates outright by giving the cache n_expert_used dummy slots (on my tested model, Qwen3.8-Flash-Next, that would be 10) instead of one. This ensures the ids are distinct again and every kernel works unmodified. To be honest, this looks like the cleanest fix, but I may be wrong.

Gist link: https://gist.github.com/Inovello/9500167e5e8dc98fabe0df0c62ba3489

[ChangXiang-SCU]

Ran this on a config I don't think has been covered yet: Vulkan, two GPUs, Qwen3.8-Flash-Next UD-Q2_K_XL, on 2x RX 6950XT 16GB over OCuLink Gen4 x4 from a Ryzen 7 7840U handheld (64 GB RAM), Windows 11 / MSVC. It works well — thanks for building this. Reporting back with numbers, two small fixes, and two bugs I hit when I tried to extend it.

Results

Decode after cache warm-up, -sm layer -ngl 99 -ot exps=CPU, one server slot:

slots/layer	cache VRAM	decode	cumulative hit rate
0 (disabled)	0	8.2 tok/s	—
128	11 GB	12–13	~85 %
256	22 GB	17.4–17.8	92 %
280	24 GB	18.9–19.1	93 % (95 % instantaneous)
300	26 GB	9.5 (allocation spilled to host)	—

The cache slots do add up across cards, as the description says. At full 262144 context, -ub 16 is worth a lot: it cuts the compute buffer from 2.5 GB to 275 MB per card, which buys ~65 extra cache slots, and it's also faster at prefill than -ub 128 (18.2 vs 15.4 tok/s on a 957-token prompt) because a smaller ubatch has better locality when the experts are in host memory. Deployed config ends up 265 slots at 256k, 17.8–19.6 tok/s.

Two fixes

1. flockfile/funlockfile don't link under MSVC. The GGML_MOE_LOG block in ggml_compute_forward_mul_mat_id needs _lock_file/_unlock_file there:

#if defined(_MSC_VER)
    _lock_file(moe_log_file);
#else
    flockfile(moe_log_file);
#endif

2. Letting the cache serve small batches makes speculative decoding a win instead of a loss. With the n_tokens == 1 gate, MTP verification batches of 2–4 tokens fall back to computing every expert on the CPU, which on a weak host is slower than not drafting at all. Replacing the gate with an opt-in env limit (default 1, so behaviour is unchanged unless asked for) and fixing the id→slot lookup for n_tokens > 1:

static const int64_t moe_cache_max_tokens = []() {
    const char * e = getenv("LLAMA_MOE_CACHE_MAX_TOKENS");
    return e ? (int64_t) atoi(e) : (int64_t) 1;
}();
if (n_tokens >= 1 && n_tokens <= moe_cache_max_tokens && ...) {
    mcache = llama_moe_cache_lookup(up_exps);
}
if (mcache) {
    ggml_tensor * mc_tbl = mcache->dev_table;              // [1, n_expert]
    if (n_tokens > 1) {
        mc_tbl = ggml_repeat_4d(ctx0, mc_tbl, 1, mc_tbl->ne[1], n_tokens, 1);
    }
    mc_slot_ids = ggml_get_rows(ctx0, mc_tbl, selected_experts);
    mc_slot_ids = ggml_reshape_2d(ctx0, mc_slot_ids, n_expert_used, n_tokens);
    ...

(ggml_get_rows asserts a->ne[2] == b->ne[1], hence the broadcast.) With LLAMA_MOE_CACHE_MAX_TOKENS=4 and --spec-draft-n-max 3 on top of #28243: decode 18.9–19.1 → 21–24 tok/s, even after giving up ~50 cache slots to the draft head.

I verified this rather than assuming: greedy decoding (temperature 0, top_k 1, fixed seed) on four fixed prompts, comparing output hashes against the same build with LLAMA_MOE_CACHE_MAX_TOKENS=1. Batches of 2–4 are byte-identical to the reference on all four.

Both patches, with full diffs: https://github.com/ChangXiang-SCU/dual-egpu-moe-llm/tree/main/patches

Two bugs, if you want to take the cache further

Raising that limit past 16 produces wrong output — a short arithmetic prompt gets misread, longer prompts emit !!!!!!…. Same greedy hash method:

max batch through the cache	result
1	reference
2–4	byte-identical
16	correct, not byte-identical
32, 64, 128	corrupted

Two independent causes, as far as I could isolate them:

(a) The Vulkan topk-moe fusion misfires. ggml_vk_can_fuse_topk_moe identifies its pattern by fixed node offsets (cgraph->nodes[node_idx + 4] and similar), and the cache path inserts a repeat and a get_rows into that region. Running with GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 fixes the short prompts at batch 128, which is what convinced me. This may be worth a note even for the current single-token path if anyone adds nodes near there.

(b) The expert tables mutate mid-prefill. A long prompt spans many ubatches, and llama_moe_cache_step() between them publishes uploads queued during earlier decoding. The CPU chain reads host_table at compute time; the device chain reads dev_table when the GPU op runs. They can see different table versions, and then the partition between the two chains is neither exclusive nor exhaustive. GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 does not help here, and neither does LLAMA_GRAPH_REUSE_DISABLE=1. This looks like it needs the tables frozen or double-buffered for the duration of a multi-ubatch prefill, which is a change to the synchronisation design rather than a patch, so I stopped there.

Worth fixing if you're inclined, because the prize is large — with the cache serving prefill batches, prefill went 17.4 → 30.5 tok/s on a 3722-token prompt (+75 %, and the gain grows with prompt length). Prefill is by far the dominant cost in this setup, since 100 % of it currently runs on the host CPU.

Happy to re-run anything on this hardware if it's useful. Full measurements, correctness methodology and the Windows/Vulkan build notes are at https://github.com/ChangXiang-SCU/dual-egpu-moe-llm

[XBold]

Huge thank you to @csantiago78 and the entire llama.cpp community — this LRU expert cache is exactly the kind of optimization that makes CPU-offloaded MoE actually usable. The 31% throughput improvement (18.4 → 24.2 tok/s) on Qwen3.8-Flash-Next is impressive, and the fact that it uses a second mul_mat_id chain without new CUDA kernels is elegant engineering.

As someone following the broader NVMe/MoE offloading landscape, I wanted to highlight a convergence I'm seeing:

1. Complementary, not competing — PR llama : stream MoE routed experts from disk  #25294 (SSD expert streaming) and this PR solve different tiers of the same problem. llama : stream MoE routed experts from disk  #25294 handles models larger than RAM by streaming from disk; llama: GPU-resident LRU cache for host-offloaded MoE expert weights #27861 optimizes the GPU-resident cache for experts pinned to host RAM. Together they form a complete hierarchy: SSD → RAM → VRAM.

2. Unified eviction policy — Both this PR's LRU and llama : stream MoE routed experts from disk  #25294's slot cache use temporal locality. A shared abstraction (e.g., --moe-residency lru|lfu|readahead with configurable tier sizes) would let the engine apply the same policy across tiers rather than managing separate caches.

3. Expert-contiguous layout — The recent work on expert-contiguous GGUF re-layout (cutting page faults 36×) would benefit any implementation that reads experts from disk. If this becomes a standard export option, it would help every streaming implementation.

The community now has multiple independent implementations of the same core idea (MoE expert caching/streaming). At this point it feels less like competing approaches and more like different modules of a single system. Would love to see these converge.

Thank you for the thorough implementation and benchmarks. This work — and llama.cpp itself — is what keeps local AI accessible. 🙏

[nasone32]

I think there are a lot of optimizations that could be done here, so let me explain a little bit what I did.
i am working on a intensely optimized version explicitly for RDNA3 cards, so my build is different from current main, but the part touched by this patch is not, so I think what i'm writing below is still relevant.

i noticed speeds with MTP only and MTP + this PR and this PR alone, are not that different even when the cache hit rate is very high. this is when comparing an MTP-optimized launch (squeeze max into VRAM using --fit and a memory target) vs a cache-optimized launch (about 300+ experts of 512 in cache, to reach >90% hit rate )

System: qwen 3.8 next UD_Q3_K_XL, two AMD 7900 xtx cards, 96gb vram, ubuntu 24, rocm 7.14

I am using these prompts as test, since they produce a short reliable output
for code: -> "hello! write an example of bubblesort code in C"
for prose: -> "hello! i'm testing your ability on prose! explain how a PID controller works in ELI5 simple terms."

what I get is for CODE:
ECACHE+mtp 27 tk/s cold
ECACHE+mtp 39.3 tk/s hot (peak 46)
MTP ONLY: 38.08 tk/s

for PROSE:
ECACHE+mtp: 29.55 tk/s hot
MTP only: 25.62 tk/s

Yes Ecache is faster but based on simple math I expected MUCH higher honestly. SO after that I run some profiling with ROCPROFv3 and I found out:

Metric	MTP code	Cache hot code	MTP prose	Cache hot prose
Request GPU span	10.080 s	9.442 s	12.724 s	9.532 s
Kernel dispatches	398,803	441,804	482,458	516,619
Summed kernel time	3.270 s	3.865 s	4.253 s	4.096 s
mul_mat_vec_q_moe time	0.512 s	0.899 s	0.680 s	0.967 s
mul_mat_vec_q_ksplit time	1.048 s	1.109 s	1.386 s	1.138 s
Copy-buffer kernel time	0.141 s	0.321 s	0.181 s	0.261 s
H2D operations	2,179	21,346	2,640	8,441
H2D engine time	0.108 s	1.383 s	0.135 s	0.405 s
D2H operations	1,866	4,327	2,362	5,070
D2H engine time	0.080 s	0.105 s	0.111 s	0.118 s
hipStreamSynchronize calls	28,528	85,393	34,012	78,726
Host time in hipStreamSynchronize	3.496 s	5.942 s	4.426 s	4.733 s
hipLaunchKernel calls	329,439	359,653	398,442	424,360
Host time in hipLaunchKernel	1.247 s	1.204 s	1.561 s	1.792 s
hipMemcpyAsync calls	34,269	70,344	41,573	61,479

The cache hot code path therefore has:

• 2.99x as many stream synchronizations as MTP-only;
• 9.80x as many H2D operations and 12.8x as much H2D engine time;
• 1.76x as much mul_mat_vec_q_moe kernel time;
• 10.8% more kernel dispatches and 18.2% more summed GPU work.

and this is the result (written with AI help):

For every eligible small decode batch, llama-graph.cpp enables the cache graph. It then:

1. Maps selected expert IDs to real cache slots or zero-filled dummy slots at llama-graph.cpp:2121-2138.
2. Keeps the original host expert up/gate/down operations at llama-graph.cpp:2173-2304.
3. Builds another cache-side up/gate/down chain at llama-graph.cpp:2306-2333.
4. Adds the host and cache outputs at llama-graph.cpp:2335-2336.

The CPU MUL_MAT_ID implementation skips the expert dot products for hits, but it still performs setup, input conversion, route scanning, zero filling, barriers, and the host activation path.

The GPU cache path executes all selected routes. Cache misses are mapped to dummy zero experts, **but those dummy routes still consume GPU kernels. A higher hit rate therefore does not reduce cache-side GPU work proportionally. **

The static dual-backend graph also continues to copy activations, route IDs, or branch outputs across CPU and GPU boundaries when all selected experts hit.

So essentially there is still a LOT of optimization that can be done under the hood, because this expert PR still has a lot of overhead even when there is a high hit rate!

[sissyhistorian-a11y]

A follow-up from my Gemma4/Vulkan testing. I extended the #27861 cache path to Gemma4-26B-A4B and small MTP verification batches, and found a few implementation details that may be useful here.

Gemma4 support

Gemma4 uses fused gate_up_exps + GEGLU, so I added a fused cached path:

gate_up_g = ggml_mul_mat_id(cache_gate_up, input, slot_ids);
act_g     = ggml_geglu_split(...);
down_g    = ggml_mul_mat_id(cache_down, act_g, slot_ids);
experts   = ggml_add(cpu_experts, down_g);

Gemma4 also has per-expert down scales. With down_exps_s, build_lora_mm_id() returns a scale MUL around the underlying MUL_MAT_ID.

For the CPU cached-row skip, I attach the cache table to the inner MMID:

ggml_tensor * experts_mmid = experts;

if (down_exps_s) {
    experts_mmid = experts->src[0];
}

experts_mmid->src[3] = mcache->host_table;
experts_mmid->op_params[0] = mcache->n_slots;

This is currently my strongest candidate for the large difference between my early Gemma4 results and the current implementation. Early C24 runs had ~63% hit rate with almost no throughput gain; the current path gives a clear gain.

I still want to isolate this single change in a clean A/B before attributing the improvement to it. It looks worth checking on any MoE architecture where expert scaling wraps the raw MMID.

Small-batch / MTP support

I also extended cache use from single-token decode to small MTP verification batches.

The selected expert IDs are made contiguous, flattened for the device-table lookup, then reshaped back to [n_expert_used, n_tokens].

The most useful direct comparisons at 64K are:

Mode	Cache	Gen t/s	Gain vs same-mode EC OFF	Hit rate	Reported free VRAM
MTP off	OFF	16.40	reference	—	—
MTP off	C24	18.33	+11.7%	63.3%	—
MTP2	OFF	22.88	reference	—	4397 MiB
MTP2	C20	26.48	+15.7%	54.3%	763 MiB
MTP2	C24	27.35	+19.6%	59.8%	321 MiB
MTP2	C28	28.65	+25.3%	65.1%	0 MiB

These were balanced same-binary comparisons. For example, the four C24 MTP2 runs were 26.9–28.2 t/s versus 22.6–23.4 t/s for EC OFF.

A separate balanced 64K capacity sweep gave the following broader curve:

Cache	Gen t/s	Hit rate	Reported free VRAM
C8	21.93	28.0%	3383 MiB
C12	22.87	39.6%	2932 MiB
C16	24.33	47.9%	1204 MiB
C20	24.93	54.3%	755 MiB
C24	26.20	60.1%	302 MiB
C28	27.07	64.7%	0 MiB

Absolute throughput varies somewhat between benchmark sessions, while the capacity ranking is very consistent: more resident experts increase both hit rate and decode throughput until the GPU memory budget is exhausted.

Cache size / VRAM residency

The memory boundary itself seems important.

In a separate 16K diagnostic sweep, C24 and C32 were still in the fast regime, while increasing the cache further produced a large performance drop despite improving cache hit rate:

Cache	Gen t/s	Hit rate	Reported free VRAM
C24	24.70	59.8%	~870 MiB
C32	26.70	68.3%	0–28 MiB
C36	18.73	71.8%	0 MiB
C38	21.33	73.2%	0 MiB
C40	17.57	74.9%	0 MiB

Target and draft logical layer placement stayed unchanged at 30/30 and 4/4 GPU layers, and MTP acceptance stayed around 66–68%.

That makes the drop look like a physical residency/paging threshold rather than an expert-routing or MTP-acceptance effect.

For small-VRAM systems, slot count therefore behaves as a practical VRAM budget: higher hit rates help until cache residency starts competing with the rest of the inference workload.

Overall, the #27861 mechanism appears to extend well to Gemma4 + Vulkan + MTP. The two implementation details that stood out most were:

• following fused/scaled expert graphs down to the actual CPU MUL_MAT_ID when applying cached-row skips;
• allowing the cache to participate in small speculative verification batches rather than limiting it to n_tokens == 1.

I can clean these changes up into a smaller patch once the remaining scaled-MMID A/B is finished.

[mobilinkd]

Problem with this patch on SYCL:

Symptom: --moe-expert-cache N produced deterministic garbage at ANY cache size —
"The user is!!!!!" instead of "The user is asking..." — while cache 0 was correct. Reproduced
on both prompts, with and without reasoning mode (the France/reasoning-off case only looked
fine because it never hit the buggy path within 32 tokens).

Root cause: the SYCL in-place weight reorder. opt_for_reorder_id() reorders Q5_K/Q6_K MoE
expert tensors in place on first dispatch (reorder_qw_q5_k_moe etc.), but the moecache upload
worker keeps writing expert slices in the ORIGINAL per-expert layout afterward (upload_slice
at slotdst_c->nb[2]). So once the cache started serving experts (token ~4-5), it read
reordered-layout data = garbage. The main expert tensors never get written after reorder, so
they were fine — which is why this only bites the cache tensors.

Isolation: GGML_SYCL_ENABLE_OPT=0 made the bug disappear → reorder was the culprit. Also
confirmed SYCL Q5_K/Q6_K MUL_MAT_ID itself is correct (3/3 isolated test pass).

The cache tensors must either be excluded from reorder or the upload worker must write in reordered layout.

I think this is the same that @ChangXiang-SCU posted above.

[arthw]

Yes, it will be impacted by the reorder feature of SYCL.
Maybe call a callback function/API provided by SYCL backend.
The callback function will check and execute reorder the weight in the cache.
For other backends, skip to call it when it's NULL.