Vulkan DeepSeek MLA+MoE load OOMs with an unactionable OutOfMemoryException instead of preflighting the ~57 GB host F32 expert dequant

[jamesburton]

Summary

Loading a DeepSeek-V2-family GGUF (MLA + MoE) on Vulkan dies with a bare, unactionable

System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
   at DotLLM.Models.Architectures.TransformerWeights.SliceExpertsToF32(...) TransformerWeights.cs:line 2291
   at DotLLM.Models.Architectures.TransformerWeights.LoadDeepSeekMoeLayer(...)
   at DotLLM.Models.Architectures.TransformerWeights.LoadMlaLayer(...)
   at DotLLM.Models.Architectures.TransformerWeights.LoadFromGguf(...)
   at DotLLM.Vulkan.VulkanTransformerModel.LoadFromGguf(...) VulkanTransformerModel.cs:line 1156

on a 128 GB box. Nothing in the message says why ~57 GB of host F32 was being allocated, or which tensors caused it. The footprint is knowable before a single byte is allocated — VulkanWeights.CanSkipMoeF32HostDequant already walks exactly the tensor descriptors that decide it.

Evidence — 2026-08-11 parity sweep

model	result	GGUF
deepseek-coder-v2-lite-q2_k	FAIL / OOM in 112 s	bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF Q2_K, 6.43 GB
deepseek-v2-lite-q4_k_m	FAIL / OOM in 144 s	mradermacher/DeepSeek-V2-Lite-GGUF Q4_K_M, 10.36 GB

Both reach the Vulkan load only after the CPU reference model has already loaded successfully (54.5 s / 105.4 s), i.e. two ~57 GB host F32 expert copies are attempted, since the CPU path (TransformerModel.LoadFromGguf → TransformerWeights.LoadFromGguf(gguf, config)) defaults skipF32MoeDequant: false. TransformerWeights.cs documents the size itself: "64 experts x 2048 hidden x 1408 intermediate x 3 mats x 4 bytes ~ 2.2 GB per layer x 26 MoE layers ~ 57 GB of F32 host RAM".

Root cause (verified against the actual GGUF tensor descriptors)

VulkanWeights.CanSkipMoeF32HostDequant returns false the moment any one routed bank on any MoE layer would fall back to F32, so every bank of every layer gets host-dequantised. MoeRoutedRawDeviceQuantType covers only Q8_0, Q4_K, Q5_K, Q6_K and (with coopmat) F16.

DeepSeek-V2-Lite's moe_intermediate_size is 1408, which is not a multiple of QK_K = 256, so llama.cpp is forced to quantise every ffn_down_exps bank to a non-K format. Census of the real files:

mradermacher/DeepSeek-V2-Lite-GGUF Q4_K_M (parsed with gguf.GGUFReader):

26 ffn_gate_exps.weight  Q4_K  (2048, 1408, 64)   <- supported
26 ffn_up_exps.weight    Q4_K  (2048, 1408, 64)   <- supported
14 ffn_down_exps.weight  Q5_0  (1408, 2048, 64)   <- NOT supported
12 ffn_down_exps.weight  Q8_0  (1408, 2048, 64)   <- supported

14 unsupported banks out of 78 force the other 64 to F32 too.

bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF Q2_K (header range-fetched and parsed):

26 ffn_gate_exps.weight  Q2_K    (2048, 1408, 64)  <- NOT supported
26 ffn_up_exps.weight    Q2_K    (2048, 1408, 64)  <- NOT supported
26 ffn_down_exps.weight  IQ4_NL  (1408, 2048, 64)  <- NOT supported

78/78 unsupported.

This issue is the diagnostics half

The capacity limit itself is real (two 57 GB copies do not fit in 128 GB, and would not fit for full DeepSeek-V2 on any box). Closing the coverage gap so the weights stay quantised on device is tracked separately in #327.

What is a bug here is that we discover it by exhausting the machine's RAM and reporting a stack frame instead of preflighting a number we can compute for free.

Proposed fix

• Extend the preflight to report why, not just whether: which layers/banks resolve to F32, their quant types, and the total F32 host footprint.
• In VulkanTransformerModel.LoadFromGguf (both overloads), when the F32 fallback is taken and the computed footprint exceeds available memory, throw a typed exception (InsufficientMemoryException) carrying that itemisation — before allocating.
• Gate on the footprint, not merely on the flag, so small MoE models that genuinely fit keep working unchanged.
• Have RealGgufVulkanParityTests convert that typed exception into Skip.If with the full message, matching its existing VRAM-exhaustion skip. A capacity limit is not a parity failure, and reporting it as FAIL misrepresents sweep results.

Acceptance criteria

• Both DeepSeek models produce an actionable message naming the offending banks/quants and the required host F32 bytes, instead of OutOfMemoryException.
• The error is raised before any large allocation (fast, not after 112 s of dequant).
• Unit test over the preflight itemisation using synthetic tensor descriptors (no GPU, no model file).
• Parity sweep reports these two models as SKIP-with-reason, not FAIL.

[jamesburton]

Correction to the framing above, from measurement. I wrote "on a 128 GB box" and "two ~57 GB copies". Both are wrong, and the real numbers make the finding sharper rather than softer:

GC.GetGCMemoryInfo() on this host
  TotalAvailableMemoryBytes = 31.6 GiB
  MemoryLoadBytes           =  0.0 GiB   (before any GC — it is a last-GC snapshot)
  MemoryLoadBytes           = 21.5 GiB   (after GC.Collect())

So the host has 31.6 GiB of RAM visible to the runtime, and the F32 fallback is 53.6 GiB — a single copy does not fit, let alone two. The CPU reference model's apparent success (54.5 s / 105.4 s) is the Windows pagefile absorbing the commit, not the allocation fitting.

That is why PR #339 bounds the preflight on total physical memory rather than free memory: exceeding total is an impossibility that no amount of freeing fixes, so refusing is safe, whereas exceeding free memory is survivable via pagefile and refusing there would break loads that currently work.

Also note MemoryLoadBytes reading 0 before the first GC — a free-memory rule would have been silently inert depending on the caller's GC history.

[jamesburton]

Correction to my own correction — "this host has 31.6 GiB of RAM" conflated installed with OS-visible. Measured both:

Win32_PhysicalMemory capacity sum (installed)      = 128.0 GiB
Win32_ComputerSystem.TotalPhysicalMemory (visible) =  31.6 GiB
GC.GetGCMemoryInfo().TotalAvailableMemoryBytes     =  31.6 GiB

The box really does have 128 GB; ~96 GB of it is the BIOS UMA frame-buffer carve-out for the Radeon 8060S, so only 31.6 GiB is visible to the OS and to any host allocation. The conclusion is unchanged — a 53.6 GiB host F32 fallback cannot fit in 31.6 GiB, and TotalAvailableMemoryBytes is exactly the right bound for PR #339 to use — but the remedy line is richer than "run on a host with more RAM": on this box, reducing the BIOS GPU carve-out would also fix it.

[jamesburton]

Likely resolved as a side effect of PR #426

PR #426 (issue #407) adds moe_indexed_matmul_q5_0_f32 and moe_indexed_matmul_iq4_nl_f32 Vulkan shaders plus the residency wiring, so routed MoE expert banks of those types stay packed on device instead of expanding to F32.

DeepSeek-V2-Lite Q4_K_M is the worked example in #344: moe_intermediate_size = 1408, which is not a multiple of QK_K = 256, so llama.cpp emits 14 of 26 ffn_down_exps banks as Q5_0 — and dotLLM expanded all of them.

Verified independently during review of #426 (not taken from the PR body):

• 1408 % 32 == 0, so the Q5_0 block constraint is satisfied and the resolver returns Q5_0 for those banks.
• 64 experts x 4 B x 1408 x 2048 = 738 MB per bank, x 14 = 9.6 GiB — matching the host-F32 blow-up recorded here.

Both the arithmetic and the mechanism check out, so this should no longer occur once #426 lands.

Please confirm before closing

This was not verified against a real DeepSeek-V2-Lite GGUF — no copy was available locally during the review, and #426's own primary evidence is real-GGUF block parity on SmolLM2-135M pure-Q5_0/IQ4_NL fixtures plus synthetic MoE banks. The reasoning is sound and the numbers reconcile, but an actual load of DeepSeek-V2-Lite Q4_K_M showing zero host-F32 expansion is what should close this.

Note also a related finding from that review, which may matter here: PlanMoeF32HostDequant and ResolveMoeBankResidency (VulkanWeights.cs:2338, 2445) both early-return unless config.MlaConfig is not null — so the residency-report evidence is structurally MLA/DeepSeek-shaped. That is fine for this issue (DeepSeek is MLA), but it means the same evidence cannot speak for non-MLA MoE models.

Related: #344 (umbrella), #407, #426.