Title
RTX PRO 6000 Single-GPU DeepSeek-V4-Flash: 75–80 tok/s with 260K context, 100% planes in VRAM, 120s load
Body
Test date
August 6, 2026
Tested configuration
1x NVIDIA RTX PRO 6000 Blackwell (96 GB), DeepSeek-V4-Flash-0731-NVFP4, vLLM-MoET (W2 base + FP4 delta tier), all 44 served W2 layers resident in GPU VRAM, no offload. Decode: 75–80 tok/s. Load time: ~100 s (down from 300+ s).
Problem 1: VRAM headroom — the plane set barely fits
On RTX PRO 6000 with 96 GB VRAM, the W2 planes for all 44 served layers (~72+ GiB) plus dense weights (~19 GiB) plus KV and the delta tier sit right at the edge of the card: 96 GB is an awkward budget, right at the threshold. Before this work, the practical choices were either offload the plane layers to host memory, or run with a much smaller context so the whole set fits. Both cost speed or context — and that was the real problem.
So the question I kept coming back to was: how do I free VRAM without giving up quality?** The answer is to keep all planes resident in VRAM, no offload. With CUDA graphs only capture 4 sizes and speculative decoding disabled , the full plane set now fits on 96 GB. Decode improves from ~50 tok/s to 75–80 tok/s at 260K configured context. Notably, this is faster than running with DSpark or MTP speculative decoding: those consume so much VRAM that a large fraction of layers has to be offloaded to host memory, and the offload overhead costs more than speculative tokens save.
Problem 2: Load time — painful daily starts
The original load took close to 5 minutes (300+ s), which is painful for daily use. Once the planes cache and the delta cache are fully built, the loader should not need to re-read fp13/fp2 on every layer — each layer carries roughly 3 GiB of fp13/fp2, and reading them every boot is ~200 GB of pointless I/O. Skipping those reads when the delta pack is verified complete is a huge speedup.
To go further, I had the agent write a concurrent cache-backed loader: the load runs 4 workers on 12 cache-backed layers per batch, with per-batch synchronize + gc.collect + empty_cache. Since the final 7–8 layers don't have enough cache-backed layers to fill a batch, they are loaded serially. Combined with skipping already-cached fp13/fp2 reads, the loading goes from 300+ s down to roughly 100 s.
Problem 3: The model goes off the rails — gibberish output, meaningless symbols, and hard loops
The model leaked stray content into the response body and failed tool-call invocations; the root cause was the vLLM reasoning parser and tool-call parser. After fixing, hard loops and gibberish largely disappeared.
MoET and serving settings
- GPU-memory utilization: 0.96
- Model runner: V1
- Tensor parallelism: 1
- Speculative decoding: disabled
- Reasoning mode: deepseek_v4 parser, thinking enabled, reasoning_effort max
- KV cache: DeepSeek fp8_ds_mla, block size 256
- Maximum model length: 260,000 tokens
- Batch size: 12 layers parallel, 4 workers
- Delta tier: ~4.5 GB, full layer coverage verified
Performance
- Decode: 75–80 tok/s (up from ~50 tok/s)
- Load time: ~100 s (down from 300+ s)
- Skipped fp13/fp2 reads: ~200 GB
Remaining bottleneck
At 260K context the card is essentially maxed out in VRAM: concurrency is limited, and there is no room left to enable speculative decoding. If there were a way to compress the VRAM footprint further (less per-layer bytes in the delta tier or base planes), it would open room for more concurrent requests or speculative decoding.
Many thanks to the brilliant author @kacper for the design, and to @jpezzulli for the exploration.
My code is a bit of a mess, so rather than trying to diff and patch everything again, I'm sharing the approach here — if you're interested, give it a try.
Title
RTX PRO 6000 Single-GPU DeepSeek-V4-Flash: 75–80 tok/s with 260K context, 100% planes in VRAM, 120s load
Body
Test date
August 6, 2026
Tested configuration
1x NVIDIA RTX PRO 6000 Blackwell (96 GB), DeepSeek-V4-Flash-0731-NVFP4, vLLM-MoET (W2 base + FP4 delta tier), all 44 served W2 layers resident in GPU VRAM, no offload. Decode: 75–80 tok/s. Load time: ~100 s (down from 300+ s).
Problem 1: VRAM headroom — the plane set barely fits
On RTX PRO 6000 with 96 GB VRAM, the W2 planes for all 44 served layers (~72+ GiB) plus dense weights (~19 GiB) plus KV and the delta tier sit right at the edge of the card: 96 GB is an awkward budget, right at the threshold. Before this work, the practical choices were either offload the plane layers to host memory, or run with a much smaller context so the whole set fits. Both cost speed or context — and that was the real problem.
So the question I kept coming back to was: how do I free VRAM without giving up quality?** The answer is to keep all planes resident in VRAM, no offload. With CUDA graphs only capture 4 sizes and speculative decoding disabled , the full plane set now fits on 96 GB. Decode improves from ~50 tok/s to 75–80 tok/s at 260K configured context. Notably, this is faster than running with DSpark or MTP speculative decoding: those consume so much VRAM that a large fraction of layers has to be offloaded to host memory, and the offload overhead costs more than speculative tokens save.
Problem 2: Load time — painful daily starts
The original load took close to 5 minutes (300+ s), which is painful for daily use. Once the planes cache and the delta cache are fully built, the loader should not need to re-read fp13/fp2 on every layer — each layer carries roughly 3 GiB of fp13/fp2, and reading them every boot is ~200 GB of pointless I/O. Skipping those reads when the delta pack is verified complete is a huge speedup.
To go further, I had the agent write a concurrent cache-backed loader: the load runs 4 workers on 12 cache-backed layers per batch, with per-batch synchronize + gc.collect + empty_cache. Since the final 7–8 layers don't have enough cache-backed layers to fill a batch, they are loaded serially. Combined with skipping already-cached fp13/fp2 reads, the loading goes from 300+ s down to roughly 100 s.
Problem 3: The model goes off the rails — gibberish output, meaningless symbols, and hard loops
The model leaked stray content into the response body and failed tool-call invocations; the root cause was the vLLM reasoning parser and tool-call parser. After fixing, hard loops and gibberish largely disappeared.
MoET and serving settings
Performance
Remaining bottleneck
At 260K context the card is essentially maxed out in VRAM: concurrency is limited, and there is no room left to enable speculative decoding. If there were a way to compress the VRAM footprint further (less per-layer bytes in the delta tier or base planes), it would open room for more concurrent requests or speculative decoding.
Many thanks to the brilliant author @kacper for the design, and to @jpezzulli for the exploration.
My code is a bit of a mess, so rather than trying to diff and patch everything again, I'm sharing the approach here — if you're interested, give it a try.