Environment
RTX 5090 32GB, DSV4-Flash (61 MoE layers x 256 experts), --moe-cache-size 700,
real workload = long-form Chinese financial analysis, measured via
collect_stats + collect_decode_freq over a ~12 min serving window.
Measurements
- Live experts per layer during decode: 240-248 of 256 — routing spreads over
~97% of experts, so the working set is effectively the whole model.
- Realized decode miss rate under LRU: 82-92%.
- Stationary-oracle hit rate (top-C experts per layer by observed frequency,
C = cache_size/num_layers): 46% at 700 slots — vs the 8-18% LRU realizes.
- The oracle capacity curve is nearly linear (500 slots: 40%, 700: 46%,
900: 52%): growing the cache barely helps; the eviction policy is the gap.
Caveat we verified ourselves (important)
We prototyped frequency-pinning at the tensor level (shielding hot slots'
usage from argmin eviction). It mechanically works, but in hybrid decode
mode the impact on throughput is small: only the PCIe-fetched share of misses
(auto-benched 29.4% here) ever enters the cache, and the CPU lane absorbs the
rest in parallel, so cache hit rate is not the critical path. The full ~30pp
opportunity applies to the pure GPU offload decode path (decode_target
"gpu"), where every miss is a synchronous PCIe stall.
Why LRU loses
With fine-grained MoE the per-layer routing rotates rapidly across hundreds of
experts; recency is a poor predictor, so LRU keeps evicting experts that
return a few steps later. A frequency-informed policy (pin top-K by decayed
frequency, LRU only for the tail) would close most of the ~30pp gap; on
PCIe-bandwidth-bound decode that translates to a large throughput gain.
Request
- Expose an LFU / frequency-pinned option for --moe-cache-policy ({"lru"} is
currently the only entry), or
- a hybrid: static pin of top-K experts per layer (K configurable or derived
from collect_decode_freq) + LRU for the remainder.
Happy to share our raw stats dumps.
Environment
RTX 5090 32GB, DSV4-Flash (61 MoE layers x 256 experts), --moe-cache-size 700,
real workload = long-form Chinese financial analysis, measured via
collect_stats + collect_decode_freq over a ~12 min serving window.
Measurements
~97% of experts, so the working set is effectively the whole model.
C = cache_size/num_layers): 46% at 700 slots — vs the 8-18% LRU realizes.
900: 52%): growing the cache barely helps; the eviction policy is the gap.
Caveat we verified ourselves (important)
We prototyped frequency-pinning at the tensor level (shielding hot slots'
usagefrom argmin eviction). It mechanically works, but in hybrid decodemode the impact on throughput is small: only the PCIe-fetched share of misses
(auto-benched 29.4% here) ever enters the cache, and the CPU lane absorbs the
rest in parallel, so cache hit rate is not the critical path. The full ~30pp
opportunity applies to the pure GPU offload decode path (decode_target
"gpu"), where every miss is a synchronous PCIe stall.
Why LRU loses
With fine-grained MoE the per-layer routing rotates rapidly across hundreds of
experts; recency is a poor predictor, so LRU keeps evicting experts that
return a few steps later. A frequency-informed policy (pin top-K by decayed
frequency, LRU only for the tail) would close most of the ~30pp gap; on
PCIe-bandwidth-bound decode that translates to a large throughput gain.
Request
currently the only entry), or
from collect_decode_freq) + LRU for the remainder.
Happy to share our raw stats dumps.