Monte Carlo simulation benchmark for MoE expert activation memory variability and admission control under non-deterministic routing.
How should an LLM serving scheduler account for the fact that MoE expert activation memory is non-deterministic at request admission time?
In standard MoE architectures like Mixtral-8x7B:
- Attention layers are dense — all tokens, all heads, full KV state
- MoE routing affects only the FFN / expert layers
- The KV cache does not vary with expert routing
What does vary with routing:
peak_expert_buffer = max_over_experts(tokens_routed * activation_size)
This buffer is non-deterministic at admission time because routing depends on token content, which is unknown until inference begins.
| Routing | mixtral mean_peak | mixtral p95_peak | Calibration error |
|---|---|---|---|
| uniform | 187 MB | 271 MB | baseline |
| skewed | 288 MB | 432 MB | -7% (underestimates) |
| very_skewed | 408 MB | 516 MB | -52% (severe underestimate) |
Uniform-calibrated worst-case assumptions fail under skewed routing. A scheduler assuming uniform traffic will underestimate buffer pressure by 52%.
| top_k | mean_peak | p95_peak | CV |
|---|---|---|---|
| 1 | 94 MB | 143 MB | 0.26 |
| 2 | 188 MB | 286 MB | 0.26 |
| 3 | 281 MB | 428 MB | 0.26 |
| 4 | 375 MB | 571 MB | 0.26 |
Relative variability (CV) stays constant. Only absolute magnitude grows.
| Model | Routing | worst_case admits | expected_case admits | Overflow |
|---|---|---|---|---|
| Mixtral-8x7B | uniform | 3 | 8 | 0% |
| Mixtral-8x7B | skewed | 2 | 5 | 0% |
| Mixtral-8x7B | very_skewed | 1 | 3 | 0% |
| MoE-16x3B | uniform | 7 | 16 | 0% |
| MoE-16x3B | very_skewed | 3 | 9 | 0% |
Expected-case admits 2-3x more requests than worst-case with zero overflow.
| Model | Routing skew | Recommended policy | Note |
|---|---|---|---|
| Mixtral-8x7B | uniform | expected_case | highest throughput |
| Mixtral-8x7B | skewed | expected_case | reduced batch, no overflow |
| Mixtral-8x7B | very_skewed | expected_case | monitor hot expert load |
| MoE-16x3B | any | expected_case | consistently best |
Expected-case admission is the best default policy, but it must be paired with routing-skew monitoring because uniform-calibrated worst-case assumptions fail under skewed expert activation.
| Model | Experts | top_k | Layers |
|---|---|---|---|
| dense_7b | 1 | 1 | 32 |
| mixtral_8x7b | 8 | 2 | 32 |
| moe_16x3b | 16 | 2 | 24 |
| Distribution | Concentration | Description |
|---|---|---|
| uniform | 1.0 | all experts equally likely |
| skewed | 0.30 | some experts preferred ~3x |
| very_skewed | 0.10 | 2-3 experts dominate |
cd ~/dev/moe-kv-cache-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 2-3 minutes. No GPU required.
results/
activation_variability.csv
topk_sweep.csv
admission_analysis.csv
decision_map.csv
plots/
01_activation_variability.png
02_imbalance_vs_cv.png
03_admission_capacity.png
04_overflow_vs_waste.png
05_topk_sweep.png
06_budget_utilization.png
moe-activation-memory-bench/
+-- src/
| +-- config.py
| +-- activation_model.py
| +-- admission_model.py
| +-- bench.py
| +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt
Expert routing per request is modeled as:
routing_probs ~ Dirichlet(concentration)
tokens_per_expert ~ Multinomial(seq_len * top_k, routing_probs)
peak_expert_mb = max(tokens_per_expert) * activation_bytes_per_token
Each configuration is evaluated over multiple seeds and repetitions to estimate mean, p95, and max of peak expert buffer.
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
- design.md -- detailed design and clarification on KV vs activation memory
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza 2026