capacity-planning-sim

Discrete-event simulation for LLM serving capacity planning.

Answers the operational question that closes 40+ prior benchmarks in this portfolio:

How many GPUs are needed to serve X requests per second with a p99 TTFT SLO under realistic traffic, fragmentation, and scaling lag?

---

Why This Exists

Prior benchmarks answered isolated questions:

• serving-cost-model-v2: cost per token
• disaggregated-prefill-decode-sim: throughput per instance
• continuous-batching-fragmentation-sim: fragmentation reduces throughput by 28-44%
• slo-aware-autoscaling-sim: startup delay is 35 seconds
• sharegpt-workload-bench: realistic arrival and token distributions

This project asks the integrating question:

Given all of the above combined, how many GPUs and at what monthly cost?

---

Key Results

The latency cliff is the central feature of LLM capacity planning

In sharegpt_diurnal / static_capacity:

GPU count	p99 TTFT	Gain from +1 GPU
3	9,139 ms	-
4	895 ms	-8,245 ms
5	357 ms	-538 ms
6	189 ms	-168 ms
7	138 ms	-51 ms
8+	~128 ms	<11 ms

Near the cliff, one GPU recovers thousands of milliseconds. After the cliff, marginal returns collapse.

Sizing for mean throughput is dangerously optimistic

Planning level	Criterion	sharegpt_diurnal GPU floor
Optimistic	majority of seeds meet SLO	5
Conservative	all seeds meet SLO	5
Robust	all seeds meet SLO with >= 50ms headroom	6

The conservative plan leaves only 3.75ms of worst-case headroom. One extra GPU increases that to 172ms at a cost of ~$500/month.

Oracle schedule is the global cost lower bound

Workload	Oracle avg cost	Scheduled avg cost	Static cost
sharegpt_diurnal	$10,707	$12,042	$16,800
fragmentation_heavy	$11,079	$14,321	$16,800
long_output_tail	$12,888	$14,321	$16,800
steady_high	$28,000	$28,000	$28,000

Dynamic scheduling saves average cost by scaling down during valley hours. It does not reduce the required peak fleet size.

Reactive autoscaling is viable but expensive

Startup delay of 35 seconds forces reactive policies to over-provision to avoid queue buildup during peaks. Average monthly cost is consistently higher than scheduled scaling.

Under constant high load, dynamic scaling offers no advantage

When traffic valleys disappear, all policies converge to the same GPU count and cost.

---

Policy Role Taxonomy

Policy	Role	Best for	Avoid when
static_capacity	robust baseline	no autoscaling infra	strong diurnal variation
static_with_headroom	safe baseline	simple deployments	cost is a priority
reactive_autoscaling	reactive scale-out	unpredictable bursts	startup delay > 10s
scheduled_scaling	best deployable	predictable diurnal traffic	irregular patterns
oracle_schedule	cost lower bound	benchmarking savings	future traffic unknown

---

Three Planning Levels

Level	Criterion	Use case
Optimistic	majority of seeds meet SLO	academic comparison only
Conservative	all seeds meet SLO	minimum production bar
Robust	all seeds meet SLO with >= 50ms headroom	production recommendation

---

Workloads

Workload	Description
sharegpt_diurnal	ShareGPT token distribution, 10x diurnal peak
burst_ramp	sudden 5x traffic ramp in 60 seconds
steady_high	constant load at 85% of peak
fragmentation_heavy	44% fragmentation penalty
long_output_tail	heavy-tail output length distribution

---

Quick Start

cd ~/dev/capacity-planning-sim

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python -u run.py

Runtime: approximately 10-20 minutes. No GPU required.

---

Output Files

results/
  summary.csv
  min_gpu_for_slo.csv
  min_gpu_for_slo_conservative.csv
  risk_adjusted_capacity.csv
  decision_map.csv
  decision_map_conservative.csv
  sensitivity.csv
  policy_roles.csv

plots/
  01_gpu_sweep_p99.png
  02_cost_vs_headroom.png
  03_min_gpu_by_workload.png
  04_sensitivity_cliff.png
  05_utilization_vs_headroom.png
  06_optimistic_vs_conservative.png
  07_risk_adjusted_capacity.png

---

Project Structure

capacity-planning-sim/
+-- src/
|   +-- config.py
|   +-- workload.py
|   +-- simulator.py
|   +-- bench.py
|   +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt

---

Methodology

This project is a calibrated discrete-event simulation.

It integrates measured parameters from prior benchmarks:

Hardware throughput:         from disaggregated-prefill-decode-sim
Fragmentation penalty:       from continuous-batching-fragmentation-sim
Startup delay:               from slo-aware-autoscaling-sim
Token distributions:         from sharegpt-workload-bench
Cost calibration:            from serving-cost-model-v2

TTFT is decomposed as:

TTFT = queue_wait_ms + prefill_ms

Effective throughput includes fragmentation:

effective_ms_per_token = base_ms_per_token / (1 - fragmentation_penalty)

For full design details, see design.md.

---

Requirements

• Python 3.10+
• NumPy >= 1.26.0
• Pandas >= 2.0.0
• Matplotlib >= 3.8.0

No GPU required.

---

Practical Recommendation

If you need a simple default:

• use scheduled_scaling as the primary policy for diurnal workloads
• plan capacity at the robust level (all seeds + >= 50ms headroom)
• avoid reactive autoscaling if startup delay exceeds 10 seconds
• treat the oracle schedule as a cost benchmark, not a deployment target
• monitor headroom_ms as the primary operational safety signal

---

Documentation

• design.md -- detailed design rationale and calibrated parameters
• summary.txt -- concise high-level summary of findings
• LICENSE -- MIT License

---

License

MIT License -- Copyright (c) 2026 Joao Felipe De Souza

---

Author

Joao Felipe De Souza 2026