A benchmark comparing LLM serving behavior under real ShareGPT conversation traces versus synthetic Poisson workloads. Measures TTFT, SLO compliance, and scheduling policy effectiveness under realistic versus synthetic traffic.
For full methodology and design decisions see design.md.
How does serving behavior change when we use real conversation traces instead of Poisson arrivals with fixed lengths?
Answer: significantly worse P99 latency and earlier SLO violations, with scheduling policies that only differentiate under real distributions.
ShareGPT V3 — 9562 real conversation turns between users and ChatGPT.
Distribution: Prompt: mean=119 median=22 p99=1872 max=22579 Output: mean=317 median=290 p99=916 max=1306
The prompt distribution is extremely right-skewed. The median is 85x smaller than the p99.
sharegpt Real length distribution, Poisson arrivals poisson_matched Normal(mean=119), Poisson arrivals poisson_p50 Normal(median=22), Poisson arrivals
poisson_p50: 5ms poisson_matched: 22ms sharegpt: 212ms
sharegpt is 9.6x worse than poisson_matched sharegpt is 42x worse than poisson_p50
sharegpt: arr=2.0 poisson_matched: arr=3.0 poisson_p50: arr=3.0
ShareGPT enters SLO violation one full step before synthetic workloads. A system sized on synthetic benchmarks will violate SLO earlier than predicted.
fcfs: SLO=0.017 TTFT_p99=2280ms shortest_first: SLO=0.012 TTFT_p99=2210ms (-29%) priority_short: SLO=0.012 TTFT_p99=2210ms (-29%) longest_first: SLO=0.017 TTFT_p99=2298ms
Under Poisson workloads, all scheduling policies are identical (SLO=0.000). Scheduling only matters under real heavy-tailed distributions.
Real ShareGPT prompts are extremely right-skewed. Median=22 tokens, p99=1872 tokens (85x difference). Matching the mean (119) still underestimates tail latency by 10x. Matching the median (22) underestimates by 42x.
SLO violations appear one arrival-rate step earlier under real traces. A system that passes synthetic benchmarks may fail in production.
Scheduling policies only differentiate under real distributions. Under Poisson workloads, FCFS and shortest-first produce identical results. Under ShareGPT, shortest-first reduces SLO violations by 29%.
Production serving must be validated against real trace distributions. Synthetic Poisson benchmarks give a false sense of SLO safety.
sharegpt-workload-bench/ ├── src/ │ ├── init.py │ ├── config.py │ ├── download.py │ ├── workload.py │ ├── simulator.py │ └── analysis.py ├── data/ ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
The dataset is downloaded automatically on first run.
Outputs: results/results.csv results/summary.txt plots/length_distributions.png plots/ttft_vs_arrival.png plots/slo_violations.png plots/scheduling_impact.png
This project validates the workload assumptions used across the portfolio:
llm-inference-scheduler tested with Poisson arrivals request-routing-sim tested with synthetic prefix distributions disaggregated-prefill-sim tested with fixed prompt/output lengths slo-aware-autoscaling-sim tested with synthetic arrival patterns
All prior results are conservative estimates of worst-case behavior under actual LLM serving traffic.
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026