tenantbench

A multi-tenant goodput and fairness benchmark for LLM inference engines.

Existing serving benchmarks (vllm bench serve, genai-perf, GuideLLM, InferenceMAX) drive one aggregate request stream and report throughput and latency. Production endpoints serve many tenants with different traffic patterns and different SLOs, and the questions that matter there are different: does one tenant's burst destroy another tenant's p99, does priority scheduling actually prevent starvation, and what does isolation cost in aggregate goodput. Schedulers that target these questions keep appearing (VTC, FairBatching, FlowPrefill, GoodServe), each evaluated with bespoke scripts. There is no shared benchmark. The Kubernetes inference-perf project has an open feature request (#524) for exactly this measurement capability.

tenantbench fills that gap: seeded multi-tenant workloads with per-tenant SLOs, run open-loop against any OpenAI-compatible endpoint, with per-tenant goodput, fairness, and starvation metrics out of the box.

What it measures

Per tenant:

• TTFT / TPOT / E2E latency distributions (p50, p90, p99, bootstrap CIs).
• SLO-attainment goodput: requests per second that completed and met both the tenant's TTFT and TPOT SLOs. Throughput counts tokens; goodput counts kept promises.
• Starvation incidence: fraction of requests with TTFT beyond 10x the tenant's SLO.

Per run:

• Jain fairness index over token-weighted per-tenant service (VTC-style service function, prefill weight 1, decode weight 2), plus a windowed Jain time series and max/min service ratio.
• Victim degradation ratio: a tenant's p99 and goodput under contention divided by the same tenant alone at the same rate.
• Loadgen validity: p99 schedule lag between intended and actual send times. If the client could not keep up, the run is marked invalid instead of silently reporting server numbers that are really client numbers.

Quickstart (no GPU needed)

pip install -e .
tenantbench mockserve --port 8100 --batch-slots 8 &
tenantbench run --scenario configs/scenarios/noisy_neighbor.toml \
    --engine configs/engines/mock.toml --tokenizer whitespace \
    --total-rps 6 --limit 300
tenantbench analyze --run results/noisy_neighbor/mock/seed0_* --out report.md --plots

The mock engine is a queueing test double, not a simulator. Its numbers demonstrate the harness, never an engine.

Real engines

# 1. Start vLLM (see RUNBOOK.md for the RTX 5070 Ti / WSL2 path)
vllm serve Qwen/Qwen3-8B-FP8 --port 8000 --max-model-len 8192

# 2. Measure sustainable capacity once per engine config
tenantbench probe --engine configs/engines/vllm.toml --out results/capacity/vllm

# 3. Run scenarios at capacity-relative load, 3 seeds each
tenantbench run --scenario configs/scenarios/noisy_neighbor.toml \
    --engine configs/engines/vllm.toml \
    --capacity-file results/capacity/vllm/capacity.json --trials 3

# 4. Report
tenantbench analyze --run results/noisy_neighbor/vllm-fcfs/* --out report.md --plots

Every run directory contains records.jsonl (raw per-request timings), summary.json (all metrics), config.json (the fully resolved configuration), env.json (package, driver, and engine versions), and optionally server_metrics.jsonl (engine queue depth and preemption counters sampled at 1 Hz).

Scenarios

scenario	question it answers
fair_share	Do identical tenants get identical service, or does the scheduler pick favorites?
noisy_neighbor	How much does a bursting batch tenant degrade an interactive tenant's p99 and goodput?
victim_baseline	Control for the above: the victim alone at the same rate.
slo_mix	What does priority scheduling buy the interactive class and cost the batch class?
overload	At 1.4x capacity, who pays: even degradation or starvation?
prefill_mix	How much does chunked prefill protect short requests from long-prefill head-of-line blocking?

Methodology

• Open-loop arrivals. Issuance follows a precomputed schedule and never waits on responses, so server queueing cannot suppress offered load (the coordinated-omission trap in closed-loop load generators).
• Multi-process load generation. Single-process asyncio clients inflate tail latency at high concurrency (arXiv:2605.24217). Work is sharded across processes, and residual client lag is measured and enforced per run, not assumed away.
• Capacity-relative load. Scenarios specify utilization (rho) against a measured single-tenant capacity, so comparisons across engines and configs happen at matched operating points.
• Controlled token counts. Prompts are corpus windows sized in the model's own tokenizer; outputs are forced with ignore_eos and exact max_tokens. Scheduler cost depends on token counts, not content. Unique per-request prefixes prevent prefix-cache hits from contaminating scheduling results.
• Seeded everything. Arrivals, lengths, and prompt content derive from per-tenant, per-purpose RNG streams keyed by the run seed. Two runs with the same seed issue byte-identical request schedules.

See DESIGN_NOTES.md for the reasoning behind each choice and the alternatives rejected.

Results

First measured results (vLLM FCFS vs priority scheduling, SGLang, on RTX 5070 Ti with Qwen3-8B-FP8) land here with the v0.2 tag. Raw run directories will be committed alongside the tables.

Limitations

• One consumer GPU (16 GB), no tensor parallelism. Scheduler phenomena are queue-level and load is capacity-normalized, but absolute concurrency is modest compared to multi-GPU production replicas. A single-day H100 replication is planned.
• The mock engine's slot model is deliberately simple. It exists for tests and demos.
• IID bootstrap CIs ignore autocorrelation in queueing time series; treat CIs as optimistic lower bounds on uncertainty.
• Tenant classes are synthetic distributions informed by public traces, not trace replay. Trace replay is on the roadmap.

Roadmap

• SGLang priority field verification and cross-engine priority comparison.
• Trace replay mode (BurstGPT, Azure LLM inference traces, Mooncake).
• Prefix-cache fairness scenario: does cache-aware scheduling make cache-hot tenants richer?
• Cost per SLO-met request at fixed $/GPU-hour across engines.

License

Apache-2.0.