Find better vLLM serving settings for your workload. Define a search space in
YAML; vllm-opt runs the benchmarks and reports the best observed settings.
Run your first experiment · Documentation · PyPI
This is independent community alpha software, not affiliated with vLLM. Execution needs Python 3.11–3.12, Linux/WSL, an NVIDIA GPU, and a local model. Native Linux, L40/H100, tensor-parallel, and multi-GPU evidence remain hardware validation pending. See the compatibility matrix.
| Use | Command |
|---|---|
| Run GPU experiments on Linux/WSL | pip install "vllm-optimizer[runtime]" |
| Inspect configs and saved results on any supported OS | pip install vllm-optimizer |
Use a virtual environment. The runtime extra installs vLLM and GuideLLM; the core package requires an existing runtime to execute experiments. See installation choices for setup and existing CUDA environments.
The quick start contains one complete example:
- Install the runtime in a virtual environment.
- Copy the YAML and set your local model path. Models are not downloaded.
- Validate and run it; open the printed
report.htmlpath.
Read the walkthrough in 5–10 minutes. Installation and benchmark time depend on your machine and model.
- Fixed or tunable vLLM flags and environment variables.
- Grid, Random, or TPE search with a baseline comparison.
- GuideLLM or native vLLM Bench Serve workloads.
- Throughput objectives subject to latency/throughput targets, such as
p99 TTFT ≤
"200 ms"or request throughput ≥"50 req/s".
Search keeps its configured budget after meeting a target. Only configurations satisfying every limit can win. See search and scoring.
Open the interactive RTX 3080 report · Experiment YAML · Baseline-control example
This deliberately small workload measured 194.34 → 1,082.27 output tokens/s. It demonstrates reporting and latency trade-offs; it is not a production benchmark or an expected improvement for other workloads. The published example used vLLM 0.28.0 and GuideLLM 0.7.3; dated runtime details and required WSL flags are in compatibility.
| Task | Guide |
|---|---|
| Set fixed values and search ranges | Configuration |
| Choose workloads and datasets | Benchmarking |
| Use native vLLM benchmarking | vLLM Bench Serve example |
| Understand confidence and target eligibility | Reports |
| Retry, reproduce, or regenerate results | CLI reference |
| Run independent trials on separate GPUs | Parallel trials |
| Resolve runtime failures | Troubleshooting |
| Find every YAML section | Complete YAML |
Use vllm-opt --config experiment.yaml --verbose to stream logs. Full trial
logs are saved regardless of verbosity; NO_COLOR disables terminal colors.
Install vllm-optimizer, import vllm_optimizer, and run vllm-opt.
The former vtune aliases are deprecated; see migration.
