Skip to content

About

Benchmark-driven configuration optimization for vLLM inference servers

Topics

Resources

Contributing

Security policy

Stars

11 stars

Watchers

0 watching

Forks

Repository files navigation

vLLM Optimizer

Find better vLLM serving settings for your workload. Define a search space in YAML; vllm-opt runs the benchmarks and reports the best observed settings.

Run your first experiment · Documentation · PyPI

This is independent community alpha software, not affiliated with vLLM. Execution needs Python 3.11–3.12, Linux/WSL, an NVIDIA GPU, and a local model. Native Linux, L40/H100, tensor-parallel, and multi-GPU evidence remain hardware validation pending. See the compatibility matrix.

Install

Use Command
Run GPU experiments on Linux/WSL pip install "vllm-optimizer[runtime]"
Inspect configs and saved results on any supported OS pip install vllm-optimizer

Use a virtual environment. The runtime extra installs vLLM and GuideLLM; the core package requires an existing runtime to execute experiments. See installation choices for setup and existing CUDA environments.

First experiment

The quick start contains one complete example:

  1. Install the runtime in a virtual environment.
  2. Copy the YAML and set your local model path. Models are not downloaded.
  3. Validate and run it; open the printed report.html path.

Read the walkthrough in 5–10 minutes. Installation and benchmark time depend on your machine and model.

What you can tune

  • Fixed or tunable vLLM flags and environment variables.
  • Grid, Random, or TPE search with a baseline comparison.
  • GuideLLM or native vLLM Bench Serve workloads.
  • Throughput objectives subject to latency/throughput targets, such as p99 TTFT ≤ "200 ms" or request throughput ≥ "50 req/s".

Search keeps its configured budget after meeting a target. Only configurations satisfying every limit can win. See search and scoring.

See a real report

Open the interactive RTX 3080 report · Experiment YAML · Baseline-control example

Report showing throughput, confidence, settings, and all trials

This deliberately small workload measured 194.34 → 1,082.27 output tokens/s. It demonstrates reporting and latency trade-offs; it is not a production benchmark or an expected improvement for other workloads. The published example used vLLM 0.28.0 and GuideLLM 0.7.3; dated runtime details and required WSL flags are in compatibility.

Next steps

Task Guide
Set fixed values and search ranges Configuration
Choose workloads and datasets Benchmarking
Use native vLLM benchmarking vLLM Bench Serve example
Understand confidence and target eligibility Reports
Retry, reproduce, or regenerate results CLI reference
Run independent trials on separate GPUs Parallel trials
Resolve runtime failures Troubleshooting
Find every YAML section Complete YAML

Use vllm-opt --config experiment.yaml --verbose to stream logs. Full trial logs are saved regardless of verbosity; NO_COLOR disables terminal colors.

Install vllm-optimizer, import vllm_optimizer, and run vllm-opt. The former vtune aliases are deprecated; see migration.

Contribute

Contributing · Architecture · Roadmap · Release notes

About

Benchmark-driven configuration optimization for vLLM inference servers

Topics

Resources

Contributing

Security policy

Stars

11 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages