Automatic performance tuning for vLLM deployments — Optuna-driven search over vLLM server flags, benchmarked with GuideLLM (throughput, TTFT, ITL, trace replay, speculative decoding). Runs on a single GPU node; Ray optional.
- 🎯 Flexible backends: Local execution by default; optional Ray for distributed runs
- 📊 GuideLLM benchmarking: Warmup/cooldown, ramp-up, trace replay, multimodal VLM, synthetic or custom datasets
- 🧮 Rich objectives: Multi-objective optimization with arithmetic metric expressions
- 🔀 Smart sampler selection: Automatic grid or random mode based on search-space size
- ⚡ Speculative decoding: Tune EAGLE3, ngram, Medusa, and MTP parameters from YAML
- 📈 Optuna integration: User attributes for extra metrics; dashboard launcher included
- 🗄️ Flexible storage: SQLite for local use, PostgreSQL for production (optional)
- ⚙️ Easy configuration: YAML-based study and parameter configuration
- ✅ Tested: Unit tests and CI on Python 3.10–3.12
For a detailed starter guide, see the Quick Start Guide.
From PyPI (after the first release):
pip install auto-tune-vllmInstall the base package for local execution. Add the optional ray extra only if you want distributed execution.
git clone https://github.com/InseeFrLab/auto-tuning-vllm.git
cd auto-tuning-vllm
# Basic installation (local execution only)
pip install -e .
# Optional: Install with Ray support for distributed execution
pip install -e ".[ray]"
# Optional: Install with PostgreSQL support
pip install -e ".[postgresql]"# Run optimization study locally (default backend)
auto-tune-vllm optimize --config config.yaml --max-concurrent-trials 2
# Run optimization study on Ray
auto-tune-vllm optimize --config config.yaml --backend ray --venv-path ./venv --max-concurrent-trials 2
# Validate config and preview grid cardinality / sampler auto-switch
auto-tune-vllm validate --config config.yaml
# Resume interrupted study
auto-tune-vllm resume --study-name study_35884
# Stream live logs
auto-tune-vllm logs --study-name study_35884
# Explore results with Optuna Dashboard (sample database included)
./optuna_dashboard/start_optuna_dashboard.sh- Quick Start Guide — Get running in 5 minutes
- Architecture overview — How the framework works (diagrams)
- Configuration Reference — Complete YAML configuration guide
- Examples — Sample study YAMLs and Python demos
- Ray Cluster Setup — For distributed optimization (optional)
- AGENTS.md — Guide for coding assistants and maintainers
- Python 3.10+
- NVIDIA GPU with CUDA support (for running vLLM)
- SQLite (included) or PostgreSQL (optional)
Core dependencies are installed with pip install -e .. Ray is optional and available via pip install -e ".[ray]".
- Expand test coverage (controller, backends, trial lifecycle)
- Make CI fail strictly on pytest errors (remove
|| trueworkaround) - Dependency hygiene — pin versions, reduce heavy core dependencies
- Improve CLI error messages and validation
- Additional benchmark providers beyond GuideLLM
- Support for alternative inference engines (e.g., SGLang)
- Better parameter validation against vLLM CLI args
-
optimization.n_repeatsandoptimization.no_repeat(PRs #28, #32 open)
Contributions are welcome. Priority areas:
- Testing — Extending coverage for controllers, backends, and edge cases
- Documentation — Improving guides and examples
- Core stability — Bug fixes and edge case handling
Before the first release, create the GitHub environment pypi, restrict it to
tags matching v*, and configure required reviewers where available. On PyPI,
add a GitHub Trusted Publisher with owner InseeFrLab, repository
auto-tuning-vllm, workflow filename publish.yml and environment pypi.
For a new project, register a pending publisher for auto-tune-vllm in
PyPI account publishing.
No permanent PyPI token is needed. See the
official Trusted Publishing guide.
-
Update the version in
pyproject.tomlandauto_tune_vllm/__init__.pytogether, update the changelog, and open a PR againstmain. -
Wait for Package CI to pass, review and merge the PR.
-
From an up-to-date, clean
main, create and push the version tag. For the initial version (already0.1.0):git tag -a v0.1.0 -m "Release 0.1.0" git push origin v0.1.0
Pushing the tag triggers publish.yml. It rejects tags that differ from
v followed by the exact version in pyproject.toml, builds and checks both
distributions, then publishes through OIDC in the pypi environment.
Approve the deployment if required and check the workflow result on GitHub.
This project originated as a fork of openshift-psap/auto-tuning-vllm. We are grateful to the original authors for providing the foundation. This fork focuses on simpler single-node deployment (Ray optional), active maintenance, test coverage, and feature expansion for production workloads.
See CHANGELOG.md for the full list of changes since the fork.
Apache License 2.0 — see LICENSE file for details.