Repository navigation
Add experimental automatic tuning with calibrated workloads - #34
Draft
brtydse100 wants to merge 10 commits into
Draft
brtydse100 wants to merge 10 commits into
brtydse100 wants to merge 10 commits into
Conversation
This was referenced Nov 17, 2026
Objectives (#38) and the runtime update (#39) are now on main. Take main's versions of that code, keep only the auto-mode modules and hooks, drop the planning document, move the two auto tests to their own file, and mark the guide experimental. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto mode runs a bounded search over serving parameters, starting from a local model and the average request size.
This PR now contains only auto mode. The objective syntax merged in #38 and the runtime update in #39; the branch takes both from
main. The planning document is removed.Behaviour
optimization.objectivecan be selected.benchmark.runskeep their data and load settings and skip calibration.max-num-seqs,max-num-batched-tokens, andgpu-memory-utilization. Settings fixed underserverare excluded.auto.json,resolved.yaml, and, when a configuration is eligible,best-server.yamlandbest-environment.yaml.Validation at b3ed7c1
main; they come from hardcoded local paths and a missing helper module.mkdocs build --strictpass.Real GPU
RTX 3080 10 GB, WSL2, CUDA 13.0, torch 2.13.0, vLLM 0.32.0, GuideLLM 0.8.1, local OPT-125M FP16. Average request 16 input / 8 output tokens, generating 4 / 16 / 48 input tokens; calibration cap 4; 2 TPE trials; 1 repeat; GPU memory fixed at 0.4.
max-num-seqsmax-num-batched-tokensAll three workloads calibrated to concurrency 4 with reason
limit_reached, so saturation was not found. No trial failed; exports were written and the server port was released.This confirms the reduced branch runs end to end. A 3.42% difference from one repeat on a small model is not evidence that auto mode improves on a baseline. Evidence is private at
vTune-private-tests/auto-tuning-evidence/rebased-b3ed7c1/.Earlier evidence on the pre-split branch (9bd0230)
Details are private under
vTune-private-tests/auto-tuning-evidence/.Before this leaves draft
Limitations
🤖 Generated with Claude Code