Skip to content

Add experimental automatic tuning with calibrated workloads - #34

Draft
brtydse100 wants to merge 10 commits into
mainfrom
feat/automatic-tuning
Draft

brtydse100 wants to merge 10 commits into
mainfrom
feat/automatic-tuning

Conversation

@brtydse100

@brtydse100 brtydse100 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Auto mode runs a bounded search over serving parameters, starting from a local model and the average request size.

server:
  model: /models/my-model
optimization:
  mode: auto
benchmark:
  average_request:
    input_tokens: 20480
    output_tokens: 128

This PR now contains only auto mode. The objective syntax merged in #38 and the runtime update in #39; the branch takes both from main. The planning document is removed.

Behaviour

  • Defaults to 20 TPE trials maximizing output tokens/s; any optimization.objective can be selected.
  • Generated small, medium, and large workloads use 1/4x, 1x, and 3x the average input length with a fixed output length. Insufficient model context fails validation; lengths are never reduced.
  • One baseline server calibrates concurrency per workload, then that load is frozen for every trial and finalist.
  • Explicit benchmark.runs keep their data and load settings and skip calibration.
  • The catalog is max-num-seqs, max-num-batched-tokens, and gpu-memory-utilization. Settings fixed under server are excluded.
  • Auto runs add auto.json, resolved.yaml, and, when a configuration is eligible, best-server.yaml and best-environment.yaml.

Validation at b3ed7c1

  • Private and public suites on vLLM 0.32.0 / GuideLLM 0.8.1 in WSL: 1,687 passed, 7 failed, 2 collection errors. The 7 failures and 2 errors also occur on main; they come from hardcoded local paths and a missing helper module.
  • Public tests: 171 passed. Coverage 81%; all 34 module gates pass.
  • Ruff lint and format, mypy, and mkdocs build --strict pass.

Real GPU

RTX 3080 10 GB, WSL2, CUDA 13.0, torch 2.13.0, vLLM 0.32.0, GuideLLM 0.8.1, local OPT-125M FP16. Average request 16 input / 8 output tokens, generating 4 / 16 / 48 input tokens; calibration cap 4; 2 TPE trials; 1 repeat; GPU memory fixed at 0.4.

Configuration max-num-seqs max-num-batched-tokens Output tok/s
Baseline default default 551.54
Trial 64 2048 559.58
Trial (selected) 128 4096 570.43

All three workloads calibrated to concurrency 4 with reason limit_reached, so saturation was not found. No trial failed; exports were written and the server port was released.

This confirms the reduced branch runs end to end. A 3.42% difference from one repeat on a small model is not evidence that auto mode improves on a baseline. Evidence is private at vTune-private-tests/auto-tuning-evidence/rebased-b3ed7c1/.

Earlier evidence on the pre-split branch (9bd0230)

  • A matched comparison on OPT-125M found no reliable advantage: in a longer holdout, auto was 0.73% below an explicit grid, and baseline drift was +2.48%. Both selection decisions were inconclusive.
  • Every calibrated load reached its cap, and six trials stayed within TPE's startup sampling.
  • A long-request shape check ran Qwen3.5-4B-AWQ at 5,120 / 20,480 / 61,440 input tokens with one request per size.

Details are private under vTune-private-tests/auto-tuning-evidence/.

Before this leaves draft

  • A validated gain over baseline on a realistic model with a saturating load and the full trial budget.
  • A wider catalog; three knobs gave no measurable signal on a small model.
  • The unrestricted/EOS output default and parameter-name-only selection syntax are not implemented; implement or drop them.

Limitations

  • Frozen calibrated concurrency does not measure each candidate's maximum capacity.
  • Percentile scores are means of backend-reported percentiles, not pooled request percentiles.
  • Native Linux and multi-GPU were not exercised.

🤖 Generated with Claude Code

Objectives (#38) and the runtime update (#39) are now on main. Take main's
versions of that code, keep only the auto-mode modules and hooks, drop the
planning document, move the two auto tests to their own file, and mark the
guide experimental.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@brtydse100 brtydse100 changed the title Add automatic tuning with calibrated workloads and selectable objectives Add experimental automatic tuning with calibrated workloads Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant