Skip to content

Add selectable latency and throughput objectives for manual search spaces - #38

Merged
brtydse100 merged 3 commits into
mainfrom
feat/objectives-percentiles
Nov 17, 2026
Merged

brtydse100 merged 3 commits into
mainfrom
feat/objectives-percentiles

Conversation

@brtydse100

Copy link
Copy Markdown
Owner

Split out of #34 so the objective syntax can ship without auto mode. It works with existing manual tune spaces.

optimization:
  objective:
    metric: end_to_end_latency_ms
    percentile: 99
  sampler: tpe
  trials: 20

Behaviour

  • Latency metrics are minimized, as a mean or an integer percentile. Throughput metrics are maximized and take no percentile.
  • Direction is carried through search, ranking, finalist selection, persistence, live and offline reports, retries, reclassification, and redaction.
  • vLLM Bench Serve supports percentiles 1–100; the selected percentile is added to each run's metric-percentiles. GuideLLM supports the percentiles it emits; others fail validation.
  • optimization.maximize keeps its behaviour. Latency statistic keys such as end_to_end_latency_ms.p99 are now rejected under maximize with a pointer to objective; the sampler would have maximized them while ranking minimized them.

Differences from the code in #34

  • The percentile-request logic moved from the auto-mode module to config/objective_percentiles.py. It now merges into whichever argument spelling was written, treats false as an omitted flag, and rejects duplicate aliases and non-boolean inline values.
  • Mean extraction keeps the earlier order, so runs stored by older versions regenerate with the same score.
  • No auto-mode code, no runtime version change, and no planning document.

Validation at 88c9e6e

  • Private and public suites on vLLM 0.30.0 / GuideLLM 0.7.4 in WSL: 1,635 passed, 7 failed, 2 collection errors. The same 7 failures and 2 errors occur on main at e02d8d9; they come from hardcoded local paths and a missing helper module. Six private auto-mode test files were excluded because that code is not on this branch.
  • Public tests: 157 passed. Coverage 81%; all 34 module gates pass.
  • Ruff lint and format, mypy, and mkdocs build --strict pass.
  • An independent read-only review found four issues; all are fixed in the second commit.

Real GPU

RTX 3080 10 GB, WSL2, CUDA 13.0, torch 2.13.0, vLLM 0.30.0, GuideLLM 0.7.4, local OPT-125M FP16. Manual space max-num-seqs in [2, 4] with baseline 8, TPE with 2 trials, 16 requests of 32 input / 16 output tokens, 2 repeats, temperature 0.

Case Baseline (ms) Tuned trials (ms) Selected
vLLM Bench Serve, P55 end-to-end 90.18 90.25, 90.88 baseline
vLLM Bench Serve, P99 end-to-end 256.30 254.92, 269.96 254.92
GuideLLM, P99 end-to-end 265.30 246.45, 262.18 246.45

Every run recorded a minimize objective, ranked lowest-first, and selected the lowest score, including the case where the baseline won. Report regeneration, reclassification, and retry planning kept the objective, and the server port was released after each case.

This is a behaviour check on a small model with 16 requests. The latency differences are not evidence of a tuning gain. Evidence is private at vTune-private-tests/objective-percentiles-evidence/88c9e6e/.

Limitations

  • A percentile score is the mean of backend-reported percentiles across workloads and repeats, not a percentile of pooled requests.
  • Reports label legacy maximize runs as Maximize X (mean).
  • Six files taken from Add experimental automatic tuning with calibrated workloads #34 are above the 150-line target; none exceeds 200.
  • Native Linux and multi-GPU were not exercised.

🤖 Generated with Claude Code

brtydse100 and others added 3 commits November 17, 2026 00:30
…aces

Split from the automatic-tuning work so the objective syntax can ship on its
own. `optimization.objective` minimizes a latency mean or percentile, or
maximizes a throughput metric; direction is carried through search, ranking,
finalist selection, persistence, reports, retries, reclassification, and
redaction.

- Move the percentile-request logic into config/objective_percentiles.py and
  merge into whichever argument spelling the user wrote.
- Reject latency statistic keys under `optimization.maximize`, which the
  sampler would maximize while ranking minimized.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Address independent review findings on the objective change:

- Mean extraction reuses the earlier order (outer average, successful mean,
  outer mean) so runs stored by older versions regenerate with the same score.
- Percentile argument merging treats `false` as an omitted flag and rejects
  duplicate aliases and non-boolean inline values, matching command building.
- A zero improvement is reported as 0.0 rather than -0.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@brtydse100
brtydse100 marked this pull request as ready for review November 17, 2026 06:41
@brtydse100
brtydse100 merged commit 0af2c14 into main Nov 17, 2026
5 checks passed
brtydse100 added a commit that referenced this pull request Nov 17, 2026
Objectives (#38) and the runtime update (#39) are now on main. Take main's
versions of that code, keep only the auto-mode modules and hooks, drop the
planning document, move the two auto tests to their own file, and mark the
guide experimental.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant