feat: add OpenAI-compatible serving and reuse vllm bench serve

[hsliuustc0106]

Problem and goal

vllm-lt has an offline M1 engine benchmark, but no HTTP server. We cannot yet measure end-to-end serving throughput and latency under controlled request rates and concurrency using vllm bench serve.

Add a minimal OpenAI-compatible serving frontend for Ouro and reuse the upstream benchmark client unchanged. Keep the M1 harness for loop-depth, scheduler, KV, and profiler analysis; HTTP measurements include a different timing boundary.

Related: #2 (serving frontend roadmap), #3 (offline benchmark). This frontend can wrap the existing synchronous engine; it does not require the learned lookahead gate or asynchronous GPU routing in #9.

Proposed scope

• Serve one resident model on one device. Use BF16 as the primary GPU target under the current precision policy, with accumulation precision specified independently.
• Implement /v1/completions, /v1/models, and /health. Support text prompts, ordinary and SSE streaming responses, incremental text deltas, finish reasons, accurate prompt/completion token usage, stream_options.include_usage, and the terminal [DONE] event. Preserve tokenizer behavior across partial text and special tokens.
• Map supported generation parameters to SamplingParams, including max_tokens, temperature, top-p/top-k, seed, and ignore_eos. Expose Ouro loop bounds and exit threshold as documented extension fields usable through the client's --extra-body. Honor neutral/default fields emitted by the pinned client, such as repetition_penalty=1 and logprobs=null; reject unsupported behavior explicitly.
• Give one worker exclusive ownership of add_request(), step(), and abort_request(). Accept new requests between steps so independent HTTP requests share continuous batching. Keep the HTTP event loop responsive while engine work runs; the engine is synchronous and not thread-safe.
• Bound pending requests and slow-client output buffering, define overload responses, and route disconnects/cancellation through the engine owner. Cover EOS, errors, shutdown, and request/KV cleanup without affecting other requests.
• Mark health ready only when model, tokenizer, and engine initialization have completed. Validate the first actual inference after readiness and report preparation separately from process-to-readiness.

The first delivery covers completions on the existing single-model engine. Chat templates, distributed serving, additional models, and a replacement benchmark client can be separate follow-ups.

Benchmark integration

Pin and record a compatible vLLM client version and document a command targeting the new server, for example:

vllm bench serve \
  --backend openai \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ByteDance/Ouro-1.4B \
  --tokenizer /path/to/prepared/ouro-tokenizer \
  --dataset-name random \
  --random-input-len 128 --random-output-len 64 \
  --num-prompts 100 --max-concurrency 8 --request-rate inf \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result --save-detailed

This is a proposed compatibility example, not an executed result or a tail-latency qualification. Include finite request-rate and concurrency-one examples as well. Report successful/failed requests, request/token throughput, TTFT, TPOT, ITL, and end-to-end latency. Verify stream framing does not turn a terminal/usage-only event into an extra token timing sample, and document any chunk-versus-token timing limitations of the pinned client.

For measured comparisons, freeze the hypothesis, isolated variable, controls, success criteria, workload, and stop budget before execution. Start with a CPU probe and one excluded feasibility run, then default to two measured runs per configuration. Keep exact reserved GPU IDs, model/tokenizer revisions, dtype, generation/depth policy, prompts, NUMA/thread placement, and cache state fixed. Reuse one server per configuration; separate downloads, preparation, warmups, and startup from measured requests. Preserve source SHA/modifications, commands, raw client/server results, variability, and task-owned cleanup records. No speedup is required to complete this feature.

Acceptance criteria

• CPU tests cover request validation, streaming/non-streaming output, Unicode/special-token handling, usage, and startup/failure readiness.
• Overlapping arrivals are admitted to the shared engine; cancellation, overload, slow-client handling, engine failure, and shutdown have bounded cleanup and preserve unrelated requests.
• A reserved BF16 real-model smoke run validates the first request after readiness and concurrent requests, with outputs checked against direct engine execution under the same declared generation contract.
• The pinned, unmodified vllm bench serve client completes both serial and concurrent workloads; saved results have correct token counts, usable timing metrics, and explicit failure accounting.
• Documentation includes server launch/configuration, reproducible benchmark commands, retained evidence, supported API limits, and the distinction between HTTP and M1 engine timings.

References

• Current engine API and server limitation
• Existing M1 harness
• Upstream benchmark CLI