feat(api): OpenAI-compatible logprobs for /v1/chat/completions (#1549)

[popfido]

Summary

Since I also think that logprobs is necessary for using oMLX as an local RL evaluation API source, I took a look at the influence scale of adding this functionality and found it can be quite a little.

Implements OpenAI-compatible per-token logprobs for /v1/chat/completions(streaming and non-streaming), closing #1549. Previously the server accepted logprobs: true / top_logprobs: N but always returned an empty
choices[].logprobs.content. The engine already computed a per-token logprob distribution for sampling; this PR plumbs it through the request → engine → scheduler → API layers and serializes it in the OpenAI shape ({token, logprob, bytes, top_logprobs[]}). Ported from mlx-vlm's implementation (the VLM-capable reference).

What changed

• Request/response schema (api/openai_models.py): logprobs/top_logprobs request fields + validator (range 0–20, requires logprobs); TopLogprob / ChatCompletionTokenLogprob / ChoiceLogprobs; logprobs on the choice and streaming-chunk-choice models.
• Config (settings.py, admin): SamplingSettings.top_logprobs_k server-side cap (default 20, clamped 0–20).
• Engine path (logprobs.py, request.py, scheduler.py, engine/*): early top-K extraction from the full-vocab vector (frees the ~800 KB array), RequestOutput.logprobs, accumulation across collector merges, GenerationOutput.logprobs.
• API serialization (server.py, api/utils.py): tokenizer-based token/bytes decoding; non-streaming attaches ChoiceLogprobs; streaming attaches aligned per-chunk logprobs.
• Streaming alignment (api/thinking.py): logprob-aware thinking parser so per-chunk logprobs.content stays aligned to content tokens through tag-lookahead buffering (fixes a <-in-content drift). Text-only path is byte-identical.

Design decisions

• Token order under thinking/tools: raw generation order (content tokens).
• top_logprobs cap: OpenAI's 0–20, plus a configurable server cap (top_logprobs_k, default 20); over-cap requests clamp.
• Speculative decoding: MTP and native DFlash report the proposing head's distribution, not the target model's, so logprobs are suppressed (null) there for now (DFlash fallback to the standard engine returns them).
• Performance: all logprobs work is gated behind the per-request flag; the disabled path is byte-for-byte unchanged.
• Invalid params return 422 via the repo's existing OpenAI-formatted RequestValidationError handler (house convention).

Verification

• Unit + integration: new tests in tests/test_logprobs_extraction.py and tests/integration/test_logprobs_api.py (extraction, formatter, thinking-parser alignment incl. the a<b case, non-streaming shape, streaming alignment, validation, server-cap clamp, disabled-path-forwards-no-kwargs). Full affected suite green (800+ tests).
• Live, real model (Qwen3.6-27B-UD-MLX-4bit):
  • Non-streaming logprobs:true, top_logprobs:2 → populated content with real top_logprobs and correct UTF-8 bytes.
  • Alignment: len(logprobs.content) == completion_tokens (99 == 99).
  • Streaming: per-chunk logprobs, aligned to content tokens.
  • Perf (128 tokens, temp 0): logprobs off ≈ 12.5 tok/s, on/top5 ≈ 12.9, on/top20 ≈ 11.9 — negligible overhead; off-path unaffected.

Limitations / follow-ups

• Streaming + active tool-call filtering omits logprobs (the filter buffers
  independently); non-streaming with tools returns them.
• True logprobs for MTP/DFlash speculative decoding.

Test plan

• pytest tests/test_logprobs_extraction.py tests/integration/test_logprobs_api.py
• Real-model curl smoke test (non-streaming + streaming)
• Perf check (off vs on)
• Reviewer: confirm 422 (vs OpenAI 400) is acceptable for invalid params