From 73566588538e4956141be640a9a1603280785f11 Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Sat, 1 Aug 2026 00:09:31 +0000 Subject: [PATCH] docs(cli): spike chat and complete parity Correct the pinned-vLLM CLI inventory and define CPU-verifiable implementation leaves for remote chat/complete commands while preserving the existing local invocation. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/coordination.md | 15 ++ .agents/engine-matrix.md | 6 +- .agents/feature-matrix.md | 2 +- .agents/parity-ledger.md | 1 + .agents/porting-inventory.md | 2 +- .agents/roadmap_v1.md | 2 +- .agents/specs/cli-chat-complete.md | 243 +++++++++++++++++++++++++++++ .agents/state.md | 28 ++++ docs/BENCHMARKS.md | 15 ++ docs/STATUS.md | 1 + 10 files changed, 309 insertions(+), 6 deletions(-) create mode 100644 .agents/specs/cli-chat-complete.md diff --git a/.agents/coordination.md b/.agents/coordination.md index 96f6f388..842151c2 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,6 +118,20 @@ without the selected contention proof for their entire run are discarded. ## Active claims +**Interactive CLI chat/complete spike (`SERVE-CLI-CHAT`, 2026-08-01, +`CLAIM-SERVE-CLI-CHAT-SPIKE`).** Codex (GPT-5), isolated worktree +`/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`, +branch `codex/serve-cli-chat-spike`, base `upstream/main` `1448e981`. CPU-only, +records/spec checkpoint: inventory the pinned vLLM CLI surface and the local +CLI/engine/serving seams, define exact interactive-chat and one-shot-complete +contracts, ported-test inventory, error/exit behavior, and row-sized follow-on +leaves. Owns only NEW `.agents/specs/cli-chat-complete.md`, the +`SERVE-CLI-CHAT` row, this claim, the matching roadmap/status checkpoint text, +`docs/BENCHMARKS.md`, `.agents/parity-ledger.md`, and append-only +`.agents/state.md`. No source, header, CMake, test, model, kernel, generated +fixture, or GPU/model-download change; verification is the CPU record/doc +checker suite. The row and open-PR list were unclaimed at selection time. + **Canonical DONE-owner reachability repair (`KV-PREFIX-CACHE`, `SAMPLE-LOGPROBS`, `SPEC-DFLASH`, `MODEL-SPEC-qwen3-dflash-dflash-qwen3-for-causal-lm`, @@ -1313,6 +1327,7 @@ table, tests, CMake. Details in the state-log entry of the same date. | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| +| `CLAIM-SERVE-CLI-CHAT-SPIKE` | `SERVE-CLI-CHAT` (`INVENTORIED` -> `SPIKE`) | Codex (GPT-5) | isolated worktree `/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`; CPU-only records/spec, no GPU/model/download | branch `codex/serve-cli-chat-spike`, base `upstream/main` `1448e981` | Owns only NEW `.agents/specs/cli-chat-complete.md`; the `SERVE-CLI-CHAT` engine-matrix row and rollup; matching feature-matrix, porting-inventory, roadmap, STATUS, BENCHMARKS, coordination, ledger, and append-only state entries. No source/header/CMake/test/README/model/kernel/fixture edits. | `SPIKE` | 2026-08-01 - pinned-source inventory corrected: vLLM does ship remote `chat`/`complete`; accepted dual-mode design preserves the existing local invocation. CPU record/doc checker suite is the closing gate. | | `CLAIM-LAGUNA-W1W2` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (INVENTORIED→ACTIVE) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_2ae79b7a-246-1`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release `-Werror`); NO GPU, NO 73 GB download — structural bring-up + oracle DECISION only | branch `spike/laguna-s21-w1w2`, base `main` `5c3da2f1` | Laguna-S-2.1 W1 oracle-decision + W2 structural bring-up. Owns ONLY: NEW `include/vllm/model_executor/models/laguna.h`, NEW `src/vllm/model_executor/models/{laguna_registry,laguna_weights,laguna}.cpp`, NEW `tests/vllm/models/test_laguna_scaffold.cpp`, NEW `.agents/specs/laguna-s21-w1w2-2026-07-30.md`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `LagunaForCausalLM` sorted-set + error-message insert in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row (INVENTORIED→ACTIVE) + checklist rollup (ACTIVE 23→24 / INVENTORIED 285→284, engaged 42→43), and the record surfaces (this claim, roadmap breadth, docs/STATUS, docs/BENCHMARKS, parity-ledger, state). **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` W3 stub so no production path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-07-30 — **W1 oracle-decision + W2 structural bring-up LANDED (foreground, NOT pushed).** Registry (`laguna`/`LagunaForCausalLM`) + `ParseLagunaParams` (nested dual-rope + variable Q-head + ungrouped sigmoid-noaux MoE) + GGUF `blk.N.*` name-map + UD-Q4_K_XL quant-mix (ZERO new decode kernel) + KV-cache spec + per-layer forward-composition scaffold with reuse citations. `test_laguna_scaffold` 3/3·40 + `test_model_registry` 24/24; CPU full-library `-Werror` clean; record checkers rc=0. RESIDUALS (W3/W4): device materialization + real forward + the 3 new ops + strict dual-oracle gate on a fetched checkpoint. **SUPERSEDED by `CLAIM-LAGUNA-W3` (2026-07-31) which landed the real forward + the 3 new ops.** | | `CLAIM-LAGUNA-W3` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (stays `ACTIVE`; W3 real forward + the 3 new ops landed; real-model dual-oracle gate still PENDING W4) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_43e61a78-0e2-1`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release `-Werror`); NO GPU, NO 73 GB download — real forward CODE + unit gates only | branch `laguna-s21-w3`, base `main` `f2e463d5` (confirmed via `git rev-parse HEAD`) | Laguna-S-2.1 W3 — turn the W1/W2 `VT_CHECK(false)` forward stub into a REAL runnable host-reference composition + land the 3 genuinely-NEW small host ops. Owns ONLY: NEW `include/vllm/model_executor/models/laguna_ops.h` + `src/vllm/model_executor/models/laguna_ops.cpp` (softplus head-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders), the rewritten `src/vllm/model_executor/models/laguna.cpp` (`LagunaModel::Forward` real composition), the `LagunaParams` per-layer variable-Q-head helpers in `include/vllm/model_executor/models/laguna.h`, the new-op + forward unit cases appended to `tests/vllm/models/test_laguna_scaffold.cpp`, the `laguna_ops.cpp` line in `CMakeLists.txt`, NEW `.agents/specs/laguna-s21-w3-2026-07-31.md`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row cells + this claim, and docs/STATUS + docs/BENCHMARKS pointers. **NON-COLLISION:** file-disjoint from the concurrent MLA-fold lane (`mla_attention.cpp`/`deepseek_v2.cpp` untouched); additive `laguna_ops` TU + one CMake line; the loaders still `VT_CHECK(false)` so NO production/device path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `ACTIVE` | 2026-07-31 — **W3 REAL forward + 3 new ops LANDED + UNIT-GATED (foreground, NOT pushed).** `laguna_ops.cpp`: `LagunaSoftplusHeadGate` (per-head softplus out-gate), `LagunaUngroupedRouterTopK` (sigmoid noaux_tc MINUS the group step + tie-break razor: lower index on equal choice, UNBIASED weights, renorm, routed_scaling), `BuildLaguna{FullYarn,Sliding}CosSin` (dual per-layer RoPE, reusing the pinned `compute_yarn_inv_freq` over the partial-64 dims). `LagunaModel::Forward` is now a REAL runnable f32 host-reference composition (variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head). `test_laguna_scaffold` **8/8·166** (softplus math; router selection + tie-break RED-first; dual-RoPE cos/sin bit-match vs hand ref; variable-Q-head shapes; forward composition on synthetic weights — RUNS, deterministic, gather==full-row, softplus gate wired) + `test_model_registry` 24/24; CPU full-library `-DVLLM_CPP_CUDA=OFF` `-Werror` clean; all record checkers rc=0. **HONEST residual (DEFERRED W4, needs the 73 GB checkpoint):** GGUF keep-quant tower materialization (loaders still LOUDLY throw) + device/paged production forward (runner variable-Q-head device wiring) + the strict dual-oracle greedy gate (llama.cpp-Q4_K token-exact + vLLM-NVFP4 near-tie). Risks: dual-RoPE numerics vs the fork on the real config; the reference forward is f32 whole-sequence (bf16 paged token-exactness is a W4 boundary); router tie-break vs the oracle's actual greedy selection. Row stays `ACTIVE`. | | `CLAIM-LAGUNA-W4` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (stays `ACTIVE`; checkpoint fetched + fidelity corrected; real-model greedy gate is the W5 close) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_63140a7b-03e-1`; DGX `dgx.casa` GB10 for the 73.4 GiB fetch + GGUF metadata read + llama.cpp oracle build (foreground); CPU-verified fidelity corrections; NOT pushed | branch `worktree-wf_63140a7b-03e-1`, base `main` `570510a9` | Laguna-S-2.1 W4 — FETCH the UD-Q4_K_XL GGUF, read its metadata + tensor map AUTHORITATIVELY, and correct the fidelity errors the W1-W3 scaffold made from config.json guesses. Owns ONLY: `laguna.h`/`laguna_ops.{h,cpp}`/`laguna.cpp`/`laguna_weights.cpp` (QK-RMSNorm + `LagunaYarnMscale` + separate gate/up + verified name-map/quant-mix), the `test_laguna_scaffold.cpp` cases for those, NEW `.agents/specs/laguna-s21-w4-2026-07-31.md`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row cells + this claim, docs/STATUS + docs/BENCHMARKS pointers. **NON-COLLISION:** laguna-only additive edits; loaders still throw the keep-quant residual so NO production/device path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `ACTIVE` | 2026-07-31 — **Checkpoint FETCHED + arch grounded in REAL bytes + 3 fidelity bugs fixed.** UD-Q4_K_XL GGUF (73.4 GiB, 3 shards, 814 tensors) fetched to dgx; metadata read authoritatively (arch `laguna`, `expert_gating_func=2` sigmoid, `expert_weights_scale=2.5`, `leading_dense_block_count=1`, rope factor 32/yarn_attn_factor 1.0, per-layer head_count `[48,72,72,72]`, quant mix: attn Q8_0 / experts gate-up Q4_K + down Q5_K / shared Q8_0 / router+norms F32). Fixed: (a) per-head QK-RMSNorm `attn_q/k_norm` (scope MISSED it — no config flag), (b) dual-RoPE mscale via llama.cpp `yarn_attn_factor·(1+0.1·ln(factor))` off GGUF factor 32 (not HF 128/1.4852), (c) SEPARATE `ffn_gate/up_exps`. Oracle = `poolsideai/llama.cpp@laguna` (mainline b10087+) same-quant; no vLLM-GGUF path for `laguna`. **HONEST residual (W5 close, needs the resident 73 GB run):** keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + host-orchestrated `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + dual-RoPE inv_freq ramp bit-match + the real greedy run vs the llama.cpp same-quant oracle (token-exact or characterized near-tie). Row stays `ACTIVE`. | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 9d060040..08ff9768 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 0 | 2 | 0 | 0 | 7 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 12 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 2 | -| Serving, API, CLI, library | 22 | 3 | 2 | 1 | 0 | 7 | 2 | 1 | 6 | +| Serving, API, CLI, library | 22 | 3 | 2 | 2 | 0 | 7 | 2 | 1 | 5 | | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 0 | 0 | 0 | 1 | 5 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 1 | 3 | 0 | 0 | 2 | 1 | 1 | 1 | -| **Total** | **131** | **8** | **16** | **4** | **4** | **48** | **8** | **9** | **33** | +| **Total** | **131** | **8** | **16** | **5** | **4** | **48** | **8** | **9** | **32** | ## Engine core and scheduling @@ -198,7 +198,7 @@ claims it. | `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; `examples/bench/main.cpp:40,109`; `examples/bench/bench_core.h:96,468` | server help contract `examples/CMakeLists.txt:34`; benchmark `tests/examples/test_bench.cpp:15,48` | `planned: specs/cli-serve-bench.md` | `PARTIAL` | - | | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ACTIVE` | CLAIM-SERVE-GATE-1 | | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - | -| `SERVE-CLI-CHAT` | Interactive chat and complete commands | T1 | `vllm/entrypoints/cli/main.py:18-34` has no direct chat/complete command at the pin; project extension | - | - | `planned: specs/cli-chat-complete.md` | `INVENTORIED` | - | +| `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `SPIKE` | `CLAIM-SERVE-CLI-CHAT-SPIKE` | | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md) — endpoints depend on the pooling RUNNER (`ENG-POOLING-RUNNER`, W3 LANDED 2026-07-29 — `PoolingRunner` returns pooled data via the model `Pooler`) and a concrete pooling model. Endpoint protocol + handler port is the W4 brick | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | - | - | [pooling-task-class.md](specs/pooling-task-class.md) | `SPIKE` | `CLAIM-POOLING` | | `ENG-POOLER-SEQ` | The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. **W1 LANDED + CPU-GATED 2026-07-28 (`CLAIM-POOLING`, NOT pushed):** the sequence pooling methods `CLSPool`/`LastPool`/`MeanPool` (+ `GetSeqPoolingMethod` factory) over a packed `[num_tokens, hidden]` CPU buffer keyed by a minimal `PoolingCursor` (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads `PoolerIdentity`/`PoolerNormalize` (L2 `F.normalize`)/`PoolerMultiLabelClassify` (sigmoid)/`PoolerClassify` (sigmoid if `num_labels<2` else `softmax`). Unit-gated vs DOUBLE-PRECISION references, RED-first. **W2 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** the pooler HEADS composite (`EmbeddingPoolerHead` = projector→matryoshka→normalize; `ClassifierPoolerHead` = classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` (method∩head task intersection) + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` groupby-task routing (`ForEmbedding`/`ForSeqCls` + a mixed embed+classify batch + ctor task-support validation), and the `PoolerConfig`/`PoolingParams`/`PoolingParamsUpdate` structs; `test_pooler_heads` 27/27 (240 asserts) vs double-precision refs, RED-first (disable matryoshka slice + logit_mean calibration → 8 cases / 50 asserts fail). RESIDUALS (named, spec §Work breakdown): the endpoints (W4), tokwise `AllPool`/`StepPool` (W5), a concrete pooling MODEL + real-oracle cosine gate (W3-model — see `ENG-POOLING-RUNNER`) | T2 | `vllm/model_executor/layers/pooler/seqwise/methods.py:35-121`; `vllm/model_executor/layers/pooler/activations.py:106-158`; `vllm/model_executor/layers/pooler/seqwise/heads.py:19-196`; `vllm/model_executor/layers/pooler/seqwise/poolers.py:41-138`; `vllm/model_executor/layers/pooler/special.py:23-140`; `vllm/model_executor/layers/pooler/common.py:12-30`; `vllm/pooling_params.py:35-70`; `vllm/config/pooler.py:16-90`; `vllm/v1/pool/metadata.py:13-71`; `tests/model_executor/layers/test_pooler_methods.py`, `tests/model_executor/layers/test_pooler_activations.py`, `tests/model_executor/layers/test_pooler_heads.py` | `include/vllm/model_executor/layers/pooler/{methods,activations,pooling_metadata,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h` + `src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp` | `tests/vllm/model_executor/layers/pooler/test_pooler.cpp` (CLS/LAST/MEAN + factory + activations, 50 asserts) + `test_pooler_heads.cpp` (Embedding/Classifier heads + SequencePooler + DispatchPooler, 240 asserts) | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` | | `ENG-POOLING-RUNNER` | The pooling RUNNER path — where the generation runner SAMPLES a token, the pooling runner applies the model's `Pooler` to the last hidden state and returns the POOLED DATA (embedding vector / classification logit row). **W3 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** `PoolingRunner` over a packed `[num_tokens, hidden]` last-hidden-state buffer + a `PoolingMetadata` — `Pool()` delegates to the model pooler (`DispatchPooler.ForEmbedding`), `GetSupportedTasks()`, `ComputeValid()` (`seq_lens==prompt_len`). GATE: a STRUCTURAL cosine-parity gate — the runner's embedding vs an independent double-precision LAST+normalize reference is cosine≈1 (5 cases / 14 asserts), RED-first (CLS-instead-of-LAST drops cosine <0.5; disable normalize → 2 unit-L2 asserts fail). GENERALIZATION DEVIATION: upstream `pooling_runner.py` hardcodes LAST+normalize; we route through the model `Pooler` (the general bert.py path), strictly more capable. HONEST RESIDUAL (named): the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) needs a registered concrete embedding model's forward — no such model is registered yet (W3-model), so no cosine-vs-oracle number is fabricated. The InputBatch→PoolingMetadata construction (logits_indices gather) rides the endpoint brick (W4) | T2 | `vllm/v1/worker/gpu/pool/pooling_runner.py:18-46`; `vllm/tasks.py:10`; `tests/models/language/pooling/test_embedding.py` (real-oracle gate, DEFERRED) | `include/vllm/v1/worker/gpu/pool/pooling_runner.h` + `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp` | `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp` (runner path + structural cosine gate, 14 asserts, RED-first) | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` | diff --git a/.agents/feature-matrix.md b/.agents/feature-matrix.md index 7212695e..b5fea204 100644 --- a/.agents/feature-matrix.md +++ b/.agents/feature-matrix.md @@ -235,7 +235,7 @@ is configured, exactly as upstream loads its draft model on demand. | C API library (llama.cpp-style, 17-symbol ABI) | — (our packaging) | `ANCHOR-BACKFILL` T0 | dlopen/FFI proof now includes six additive nonblocking request submit/cancel/wait/done/error/free symbols over AsyncLLM; CPU/TSan green, not a claim of LocalAI integration | `planned: specs/c-api-library.md` | | Rich C++ API (`LLM`/`AsyncLLM` mirror) | `entrypoints/llm.py` | ☐ T1 | | `planned: specs/cpp-api.md` | | CLI: `serve` + `bench {latency,throughput,serve}` | `entrypoints/cli/` | `PARTIAL` T0 | separate server/bench binaries and one in-process benchmark; server now exposes `max_num_seqs`/`max_num_batched_tokens` for reproducible operating points, but no matching command family | `planned: specs/cli-serve-bench.md` | -| CLI: `chat`, `complete` | `entrypoints/cli/` | ☐ T1 | examples/cli covers basic complete | `planned: specs/cli-chat-complete.md` | +| CLI: `chat`, `complete` | `entrypoints/cli/openai.py:30-312` | `SPIKE` T1 | The existing `vllm-cli` covers one in-process completion only. The accepted spike corrects the old inventory: pinned vLLM does ship remote `chat`/`complete`. Selected design mirrors both command contracts over OpenAI HTTP/SSE while preserving the local invocation as a compatibility alias; implementation is not yet present. | [specs/cli-chat-complete.md](specs/cli-chat-complete.md) (`SERVE-CLI-CHAT`) | | `/v1/embeddings`, `/pooling`, `/score`, `/rerank` | pooling routers | ☐ T2 | with pooling models (§4) | `planned: specs/pooling-endpoints.md` | | `/v1/responses`, `/v1/messages` (Anthropic-style), audio | responses/messages routers | ☐ T2 | includes `/v1/responses`(+retrieve/cancel) `responses/api_router.py:48`, `/v1/messages`(+count_tokens) `anthropic/api_router.py:49`, and the audio `/v1/audio/{transcriptions,translations}` `speech_to_text/transcription/api_router.py:31` — we have Whisper/Voxtral encode+decode but the transcription ENDPOINT is unwired (`SERVE-RESPONSES-MESSAGES`) | `planned: specs/responses-messages-endpoints.md` | | Offline Batch API (JSONL file runner) | `entrypoints/openai/run_batch.py:793` (`run_batch`); schema `:148-228`; batched chat/embed/score/transcription/translation | ◑ **ACTIVE** T2 (`SERVE-BATCH-API`) | ROW CREATED 2026-07-29 (`CLAIM-BATCH-API`): `RunBatch`/`RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map. `/v1/chat/completions` wired; `BatchRequestOutput` schema + custom_id echo + per-line error isolation; unit-gated RED-first (7/80). Recorded deviation: a malformed line is isolated into an error row (batch continues) where upstream aborts. Residuals: the `vllm run-batch` CLI, embeddings/score/rerank + audio dispatch, http(s)/data-URL I/O | [batch-api.md](specs/batch-api.md) (`SERVE-BATCH-API`) | diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index b8028442..1156d192 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -885,3 +885,4 @@ Columns: | 2026-07-29 (**Post-rebase RE-VALIDATION of the 4B lever onto `main` `f3ecbe70d`, +139 upstream commits: nothing moved**; `CLAIM-POSTREBASE-REVALIDATE`; local working branch `bench-lever-sampled-token-20260727` rebased from `7f620e74`, build `build-nix-cuda-transplant-triton` clean 925/925 with 0 warnings; ALL GPU under one `flock /tmp/gpu`, all 9 performance legs at 0% utilization; NOT pushed; closing commit: this checkpoint) | **What it does.** Re-measures the Qwen3.5-4B matched comparison after the branch was rebased across 139 upstream commits, against the SAME source-built oracle at the parity pin (`0.23.1rc1.dev1511+g555967922`; pin re-checked and unchanged before the run). Result is a null: 0.9972x total throughput (was 0.9970x), TTFT 0.7701x PASS, TPOT 1.1247x FAIL — every axis inside noise. Records-and-evidence only; no engine change. | vLLM `555967922` (0.26.0.dev0) built from source at the pin, run through the same `run_qwen35_4b_compare.sh` harness as the 2026-07-28 series; no upstream code ported. | **VERIFIED, and the control is the point.** Our output is token-identical 128/128 per repetition to the pre-rebase series on BOTH the direct-ON and direct-OFF arms, which is the semantic check on three rebase conflict resolutions (two append-only record files, plus a real `ModelForwardInput` field collision with upstream's `mm`). All three arms drifted down ~0.13% (ours 0.9988x, OFF 0.9983x, pin 0.9986x) — the pin arm is an UNCHANGED binary, so that drift is ambient, not code. Gates re-run green: `test_qwen35_plain_weights --no-skip` 3/3·1672/1672, `test_input_batch` 25/25·183/183, `test_combine_tokens` 7/7·14/14. Evidence: [docs/bench-evidence/qwen35-4b-postrebase-20260729.md](../docs/bench-evidence/qwen35-4b-postrebase-20260729.md). | | 2026-07-31 (`SERVE-C-ABI` W0 contract spike; `CLAIM-SERVE-C-ABI-SPIKE`; CPU-only records/docs) | Accepted `.agents/specs/c-api-library.md` for the already-shipped original C packaging layer: complete scope, vLLM semantic chain/deviation, ABI v10/19-symbol baseline, ownership/error/version/dispatch rules, exact code/test anchors, gates, dependencies, risks, and W1-W5 follow-ons. Also fixes the verified stale public `VLLM_ABI_VERSION 9` labels in README/USAGE to the source-of-truth v10 and adds the missing v10 usage-table entry. No production/test/CMake source changed. | Pinned vLLM `555967922` has no C ABI; behavior beneath the adapter remains owned by its vLLM-derived engine rows. The flat ABI is the recorded llama.cpp-style packaging deviation and may translate, never reimplement, policy. | **CPU/records gate only; benchmark NOT APPLICABLE.** Focused C11/C++/dlopen/export gate passed 3/3 after explicitly building `vllm_shared`; five record checkers pass. `check-agent-record` reports the base tree's same six missing closing-commit objects (`444ea9d7`, `7a3f04b2`, `164453a2`), none in this row/diff. Row stays `ANCHOR-BACKFILL` because all-symbol dlsym coverage (chat symbols currently omitted), historical-layout compatibility, allocation-failure no-throw proof, lifetime sanitizer stress, and a standalone real-model C consumer remain W1-W5. | | 2026-07-31 (`CLAIM-CPU-GCC12-WERROR-PORTABILITY`; maintenance, rows `QUANT-GGUF-KEEPQ-LOADER` + `KV-OFFLOAD`; lifecycle unchanged) | Removes two GCC 12 production-library `-Werror` blockers without suppressions: the GGUF prefault keeps the same one-byte-per-page volatile XOR but uses simple assignment, and the KV filesystem tier builds the identical `...tmp` suffix with append operations inside its thread-local initializer. No API, algorithm, default, CUDA, fixture, or golden change. | Behavior remains grounded in the accepted loader and KV-persistence leaf specs: llama.cpp mmap prefault intent and vLLM `tiering/fs/io.py` unique temporary-file publication. This is compiler portability, not a parity-surface change. | RED: GCC 12 failed first at `qwen3_5_gguf_weights.cpp:49` (`-Wvolatile`), then at `fs_io.cpp:66` (`-Wrestrict`). GREEN: production `vllm` and focused test targets build clean; focused CTest 2/2 (`test_gguf_keep_quant`, `test_kv_offload_fs`). Full all-target build is PARTIAL at 42% on unrelated test-only GCC 12 `-Wrestrict` diagnostics in `test_deepseek_v2_paged_engine.cpp` and `test_glm4_moe_lite_paged_engine.cpp`; no full-CTest claim. Benchmark NOT APPLICABLE. | +| 2026-08-01 (`SERVE-CLI-CHAT` W0 contract spike; `CLAIM-SERVE-CLI-CHAT-SPIKE`; CPU-only records/spec) | Accepts `.agents/specs/cli-chat-complete.md`, corrects the inventory from “no direct commands” to the actual pinned `chat`/`complete` surface, and decomposes a dual-mode port: exact remote OpenAI HTTP/SSE commands plus preservation of the existing in-process invocation as a compatibility alias. No production, test, CMake, model, kernel, fixture, or generated file changes. | Pinned vLLM `5559679229`: command registration `vllm/entrypoints/cli/main.py:17-37,73-98`; model/auth resolution and stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312`. The local compatibility baseline is `examples/cli/main.cpp:1-207`. | CPU record/doc gates only; benchmark `NOT APPLICABLE`, `benchmark_binding=false`. Implementation remains absent and the row moves `INVENTORIED` -> `SPIKE`. W1-W5 name parse, transport, complete, chat, and packaging gates, including fake-server request/SSE transcript parity, Release `-Werror`, ASan+UBSan, and TSan. | diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index fb6db422..c41c496a 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -319,7 +319,7 @@ before this becomes supported. | `/v1/embeddings`, `/pooling`, `/score`, `/rerank` | pooling routers | T2 | | `/v1/responses`, `/v1/messages` (Anthropic-style), audio endpoints | responses/messages routers | T2 | | Sleep/pause/resume, LoRA load/unload, profiling, RL weight-update endpoints | various | T2–T3 | -| CLI: `serve`, `bench {latency,throughput,serve}`, `chat`, `complete` | `entrypoints/cli/` | T0 **partial** (separate example server + one in-process bench), T1 matching command families/rest | +| CLI: `serve`, `bench {latency,throughput,serve}`, `chat`, `complete` | `entrypoints/cli/` | T0 **partial** (separate example server + one in-process bench); T1 `chat`/`complete` spike accepted at [cli-chat-complete.md](specs/cli-chat-complete.md), correcting the prior false claim that the pin lacked those commands. Remote command implementation remains open under `SERVE-CLI-CHAT` | | Offline API: `generate`, `chat`, streaming enqueue; `get_metrics` | `entrypoints/llm.py` | T1 parity surface; bounded synchronous C/C++ entry points exist | | Prometheus metric names **1:1** (`vllm:num_requests_running`, `vllm:time_to_first_token_seconds`, `vllm:kv_cache_usage_perc`, …) | `v1/metrics/` | T0 (core set), T1 (full set) | | OTLP tracing | `config/observability.py` | T2 | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index e1362df7..ee8d437e 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -533,7 +533,7 @@ diagnostics recorded in `docs/BENCHMARKS.md`. | A3 | `test_ops_fused_chain` FMA-contraction fix | ✅ merged bf48edb (`-ffp-contract=off` host-wide) | | A4 | De-Python the build: vendor Triton AOT artifacts per-arch (`triton_aot_vendored//` + MANIFEST; `VLLM_CPP_TRITON_REGEN` = maintainer-only Python) | ✅ **DONE** (54367cc..a432461; reproducibility hardening `09f1d23`) — `sm_121a` now has 48 generated C/H files + MANIFEST, including both bf16 `chunk_o` shapes; normal builds remain Python-free. Regen is explicit-target (`cuda:121:32`), line-info-disabled and byte-reproducible across source paths; the pure checker makes source/contract/artifact drift fatal and mutation-tests missing/extra/changed artifacts. A4 remains closed; fresh current-main CUDA/runtime/performance validation belongs to the two ACTIVE `CLAIM-PR3` kernel rows (evidence: porting-inventory §9). | | `SERVE-E2E-NIGHTLY` (formerly A5) | e2e suites per gates.md (server conformance nightly on dgx etc.) | ☐ next; leaf spike required | -| A6 | **User-facing surface closure** (audit `CLAIM-DOCS-COVERAGE`, 2026-07-24): make what SHIPPED reachable and truthfully documented. The audit found 2 class-D overstatements, 5 stale doc lines, 4 shipped-but-unreachable features and 153 undocumented env vars | 🚧 Tier 1 (falsehood repair) DONE in the audit commit. **Tier 2 DONE 2026-07-24 (`CLAIM-DOCS-T2-FIXES`)**, 3 of its 4 items: the D1 guard (a per-connector worker-half capability predicate; the disk connector is now REFUSED at engine construction on every device instead of silently shortcutting prefill for KV nothing writes — **its worker half remains unimplemented and is NOT claimed**), server `--tool-call-parser`/`--reasoning-parser` (40 tool names / 7 reasoning parsers now reachable over HTTP; defaults reproduce the old hardcode exactly), and server `--kv-transfer-config` mirroring vLLM's own CLI + JSON shape. Also landed from Tier 3/4: `docs/KV-OFFLOAD.md`, the README flag rows, the three missing Gemma model rows, and the `SERVE-C-ABI` 17→19 record repair. **Comprehensive refresh DONE 2026-07-25 (`CLAIM-DOCS-REFRESH`)**: **Tier 4** `docs/ENVIRONMENT.md` (26 user-facing env vars documented of 153 swept, kernel-internal tail on `scripts/env-doc-allowlist.txt`) + its CI checker `scripts/check-env-doc.py` (mutation-tested, fails on a new undocumented+unallowlisted production env var); the **Tier 3** README refresh to current git history (spec-decode `--speculative-config` now documented as shipped, the "not user-visible" class-B falsehood fixed, C ABI `VLLM_ABI_VERSION 6`, CLI spec-config flag); new usage guide `docs/SPECULATIVE-DECODING.md`. STILL OPEN: `vllm-cli` structured-output/sampling flags (Tier 2 item 4, code, not docs). **C-ABI contract W0 2026-07-31 (`CLAIM-SERVE-C-ABI-SPIKE`):** accepted [c-api-library.md](specs/c-api-library.md), corrected public ABI v9 labels to v10, and named W1-W5 compatibility/no-throw/lifetime gates; `SERVE-C-ABI` honestly remains `ANCHOR-BACKFILL`. Full table: [specs/docs-coverage-audit.md](specs/docs-coverage-audit.md) | +| A6 | **User-facing surface closure** (audit `CLAIM-DOCS-COVERAGE`, 2026-07-24): make what SHIPPED reachable and truthfully documented. The audit found 2 class-D overstatements, 5 stale doc lines, 4 shipped-but-unreachable features and 153 undocumented env vars | 🚧 Tier 1 (falsehood repair) DONE in the audit commit. **Tier 2 DONE 2026-07-24 (`CLAIM-DOCS-T2-FIXES`)**, 3 of its 4 items: the D1 guard (a per-connector worker-half capability predicate; the disk connector is now REFUSED at engine construction on every device instead of silently shortcutting prefill for KV nothing writes — **its worker half remains unimplemented and is NOT claimed**), server `--tool-call-parser`/`--reasoning-parser` (40 tool names / 7 reasoning parsers now reachable over HTTP; defaults reproduce the old hardcode exactly), and server `--kv-transfer-config` mirroring vLLM's own CLI + JSON shape. Also landed from Tier 3/4: `docs/KV-OFFLOAD.md`, the README flag rows, the three missing Gemma model rows, and the `SERVE-C-ABI` 17→19 record repair. **Comprehensive refresh DONE 2026-07-25 (`CLAIM-DOCS-REFRESH`)**: **Tier 4** `docs/ENVIRONMENT.md` (26 user-facing env vars documented of 153 swept, kernel-internal tail on `scripts/env-doc-allowlist.txt`) + its CI checker `scripts/check-env-doc.py` (mutation-tested, fails on a new undocumented+unallowlisted production env var); the **Tier 3** README refresh to current git history (spec-decode `--speculative-config` now documented as shipped, the "not user-visible" class-B falsehood fixed, C ABI `VLLM_ABI_VERSION 6`, CLI spec-config flag); new usage guide `docs/SPECULATIVE-DECODING.md`. STILL OPEN: `vllm-cli` structured-output/sampling flags (Tier 2 item 4, code, not docs). **C-ABI contract W0 2026-07-31 (`CLAIM-SERVE-C-ABI-SPIKE`):** accepted [c-api-library.md](specs/c-api-library.md), corrected public ABI v9 labels to v10, and named W1-W5 compatibility/no-throw/lifetime gates; `SERVE-C-ABI` honestly remains `ANCHOR-BACKFILL`. **CLI chat/complete W0 2026-08-01 (`CLAIM-SERVE-CLI-CHAT-SPIKE`):** [cli-chat-complete.md](specs/cli-chat-complete.md) corrects the inventory (the pin does ship both remote commands), selects exact OpenAI HTTP/SSE parity plus a compatibility-preserved local invocation, and decomposes CPU-only W1-W5 implementation leaves; `SERVE-CLI-CHAT` is `SPIKE`, no command code has landed. Full table: [specs/docs-coverage-audit.md](specs/docs-coverage-audit.md) | ## B. Research tracks (complete) diff --git a/.agents/specs/cli-chat-complete.md b/.agents/specs/cli-chat-complete.md new file mode 100644 index 00000000..39c48ae1 --- /dev/null +++ b/.agents/specs/cli-chat-complete.md @@ -0,0 +1,243 @@ +# `SERVE-CLI-CHAT`: interactive chat and completion commands + +Date: 2026-08-01 +Row: `SERVE-CLI-CHAT` +Claim: `CLAIM-SERVE-CLI-CHAT-SPIKE` +Pinned vLLM: `5559679229bc961848b121ccdeaa8fa5d79bec98` + +## Scope + +Port the pinned vLLM `chat` and `complete` command contracts into +`vllm-cli`. Both commands target a running OpenAI-compatible server, stream +output, support one-shot and interactive modes, select an explicit model or +the first model returned by `/v1/models`, and optionally print TTFT/TPS. Keep +the current in-process `vllm-cli --model DIR --prompt TEXT` form as a deprecated +compatibility alias for `vllm-cli complete --local-model DIR --quick TEXT`. + +In scope: + +- `vllm-cli chat [--url URL] [--model-name NAME] [--api-key KEY] + [--system-prompt TEXT] [-q MESSAGE] [--stats]`; +- `vllm-cli complete [--url URL] [--model-name NAME] [--api-key KEY] + [--max-tokens N] [-q PROMPT] [--stats]`; +- interactive stdin loops, EOF and interrupt handling, streamed content, + usage-derived statistics, model discovery, HTTP/SSE failure reporting, and + stable exit codes; +- preservation of the existing local-model completion flags and C-ABI-only + implementation path. + +Out of scope: + +- starting or managing a server from the client; +- adding a new public C ABI or changing generation semantics; +- terminal editing/history, markdown rendering, tool execution, multimodal + file attachment, TLS policy, or non-OpenAI endpoints; +- model correctness or throughput claims. This is a client/packaging row. + +The current matrix statement that the pin has no direct commands is false. +The pin imports `vllm.entrypoints.cli.openai` and registers its commands in +`vllm/entrypoints/cli/main.py:17-37,83-95`; `openai.py:155-234` implements +`chat`, and `openai.py:237-312` implements `complete`. + +## Upstream chain + +All upstream anchors below are at the pinned SHA. + +| Surface | Pinned behavior | +|---|---| +| CLI registration | `vllm/entrypoints/cli/main.py:17-37,73-98` lazily imports command modules, registers each `CLISubcommand`, validates, then dispatches. | +| Command interface | `vllm/entrypoints/cli/types.py:13-29` defines the name, argument-registration, validation, and dispatch contract. | +| Client/model resolution | `vllm/entrypoints/cli/openai.py:30-45` uses `--url`, resolves `--api-key` over `OPENAI_API_KEY` over `EMPTY`, and selects `--model-name` or the first `/models` result. | +| Stream rendering | `vllm/entrypoints/cli/openai.py:48-100` prints chat content or completion text as it arrives, collects usage, and prints TTFT/TPS only when measurable. | +| Chat loop | `vllm/entrypoints/cli/openai.py:155-234` preserves system/user/assistant turns, supports `--quick`, streams every request, and exits interactive mode on EOF. | +| Complete loop | `vllm/entrypoints/cli/openai.py:237-308` supports `--max-tokens`, `--quick`, streamed independent prompts, and EOF exit. | +| Server endpoints | `src/vllm/entrypoints/openai/api_server.cpp` locally exposes `/v1/models`, `/v1/chat/completions`, and `/v1/completions`; `serving_chat.cpp` and `serving_completion.cpp` own response semantics. | + +Dispatch is dynamic at the HTTP boundary, not at a device kernel. Runtime trace +is therefore a deterministic fake-server transcript, not `nsys`: record the +request method/path/headers/body and the exact SSE frames consumed. No GPU or +dependency kernel chain participates. + +## Our baseline + +`examples/cli/main.cpp:1-207` is a single in-process completion program. It +loads a model through the stable C ABI, maps sampling flags, and calls +`vllm_complete` or `vllm_complete_stream`. It has no subcommands, remote URL, +model discovery, chat conversation state, OpenAI JSON/SSE client, system +prompt, quick mode, or TTFT/TPS display. + +Useful existing seams: + +- `include/vllm.h:370-453` provides local streaming completion and chat; +- `src/capi/vllm_c.cpp:604-951` owns their no-throw implementations; +- `third_party/httplib/httplib.h` is the already-vendored HTTP transport used + by the server and can also drive a client without adding a dependency; +- `nlohmann::json` is already vendored and used throughout the entrypoints; +- `tests/capi/test_capi.cpp:567-711` proves completion stream concatenation, + early stop, recovery, determinism, and argument errors; +- `tests/capi/test_chat_prompt.cpp:37-89` proves template-backed chat prompt + construction. + +Honest gaps: there is no CLI unit target, parsing and execution are fused in +`main`, numeric parsing uses permissive `atoi`/`atof`, `NextArg` exits from a +helper, and the current help smoke test covers only the server. These are +testability defects to fix as part of the port, not reasons to fork behavior. + +## Design decision and alternatives + +Three approaches were considered: + +1. **Remote-only rewrite.** This is the closest vLLM mirror but would remove + the documented local, library-first example and break existing scripts. +2. **In-process-only subcommands.** This reuses the C ABI but does not mirror + vLLM's running-server commands, model discovery, or OpenAI/SSE boundary. +3. **Dual-mode command family (selected).** Add exact remote `chat` and + `complete` subcommands, keep the existing invocation as a compatibility + alias, and factor parsing/stream shaping behind testable helpers. This + mirrors upstream without regressing the project's existing consumption + example. + +Remote mode is the default for named subcommands. Local mode is explicit via +`--local-model`; `--url` and `--local-model` are mutually exclusive. The legacy +top-level form remains local. No implicit server start, silent fallback, or +network-to-local mode switch is allowed. + +The remote client uses the vendored `httplib` and `nlohmann::json`; it does not +vendor the OpenAI SDK. It accepts `http://` URLs in W2. An `https://` URL fails +clearly until the existing optional OpenSSL build can provide a verified TLS +client; it must never downgrade to HTTP. + +## Port map + +| Upstream | Local target | Port/deviation | +|---|---|---| +| `entrypoints/cli/types.py:13-29` | NEW `examples/cli/cli_app.{h,cpp}` | `Command` enum plus parse/validate/run result, no process exit in helpers. | +| `entrypoints/cli/main.py:73-98` | `examples/cli/main.cpp` | Dispatch `chat`/`complete`; retain legacy compatibility translation. | +| `entrypoints/cli/openai.py:30-45` | NEW `examples/cli/openai_client.{h,cpp}` | URL/key/model resolution; dependency-free transport. | +| `entrypoints/cli/openai.py:48-100` | NEW `examples/cli/stream_printer.{h,cpp}` | Incremental SSE JSON shaping, content/text extraction, usage, TTFT/TPS. | +| `entrypoints/cli/openai.py:155-234` | `cli_app.cpp` chat runner | Same conversation accumulation, quick/interactive behavior, EOF exit. | +| `entrypoints/cli/openai.py:237-308` | `cli_app.cpp` complete runner | Same quick/interactive behavior and `max_tokens` omission semantics. | +| Existing project-only local CLI | `cli_app.cpp` local runner | Preserve flags and C ABI calls; documented compatibility deviation. | + +## Tests to port + +The pin has no dedicated `tests/entrypoints/cli` module for these commands, so +the executable spec is `openai.py` plus endpoint tests. Port the following +behaviors into NEW `tests/examples/test_cli.cpp` with a fake OpenAI server and +in-memory stdin/stdout/stderr: + +| Case | Source | Local assertion | +|---|---|---| +| model selection | `openai.py:30-45` | explicit name avoids `/models`; absent name uses the first returned ID; empty list is a clear error | +| auth precedence | `openai.py:33-35` | flag beats environment; absent values send `Bearer EMPTY` | +| chat quick | `openai.py:163-182` | system+user request, streamed text, one request, exit 0 | +| chat interactive | `openai.py:184-196` | second request contains prior assistant output; EOF exits cleanly | +| complete quick/interactive | `openai.py:243-268` | correct endpoint/body, streamed text, independent prompts | +| optional max tokens | `openai.py:247-258` | omitted when unset, present with exact integer when set | +| stats | `openai.py:48-100,170-173,251-254` | requests usage frames and prints TTFT/TPS only with usable timing+usage | +| malformed/failed stream | endpoint contract | non-2xx, invalid JSON, truncated SSE, missing choices, and `[DONE]` handling are deterministic and never crash | +| CLI validation | local contract | help 0; unknown/missing/invalid/conflicting args 2; network/runtime failure 1 | +| compatibility | `examples/cli/main.cpp:68-207` | legacy argv translates to the local complete configuration byte-for-byte | + +Tests use real parsing, JSON, HTTP, and SSE code. Transport injection is only +for deterministic clock/input/output control; assertions target emitted +requests and user-visible output, not mock call counts. No upstream test is +dropped. The absence of upstream command tests is recorded rather than hidden. + +## Error and lifecycle contract + +- Parse/help errors happen before network or model work. Help exits 0; invalid + invocation exits 2. +- Connection, HTTP, JSON/SSE, model-discovery, and generation failures include + endpoint context on stderr and exit 1. API keys are never printed. +- EOF ends an interactive session with exit 0. SIGINT cancels the current + request or exits the prompt loop without a stack trace; it never leaves a + local C-ABI request alive. +- Each remote request has a finite connect/read timeout. A stalled peer cannot + hang CI or the terminal forever. +- The chat history appends the assistant turn only after a successful complete + stream. A failed/aborted response is not committed to history. +- Local engine ownership remains load once, free once. All helper APIs return + status objects and are no-throw across `main`. + +## Gates + +Spike checkpoint (this change, CPU-only): + +```bash +python3 scripts/check-agent-record.py +python3 tests/scripts/test_agent_record.py +python3 scripts/check-doc-checkpoint.py --staged +python3 tests/scripts/test_doc_checkpoint.py +python3 scripts/check-readme-structure.py +python3 tests/scripts/test_check_readme_structure.py +git diff --check +``` + +Implementation gates: + +```bash +cmake -S . -B build-cli-cpu -DVLLM_CPP_CUDA=OFF \ + -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_SERVER=ON \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS='-Wall -Wextra -Werror' +cmake --build build-cli-cpu --target vllm-cli test_cli -j2 +ctest --test-dir build-cli-cpu --output-on-failure \ + -R 'test_cli|test_capi|test_chat_prompt|test_openai_(api_server|serving|conformance)' +``` + +Correctness is exact request/transcript and stdout/stderr parity for the +inventoried cases. Sanitizer gates run `test_cli` under ASan+UBSan and TSan. +End-to-end uses the CPU synthetic server/engine so no checkpoint or GPU is +needed. A later real-server smoke on a gate model is useful packaging evidence +but does not block the CPU-complete client row. + +Performance and memory are `NOT APPLICABLE` for model inference: no compute +path changes. The client gate instead enforces bounded buffering (stream +content is appended only where chat history requires it), finite timeouts, and +no per-turn history duplication. Backend/architecture matrix: CPU execution +for the client; CUDA/ROCm/Metal are inherited from the independently running +server and are not exercised or claimed here. + +## Dependencies + +- Row dependency: the existing OpenAI completion/chat serving and C ABI seams; + no open lifecycle row must advance for W1/W2. +- Toolchain: C++17, vendored `httplib`, vendored `nlohmann::json`, doctest, + CMake. No new package or license. +- Hardware/data: CPU and loopback TCP only; no GPU, model, oracle venv, or + downloaded checkpoint. +- The HTTP client must coordinate with `SERVE-HTTP-TRANSPORT` only if shared + transport code is edited. The plan avoids that by keeping client code under + `examples/cli/`. + +## Work breakdown + +| Leaf | Files/ownership | Deliverable | Gate | +|---|---|---|---| +| W1 parse/dispatch | `examples/cli/{main,cli_app}.{cpp,h}`, `tests/examples/test_cli.cpp` | subcommands, validation, legacy translation, injectable IO/clock | parse/exit-code tests, RED first | +| W2 OpenAI transport | `examples/cli/openai_client.{h,cpp}`, same test | URL/auth/model discovery, POST, incremental SSE, timeouts/errors | fake-server transcript and failure matrix | +| W3 complete | `cli_app.cpp`, same test | quick+interactive completion, max-tokens, stats | exact bodies/output, EOF/SIGINT | +| W4 chat | `cli_app.cpp`, same test | system prompt, conversation history, quick+interactive, stats | multi-turn transcript and failed-turn rollback | +| W5 packaging | `examples/CMakeLists.txt`, README/help docs, record surfaces | build wiring, CPU sanitizer/conformance closure | clean Release `-Werror`, focused CTest, record checks | + +W1 and W2 may be claimed separately because their owned production files do +not overlap; both share the test file only under an explicit lead claim. W3 +depends on W1+W2. W4 depends on W1+W2 and can run alongside W3 with split test +sections. W5 closes after W3+W4. + +## Risks and decisions + +- **Corrected inventory:** chat/complete are present at the pin. Future work + mirrors them; it is not a greenfield UX design. +- **Compatibility:** removing the current local CLI would be an unnecessary + breaking change. The compatibility alias is explicit and test-pinned. +- **SSE boundaries:** arbitrary TCP chunks do not equal SSE frames. W2 buffers + through the blank-line delimiter and parses multi-line `data:` fields. +- **Secrets:** authorization values never appear in diagnostics or captured + golden output. +- **Unbounded sessions:** conversation history necessarily grows by turns; + no hidden truncation policy is invented. Model context-limit errors surface + from the server exactly as errors. +- **Product calls:** none remain. Remote behavior mirrors vLLM; preserving the + already-shipped local invocation is the only project-specific compatibility + decision. diff --git a/.agents/state.md b/.agents/state.md index 7dfc8423..7a31fb4a 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -34432,3 +34432,31 @@ PARTIAL on two further unrelated test-only GCC 12 `-Wrestrict` diagnostics in CTest cannot run from this incomplete build. No GPU, model, or benchmark ran. **Canonical DONE-owner reachability repaired (2026-07-31, `CLAIM-RECORD-DONE-OWNER-REPAIR`).** Records-only correction: the six checker-reported `DONE` owner cells now name the reachable commits that introduced their binding closures (`KV-PREFIX-CACHE` `a41af480`, `SAMPLE-LOGPROBS` `ecda3ce1`, and the four DFlash rows `489a7544`). Lifecycle states, evidence, support claims, source, tests, and benchmark results are unchanged. + +## 2026-08-01 - `SERVE-CLI-CHAT` contract spike + +`CLAIM-SERVE-CLI-CHAT-SPIKE`, isolated worktree +`/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`, +branch `codex/serve-cli-chat-spike`, base `upstream/main` `1448e981`, CPU-only. + +The pinned-source read found and corrected a material inventory error: vLLM +`5559679229` does have direct `chat` and `complete` commands. They are registered +through `entrypoints/cli/main.py` and implemented in `entrypoints/cli/openai.py` +as clients of a running OpenAI-compatible server. The old row described them +as a project extension based only on the import list in `main.py`, without +following the imported `openai` command module. + +Accepted `.agents/specs/cli-chat-complete.md`. The selected design mirrors the +remote URL/auth/model-discovery, quick/interactive, SSE, conversation-history, +and TTFT/TPS contracts while retaining today's in-process `--model --prompt` +form as an explicit compatibility alias. Alternatives rejected: remote-only +would break the shipped library-first example; in-process-only would fail +upstream parity. The implementation is split into W1 parse/dispatch, W2 OpenAI +transport, W3 complete, W4 chat, and W5 packaging/gates. No source, tests, +CMake, model, GPU, checkpoint, or benchmark changed. Resume with W1/W2 after +the spike merges; both are fully CPU-gateable via a loopback fake server. + +The required Slack selection notification was attempted through the bundled +secret-safe sender to the only conventional target available, `#general`, but +Slack returned `channel_not_found`. No channel ID/name is configured and no +credential was inspected or exposed. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 17c962c1..d2cb8e05 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -10974,3 +10974,18 @@ Six existing `DONE` matrix rows now point to reachable commits that introduced their binding closure states. This records-only metadata repair changes no lifecycle state, implementation, workload, or accepted benchmark result, so no new performance number is applicable or claimed. + +## CLI chat/complete contract spike (2026-08-01) - NOT APPLICABLE + +`SERVE-CLI-CHAT` W0 (`CLAIM-SERVE-CLI-CHAT-SPIKE`) is a records/spec-only +checkpoint. No production, test, model, scheduler, memory, latency, or +throughput path changed, so `benchmark_binding=false`. The pinned source proves +that vLLM registers remote `chat` and `complete` commands in +`entrypoints/cli/main.py:17-37,73-98` and defines their full client behavior in +`entrypoints/cli/openai.py:30-100,155-312`; the local baseline remains the +in-process-only `examples/cli/main.cpp:1-207`. + +Implementation is `PENDING`. The next reproduction is the W1/W2 CPU fake-server +transcript suite in +[the accepted spec](../.agents/specs/cli-chat-complete.md), followed by clean +Release `-Werror` and sanitizer gates. No GPU or model is required. diff --git a/docs/STATUS.md b/docs/STATUS.md index 102532af..19a5a82e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -79,6 +79,7 @@ token-for-token correctness against the pinned oracle. | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | | Prefix-cache matching unit (`--prefix-match-unit`) | Partial (resolver landed, config/scheduler wiring pending) | 0.26-new knob setting the finest token boundary a prefix-cache hit can land on (the `hash_block_size`). W1: `resolve_kv_cache_block_sizes` ported 1:1 (hybrid `hash_block_size = prefix_match_unit if set else gcd(group block sizes)`; single-group inert; back-off on no-consumer / mamba-non-align; throws on non-divisible), CPU unit-gated RED-first (default gcd vs `=16`). Pending: the config/CLI/ABI field (W2), scheduler threading of a finer-than-block matching unit (W3, needs the KV-block-pool align path), and the benchmark (W4). Default path byte-identical (dense single-group models ignore it). | +| CLI `chat` / `complete` (`SERVE-CLI-CHAT`) | `SPIKE` (implementation pending) | The accepted CPU-only spike corrects a stale inventory claim: pinned vLLM 0.26 ships both commands as clients of a running OpenAI-compatible server. The selected design mirrors their URL/auth/model-discovery, quick/interactive, SSE, history, and TTFT/TPS contracts while preserving the existing in-process `--model --prompt` invocation as a compatibility alias. No remote command code or support claim has landed; next gate is W1 parse/dispatch plus W2 fake-server transcript tests. See [the spike](../.agents/specs/cli-chat-complete.md). | | KV offload to CPU / disk | Built, opt-in, off by default; the disk connector is engine-refused | CPU and disk tiers with identity-checked blocks, selected by `--kv-transfer-config` (or programmatically) over one abstract KVConnector ABI. Worker-side KV store/load is implemented for the LMCache connector only; the CPU/disk connector is scheduler-side only, so the engine now REFUSES it at construction (a loud error, not silently wrong output). Guide: [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | LMCache client (`lm://` remote KV) | Built, opt-in, off by default; a working, verified external KV cache | Pure-C++ `lm://` client wired as an `LMCacheConnector`, no `lmcache` in-process; keys agree byte-for-byte with a real vLLM+LMCache peer, mismatched blocks refused. Proven in a real OPT-125m loop vs a live `lmcache.v1.server`: connector-ON tokens are BIT-IDENTICAL to the connector-OFF cold run (both after an in-process restart and from a cold second process). See docs/BENCHMARKS.md | | KV-cache events (for external routers) | Built, off by default; generation and payload gated, live ZMQ transport deferred | Block store/remove/clear events (`BlockStored`/`BlockRemoved`/`AllBlocksCleared`) emitted at the prefix-cache sites when enabled, with a `msgpack` payload byte-identical to vLLM's `msgspec` encoding. Behind a publisher seam faithful to `--kv-events-config`; the live ZMQ transport is not wired yet. Off by default, so the prefix-cache path is byte-identical. |