mudler · mudler · Aug 1, 2026 · Aug 1, 2026
diff --git a/.agents/coordination.md b/.agents/coordination.md
@@ -118,6 +118,20 @@ without the selected contention proof for their entire run are discarded.
 
 ## Active claims
 
+**Interactive CLI chat/complete spike (`SERVE-CLI-CHAT`, 2026-08-01,
+`CLAIM-SERVE-CLI-CHAT-SPIKE`).** Codex (GPT-5), isolated worktree
+`/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`,
+branch `codex/serve-cli-chat-spike`, base `upstream/main` `1448e981`. CPU-only,
+records/spec checkpoint: inventory the pinned vLLM CLI surface and the local
+CLI/engine/serving seams, define exact interactive-chat and one-shot-complete
+contracts, ported-test inventory, error/exit behavior, and row-sized follow-on
+leaves. Owns only NEW `.agents/specs/cli-chat-complete.md`, the
+`SERVE-CLI-CHAT` row, this claim, the matching roadmap/status checkpoint text,
+`docs/BENCHMARKS.md`, `.agents/parity-ledger.md`, and append-only
+`.agents/state.md`. No source, header, CMake, test, model, kernel, generated
+fixture, or GPU/model-download change; verification is the CPU record/doc
+checker suite. The row and open-PR list were unclaimed at selection time.
+
 **Canonical DONE-owner reachability repair (`KV-PREFIX-CACHE`,
 `SAMPLE-LOGPROBS`, `SPEC-DFLASH`,
 `MODEL-SPEC-qwen3-dflash-dflash-qwen3-for-causal-lm`,
@@ -1313,6 +1327,7 @@ table, tests, CMake. Details in the state-log entry of the same date.
 
 | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update |
 |---|---|---|---|---|---|---|---|
+| `CLAIM-SERVE-CLI-CHAT-SPIKE` | `SERVE-CLI-CHAT` (`INVENTORIED` -> `SPIKE`) | Codex (GPT-5) | isolated worktree `/home/mudler/.cache/sdd/localai-org-maint-bot-vllm.cpp/codex-serve-cli-chat-spike`; CPU-only records/spec, no GPU/model/download | branch `codex/serve-cli-chat-spike`, base `upstream/main` `1448e981` | Owns only NEW `.agents/specs/cli-chat-complete.md`; the `SERVE-CLI-CHAT` engine-matrix row and rollup; matching feature-matrix, porting-inventory, roadmap, STATUS, BENCHMARKS, coordination, ledger, and append-only state entries. No source/header/CMake/test/README/model/kernel/fixture edits. | `SPIKE` | 2026-08-01 - pinned-source inventory corrected: vLLM does ship remote `chat`/`complete`; accepted dual-mode design preserves the existing local invocation. CPU record/doc checker suite is the closing gate. |
 | `CLAIM-LAGUNA-W1W2` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (INVENTORIED→ACTIVE) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_2ae79b7a-246-1`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release `-Werror`); NO GPU, NO 73 GB download — structural bring-up + oracle DECISION only | branch `spike/laguna-s21-w1w2`, base `main` `5c3da2f1` | Laguna-S-2.1 W1 oracle-decision + W2 structural bring-up. Owns ONLY: NEW `include/vllm/model_executor/models/laguna.h`, NEW `src/vllm/model_executor/models/{laguna_registry,laguna_weights,laguna}.cpp`, NEW `tests/vllm/models/test_laguna_scaffold.cpp`, NEW `.agents/specs/laguna-s21-w1w2-2026-07-30.md`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `LagunaForCausalLM` sorted-set + error-message insert in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row (INVENTORIED→ACTIVE) + checklist rollup (ACTIVE 23→24 / INVENTORIED 285→284, engaged 42→43), and the record surfaces (this claim, roadmap breadth, docs/STATUS, docs/BENCHMARKS, parity-ledger, state). **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` W3 stub so no production path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-07-30 — **W1 oracle-decision + W2 structural bring-up LANDED (foreground, NOT pushed).** Registry (`laguna`/`LagunaForCausalLM`) + `ParseLagunaParams` (nested dual-rope + variable Q-head + ungrouped sigmoid-noaux MoE) + GGUF `blk.N.*` name-map + UD-Q4_K_XL quant-mix (ZERO new decode kernel) + KV-cache spec + per-layer forward-composition scaffold with reuse citations. `test_laguna_scaffold` 3/3·40 + `test_model_registry` 24/24; CPU full-library `-Werror` clean; record checkers rc=0. RESIDUALS (W3/W4): device materialization + real forward + the 3 new ops + strict dual-oracle gate on a fetched checkpoint. **SUPERSEDED by `CLAIM-LAGUNA-W3` (2026-07-31) which landed the real forward + the 3 new ops.** |
 | `CLAIM-LAGUNA-W3` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (stays `ACTIVE`; W3 real forward + the 3 new ops landed; real-model dual-oracle gate still PENDING W4) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_43e61a78-0e2-1`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release `-Werror`); NO GPU, NO 73 GB download — real forward CODE + unit gates only | branch `laguna-s21-w3`, base `main` `f2e463d5` (confirmed via `git rev-parse HEAD`) | Laguna-S-2.1 W3 — turn the W1/W2 `VT_CHECK(false)` forward stub into a REAL runnable host-reference composition + land the 3 genuinely-NEW small host ops. Owns ONLY: NEW `include/vllm/model_executor/models/laguna_ops.h` + `src/vllm/model_executor/models/laguna_ops.cpp` (softplus head-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders), the rewritten `src/vllm/model_executor/models/laguna.cpp` (`LagunaModel::Forward` real composition), the `LagunaParams` per-layer variable-Q-head helpers in `include/vllm/model_executor/models/laguna.h`, the new-op + forward unit cases appended to `tests/vllm/models/test_laguna_scaffold.cpp`, the `laguna_ops.cpp` line in `CMakeLists.txt`, NEW `.agents/specs/laguna-s21-w3-2026-07-31.md`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row cells + this claim, and docs/STATUS + docs/BENCHMARKS pointers. **NON-COLLISION:** file-disjoint from the concurrent MLA-fold lane (`mla_attention.cpp`/`deepseek_v2.cpp` untouched); additive `laguna_ops` TU + one CMake line; the loaders still `VT_CHECK(false)` so NO production/device path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `ACTIVE` | 2026-07-31 — **W3 REAL forward + 3 new ops LANDED + UNIT-GATED (foreground, NOT pushed).** `laguna_ops.cpp`: `LagunaSoftplusHeadGate` (per-head softplus out-gate), `LagunaUngroupedRouterTopK` (sigmoid noaux_tc MINUS the group step + tie-break razor: lower index on equal choice, UNBIASED weights, renorm, routed_scaling), `BuildLaguna{FullYarn,Sliding}CosSin` (dual per-layer RoPE, reusing the pinned `compute_yarn_inv_freq` over the partial-64 dims). `LagunaModel::Forward` is now a REAL runnable f32 host-reference composition (variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head). `test_laguna_scaffold` **8/8·166** (softplus math; router selection + tie-break RED-first; dual-RoPE cos/sin bit-match vs hand ref; variable-Q-head shapes; forward composition on synthetic weights — RUNS, deterministic, gather==full-row, softplus gate wired) + `test_model_registry` 24/24; CPU full-library `-DVLLM_CPP_CUDA=OFF` `-Werror` clean; all record checkers rc=0. **HONEST residual (DEFERRED W4, needs the 73 GB checkpoint):** GGUF keep-quant tower materialization (loaders still LOUDLY throw) + device/paged production forward (runner variable-Q-head device wiring) + the strict dual-oracle greedy gate (llama.cpp-Q4_K token-exact + vLLM-NVFP4 near-tie). Risks: dual-RoPE numerics vs the fork on the real config; the reference forward is f32 whole-sequence (bf16 paged token-exactness is a W4 boundary); router tie-break vs the oracle's actual greedy selection. Row stays `ACTIVE`. |
 | `CLAIM-LAGUNA-W4` | `MODEL-TEXT-laguna-laguna-for-causal-lm` (stays `ACTIVE`; checkpoint fetched + fidelity corrected; real-model greedy gate is the W5 close) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/wf_63140a7b-03e-1`; DGX `dgx.casa` GB10 for the 73.4 GiB fetch + GGUF metadata read + llama.cpp oracle build (foreground); CPU-verified fidelity corrections; NOT pushed | branch `worktree-wf_63140a7b-03e-1`, base `main` `570510a9` | Laguna-S-2.1 W4 — FETCH the UD-Q4_K_XL GGUF, read its metadata + tensor map AUTHORITATIVELY, and correct the fidelity errors the W1-W3 scaffold made from config.json guesses. Owns ONLY: `laguna.h`/`laguna_ops.{h,cpp}`/`laguna.cpp`/`laguna_weights.cpp` (QK-RMSNorm + `LagunaYarnMscale` + separate gate/up + verified name-map/quant-mix), the `test_laguna_scaffold.cpp` cases for those, NEW `.agents/specs/laguna-s21-w4-2026-07-31.md`, the `MODEL-TEXT-laguna-laguna-for-causal-lm` row cells + this claim, docs/STATUS + docs/BENCHMARKS pointers. **NON-COLLISION:** laguna-only additive edits; loaders still throw the keep-quant residual so NO production/device path changes; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `ACTIVE` | 2026-07-31 — **Checkpoint FETCHED + arch grounded in REAL bytes + 3 fidelity bugs fixed.** UD-Q4_K_XL GGUF (73.4 GiB, 3 shards, 814 tensors) fetched to dgx; metadata read authoritatively (arch `laguna`, `expert_gating_func=2` sigmoid, `expert_weights_scale=2.5`, `leading_dense_block_count=1`, rope factor 32/yarn_attn_factor 1.0, per-layer head_count `[48,72,72,72]`, quant mix: attn Q8_0 / experts gate-up Q4_K + down Q5_K / shared Q8_0 / router+norms F32). Fixed: (a) per-head QK-RMSNorm `attn_q/k_norm` (scope MISSED it — no config flag), (b) dual-RoPE mscale via llama.cpp `yarn_attn_factor·(1+0.1·ln(factor))` off GGUF factor 32 (not HF 128/1.4852), (c) SEPARATE `ffn_gate/up_exps`. Oracle = `poolsideai/llama.cpp@laguna` (mainline b10087+) same-quant; no vLLM-GGUF path for `laguna`. **HONEST residual (W5 close, needs the resident 73 GB run):** keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + host-orchestrated `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + dual-RoPE inv_freq ramp bit-match + the real greedy run vs the llama.cpp same-quant oracle (token-exact or characterized near-tie). Row stays `ACTIVE`. |

diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md
@@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger.
 | Sampling and generation | 15 | 0 | 2 | 0 | 0 | 7 | 0 | 1 | 5 |
 | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 |
 | Speculative decoding | 12 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 2 |
-| Serving, API, CLI, library | 22 | 3 | 2 | 1 | 0 | 7 | 2 | 1 | 6 |
+| Serving, API, CLI, library | 22 | 3 | 2 | 2 | 0 | 7 | 2 | 1 | 5 |
 | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
 | Long context and attention | 10 | 0 | 0 | 0 | 1 | 5 | 1 | 0 | 3 |
 | Loading, tokenizer, config | 9 | 1 | 3 | 0 | 0 | 2 | 1 | 1 | 1 |
-| **Total** | **131** | **8** | **16** | **4** | **4** | **48** | **8** | **9** | **33** |
+| **Total** | **131** | **8** | **16** | **5** | **4** | **48** | **8** | **9** | **32** |
 
 ## Engine core and scheduling
 
@@ -198,7 +198,7 @@ claims it.
 | `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; `examples/bench/main.cpp:40,109`; `examples/bench/bench_core.h:96,468` | server help contract `examples/CMakeLists.txt:34`; benchmark `tests/examples/test_bench.cpp:15,48` | `planned: specs/cli-serve-bench.md` | `PARTIAL` | - |
 | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ACTIVE` | CLAIM-SERVE-GATE-1 |
 | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - |
-| `SERVE-CLI-CHAT` | Interactive chat and complete commands | T1 | `vllm/entrypoints/cli/main.py:18-34` has no direct chat/complete command at the pin; project extension | - | - | `planned: specs/cli-chat-complete.md` | `INVENTORIED` | - |
+| `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `SPIKE` | `CLAIM-SERVE-CLI-CHAT-SPIKE` |
 | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md) — endpoints depend on the pooling RUNNER (`ENG-POOLING-RUNNER`, W3 LANDED 2026-07-29 — `PoolingRunner` returns pooled data via the model `Pooler`) and a concrete pooling model. Endpoint protocol + handler port is the W4 brick | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | - | - | [pooling-task-class.md](specs/pooling-task-class.md) | `SPIKE` | `CLAIM-POOLING` |
 | `ENG-POOLER-SEQ` | The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. **W1 LANDED + CPU-GATED 2026-07-28 (`CLAIM-POOLING`, NOT pushed):** the sequence pooling methods `CLSPool`/`LastPool`/`MeanPool` (+ `GetSeqPoolingMethod` factory) over a packed `[num_tokens, hidden]` CPU buffer keyed by a minimal `PoolingCursor` (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads `PoolerIdentity`/`PoolerNormalize` (L2 `F.normalize`)/`PoolerMultiLabelClassify` (sigmoid)/`PoolerClassify` (sigmoid if `num_labels<2` else `softmax`). Unit-gated vs DOUBLE-PRECISION references, RED-first. **W2 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** the pooler HEADS composite (`EmbeddingPoolerHead` = projector→matryoshka→normalize; `ClassifierPoolerHead` = classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` (method∩head task intersection) + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` groupby-task routing (`ForEmbedding`/`ForSeqCls` + a mixed embed+classify batch + ctor task-support validation), and the `PoolerConfig`/`PoolingParams`/`PoolingParamsUpdate` structs; `test_pooler_heads` 27/27 (240 asserts) vs double-precision refs, RED-first (disable matryoshka slice + logit_mean calibration → 8 cases / 50 asserts fail). RESIDUALS (named, spec §Work breakdown): the endpoints (W4), tokwise `AllPool`/`StepPool` (W5), a concrete pooling MODEL + real-oracle cosine gate (W3-model — see `ENG-POOLING-RUNNER`) | T2 | `vllm/model_executor/layers/pooler/seqwise/methods.py:35-121`; `vllm/model_executor/layers/pooler/activations.py:106-158`; `vllm/model_executor/layers/pooler/seqwise/heads.py:19-196`; `vllm/model_executor/layers/pooler/seqwise/poolers.py:41-138`; `vllm/model_executor/layers/pooler/special.py:23-140`; `vllm/model_executor/layers/pooler/common.py:12-30`; `vllm/pooling_params.py:35-70`; `vllm/config/pooler.py:16-90`; `vllm/v1/pool/metadata.py:13-71`; `tests/model_executor/layers/test_pooler_methods.py`, `tests/model_executor/layers/test_pooler_activations.py`, `tests/model_executor/layers/test_pooler_heads.py` | `include/vllm/model_executor/layers/pooler/{methods,activations,pooling_metadata,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h` + `src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp` | `tests/vllm/model_executor/layers/pooler/test_pooler.cpp` (CLS/LAST/MEAN + factory + activations, 50 asserts) + `test_pooler_heads.cpp` (Embedding/Classifier heads + SequencePooler + DispatchPooler, 240 asserts) | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` |
 | `ENG-POOLING-RUNNER` | The pooling RUNNER path — where the generation runner SAMPLES a token, the pooling runner applies the model's `Pooler` to the last hidden state and returns the POOLED DATA (embedding vector / classification logit row). **W3 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** `PoolingRunner` over a packed `[num_tokens, hidden]` last-hidden-state buffer + a `PoolingMetadata` — `Pool()` delegates to the model pooler (`DispatchPooler.ForEmbedding`), `GetSupportedTasks()`, `ComputeValid()` (`seq_lens==prompt_len`). GATE: a STRUCTURAL cosine-parity gate — the runner's embedding vs an independent double-precision LAST+normalize reference is cosine≈1 (5 cases / 14 asserts), RED-first (CLS-instead-of-LAST drops cosine <0.5; disable normalize → 2 unit-L2 asserts fail). GENERALIZATION DEVIATION: upstream `pooling_runner.py` hardcodes LAST+normalize; we route through the model `Pooler` (the general bert.py path), strictly more capable. HONEST RESIDUAL (named): the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) needs a registered concrete embedding model's forward — no such model is registered yet (W3-model), so no cosine-vs-oracle number is fabricated. The InputBatch→PoolingMetadata construction (logits_indices gather) rides the endpoint brick (W4) | T2 | `vllm/v1/worker/gpu/pool/pooling_runner.py:18-46`; `vllm/tasks.py:10`; `tests/models/language/pooling/test_embedding.py` (real-oracle gate, DEFERRED) | `include/vllm/v1/worker/gpu/pool/pooling_runner.h` + `src/vllm/v1/worker/gpu/pool/pooling_runner.cpp` | `tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp` (runner path + structural cosine gate, 14 asserts, RED-first) | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` |