Skip to content

docs(cli): spike chat and complete parity - #13

Merged
mudler merged 1 commit into
mudler:mainfrom
localai-org-maint-bot:codex/serve-cli-chat-spike
Aug 1, 2026
Merged

docs(cli): spike chat and complete parity#13
mudler merged 1 commit into
mudler:mainfrom
localai-org-maint-bot:codex/serve-cli-chat-spike

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Contributor

Outcome

Moves SERVE-CLI-CHAT from INVENTORIED to SPIKE with an accepted, CPU-only implementation design. No runtime command support is claimed by this PR.

The pinned-source audit found a material inventory bug: vLLM 5559679229 does ship both chat and complete through vllm/entrypoints/cli/openai.py. The prior matrix had stopped at the import list in main.py and incorrectly called them project extensions.

Design selected

The spike chooses a dual-mode command family:

  • mirror vLLM remote chat and complete over OpenAI HTTP/SSE, including URL/auth/model discovery, quick and interactive modes, chat history, streaming, and TTFT/TPS
  • preserve the shipped in-process vllm-cli --model DIR --prompt TEXT form as an explicit compatibility alias
  • use the already-vendored httplib and nlohmann::json, with no new dependency
  • split follow-on work into CPU-verifiable W1 parse/dispatch, W2 transport, W3 complete, W4 chat, and W5 packaging gates

Remote-only was rejected because it breaks the current library-first example. In-process-only was rejected because it does not mirror the pinned vLLM behavior.

Evidence

Pinned vLLM 5559679229bc961848b121ccdeaa8fa5d79bec98:

  • command registration: vllm/entrypoints/cli/main.py:17-37,73-98
  • client/model resolution and stream shaping: vllm/entrypoints/cli/openai.py:30-100
  • chat: vllm/entrypoints/cli/openai.py:155-234
  • complete: vllm/entrypoints/cli/openai.py:237-312

Local baseline:

  • in-process completion only: examples/cli/main.cpp:1-207
  • C-ABI stream semantics: tests/capi/test_capi.cpp:567-711
  • chat-template baseline: tests/capi/test_chat_prompt.cpp:37-89

Verification

All CPU-only gates pass on commit 73566588538e4956141be640a9a1603280785f11:

  • python3 scripts/check-agent-record.py
    • agent record OK: ENGINE=131 MODEL=327 QUANT=82 KERNEL=45 BACKEND=68
  • python3 tests/scripts/test_agent_record.py
    • 13 mutation tests
  • python3 scripts/check-doc-checkpoint.py
  • python3 tests/scripts/test_doc_checkpoint.py
    • 5 tests
  • python3 scripts/check-readme-structure.py
  • python3 tests/scripts/test_check_readme_structure.py
    • 18 mutation tests
  • git diff --check HEAD^

No source, header, CMake, test, README, model, kernel, fixture, generated artifact, GPU work, model download, or benchmark changed. docs/BENCHMARKS.md records NOT APPLICABLE and benchmark_binding=false.

Slack

The required selection notification was attempted with the bundled secret-safe sender. No channel is configured; the only conventional target available (#general) returned channel_not_found. No credential was read or exposed.

Correct the pinned-vLLM CLI inventory and define CPU-verifiable implementation leaves for remote chat/complete commands while preserving the existing local invocation.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5 [Codex]
@localai-org-maint-bot

Copy link
Copy Markdown
Contributor Author

Slack follow-up: the required PR-open notification was also attempted through the bundled secret-safe sender. With no configured channel, #general again returned channel_not_found; no credential was read or exposed.

@mudler
mudler merged commit 74d7bc8 into mudler:main Aug 1, 2026
5 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants