inkling: serve mode + openai_server gateway support - #539
Merged
Conversation
Stage-A inkling.c only had greedy -p / oracle modes, so the shared openai_server.py gateway had nothing to drive. Add the serve loop it speaks, byte-identical to colibri.c's protocol: - SERVE=1 -> serve_loop: READY sentinel, STAT handshake, then per request SUBMIT <id> <slot> <len> <max_tok> <temp> <top_p> -> DATA frames -> DONE <id> STAT ... + PROF (per-turn phase timings). CANCEL honored between tokens; requests run one at a time (v1). - temperature + top-p nucleus sampling (sample_logits; temp<=0 = greedy, so the oracle path is unchanged) + a light REP_PEN repeat guard. - HWINFO / TIERS / EMAP dashboard lines (same as colibri.c) so the web Brain/Profiling pages render live expert-tier state. - prompt_reject bounds the served context (CTX_MAX, default 8192). Additive: the oracle still matches 36/36 teacher-forced + 24/24 greedy, 0 warnings. POSIX serve loop (inkling targets Linux/CUDA); the select() include is guarded like colibri.c's. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27
…nel)
With inkling.c now speaking the serve protocol, teach the gateway to
drive it. Additive — the glm path is byte-unchanged (ARCH defaults to
"glm"):
- --arch {auto,glm,inkling}; auto reads model_type from the model's
config.json. Picks the chat-template renderer (render_chat_inkling)
and the model-id.
- inkling's thinking is reasoning, not answer: split the <|content_thinking|>
/ <|content_text|> channels (split_inkling / strip_inkling_markers) into
reasoning_content on both the streaming (InklingStreamSplit) and
non-streaming chat paths.
- response_format grammars are refused for inkling (its serve loop speaks
the 6-field SUBMIT header only) with a clear 400.
51 openai_server tests pass; end-to-end serve smoke against a tiny model
returns DATA/DONE/PROF and coherent reasoning/text split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
#312 landed the Inkling engine, but
inkling.conly has-pstreaming and the oracle harness — the sharedopenai_server.pygateway has nothing to drive. This adds the missing serve loop and the gateway support to actually serve it, both strictly additive.Engine (
inkling.c, +246): aSERVE=1loop byte-identical tocolibri.c's protocol, so the existing gateway drives it unchanged:SUBMIT <id> <slot> <len> <max_tok> <temp> <top_p>→DATAframes →DONE <id> STAT …+PROF(per-turn phase timings).CANCELhonored between tokens; v1 runs one request at a time.sample_logits;temp<=0= greedy, so the oracle path is untouched) + a lightREP_PENrepeat guard.HWINFO/TIERS/EMAPdashboard lines (same ascolibri.c) for the web Brain/Profiling pages.prompt_rejectbounds the served context (CTX_MAX, default 8192).Gateway (
openai_server.py, +193): teach it to speak inkling — the glm path is byte-unchanged (ARCHdefaults toglm):--arch {auto,glm,inkling};autoreadsmodel_typefrom the model'sconfig.json, then picks the chat-template renderer and model-id.<|content_thinking|>/<|content_text|>channels intoreasoning_contenton both the streaming (InklingStreamSplit) and non-streaming chat paths.response_formatgrammars are refused for inkling (its serve loop speaks the 6-fieldSUBMITheader only) with a clear 400.Validation
make_tiny_inkling.py+./inkling 8 0 ref_inkling.jsonstill 36/36 teacher-forced argmax + 24/24 greedy. Serve mode is additive;-Wall -Wextraclean.SUBMIT→ 8DATAframes →DONE req1 STAT 8 … 2 1→ 9-fieldPROF. Sampling (temp/top_p) works; theDONE/STAT/PROFlayout matches the gateway's_statsparser exactly.openai_servertests pass (glm default path intact).Follow-up worth doing: a serve-mode smoke wired into the
inkling-oracleCI job (needs the tiny fixture to also emit a tokenizer). Happy to add it here or separately.🤖 Generated with Claude Code
https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27