feat(serve): override the frontend chat template via --chat-template FILE - #183
feat(serve): override the frontend chat template via --chat-template FILE#183wojciak wants to merge 1 commit into
Conversation
…FILE Adds a startup-time chat template source override for ninfer-serve: - ServeOptions/EngineOptions/FrontendOptions carry an optional path. - make_frontend replaces resources.chat_template_jinja and syncs tokenizer_config.json.chat_template so the existing validation and SHA-256 acceptance gates run against the override unchanged. - resolve() accepts froggeric/Qwen-Fixed-Chat-Templates v22.5 (template_version qwen3.8-froggeric-v22.5) as reasoning-effort semantics; the renderer already matches its prompt surface. - Tests: --chat-template parse + usage; v22.5 fixture resolves and renders the official reasoning-effort prompts.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5e28b53f6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (digest == kReasoningEffortTemplateV225Digest) { | ||
| return CompiledChatTemplate(ChatTemplateSemantics::ReasoningEffort); |
There was a problem hiding this comment.
When --chat-template selects v22.5, thinking is enabled, and the client omits reasoning_effort, the accepted source explicitly defaults to medium (reasoning_effort_froggeric_v225_chat_template.jinja:17-29), but this branch aliases it to semantics whose capabilities and renderer default to XHigh (capabilities() at line 448 and resolve_reasoning_instructions() at line 367). Ordinary requests therefore receive an extra xhigh system instruction and are logged as xhigh instead of rendering the selected template; represent the v22.5 default and render behavior separately rather than aliasing it to the existing template.
Useful? React with 👍 / 👎.
| std::ostringstream buffer; | ||
| buffer << stream.rdbuf(); | ||
| std::string source = buffer.str(); | ||
| if (source.size() > (10U << 20)) { |
There was a problem hiding this comment.
When FILE is accidentally a very large file or an unbounded stream, the entire input is copied into the ostringstream and then again into source before the 10 MiB check runs. Pointing this option at a multi-gigabyte artifact can therefore exhaust memory or stall startup instead of producing the intended size-limit error; bound the read to 10 MiB plus one byte, optionally using an early regular-file size check.
Useful? React with 👍 / 👎.
…usion, template switch (T33-T40) - T33 (top priority): DFlash2 drafter GRAFTED onto our QUASAR artifact - operator constraint is to keep quasar weights and add the drafter on top. gpillon's graft_dflash2_module.py proves the mechanism (66 appended objects, base tensors bit-identical); the drafter checkpoint is public upstream (z-lab/Qwen3.8-27B-DFlash2), so we do NOT need gpillon's nvfp4full artifact. 27B gains a separate DFlash2Config (execution=true, max draft 7) while DFlash v1 stays permanently unsupported - that, not the artifact, was the real T8/T28 blocker. Acceptance-gated probe required: gpillon reports 3.4-3.7 tok/round but ninfer-fusion measures DFlash2 < MTP3 at every context. - T34: gpillon f4b128c proves packed rewrite-checkpoint restores from host RAM serve one request with another's state -> T31 must NOT relax the frontier invariant. - T35/T36: md's 48 branches are mostly off-lane (35B sparse-MoE); on-lane = draft window k=3->5 (+17.3%, zero code), decode-softmax-fold, 27B-measured nvfp4-TMA. - T37: chat-template switch verified - omitting --chat-template-file resolves the artifact-embedded template whose sha == kReasoningEffortTemplateDigest -> compiled-in ReasoningEffort with native xhigh. Live template is Sharp v22.4.0, not v22.1. - T38/T39/T40: upstream Neroued#183/Neroued#182/Neroued#184; ninfer-fusion WATCH (sub-floor KV reject); dylan cdd1b6c probe. - T31 pick set INVALID: gzenz force-pushed 4627561 -> 5f23c37, dropping the 5 rewrite-checkpoint commits the port was derived from.
1 participant
Closes #182
Design
--chat-template FILEis a startup-time template source override layered at frontend creation:ServeOptions.chat_template_path(flag parse + usage text) →EngineOptions.chat_template_path(generation_service.cpp) →FrontendOptions.chat_template_path(both 27B and 35B-A3BPackage::make_frontendforward it).make_frontend(andFrontendTestAccess::create_component(options)) copiesFrontendResources, replaceschat_template_jinjawith the file bytes (size-capped at 10 MiB, invalid-UTF-8 rejected with a clearinvalid_argument), and reserializestokenizer_config.json.chat_templateto the same bytes.validate_tokenizer_configandCompiledChatTemplate::resolvethen run unchanged against the consistent pair — the acceptance contract is untouched for non-users of the flag.CompiledChatTemplate::resolvegains one accepted digest:froggeric/Qwen-Fixed-Chat-Templatesv22.5 (template_version = "qwen3.8-froggeric-v22.5"), mapped to the existingReasoningEffortsemantics. The fixed C++ renderer already reproduces that template's prompt surface; deliberately unimplemented family features (in-messagethink_on/think_off/think_lowcontrol markers,auto_disable_thinking_with_tools,max_tool_arg_charscaps) are documented at the digest.Affected contract
EngineOptions/FrontendOptions: one new optionalstd::filesystem::pathfield each.Verification
Both suites exit 0 on this branch. Live serving check:
ninfer-serve --chat-template <v22.5 file>accepts the template at startup; requests carry the v22.5 system prompt (verified against our deployment on RTX 5090).