Repository navigation
fix(generate): start a chat reply on its own line after the prompt echo - #2862
Merged
Merged
Conversation
`mlxcel generate` echoes the user prompt and then streams the reply with no separator, so a chat-templated reply runs on from the prompt ("...on a laptop.Cores ablaze bright"). With a chat template the reply is a new assistant turn, so the echo now ends with a newline. Without one (`--no-chat-template`, or a checkpoint that has no template) the reply continues the prompt text and the echo is unchanged.
This matches llama.cpp: `llama-cli` echoes the prompt as `> prompt` on its own line before the reply, `llama-completion` in conversation mode echoes the rendered template so the reply follows the assistant header, and only raw `-no-cnv` completion prints prompt and continuation as one passage.
The text, pipeline, and XLA paths decide from whether the rendered prompt differs from the user prompt; the diffusion, LLaDA-2, and Florence-2 paths from `--no-chat-template`; Nemotron Parse always starts its output on a new line.
Validated with two unit tests and real runs on an M1 Ultra: Meta-Llama-3.1-8B-Instruct-4bit and gemma-4-e4b-it-4bit put the reply on the line after the prompt, and `--no-chat-template -p "The capital of France is"` still prints "The capital of France is Paris.".
inureyes
added
status:review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mlxcel generateechoes the user prompt and then streams the reply with no separator, so with a chat template the reply runs on from the prompt:A chat-templated reply is a new assistant turn, so the echo now ends with a newline and the reply starts on the next line. Raw completion (
--no-chat-template, or a checkpoint without a template) is unchanged: the continuation still follows the prompt text directly.This lines up with llama.cpp (checked at
69ee2ec):llama-cliechoes> prompton its own line before the reply,llama-completionin conversation mode echoes the rendered template so the reply follows the assistant header, and only-no-cnvraw completion prints prompt and continuation as one passage.Validation on an M1 Ultra: two new unit tests in
generate_tests.rs; Meta-Llama-3.1-8B-Instruct-4bit and gemma-4-e4b-it-4bit (with--image) now print the reply on the line after the prompt;--no-chat-template -p "The capital of France is"still printsThe capital of France is Paris..cargo fmt --checkand clippy on the binary target are clean. Not run locally: the model-gated integration tests that parse generate stdout (tests/lang_bias.rs,tests/surgery_cli.rs,tests/common/mod.rs::extract_generated_body); they read the text afterGenerating...\n, which now starts with the prompt line and a newline, so the extracted body gains one newline and their comparisons are between runs of the same build.The mlxcel.ai landing page captures were re-recorded with this change.