server: do not abort a completion when the model emits invalid UTF-8

[danielhanchen]

What happens

A request can end with no tokens at all. The server logs

W common_chat_peg_parse: unparsed Content-only output: <byte>
W srv          stop: cancel task, id_task = N
I slot      release: id X | task N | stop processing: n_tokens = 128, truncated = 0

and the client sees a 200 with a truncated SSE stream and zero content deltas. On a
non-streaming request the same thing surfaces as

{"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format","type":"server_error"}}

Why

The generated text is a raw byte stream and is not guaranteed to be valid UTF-8. A byte
fallback token, or a prompt that ends in the middle of a multi-byte character, in which case
the model correctly continues with the remaining continuation bytes, makes the generated text
start with or contain bytes that do not decode.

task_result_state::update_chat_msg() hands that text to common_chat_parse() on every
token. The PEG parsers reject malformed UTF-8 by design, tests/peg-parser/test-unicode.cpp
asserts exactly that, and the std::runtime_error thrown for it is raised inside
server_response_reader::next(), which runs on the HTTP thread inside the streaming
res->next closure. It escapes into the HTTP layer, the connection is torn down, and the
reader's destructor cancels the task. The endpoint does not matter: /completion builds the
chat message too, so a plain completion request dies the same way.

validate_utf8() in the token loop only holds back a multi-byte sequence that is cut off at
the end. A byte that can never start a codepoint passes straight through to the parser.

The fix

Normalise the text before parsing: hold back the trailing bytes of an incomplete sequence
until the next chunk completes it, and replace bytes that can never form a codepoint with
U+FFFD. That is what the JSON serialiser already substitutes on the way to the client, so the
parser now sees exactly the text the client receives. generated_text in task_result_state
is only read by update_chat_msg(), so nothing else changes.

What was measured

Reproduced deterministically with stories15M-q4_0.gguf on CPU, forcing the byte fallback
token 164 (a lone 0xA1) with a logit bias, so no GPU and no large model is needed:

curl -s http://127.0.0.1:8099/completion -d '{"prompt":"Once upon a time","n_predict":8,
  "ignore_eos":true,"stream":true,"temperature":0,"top_k":1,"logit_bias":[[164,50.0]]}'

Before: {"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format"}}, zero tokens, streaming and non-streaming alike.
After: all 8 tokens are delivered, tokens_predicted = 8, content rendered as U+FFFD.

In production traffic this was one request in 64 at 32 concurrent users on Qwen3 27B
UD-Q4_K_XL with 128 token prompts drawn as arbitrary token id windows of a corpus, so a
prompt ending mid-character is common. Across 52 server logs from unrelated benchmark
windows there are 65 of these aborted requests and every single one of them contains invalid
UTF-8 in the text the parser was given: 38 with an invalid lead byte, 27 with a bad
continuation byte, none with anything else.

test-peg-parser, test-chat-peg-parser and test-chat pass unchanged. Plain completions,
chat completions and streaming chat completions were checked for regressions on the same
tiny model, and a multi-byte character split across two tokens still arrives whole.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for security reviews. Please try again later.

[danielhanchen]

Hardware evidence, since the report above was proved on CPU with a 19 MB model and a forced logit bias.

Reproduced and cured on a DGX Spark GPU with a real quantised model, using the harness and prompt shape that produced the original observation: Qwen3.5-4B UD-Q4_K_XL, -ngl 99 -fa on -c 16384 --parallel 32 --cache-ram 0 --kv-unified, 64 requests at 32 concurrent, prompts drawn as 128 token windows of a text corpus, n_predict 256. Two CUDA builds from one base, fbf9abcc7 and the same tree plus this commit.

Before: 63 of 64 requests return tokens, in six independent arms across two cells. The server log carries the signature three times, once per arm:

0.29.450.277 W common_chat_peg_parse: unparsed Content-only output: \xa1
0.29.450.738 W srv          stop: cancel task, id_task = 310
0.29.455.385 I slot      release: id 10 | task 310 | stop processing: n_tokens = 128, truncated = 0

n_tokens = 128 is the prompt alone, against 383 for every healthy request in the same cell (189 of them in that cell, plus 67 at 135 from the serial probe).

After: 64 of 64 in every arm, and zero occurrences of the warning.

The failing byte is 0xA1 and it is now pinned down exactly. Replaying the same prompt draws serially isolates it to one request, at corpus token offset 48080. The corpus there is a markdown table of status emoji, and 🟡 is three tokens in this vocabulary: 10838 = 20 F0 9F, 253 = 9F, 94 = A1. The 128 token window ends on 253, so the prompt stops after F0 9F 9F and the model correctly emits 94, a lone 0xA1, as its first generated token. Served serially that same request is a clean 500 on the base build and a 200 on this one, so the defect needs no concurrency at all; concurrency only decides which draw happens to straddle a character.

With the fix the request returns n_predict 8 tokens and the content "� | ✅ | ✅ | ✅ |", first bytes ef bf bd 20 7c 20 e2 9c 85. The U+FFFD is in the first position and nowhere else, which is exactly right: the orphaned continuation byte can never be completed because its lead bytes are in the prompt and were never in generated_text, while the multi-byte ✅ later in the same chunk arrives intact and the model goes on producing the table row it was in the middle of.

Cost. Bracketed base / fixed / base at 32 concurrent, three arms of 64 requests per cell, one lock hold, GPU clocks pinned at 1690 MHz for the whole block, every arm 1677 to 1683 MHz, cell maxima 54 C, no clock transition:

cell	build	ok	agg tok/s	median
base	fbf9abc	63, 63, 63	336.27, 333.35, 332.86	333.35
fixed	+ this PR	64, 64, 64	336.35, 333.78, 332.36	333.78
base	fbf9abc	63, 63, 63	336.91, 333.47, 335.28	335.28

The fixed cell lands between the two base cells, which themselves differ by 0.58 percent.

The per-token figure in that bracket is confounded, because the base arm loses a slot in its first decode step and then runs 31 requests where the fixed arm runs 32. So the cost was measured again on a shape where both builds complete all 64, concurrency 8 with 8 requests per client, same pin, same block:

cell	build	ok	agg tok/s	median TPOT ms
base	fbf9abc	64, 64	204.59, 201.52	36.99, 37.38
fixed	+ this PR	64, 64	207.03, 201.64	36.47, 37.34
base	fbf9abc	64, 64	206.20, 201.81	36.85, 37.16

With identical work on both sides the fixed build is inside the 0.46 percent drift between the two base cells and the TPOT distributions are indistinguishable. append_utf8_sanitized() scans only the bytes added since the last token, so the added work is linear in the generated bytes and is dwarfed by the full re-parse of the accumulated text that update_chat_msg() already did per token before this change.