Fixed character encoding for arch-specific generate loops.

[fgheorghe]

Appears to affect: arch-specific generate loops.
Tested for Deepseek v4 flash 0713 - requires more manual testing to confirm nothing else is broken.
Supplied by GLM 5.2 and Claude. Feels like it could be done better though. Tests are dubious.

Problem

Any character whose UTF-8 spans more than one token — emoji, byte-fallback
CJK, ∑∫, accented Latin — reached the HTTP client as replacement chars:

"emoji: 🎉🔥🚀"   streamed out as   "emoji: ���������"

Tokenizer::decode() only reassembles byte fragments within a single call, so
decode(&[tok]) per streamed token runs from_utf8_lossy over half a
character. qwen35's ordinary daemon path already held the tail back correctly
with decode_bytes + valid_up_to — because that's the mainline arch, the bug
stayed invisible on every other one.

Broken — every request, all modes:
deepseek-v4-flash, gemma4, muse-glimmer, lfm2moe, minimax, cohere2moe, qwen2. Single-GPU, multi-GPU, expert-parallel and speculative decoding alike — no mode escaped it.

Broken — conditionally:
qwen3.5 / 3.6 only with serve.multi_slot = true. Off by default.

Fine:
qwen3.5 / 3.6 on ordinary serve (the default, so most traffic), the continuous-batch lane, and the pipeline-parallel path.

What actually corrupts: emoji always, maths symbols, accented Latin, and CJK on vocabs without whole-character tokens. Plain ASCII was never affected — which is why this survived so long.

Where it showed: visible content, the reasoning channel, and tool-call arguments — anywhere text crosses the wire. Not the KV cache or sampling; the model was fine, only the text handed back was mangled.

Change

Adds TokenTextStream: buffers only the trailing incomplete codepoint, O(1)
per token, ≤3 bytes held. Flushes at end of stream (generation can stop
mid-character); invalid bytes emit one replacement char and advance rather than
stalling the buffer.

Rewired: deepseek4 (AR/EP/multi-GPU/spec), gemma4 (AR + EAGLE), muse-glimmer,
lfm2moe, minimax (AR/EP), cohere2moe (AR + spec), qwen2, CLI multi-slot serve.

Channel-routed paths (cohere2moe sections, harmony router, ds4 DSML parser)
send the flushed tail down whichever channel was open at end of turn. Grammar
matchers keep per-token decode — they drive a state machine, not client output.

[Kaden-Schutt]

Triage: still relevant, but the branch predates the generate/daemon split. Please rebase and re-home TokenTextStream onto current spec_emit, dense loops, and multi-slot decoding with focused UTF-8 boundary coverage.