Summary
Gemma4 can generate a valid, coherent answer after an orphan <|channel> token, while GemmaThoughtRouter remains in AwaitingThought and drops the entire response. The daemon reports a successful generation with hundreds or thousands of tokens, but the client receives an empty or severely truncated answer.
This is a user-facing correctness issue rather than a kernel or tokenizer failure.
Reproduction
Observed with the lowered Gemma4 MoE route on an R9700 (gfx1201), using a Gemma4-26B-A4B-it artifact quantized from the ModelScope google/gemma-4-26B-A4B-it checkpoint:
- artifact SHA-256:
45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1c
- KV mode:
q8
max_seq: 16384- temperature: 0
- prompt MD5:
0c6464135367951bcee596c9a4a62211
A 256-token diagnostic run completed successfully:
- generated tokens: 256
- prefill: 58.44 tok/s
- decode: 49.99 tok/s
- runtime errors: 0
- visible output: 0 bytes
The token/logit trace showed:
- token 0: ID 100, decoded as
<|channel>, top-1 margin 28.386
- token 1: ID 107, decoded as
\n
- remaining tokens: coherent Python/Pygame snake-game implementation
- sampled token matched top-1 for 256/256 steps
- maximum non-finite-logit count: 0
- all 256 decoded fragments were non-empty
The concatenated decoded stream begins:
<|channel>
```python
import pygame
import random
import sys
...
The same prompt previously reproduced deterministically twice at 4096 generated tokens, with only 63 visible characters, ruling out request-to-request KV contamination and an intermittent GPU race.
Root cause
When thinking is enabled, GemmaThoughtRouter starts in AwaitingThought and looks for <|channel>thought.
If the pending text can no longer form that header, gemma_longest_marker_suffix() returns zero, but the current branch breaks without emitting or changing state:
if hold == 0 || hold >= self.pending.len() {
break;
}The pending answer therefore grows for the entire generation. At EOS, flush() clears all pending text while still in AwaitingThought.
There is a second data-loss hazard in gemma_is_marker_prefix(): s.starts_with(marker) classifies a marker followed by arbitrary payload as if the whole string were only a control-marker prefix.
The original checkpoint's response_schema makes the thought channel optional and recognizes reasoning only as <|channel>thought\n...<channel|>. A noncanonical orphan <|channel> should not cause the following recoverable content to disappear.
The router was introduced by commit b482d9514; this behavior predates and is independent of the HFQ4-G128 row-tail work in #667.
Proposed fix
- Hold bytes in
AwaitingThought only while they remain a prefix of the canonical thought header.
- Consume the canonical header and transition to
Reasoning when it completes.
- Otherwise transition to
Answer; if the stream begins with an orphan <|channel>, strip only that control token and an optional newline, then emit the payload.
- Treat only an exact/incomplete marker as a marker prefix; never discard marker-plus-payload as a unit.
- At EOS, suppress only an incomplete control marker, not ordinary pending content.
The same GemmaThoughtRouter is used by both eager and lowered Gemma4 generation, so the fix should cover both routes.
Six CPU regression tests covering the canonical thought path, the observed orphan-channel sequence, chunk-boundary invariance, direct answers with thinking enabled, EOS handling, and marker-plus-payload classification are passing on the #667 source branch. R9700 and W7900 runtime reruns are in progress.
Summary
Gemma4 can generate a valid, coherent answer after an orphan
<|channel>token, whileGemmaThoughtRouterremains inAwaitingThoughtand drops the entire response. The daemon reports a successful generation with hundreds or thousands of tokens, but the client receives an empty or severely truncated answer.This is a user-facing correctness issue rather than a kernel or tokenizer failure.
Reproduction
Observed with the lowered Gemma4 MoE route on an R9700 (
gfx1201), using a Gemma4-26B-A4B-it artifact quantized from the ModelScopegoogle/gemma-4-26B-A4B-itcheckpoint:45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1cq8max_seq: 163840c6464135367951bcee596c9a4a62211A 256-token diagnostic run completed successfully:
The token/logit trace showed:
<|channel>, top-1 margin 28.386\nThe concatenated decoded stream begins:
The same prompt previously reproduced deterministically twice at 4096 generated tokens, with only 63 visible characters, ruling out request-to-request KV contamination and an intermittent GPU race.
Root cause
When thinking is enabled,
GemmaThoughtRouterstarts inAwaitingThoughtand looks for<|channel>thought.If the pending text can no longer form that header,
gemma_longest_marker_suffix()returns zero, but the current branch breaks without emitting or changing state:The pending answer therefore grows for the entire generation. At EOS,
flush()clears all pending text while still inAwaitingThought.There is a second data-loss hazard in
gemma_is_marker_prefix():s.starts_with(marker)classifies a marker followed by arbitrary payload as if the whole string were only a control-marker prefix.The original checkpoint's
response_schemamakes the thought channel optional and recognizes reasoning only as<|channel>thought\n...<channel|>. A noncanonical orphan<|channel>should not cause the following recoverable content to disappear.The router was introduced by commit
b482d9514; this behavior predates and is independent of the HFQ4-G128 row-tail work in #667.Proposed fix
AwaitingThoughtonly while they remain a prefix of the canonical thought header.Reasoningwhen it completes.Answer; if the stream begins with an orphan<|channel>, strip only that control token and an optional newline, then emit the payload.The same
GemmaThoughtRouteris used by both eager and lowered Gemma4 generation, so the fix should cover both routes.Six CPU regression tests covering the canonical thought path, the observed orphan-channel sequence, chunk-boundary invariance, direct answers with thinking enabled, EOS handling, and marker-plus-payload classification are passing on the #667 source branch. R9700 and W7900 runtime reruns are in progress.