Skip to content

Commit 4963390

Browse files
fix(mac chat): strip gemma-4 reasoning-channel bleed (\nthought) from displayed answer
Cut the decoded text at the first channel marker (<|channel / <channel / \nthought / \nthink) so the chat shows only the natural-language answer. Applied to both the full fused-engine chat (harness _gen_turn) and the lightweight AR chat. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent f73f057 commit 4963390

2 files changed

Lines changed: 14 additions & 1 deletion

File tree

scripts/chat_mlx_kakeya.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -192,8 +192,14 @@ def _iter():
192192
stop_reason = "loop"
193193
break
194194
dt = max(time.time() - t0, 1e-9)
195+
_txt = tok.decode(toks, skip_special_tokens=True)
196+
# gemma-4 sometimes bleeds its reasoning channel after the answer; cut it.
197+
for _cut in ("<|channel", "<channel", "\nthought", "\nthink"):
198+
_i = _txt.find(_cut)
199+
if _i > 0:
200+
_txt = _txt[:_i]
195201
return {
196-
"text": tok.decode(toks, skip_special_tokens=True),
202+
"text": _txt.strip(),
197203
"n_tokens": len(toks),
198204
"stop_reason": stop_reason,
199205
"decode_tps": round(len(toks) / dt, 2),

scripts/research/k3_integrated_niah_eval_mac.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -808,6 +808,13 @@ def _gen_turn(pid: List[int]) -> Dict[str, Any]:
808808
txt = tokenizer.decode(res["tokens"])
809809
for marker in ("<turn|>", "<end_of_turn>", "<eos>"):
810810
txt = txt.replace(marker, "")
811+
# gemma-4 sometimes bleeds its reasoning channel after the answer
812+
# (e.g. a trailing "\nthought ...") — cut at the first channel
813+
# marker so the chat shows only the natural-language answer.
814+
for cut in ("<|channel", "<channel", "\nthought", "\nthink"):
815+
idx = txt.find(cut)
816+
if idx > 0:
817+
txt = txt[:idx]
811818
res["text"] = txt.strip()
812819
res["resident_kv_bytes"] = int(
813820
sum(int(getattr(c, "nbytes", 0)) for c in (adapter._cache or [])))

0 commit comments

Comments
 (0)