gemma-4 GGUF: <|tool_response> is not a stop token, so every tool call burns max_tokens and returns finish_reason=length

[salekseev]

Summary

On the gemma-4 GGUF path, every tool call runs to max_tokens. The model emits a valid tool
call, then repeats <|tool_response> (vocab id 50) until the budget is exhausted, because
that token is not in the stop set. Three consequences per tool call:

• the whole completion budget is spent (99 of 100 tokens in the example below)
• finish_reason is "length", not "tool_calls" — clients that branch on it mis-handle the turn
• the raw control token leaks into message.content

finish_reason: length
usage: {"prompt_tokens": 281, "completion_tokens": 99, "total_tokens": 380}
content: '<|tool_response><|tool_response><|tool_response><|tool_response>...'   (99 tokens of it)
reasoning_content: ''
tool_calls: [{"function": {"name": "get_current_weather",
                           "arguments": "{\"city\": \"Dallas\", \"state\": \"TX\", ...}"}}]

Cause

models/gguf/tokenizer.py::gguf_eos_token_ids builds the stop set from
tokenizer.ggml.eos_token_id plus two hardcoded vocab names:

for name in ("<eos>", "<turn|>"):
    try:
        ids.add(tokens.index(name))
    except ValueError:
        pass

For unsloth/gemma-4-26B-A4B-it-qat-GGUF both resolve — <eos> is 1 and <turn|> is 106 —
so the stop set is {1, 106}, which is correct for ordinary chat and matches gemma-4's
documented eos_token_id: [1, 106]. What is missing is <|tool_response> (id 50).

gemma-4's vocab uses paired markers, opener <|x> and closer <x|>:

id   46 = '<|tool>'          id   47 = '<tool|>'
id   48 = '<|tool_call>'     id   49 = '<tool_call|>'
id   50 = '<|tool_response>' id   51 = '<tool_response|>'
id  105 = '<|turn>'          id  106 = '<turn|>'

The tool-response section is supplied by the caller, so the model emitting its opener means
"my turn is done, your turn to run the tool". llama.cpp treats it as end-of-generation for
this same checkpoint — it reports special_eog_ids contains '<|tool_response>' on load.
FreeToken has no equivalent, so sampling continues and the token is re-emitted indefinitely.

Reproduction

Any tool-calling request on a gemma-4 GGUF. It is also caught by vLLM's own tool-use
conformance tests, which are engine-agnostic once pointed at a running server
(tests/tool_use/test_tool_calls.py::test_tool_call_and_choice and
test_parallel_tool_calls.py::test_parallel_tool_calls both fail with
assert 'length' == 'tool_calls').

curl -s http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' -d '{
 "model":"<served-name>",
 "messages":[{"role":"user","content":"What is the weather in Dallas TX? Use the tool."}],
 "tools":[{"type":"function","function":{"name":"get_current_weather","description":"Get the current weather",
   "parameters":{"type":"object","properties":{"city":{"type":"string"},"state":{"type":"string"},
   "unit":{"type":"string","enum":["celsius","fahrenheit"]}},"required":["city","state","unit"]}}}],
 "tool_choice":"auto","max_tokens":100}' | python3 -m json.tool

Fix

Adding the opener to the same lookup is sufficient, and is what I am running locally:

for name in ("<eos>", "<turn|>", "<|tool_response>"):

With that change the same request returns finish_reason: "tool_calls", content: "", and
spends only the tokens the tool call actually needs.

A more general version would honour the checkpoint's declared end-of-generation set rather
than a hardcoded name list — this GGUF also sets tokenizer.ggml.eot_token_id = 106, which
the current code happens to catch via the <turn|> name but does not read.

Second, separate issue in the same area: parallel_tool_calls: false is ignored

ChatCompletionRequest accepts parallel_tool_calls (server/api_models.py:89) but nothing
enforces it. A request with parallel_tool_calls: false still comes back with two tool calls,
which is what makes vLLM's
test_parallel_tool_calls.py::test_parallel_tool_calls_false fail (assert 2 == 1). Happy to
split this into its own issue if you prefer.

Environment

• FreeToken 0.1.2 (PyPI wheel + freetoken_kernel_cache-0.1.2+cu130)
• RTX 4080 SUPER (sm_89, 16 GB), driver 610.57.04, CUDA UMD 13.3
• unsloth/gemma-4-26B-A4B-it-qat-GGUF (Q4_0), --tool-call-parser auto-resolved to gemma4
• Tree also carries fix(kernel): launch GGUF MoE GEMV token axis on grid.x, not grid.z #185, feat(kvcache): 8-bit KV cache (q8_0 / fp8_e4m3) behind --kv-cache-dtype #103, feat(server): expose swa_full_tokens_ratio as a startup CLI flag #109 and a local gemma4 embedding-quant fix (gemma4 GGUF: embedding quant type hardcoded to Q6_K, so a Q4_0-embedding GGUF fails to load with a bare AssertionError #188); none touch
  tokenisation or sampling.

For context on what does work: test_tool_call_with_results,
test_parallel_tool_calls_with_results, test_chat_completion_without_tools and
test_chat_completion_with_tools all pass, so tool-call parsing and multi-turn tool results
are fine — it is only termination that is wrong.