test(llm): pin reasoning_effort transport and its guard - #17
Merged
Conversation
Chat felt like it dragged on every turn, and the suspicion was that something
was pushing the model to xhigh. It was not — the opposite. Familiar had no way
to set effort at all.
What was actually happening. The only thinking control on the wire was
chat_template_kwargs.enable_thinking, a bool. The classifier's Off/Low/Medium/
High level resolved to a TOKEN BUDGET, and on the trusted chat path even that
was discarded: pipeline.go deliberately grants the High budget (8000) every
time, because the level is a small model's one-shot guess and guessing Low
would truncate a thought mid-stream. MaxThinkingTokens was carried but, by its
own comment, "no provider reads this today". Net effect: every chat turn got
8000 tokens of headroom and reasoned at whatever --reasoning-effort the server
was launched with (medium on hive A). Trivial questions included.
Effort is the right knob precisely because it sidesteps that concern: telling a
model to think LESS cannot truncate it mid-thought, it just produces a shorter
one. So the classifier level can safely govern effort even though it must not
govern the token ceiling.
Transport matters, and this was measured rather than assumed. Against rune
(Qwen 3.8 on llama.cpp), one fixed prompt:
top-level "reasoning_effort" low 421 / medium 416 / xhigh 421 chars
chat_template_kwargs low 430 / xhigh 991 chars
enable_thinking:false 0 chars (control)
A top-level field is silently ignored. The first cut of this change sent
exactly that and would have shipped a no-op. It now rides inside
chat_template_kwargs alongside enable_thinking.
Mapping (ThinkingBudget.Effort, overridable per level via
[effort.thinking.<level>] effort = "..."):
off -> enable_thinking:false, no effort key
low -> "low"
medium -> "medium"
high -> "medium", NOT xhigh — the level is a one-shot guess and xhigh is a
large latency jump for a guess to be spending. Raise it in config
if wanted.
An empty Effort omits the key, so the server's launch default applies and
backends without an effort dial are unaffected.
Measured against rune (Qwen 3.8 / llama.cpp): a top-level reasoning_effort field is silently ignored — low/medium/xhigh all gave ~420 chars of reasoning on one fixed prompt — while inside chat_template_kwargs the same prompt gave 430 at low and 991 at xhigh. Moving it back to a top-level field would break nothing at runtime and just quietly stop working, so the test asserts BOTH that it rides in chat_template_kwargs and that it is absent as a top-level field. Also pins the guard that effort only ships when thinking is on, so backends with no effort dial (MLX/Gemma on the tier-4 path) keep receiving exactly what they got before. Verified live that MLX returns HTTP 200 and ignores the key rather than erroring, so this change cannot break tier 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured against rune (Qwen 3.8 / llama.cpp): a top-level reasoning_effort
field is silently ignored — low/medium/xhigh all gave ~420 chars of reasoning
on one fixed prompt — while inside chat_template_kwargs the same prompt gave
430 at low and 991 at xhigh. Moving it back to a top-level field would break
nothing at runtime and just quietly stop working, so the test asserts BOTH that
it rides in chat_template_kwargs and that it is absent as a top-level field.
Also pins the guard that effort only ships when thinking is on, so backends
with no effort dial (MLX/Gemma on the tier-4 path) keep receiving exactly what
they got before. Verified live that MLX returns HTTP 200 and ignores the key
rather than erroring, so this change cannot break tier 4.
Opened automatically after the
tierscheck passed on1dc356d9.The check is attached to this branch head, so this is mergeable now.