fix(console): ask chat models for minimal reasoning effort - #1875
michaelneale wants to merge 3 commits into
Conversation
Ask a reasoning model a casual question in the console today and it spends
tens of seconds deliberating before it answers. On a 9B at Q4 it often never
answers at all: the reasoning consumes the whole token budget, `finish_reason`
comes back `length` and `content` is null, so the chat window shows a growing
thinking trace and then nothing. Worse, the deliberation degenerates -- one
measured run repeated "Wait, could it be ..." 39 times before terminating.
Measured on Qwen3.5-9B-GGUF:Q4_K_M, prompt "what is updog", max_tokens 4000:
thinking off 350 tokens, 4.6 s, answered
default 4000 tokens, 38 s+, 13.5 KB of reasoning, no answer
The console now asks for a direct answer, as it did before the UI rewrite:
`00296cc3a` added `chat_template_kwargs: {enable_thinking: false}` to console
chat requests in March for exactly this symptom, and #545 dropped it when the
request builder was rewritten.
API callers are unaffected and keep full control: `reasoning_effort`,
`reasoning.enabled` and `chat_template_kwargs` all work, verified against a
live node. Only the console's own requests change.
Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>
Signed-off-by: Michael Neale <14976+michaelneale@users.noreply.github.com>
A reasoning model asked a casual question in the console deliberates for tens
of seconds before answering, because nothing in the request asks it not to. On
a 9B at Q4 the deliberation sometimes never converges: it consumes the whole
token budget, `finish_reason` is `length` and `content` is null, so the window
shows a growing thinking trace and then nothing.
The console now sends `reasoning_effort: "minimal"`. Thinking stays on and the
trace still renders -- the console has the UI for it and it is worth showing --
but the model is asked for the least of it.
This is a mitigation, not a cure, and the honest numbers say so. Measured on
Qwen3.5-9B-GGUF:Q4_K_M, "what is updog", max_tokens 4000, three runs:
minimal 2158 / 4000 / 3888 tokens -- answered, exhausted, answered
Effort is a hint to the chat template, not an enforced budget, so the failure
is less likely rather than impossible. The same prompt with reasoning off
answers every time in ~350 tokens; that was rejected because it throws away a
feature the console exists to display.
API callers are unaffected and keep full control: `reasoning_effort`,
`reasoning.enabled` and `chat_template_kwargs` all work. Only the console's own
requests change, and `chat_template_kwargs` is deliberately not sent with the
effort hint -- an explicit template setting wins over an effort alias
(`normalize_reasoning_template_options`), so sending both would silently
override the hint this change exists to send.
Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>
Signed-off-by: Michael Neale <14976+michaelneale@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review. 📝 WalkthroughWalkthroughThe chat request type now supports reasoning effort values. ChangesChat reasoning effort
Priority: ⬇️ Low Estimated code review effort: 1 (Trivial) | ~5 minutes Change: Bug fix Suggested reviewers: Merge Risk: ⚪ Minimal · up to The console requests minimal reasoning effort while retaining reasoning output, with no identified merge-blocking risk. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merge origin/main at 2d83baa without rewriting the console minimal-reasoning commits. This synchronizes the source slice catalog with the protected planner; no CI gates or console behavior are changed. Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com> Signed-off-by: Michael Neale <14976+michaelneale@users.noreply.github.com>
| // when nothing asks them not to. The console keeps thinking -- it renders | ||
| // the trace -- but asks for the least of it. API callers are unaffected: | ||
| // they choose their own reasoning controls. | ||
| reasoning_effort: 'minimal' |
There was a problem hiding this comment.
@michaelneale would low be safer here? I've infrequently seen minimal used
There was a problem hiding this comment.
hrm yeah - don't know, takes so long to test this so not sure.
|
I think this is the correct way to fix this #1876 |
i386
left a comment
i386
left a comment
There was a problem hiding this comment.
Requesting changes because the new request field does not change Qwen3.5 behavior and it bypasses the precedence required by #1876.
1. [P1] minimal is a no-op for the reported Qwen3.5 path
At build-input.ts:194, reasoning_effort: "minimal" reaches normalize_reasoning_template_options, which maps every non-none effort to enable_thinking=true and forwards the effort string as a template kwarg. The Qwen3.5 template at the tested source revision reads enable_thinking, but contains no reference to reasoning_effort, reasoning_budget, or thinking_budget. The prior/default request already gave that template enable_thinking=true, so minimal, low, and an omitted effort render the same prompt for this model.
That means the 2/3 versus 0/3 live result does not establish that this patch mitigates the failure; it is consistent with sampling variance, and one of the three patched runs still exhausted all 4,000 tokens without an answer. Changing this line to low would have the same behavior for Qwen3.5.
Source template: https://huggingface.co/Qwen/Qwen3.5-9B/blob/c202236235762e1c871ad0ccb60c8ee5ba337b9a/tokenizer_config.json
2. [P1] the unconditional request override defeats operator and package policy
chat_template_options resolves the request's enable_thinking before defaults.reasoning_enabled, and request template kwargs overwrite default kwargs. Consequently, a console request from this PR turns thinking on even when an operator configured reasoning_enabled="off". It would also override the per-model thinking/direct profiles proposed in #1876, because request values are intentionally the highest-precedence layer.
The console should omit a reasoning override unless the user explicitly selected one. Model/operator/package defaults then remain effective.
3. [P1] the original unbounded failure remains
This PR adds neither max_output_tokens nor a hard reasoning-token budget. When Qwen does not emit </think>, generation can still use the remaining context and return no visible answer. A semantic effort hint cannot reserve output tokens for the answer.
Recommended change
- Implement #1876 as the durable path: package-owned model profiles, request → operator → package → global precedence, a finite total-output fallback, and sampler-enforced reasoning budgets.
- Remove the unconditional
reasoning_effortfrom the request builder. - If a console hotfix must ship before hard budget enforcement, use the only deterministic Qwen3.5 control currently available: default thinking off with an explicit visible opt-in, and send a finite
max_output_tokens. Keep that clearly scoped as an interim console policy. - Test resolved server/template behavior for the target model and operator-default precedence, rather than only asserting that the UI object contains the field.
Validation at PR head 0bb908ba5cf8e2bd96b38dac7848b0ade1a2fa3a: UI typecheck, all 1,751 tests (3 skipped), and production build pass. Those checks establish code health; they do not cover the two behavioral failures above.
|
@i386 not sure if it would - llama.cpp and other things have same issue - its just how they work I guess? expectation is that harness would set it maybe? |
|
"minimal" does work - when I tried it, however, so not sure the agents are correct here |
llama would have the same problem as its thinking budget also defaults to -1 |
|
yeah ideally shouldn't use this I think - just use the correct recommendations? |
yep needs recommendations. thats where our model packages can come in handy. we can do that testing and encode whats right in it. |
3 participants
Ask a reasoning model a casual question in the console today and it deliberates for tens of seconds before answering — and on a 9B at Q4 it sometimes never answers at all: the reasoning consumes the whole token budget, so
finish_reasonislengthandcontentis null. The chat window shows a growing thinking trace and then nothing.The console now sends
reasoning_effort: "minimal". Thinking stays on and the trace still renders — the console has the UI for it — but the model is asked for the least of it.This is a mitigation, not a cure
Effort is a hint to the chat template, not an enforced token budget. Measured on a live node,
unsloth/Qwen3.5-9B-GGUF:Q4_K_M, prompt "what is updog",max_tokens: 4000:reasoning_effort: "minimal"So this makes the failure less likely, not impossible. Turning reasoning off is reliable but discards a feature the console exists to display, which is why it is not what this PR does.
Protocol
No API change.
reasoning_effort,reasoning.enabledandchat_template_kwargsall work today and are untouched. Only the console's own requests carry the hint.Note on why
chat_template_kwargsis not sent alongside: an explicit template setting wins over an effort alias innormalize_reasoning_template_options, so sending both would override the hint this change exists to send.Validation
pnpm run typecheckclean; chat API tests pass (9 files, 57 tests), including a new one asserting the request carriesminimaland no conflicting template kwargs. Live behaviour is the table above.Not covered: no screenshot — the visible change is timing and the absence of an unanswered thinking trace, not new UI.
Summary by CodeRabbit
New Features
Tests