Skip to content

[ENHANCEMENT] Dynamic Thinking Effort: AI self-decides per-turn reasoning effort (experimental) — design + implementation plan #28

Description

@easonLiangWorldedtech

[ENHANCEMENT] Dynamic Thinking Effort — letting the AI decide its per-turn reasoning effort (experimental)

Design + implementation plan. Research-backed (2026-08 industry + arXiv review).
Related upstream gap: Roo Code issue #7048 (effort override semantics).

1. Problem and goals

Today reasoning effort can only be fixed once in Settings (static per-profile). It is not possible to:

  • Let the model decide its effort based on task complexity
  • Raise/lower the effort mid-task as needed (e.g. start cheap, escalate to high only when tests fail)
  • Let the user adjust it in chat instantly, without going to Settings

Goals (scope of this issue):

# Requirement
F1 Fix: Anthropic adaptive models receive no thinking content + no thinking-token data + no effort envelope (see §4)
F2 An experimental settings toggle that enables the whole feature
F3 Native tool set_thinking_effort: the AI decides its effort mid-task, with a display in the chat (same pattern as switch_mode)
F4 Top dashboard (TaskHeader) shows the current effort, including the init state (the resolved default at task start)
F5 Bottom manual button (composer bottom bar): temporary, current chat only, exactly the same scope as the AI's tool use (the same task-local state)
F6 Orchestrator: new_task supports a thinking-effort input (model-specified); the user can switch it before entering the subtask (before clicking the button)

Non-goals (backlog): per-message quick chips / prompt trigger words ("ultrathink"), session-level thinking ledger / overthinking warnings, per-mode effort policies, cost-saver mode.

2. Permission question: does the AI's auto-switch need approval?

Conclusion: no. set_thinking_effort executes directly (no approval gate) and is bounded by guardrails instead:

Reason Explanation
Non-destructive It changes no files, runs no commands, and does not affect the permission context. Compare: switch_mode changes the mode (which changes the available tools/permissions), so it needs approval; new_task spawns a new task, so it needs approval. Effort is neither.
Reversible It can be changed back at any time (by the user or the model); the bottom button is the undo mechanism.
Bounded cost Effort is clamped to the levels the model supports, so it cannot escalate unbounded; plus an escalation cap (guardrails below)
Product logic Requiring human approval for every switch defeats the purpose of the dynamic loop (the model could not self-adjust while the user is away) and adds a latency round-trip each time. Industry precedent (Claude Code /effort + adaptive default, OpenCode adaptive-thinking plugin) also has no approval.

Guardrails (in place of approval):

  1. Always notify: every change by the model or the user emits a display line in the chat stream (never a silent change)
  2. Escalation cap: at most N upward adjustments per task (3 suggested); beyond that, clamp + notify
  3. Oscillation detection: low → high → low within 3 steps notifies once (possible cost spiral)
  4. Hard cap: never exceeds the highest level the model supports (per the supportsReasoningEffort capability array)
  5. (Fallback, off by default) a "effort changes require approval" setting could be added for conservative users — out of scope for v1

3. High-level UI/UX design

3.1 Single source of truth: task-level effective effort

resolution order (strongest first):
  1. task.runtimeThinkingEffort   ← written by the tool or the bottom button (same state, last-write-wins)
  2. apiConfiguration.reasoningEffort   ← the Settings persisted value
  3. model.reasoningEffort default  ← the model definition
  • runtimeThinkingEffort is task-layer: stored with the task (persisted in its history item) and restored when the task is reopened from history; never written to Settings, and never pollutes the profile. F5's "same scope as the AI's tool use" = both sides read/write this same field.
  • Every display surface shows the effective value + source (default / Zoo (auto) / you), so the user can always tell where the value came from — this is what makes "let the AI decide" trustworthy (transparency builds trust).

3.2 The surfaces

(a) Experimental settings toggle

  • Follows the existing ExperimentalSettings.tsx generic pattern: EXPERIMENT_IDS.DYNAMIC_THINKING_EFFORT + i18n (settings:experimental.DYNAMIC_THINKING_EFFORT.name/.description)
  • Toggle semantics: "Let the model decide its thinking effort per step, and let you adjust it in-chat"
  • Off = the set_thinking_effort tool is not exposed, so model-driven mid-task switching is disabled. The composer toggle and TaskHeader chip are capability-gated (they render whenever the selected model advertises per-request effort support — registry capability or an F7 declaration) and stay available regardless of this toggle

(b) Top dashboard (TaskHeader) chip

  • Placement: near the cost/tokens row, a small chip:
┌──────────────────────────────────────────────┐
│ Fix the login bug…        🧠 High · Zoo      │  ← thinking chip (new)
│ $0.05 · 2.1k in · 980 out · 42%             │
└──────────────────────────────────────────────┘
  • Init state: at task start it shows the default resolved by the resolution order (e.g. 🧠 Medium · default); a reopened task shows its restored value with the saved source — the user does not have to wait for the first switch to know the current level
  • The source badge updates with changes: · default· Zoo (model tool) → · you (manual)
  • Tooltip (shipped): "Thinking effort: {{effort}} ({{source}})" — e.g. "Thinking effort: High (you)"; source labels: default / Zoo (auto) / you; adaptive-class models additionally show "This model decides its effort automatically — your selection is soft guidance only."
  • Provider honesty: for providers without native adaptive, the tooltip truthfully states that effort is a per-request parameter (takes effect on the next request), no "adaptive" overselling

(c) Bottom manual button (ChatTextArea bottom bar)

  • Placement: between the API-profile selector and the auto-approve (⚡) control, small button in the same row (same border/hover treatment as the sibling selectors):
[ Code ▾ ]  [ api-profile ▾ ]  [ 🧠 High ▾ ]  [⚡▾]   [ Send ]
  • Click → small menu: effort levels (only values the model's supportsReasoningEffort supports, following the existing ThinkingBudget clamp logic), with an optional "Reset to default" at the top
  • Selection applies immediately: writes task.runtimeThinkingEffort + rebuilds the handler + posts to the webview (chip updates), and the chat emits a display line (source = you)
  • Persistence semantics: task-layer — the selected value is stored with the task and restored when the task is reopened from history; the Settings persisted value is never written to
  • Visibility: hidden (renders nothing) unless the selected model advertises per-request effort support (registry capability, or an F7 declaration) — no separate disabled state in v1

(d) Orchestrator (new_task) pre-enter effort

  • Add an optional thinking_effort to the new_task tool schema: the model can specify a starting effort for a subtask (e.g. "this subtask is mechanical, use low")
  • Add an effort selector to the ChatRow newTask ask block (currently has mode + message + todos + approve/enter buttons):
┌─ New task ─────────────────────────────────────┐
│ Mode: Code                                      │
│ Message: Implement the retry logic…             │
│ Thinking: [ High ▾ ]   ← prefill = model-specified value
│                                    [ Enter ]    │
└─────────────────────────────────────────────────┘
  • The user can change it before clicking Enter; after entering, this value becomes the child task's initial effective effort (the child's TaskHeader chip init state shows it directly)
  • Prefill when the model specified nothing = the parent's current effective effort

(e) Chat stream display (same as switch_mode)

🧠 Zoo raised thinking to High — "multi-file refactor, 3 modules affected"
  • Partial streaming follows the switchModeTool.handlePartial pattern (task.ask("tool", ...))
  • Manual user changes use the same say type with a different source (slight icon/text adjustment), for task-history auditability

3.3 Wireframe overview

┌─ VS Code Panel ──────────────────────────────────────────┐
│ ┌ TASK HEADER ────────────────────────────────────────┐  │
│ │ Task title…            $0.05 · tokens …   🧠 H·Zoo │  │ ← (b)
│ └────────────────────────────────────────────────────┘  │
│                                                          │
│  👤 refactor the auth module                             │
│                                                          │
│  🧠 Zoo set thinking to Medium — "scoped, low risk"     │ ← (e)
│                                                          │
│  🤖 …working…                                            │
│                                                          │
│  🧠 New task (Code) · Thinking: [High ▾]     [Enter]    │ ← (d)
│                                                          │
│ ┌ COMPOSER ──────────────────────────────────────────┐   │
│ │ [Code ▾] [profile ▾] [🧠 High ▾] [⚡▾]       [➤]  │   │ ← (c)
│ └────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────┘
Settings → Experimental → ☑ Dynamic thinking effort        ← (a)

3.4 Design decision summary (mapped to research findings)

Decision Rationale
Controls in three layers: Settings (persistent) + composer bottom bar (temp) + dashboard (display) Follows the 2026 industry convergence direction: VS Code/Cursor use a picker (≈ our bottom bar), Cline/Roo use settings; the three layers do not conflict — settings = persistent default, bottom bar = temp for this chat, tool = temp by the AI
"Auto" = the experimental feature itself (not an option inside the selector) This repo's effort already has a per-model default; "the model decides" only becomes complete with the tool + adaptive pass-through, so an experimental toggle is the clean boundary that does not fight the existing settings semantics
No approval, always notify §2
Source badges (default/Zoo/you) transparency → trust; industry patterns (VS Code hover shows the model, Cursor picker labels) all do state visibility
Only list the levels the model supports follows the supportsReasoningEffort capability array (OpenAI gpt-5.2 default none, o1 only low/med/high, etc.), avoiding sending a level that 400s
Anthropic uses native adaptive + soft envelope; the rest use a per-request parameter §5 provider matrix; UI copy is truthful (no overselling)

4. Fix part (Anthropic adaptive models)

Code evidence (SDK 0.109.1 already supports every required type — no dependency upgrade needed):

  1. Thinking content is never received: src/api/providers/anthropic.ts never sends the display parameter; Opus 4.7 / Fable 5 / Mythos class default to display: "omitted" → the stream handler has a thinking_delta case but receives nothing. Fix: the adaptive branch of getAnthropicProviderReasoning (src/api/transform/reasoning.ts) emits { type: "adaptive", display: "summarized" }
  2. No thinking-token data: usage.thinking_tokens is not parsed. Fix: add reasoningTokens to the message_start / message_delta usage chunks (0 ⇒ the model skipped thinking that turn — exactly the "how much did the model actually think" telemetry)
  3. No effort envelope: output_config.effort (low/medium/high/xhigh/max, soft guidance) is a separate top-level field (putting it inside thinking raises a ValidationException). Fix: the transform returns { thinking, outputConfig? }, and both requestParams branches in the handler merge output_config
  4. The Bedrock handler (Converse API) gets the same fix (additionalModelRequestFields)

These three are standalone fixes on their own (Opus 4.7 users seeing no thinking at all is already a bug) and can ship ahead of the feature.

5. Provider support matrix

Provider Native "model decides" Tool-driven (F3) Notes
Anthropic 4.7+/Fable 5/Opus 5 ✅ adaptive ✅ adaptive + output_config.effort envelope In adaptive mode the API has no per-turn effort parameter; interleaved thinking happens automatically between tool calls; Opus 4.7 must be explicitly sent adaptive, otherwise thinking is off
Gemini 3.x / 2.5 ✅ dynamic (default) thinkingLevel per-request 3.1 Pro cannot be fully disabled; thoughtSignature already handled
OpenAI native / OpenRouter / OpenAI-compatible ⭕ none (omit = model default) reasoning_effort per-request Clamped to the capability array; gpt-5.2 default none
DeepSeek V4 ⭕ (thinking on + default high) ✅ (coarse-grained: medium/xhigh silently map to high) reasoning_content round-trip already handled (openai-format.ts)

6. Implementation plan (file-level)

Phase 0 — Fix (independently mergeable, ~0.5d)

  • src/api/transform/reasoning.ts: new getAnthropicProviderReasoning return shape + ADAPTIVE_THINKING_EFFORT_LEVELS
  • src/api/providers/anthropic.ts / bedrock.ts / anthropic-vertex.ts: output_config merge + thinking_tokens parse
  • Unit tests: src/api/transform/__tests__/reasoning.spec.ts (adaptive + display + outputConfig)

Phase 1 — Experimental toggle + types (~0.5d)

  • packages/types/src/experiment.ts: dynamicThinkingEffort
  • src/shared/experiments.ts: DYNAMIC_THINKING_EFFORT config
  • webview-ui/src/components/settings/ExperimentalSettings.tsx + i18n (en + zh-TW, other locales follow)

Phase 2 — Tool + Task state (~2d)

  • packages/types/src/tool.ts: set_thinking_effort ToolName
  • src/shared/tools.ts: ALWAYS_AVAILABLE_TOOLS + NativeToolArgs + toolParamNames (thinking_effort)
  • src/core/prompts/tools/native-tools/set_thinking_effort.ts: schema (effort enum, reason required) + guidance copy ("pick the lowest safe effort; escalate only for ambiguity/debugging/risky changes; state your reason")
  • src/core/prompts/tools/native-tools/index.ts: registration
  • src/core/prompts/tools/filter-tools-for-mode.ts: gating = experiment on AND model supports per-request effort (per the generate_image / run_slash_command precedent; prompt-cache rule: once a task starts, the tool list is stable and never grows/shrinks with state)
  • src/core/tools/SetThinkingEffortTool.ts: executor (no approval); clamp against the capability; escalation cap + oscillation detection; write task state; say display
  • src/core/assistant-message/presentAssistantMessage.ts + NativeToolCallParser.ts: dispatch case (following switch_mode)
  • src/core/task/Task.ts: runtimeThinkingEffort + setRuntimeThinkingEffort(effort, source) (merge into the apiConfiguration copy + updateApiConfiguration() rebuilds the handler — existing profile-switch precedent; the value is written into the task's history entry, so it survives closing and reopening) + post to the webview

Phase 3 — Webview UI (~2d)

  • webview-ui/src/components/chat/TaskHeader.tsx: chip + source badge + init state
  • webview-ui/src/components/chat/ChatTextArea.tsx: ThinkingEffortButton (bottom bar)
  • webview-ui/src/components/chat/ChatRow.tsx: case "setThinkingEffort" display (ask/say both states)
  • ExtensionState / message handler plumbing (extension→webview effort change events)
  • Vitest: ChatRow case, chip, button interactions (webview-ui AGENTS.md two-layer strategy)

Phase 4 — Orchestrator (~1d)

  • src/core/prompts/tools/native-tools/new_task.ts: + optional thinking_effort
  • src/core/tools/NewTaskTool.ts: ask JSON carries the effort; pass-through
  • webview-ui/src/components/chat/ChatRow.tsx newTask block: effort selector (changeable before Enter)
  • src/core/webview/ClineProvider.ts: delegateParentAndOpenChild accepts thinkingEffort? → set at child task init

Order

P0 → P1 → P2 → P3 → P4; P0 can proceed independently. Total estimate ~6d (single developer).

7. Risks

Risk Mitigation
Cost spiral (model over-escalates / oscillates) §2 guardrails: cap + detection + always notify + user can lower it instantly
Prompt-cache bust (conditional tool injection) The tool is only exposed when experiment on + model supports it, and is stable within a task; the description is static
Anthropic granularity is coarse (adaptive is on/off + soft envelope only) UI copy is truthful; for it, a tool change is the envelope, not a hard level
Old o1-generation thinking + tool-call limitations Gated by the model capability flags; the tool is not exposed when unsupported
Users assume "temp" persists Copy is explicit: the value is stored with the task and restored when the task is reopened; the Settings value never changes

8. Test plan (per AGENTS.md test pyramid)

9. Acceptance criteria

  • Experiment on + supported model: the AI can call set_thinking_effort mid-task, the chat emits a display line, and the next request actually carries the new effort (verifiable via network/logs)
  • Top dashboard chip: shows the default as soon as the task starts; updates immediately after a model or user change + correct source
  • Bottom button: affects only the current task; the selected value is stored with the task and restored when the task is reopened from history; the Settings value is never written to
  • Opus 4.7 / Fable class: thinking text visible (display fix) + thinking-token data present (usage fix)
  • Orchestrator: newTask can carry an effort; changeable before Enter; effective at child init
  • Experiment off: no set_thinking_effort tool exposed (the model cannot switch mid-task); the composer toggle / chip are unaffected (capability-gated)
  • No approval gate; every change is notified; the escalation cap holds

Execution status (synced 2026-08-26)

Synced 2026-08-26: trial composite Zoo-Code-Org#1379 (whole series + e2e addenda + F7) pushed at 27a2e97df — all CI green (Code QA 8 jobs incl. Build test VSIX, E2E Tests (Mocked), Release Validation), CodeRabbit clean (all findings fixed, threads resolved), patch coverage 97.31% (check codecov/patchSuccessful, target 80% — pass) / webview patch 97.67% (check codecov/patch/webview-patch, target 70% — pass). Docs: Zoo-Code-Docs#48 open, Docusaurus Build Check green. Remaining (maintainer-only): merge Zoo-Code-Org#1379 + #48.

Base reference: upstream/main = 87077e1b1 (moved 2026-08-23: 11 commits since db52d7f incl. v3.80.0 release prep Zoo-Code-Org#1347, Zoo-Code-Org#1351/Zoo-Code-Org#1340 async fixes, Zoo-Code-Org#1323 task persistence; merge-tree preview dte-3+newmain=4be3fe72, dte-5+newmain=b534d097 = clean). ⚠️ Zoo-Code-Org#1345 (ViX3L feat(ollama): reasoning effort selectors, OPEN non-draft, 15pass/2fail codecov/patch+e2e-mock, REVIEW_REQUIRED, unmerged) adds webview ReasoningEffortSelector into ChatTextArea bottom bar (PR-4's specified toggle slot) + utils/reasoning-effort.ts + SettingsView refactors → PR-4 must base-refresh post-Zoo-Code-Org#1345 and place the toggle adjacent; PR-3/PR-5 zero overlap. (prior: db52d7f = Gemini Flash-Lite Zoo-Code-Org#1334 merged 2026-08-22.) PR-1/PR-2 base = 1ad8f528d (1 commit behind, gemini-only, non-conflicting → no rebase; re-triggering CI/CodeRabbit not worth it). Wave-2 (PR-3/PR-5) will base-refresh via git merge upstream/main at launch so they are truly on the latest main (gemini commit cancels out of the main..head diff as a common ancestor).

F1 fix — delivered

Feature rollout — 5 stacked PRs (one issue each, upstream):

PR Issue Branch Base State
1/5 experimental setting Zoo-Code-Org#1328PR Zoo-Code-Org/Zoo-Code#1336 (a05830c + 1cf4f0d + base-merge 5db5cf4) feat/dte-1-experiment main DONE (final, on latest main): merge 5db5cf4 = 1ad8f52/Zoo-Code-Org#1069 (clean ort, locale coexistence re-verified); new head 15/15 CI green; CodeRabbit real review new range = 'No actionable comments 🎉'; Codecov all-coverable-lines; final comment updated in place (base-update section); awaiting maintainer merge (BLOCKED = review policy)
2/5 task state + metadata + envelope Zoo-Code-Org#1329PR Zoo-Code-Org/Zoo-Code#1338 (6ea45b3 + base-merge 9275aa1 + fix 14d1f35 + docstring 90b47b0, UNDRAFTED) feat/dte-2-task-state main 🔄 MAJOR RESOLVED (noAct 🎉) + Description PASS + 100% patch cov (30/30+10/10). Premature FINAL 09:11 REJECTED (Docstring 33%<80%) → JSDoc instruction → docstring commit 90b47b0 pushed 09:38 (all 3 modified fns documented). CodeRabbit settled 01:50:35Z re-scanned 90b47b0 but docstring still 33.33% = STALE CACHE (verified: all 7 diff fns documented in code = 100%; bot noAct 🎉 + check pass = non-blocking). 09:54 override: stop docstring iteration, finalize. **10:22 agent stuck on final PATCH (~6.5min zero I/O) → interrupt + finish instruction; ALL substantive gates VERIFIED (14/14 + noAct 🎉 + 100% cov) → final comment PATCH landed 03:02:37Z (5345 chars; live-verified: 90b47b0 + 14/14 + stale-cache note + corrected title/body line + signature). FINAL report received 11:0x (all gates self-verified, hard-rule compliant). ✅ COMPLETE (awaiting maintainer merge). Docstring-stale root cause (agent closing report 11:0x): CodeRabbit's incremental review skipped both JSDoc files as "similar to previous changes (2)" so the coverage check never re-measured; the /meow full re-review (posted 02:18:55Z) never produced a fresh walkthrough (~9h later still the 01:50:35Z run) — stale heuristic proven, non-blocking (noAct + check pass).
3/5 set_thinking_effort tool Zoo-Code-Org#1330 feat/dte-3-native-tool PR1+PR2 🚀 LAUNCHED 10:22 (wave 2): wt-dte-3; fcc3cf4 feat commit (33 files +1440; SIZE approved by user — do not grow) + 0ab4a60 base-refresh to main 78c712a; DRAFT PR Zoo-Code-Org/Zoo-Code#1354 OPEN (review fixes 19954d3 landed: 24 files +266/−75 — parser strictness, 'disable'-only capability hidden, baseline-seeded oscillation guard, i18n values 17 locales, 2 out-of-scope replies → PR-1 Zoo-Code-Org#1336; parent-verified 19/19 checks green) → bot tail verified by parent 22:3x: all 6 findings confirmed/withdrawn (11:49-11:50Z) = CLEANuser directive 22:3x: e2e moves to SEPARATE addendum PR-6 (feat/dte-3-e2e from 19954d3) — undraft NOW (gates already met: 19/19 + bot clean + 100% cov) → agent e06e4e01 executing: undraft Zoo-Code-Org#1354 → addendum PR-6 DRAFT → CI → undraft → bot → final
4/5 webview UI Zoo-Code-Org#1331 feat/dte-4-webview-ui PR3 🚀 LAUNCHED (user directive 16:1x): wt-dte-4 @ 0ab4a60 (stack dte-3 head); launch re-verify done: Zoo-Code-Org#1345 still OPEN non-draft (base 78c712a, 17pass/2fail, MERGEABLE) → pending-overlap rule (note in PR body; toggle lands adjacent to their selector post-merge) + per-tab Zoo-Code-Org#977/Zoo-Code-Org#981 NOT merged → extension-state push path + follow-up note; pnpm install done; agent: implement (TaskHeader chip + composer toggle + display + message types) → JSDOM + Playwright CT + e2e Part C round-trip → DRAFT → CI → undraft → bot → 100% cov
5/5 orchestrator new_task Zoo-Code-Org#1332 feat/dte-5-orchestrator PR2 🚀 LAUNCHED 10:22 (wave 2): wt-dte-5; DRAFT PR Zoo-Code-Org/Zoo-Code#1355 OPEN @ f6410bb (feat 146c5c8 = 15 files +1041/−16, SIZE approved by user — do not grow; base-refresh to main 78c712a clean; all post-merge verification green; also fixed ChatView approval pass-through gap found in testing) → e2e files written uncommitted (16KB test + subtasks DTE markers) → user directive 22:3x: e2e moves to SEPARATE addendum PR-7 (feat/dte-5-e2e from f6410bb)parent verified 3 UNRESOLVED MAJOR CodeRabbit comments (NewTaskTool boolean capability / ClineProvider post-mode-switch child revalidation / ChatView prefill normalization — full fix specs extracted + bot's proposed ChatView diff) → agent 4c597817 executing: STEP 1 e2e → addendum branch (DRAFT PR-7) → STEP 2 fix 3 Majors (1 fix commit) → STEP 3-5 push → fresh CI green → reply to bot → THEN undraft → cov → final

Per-PR protocol: draft PR until CI fully green → undraft → CodeRabbit review fully addressed → 100% test coverage on patch lines → only then next PR. Each PR <= 1000 diff lines (standalone, vs stacked base). Max 2 implementation agents concurrent; waves [1∥2] → [3∥5] → [4].

Design updates since original post:

  • Composer button placement: between the API-profile selector and the auto-approve (⚡) control (user-visible row order, plan §3.2(c)/§3.3 wireframe).
  • §3.5 reuse design: effort body fields stay in the transform layer (RequestConfigBuilder = SDK options only); per-request override rides metadata.reasoningEffort (same pattern as abort signal / Bedrock metadata.thinking); webview effort display state goes in the per-tab view state container (sequencing dependency on the view-local-state merge noted).
  • §8/§9 updated for the webview-ui two-layer test strategy (Vitest+JSDOM behavioral + Playwright CT visual snapshots in the same PR) and Codecov gates (80% src patch / 70% webview patch; requirement: 100% on patch lines).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions