ranxianglei · ranxianglei · Sep 8, 2026 · Sep 8, 2026
diff --git a/devlog/2026-09-08_reasoning-in-context-estimate/REQ.md b/devlog/2026-09-08_reasoning-in-context-estimate/REQ.md
@@ -0,0 +1,66 @@
+# REQ - Count reasoning tokens in context-usage display estimates
+
+- Task ID: `2026-09-08_reasoning-in-context-estimate`
+- Home Repo: `opencode-acp`
+- Created: 2026-09-08
+- Status: InProgress
+- Priority: P1
+- Owner: ework-daemon
+- References: https://github.com/ranxianglei/opencode-acp/issues/371 (source: #368 secondary finding A), related PR #370
+
+## 1. Background & Problem Statement
+
+- **Context**: ACP displays context-usage percentages in two places derived from message *content*: the nudge breakdown (`Breakdown: ...` line appended to nudges) and the `acp_status` overview (`CONTEXT BREAKDOWN` line). The decision path (when to nudge) uses API-reported tokens, which include reasoning.
+- **Current behavior (symptom)**: Both content-based estimators count only `text` + `tool` parts and skip `reasoning` parts entirely:
+  - `estimateContextComposition` (`lib/messages/inject/utils.ts:586`, per-part loop :623-665) — powers the nudge breakdown.
+  - `collectVisibleMessages` + `renderOverview` (`lib/compress/status.ts:125/:185`) — powers the `acp_status` overview. (Note: `acp_status` does NOT call `estimateContextComposition`; the import at `status.ts:9` is dead.)
+  - Real usage formula includes reasoning: `lib/token-utils.ts:44` (`input + cacheRead + cacheWrite + output + reasoning`).
+  - Result: displayed percentages systematically undercount real usage, and the largest residual component (reasoning — see #368) is invisible in the display.
+  - Not affected: `/acp context` command (`lib/commands/context.ts:119`) — its TOTAL comes from API-reported tokens and already includes reasoning.
+- **Expected behavior**: Both display estimators count `reasoning` parts (same `len/4` heuristic as text parts) and show reasoning as its own breakdown category, so displayed totals align with the real-usage formula.
+- **Impact**: Users cannot perceive the reasoning floor; displayed % disagrees with billing/limit accounting.
+
+## 2. Reproduction (if applicable)
+
+- **Environment**: any session with a thinking/reasoning model (assistant messages carry `reasoning` parts).
+- **Minimal reproduction steps**:
+  1) Run a session with a reasoning model until context usage crosses the nudge threshold.
+  2) Compare the `Breakdown:` line total (and `/acp status` overview total) against the API-reported usage — the display total is lower by roughly the reasoning token count.
+- **Relevant configuration**: none (display-only paths).
+
+## 3. Constraints & Non-Goals
+
+- **Constraints**:
+  - Backward compatibility: display-only change; no persisted-state, internal-tag, config, or API changes. Breakdown lines gain a category (tests assert only `includes("Breakdown:")` / `includes("CONTEXT BREAKDOWN")` — verified safe).
+  - Performance: one extra branch in two existing per-part loops; negligible.
+  - Keep the `len/4` heuristic consistent with existing parts (no tokenizer introduction).
+- **Non-Goals** (explicitly out of scope):
+  - Stripping reasoning at request time (that is PR #370).
+  - Changing `buildCompressibleRanges` range-token semantics (ranges = compressible amounts; reasoning on protected messages is the incompressible floor).
+  - Changing the decision path (`getCurrentTokenUsage` already includes reasoning).
+  - Making `acp_status` aware of PR #370's request-time stripping (known interaction, noted in PR description).
+  - `countAllMessageTokens` fallback (first-turn only; no reasoning present at that point).
+
+## 4. Acceptance Criteria (must be testable)
+
+- **Correctness**:
+  - [ ] `estimateContextComposition` returns a `reasoningTokens` field; `total` = system + tool + summary + message + reasoning.
+  - [ ] Reasoning on protected messages is included in `protectedTokens`.
+  - [ ] Nudge breakdown line shows a `reasoning (Q%)` category.
+  - [ ] `acp_status` overview `CONTEXT BREAKDOWN` line shows a `reasoning (Q%)` category and includes it in the total.
+  - [ ] Per-message drilldown token counts include reasoning (consistent with overview total).
+- **Performance / Stability**:
+  - [ ] No change to nudge decision behavior (decision path untouched).
+- **Regression**:
+  - [ ] New/modified test cases added to test suite and passing (unit tests for both estimators + overview rendering; full suite green).
+
+## 5. Proposed Approach (optional)
+
+- **Affected modules & entry files**:
+  - `lib/messages/inject/utils.ts` — `ContextComposition` interface + `estimateContextComposition` per-part loop + total.
+  - `lib/messages/inject/inject.ts` — nudge breakdown line.
+  - `lib/compress/status.ts` — `VisibleMessageInfo` (+`reasoning` field), `collectVisibleMessages`, `renderOverview` (total + breakdown line), drilldown totals.
+  - `lib/prompts/system.ts` — CONTEXT BREAKDOWN example line + category bullets.
+  - `tests/inject-utils-pure.test.ts`, `tests/acp-status.test.ts` (+ possibly `tests/protection-aware-stats.test.ts`).
+- **Risks**: low — additive field + display line; verified no exact-format test assertions.
+- **Rollback strategy**: revert the single commit.
diff --git a/devlog/2026-09-08_reasoning-in-context-estimate/WORKLOG.md b/devlog/2026-09-08_reasoning-in-context-estimate/WORKLOG.md
@@ -0,0 +1,60 @@
+# WORKLOG - Count reasoning tokens in context-usage display estimates
+
+- Task ID: `2026-09-08_reasoning-in-context-estimate`
+- Branch: `2026-09-08_reasoning-in-context-estimate`
+- Base: `origin/master` @ `9b7adfd` (v1.14.27)
+
+## Changes
+
+### `lib/messages/inject/utils.ts`
+- `ContextComposition`: added `reasoningTokens: number`.
+- `estimateContextComposition` per-part loop: new `part.type === "reasoning"` branch — `Math.round(text.length / 4)` (same heuristic as text parts), added to `msgTotal` and `reasoningTokens` (NOT to `messageTokens`, so text/code classification stays clean).
+- `total` now = `systemTokens + toolTokens + summaryTokens + messageTokens + reasoningTokens`.
+- Consequences (intended): reasoning on protected messages flows into `protectedTokens`; reasoning-only / reasoning-heavy messages appear in `largestRanges` with full footprint.
+
+### `lib/messages/inject/inject.ts`
+- Nudge breakdown line (:604): added `| N reasoning (Q%)` category (always shown, consistent with other zero-capable categories).
+
+### `lib/compress/status.ts`
+- `VisibleMessageInfo`: added `reasoning: number` (tracked separately from `tokens` = text+tool to avoid double counting in the overview total).
+- `collectVisibleMessages`: counts reasoning parts per message; inclusion gate widened to `tokens > 0 || reasoning > 0` (reasoning-only messages now appear in the visible listing).
+- `renderOverview`: `totalReasoning` aggregate; `total` includes it; `CONTEXT BREAKDOWN` line gains `| N reasoning (Q%)`.
+- `renderUncompressedDrilldown`: `sizeOf(m) = m.tokens + m.reasoning` used for size/tool sorting, header totals, and per-message line tokens (full footprint).
+
+### `lib/prompts/system.ts`
+- CONTEXT BREAKDOWN example line + category bullets: added reasoning.
+
+### Tests
+- `tests/inject-utils-pure.test.ts`: +4 — reasoning counted in `reasoningTokens`/`total`; total formula includes reasoning; mixed message (msgTotal vs messageTokens + largestRanges footprint); no-reasoning regression guard.
+- `tests/protection-aware-stats.test.ts`: +1 — reasoning on a protected message counted in `protectedTokens` (exact `reasoningTokens === 200`).
+- `tests/acp-status.test.ts`: +3 — overview reasoning category (100 text/33% + 200 reasoning/67% of 300); reasoning-only message visible (overview 100% + drilldown line); drilldown per-message footprint includes reasoning.
+- `tests/inject.test.ts`: +1 — rendered nudge breakdown line shows `2.0K reasoning (Q%)` (added per test review).
+- `tests/acp-status.test.ts` (pre-existing fixes): 2 vacuous tests (partial mocks dropped by `filterMessages`) given complete mocks + real listing-line assertions (per both reviews).
+
+## Dual-agent review (AGENTS.md §5.3 + §5.6)
+
+Both reviewers: **APPROVE**. Findings addressed in this branch:
+
+- **Test reviewer finding (minor)**: rendered nudge breakdown line was untested → added `tests/inject.test.ts` "E2E: nudge breakdown line shows reasoning category with token count (#371)" (asserts `2.0K reasoning (\d+%)` in the injected nudge).
+- **Both reviewers (minor)**: pre-existing vacuous tests `tests/acp-status.test.ts:286-305` / `:307-328` used partial mocks (missing `info.sessionID`/`info.time.created`) that `filterMessages` drops → tests passed with ZERO visible messages. Fixed: complete mocks + assertions on the actual `m00001 (...) text|bash` listing lines.
+- **Test reviewer (nit)**: tightened `comp.reasoningTokens >= 200` → `assert.equal(comp.reasoningTokens, 200)`.
+- **Code reviewer (nit)**: system prompt example percentages now sum to 100% (were 121% pre-existing, 131% after first edit).
+
+Not addressed (documented, non-blocking):
+- Composition-vs-range divergence (code reviewer minor): `buildCompressibleRanges` range tokens still exclude reasoning (intentional — ranges = compressible amounts; the pipeline's min-size check `countMessageCharacters` also excludes reasoning, so adding it there risks phantom "Range too small" rejections per #37). Consequence: "Effective compressible: ~X" (nudge) and overview totals now include reasoning while per-range lines don't. Documented in PR description; candidate follow-up issue (source-tagged).
+- Reasoning-only messages render with `text` label in the drilldown (`toolName || "text"`); `classifyMessageType` would say `reasoning` — label-semantics change, out of scope.
+- Per-message `dcp-message-id` token annotation (`countMessageCharacters`, token-utils.ts) still excludes reasoning — pre-existing, out of scope, candidate follow-up.
+
+## Verification
+
+- `npm run typecheck` — clean.
+- `npm run test` — **1086/1086 pass** (was 1077 on master; +9 new).
+- `npm run build` — clean.
+- `npm run format:check` — repo-wide pre-existing Prettier drift (423 files fail on clean master, incl. all 7 touched files); CI does not run format checks; no reformat to keep the diff minimal.
+- Test-input fidelity note: `filterMessages` (`lib/messages/shape.ts:14-24`) drops messages lacking `info.sessionID`/`info.time.created` — all acp_status tests (new + 2 pre-existing fixed per review) use complete mocks.
+
+## Open items / known interactions
+
+- PR #370 (`stripProtectedReasoning`, open): its pass runs BEFORE `injectCompressNudges` in `lib/hooks.ts`, so post-merge the nudge-path estimator naturally matches sent content. `acp_status` reads raw DB messages and will still show request-time-stripped reasoning — #370-side concern, noted in PR description only.
+- `buildCompressibleRanges` range tokens still exclude reasoning (ranges = compressible amounts; protected-msg reasoning is the incompressible floor per #368) — intentional non-goal.
+- `countAllMessageTokens` fallback (token-utils.ts) still excludes reasoning — first-turn only, no reasoning present at that point — intentional non-goal.
diff --git a/lib/compress/status.ts b/lib/compress/status.ts
@@ -115,6 +115,9 @@ interface VisibleMessageInfo {
     tokens: number
     tool: string
     index: number
+    // Reasoning (thinking) part tokens, tracked separately from `tokens`
+    // (text+tool) so the overview can show it as its own category (#371).
+    reasoning: number
 }
 
 export interface StatusRenderContext {
@@ -153,6 +156,7 @@ function collectVisibleMessages(
         if (!ref) return
 
         let tokens = 0
+        let reasoning = 0
         let toolName = ""
 
         for (const part of msg.parts || []) {
@@ -164,11 +168,13 @@ function collectVisibleMessages(
                 if (!toolName) {
                     toolName = (part as any)?.tool || "unknown"
                 }
+            } else if (part.type === "reasoning" && typeof (part as any).text === "string") {
+                reasoning += Math.round(((part as any).text as string).length / 4)
             }
         }
 
-        if (tokens > 0) {
-            result.push({ ref, tokens, tool: toolName || "text", index: idx })
+        if (tokens > 0 || reasoning > 0) {
+            result.push({ ref, tokens, tool: toolName || "text", index: idx, reasoning })
         }
     })
 
@@ -209,16 +215,18 @@ function renderOverview(
         const totalText = visibleMessages
             .filter((m) => m.tool === "text")
             .reduce((s, m) => s + m.tokens, 0)
-        const total = systemTokens + totalTool + totalText + summaryTokens
+        const totalReasoning = visibleMessages.reduce((s, m) => s + m.reasoning, 0)
+        const total = systemTokens + totalTool + totalText + summaryTokens + totalReasoning
 
         const sysPct = pct(systemTokens, total)
         const toolPct = pct(totalTool, total)
         const textPct = pct(totalText, total)
         const summaryPct = pct(summaryTokens, total)
+        const reasoningPct = pct(totalReasoning, total)
 
         lines.push("CONTEXT BREAKDOWN")
         lines.push(
-            `  ${formatTokens(systemTokens)} system (${sysPct}%) | ${formatTokens(totalTool)} tool (${toolPct}%) | ${formatTokens(totalText)} text (${textPct}%) | ${formatTokens(summaryTokens)} summaries (${summaryPct}%)`,
+            `  ${formatTokens(systemTokens)} system (${sysPct}%) | ${formatTokens(totalTool)} tool (${toolPct}%) | ${formatTokens(totalText)} text (${textPct}%) | ${formatTokens(summaryTokens)} summaries (${summaryPct}%) | ${formatTokens(totalReasoning)} reasoning (${reasoningPct}%)`,
         )
 
         const topTypes = Array.from(toolTypeMap.entries())
@@ -357,16 +365,18 @@ function renderUncompressedDrilldown(
         filtered = filtered.filter((m) => m.tool === toolFilter)
     }
 
+    const sizeOf = (m: VisibleMessageInfo) => m.tokens + m.reasoning
+
     if (sort === "time") {
         filtered.sort((a, b) => a.index - b.index)
     } else if (sort === "tool") {
-        filtered.sort((a, b) => a.tool.localeCompare(b.tool) || b.tokens - a.tokens)
+        filtered.sort((a, b) => a.tool.localeCompare(b.tool) || sizeOf(b) - sizeOf(a))
     } else {
-        filtered.sort((a, b) => b.tokens - a.tokens)
+        filtered.sort((a, b) => sizeOf(b) - sizeOf(a))
     }
 
-    const totalTokens = filtered.reduce((s, m) => s + m.tokens, 0)
-    const allTokens = visibleMessages.reduce((s, m) => s + m.tokens, 0)
+    const totalTokens = filtered.reduce((s, m) => s + sizeOf(m), 0)
+    const allTokens = visibleMessages.reduce((s, m) => s + sizeOf(m), 0)
 
     const header = toolFilter
         ? `UNCOMPRESSED — ${toolFilter}: ${formatTokens(totalTokens)} | ${filtered.length} msgs | ${pct(totalTokens, allTokens)}% of visible`
@@ -378,7 +388,7 @@ function renderUncompressedDrilldown(
 
     const shown = filtered.slice(0, limit)
     for (const m of shown) {
-        lines.push(`  ${m.ref} (${formatTokens(m.tokens)}) ${m.tool}`)
+        lines.push(`  ${m.ref} (${formatTokens(sizeOf(m))}) ${m.tool}`)
     }
 
     if (filtered.length > shown.length) {

diff --git a/lib/messages/inject/inject.ts b/lib/messages/inject/inject.ts
@@ -601,7 +601,7 @@ export const injectCompressNudges = (
             const sysPart = composition.systemTokens > 0
                 ? `${fmt(composition.systemTokens)} system (${pct(composition.systemTokens)}%) | `
                 : ""
-            let breakdown = `${efficiencyNote}\nBreakdown: ${sysPart}${fmt(composition.toolTokens)} tool (${pct(composition.toolTokens)}%) | ${fmt(composition.summaryTokens)} summaries (${pct(composition.summaryTokens)}%) | ${fmt(composition.codeTokens)} code (${pct(composition.codeTokens)}%) | ${fmt(plainTextTokens)} text (${pct(plainTextTokens)}%)${growthStr}`
+            let breakdown = `${efficiencyNote}\nBreakdown: ${sysPart}${fmt(composition.toolTokens)} tool (${pct(composition.toolTokens)}%) | ${fmt(composition.summaryTokens)} summaries (${pct(composition.summaryTokens)}%) | ${fmt(composition.codeTokens)} code (${pct(composition.codeTokens)}%) | ${fmt(plainTextTokens)} text (${pct(plainTextTokens)}%) | ${fmt(composition.reasoningTokens)} reasoning (${pct(composition.reasoningTokens)}%)${growthStr}`
 
             const compressibleTokens =
                 composition.total -

diff --git a/lib/messages/inject/utils.ts b/lib/messages/inject/utils.ts
@@ -561,6 +561,7 @@ export interface ContextComposition {
     textTokens: number
     systemTokens: number
     protectedTokens: number
+    reasoningTokens: number
     total: number
     largestRanges: { ref: string; tokens: number }[]
     largestToolRanges: { ref: string; tokens: number; tool?: string }[]
@@ -594,6 +595,7 @@ export function estimateContextComposition(
     let summaryTokens = 0
     let messageTokens = 0
     let protectedTokens = 0
+    let reasoningTokens = 0
     const perMessage: { ref: string; tokens: number }[] = []
     const perTool: { ref: string; tokens: number; tool?: string }[] = []
     const perCode: { ref: string; tokens: number }[] = []
@@ -662,6 +664,12 @@ export function estimateContextComposition(
                 summaryTokens += summaryPartTokens
                 toolTypeMap.set(toolName, (toolTypeMap.get(toolName) || 0) + toolPartTokens)
                 if (!msgToolName) msgToolName = toolName
+            } else if (part.type === "reasoning" && typeof (part as any).text === "string") {
+                // Real usage includes reasoning (token-utils.ts) — track as its own
+                // category so display totals align with the usage formula (#371).
+                const tokens = Math.round(((part as any).text as string).length / 4)
+                msgTotal += tokens
+                reasoningTokens += tokens
             }
         }
 
@@ -700,7 +708,8 @@ export function estimateContextComposition(
         textTokens: Math.max(0, messageTokens - codeTokens),
         systemTokens,
         protectedTokens,
-        total: systemTokens + toolTokens + summaryTokens + messageTokens,
+        reasoningTokens,
+        total: systemTokens + toolTokens + summaryTokens + messageTokens + reasoningTokens,
         largestRanges: perMessage.slice(0, 15),
         largestToolRanges: perTool.slice(0, 15),
         largestCodeRanges: perCode.slice(0, 5),

diff --git a/lib/prompts/system.ts b/lib/prompts/system.ts
@@ -73,13 +73,14 @@ CONTEXT BREAKDOWN
 
 When context usage passes a threshold, the system appends a breakdown showing where your context tokens are spent:
 
-Breakdown: 5.2K system (21%) | 12.3K tool (40%) | 3.1K summaries (10%) | 8.5K code (28%) | 6.5K text (22%)
+Breakdown: 4.2K system (21%) | 8.0K tool (40%) | 2.0K summaries (10%) | 2.6K code (13%) | 2.2K text (11%) | 1.0K reasoning (5%)
 
 - "system" = system prompt tokens (AGENTS.md, tool definitions — not compressible)
 - "tool" = tool call outputs (largest category — compress first when consumed)
 - "summaries" = existing compression block summaries (already compressed; do not re-compress standalone)
 - "code" = messages containing code blocks
 - "text" = plain text messages
+- "reasoning" = model thinking blocks (counted to match real API usage; freed when their message is compressed)
 
 Below the breakdown, the system lists compressible ranges grouped by conversation turn. All listed ranges should be compressed to summary format — the only exceptions are protected content, content the current step is actively using, or critical content you cannot reconstruct. Compress the largest ranges first when the current step no longer needs them.