Repository navigation
Deferred-toolbox eviction is economically net-negative under provider prompt caching: full-prefix re-prefill + write premiums, permanent stale recovery notes, wire-byte-blind budget #6969
Description
What happened? 发生了什么？
The bounded deferred-tool toolbox (ToolActivationCache, 8 names / 16 KiB, crates/tui/src/core/session.rs:17-19 at HEAD 008d21d69) is cost-blind, and under current provider prompt-cache pricing its eviction path is economically net-negative. Concretely:
1. Eviction invalidates the entire downstream prefix — the one toolbox operation with no offsetting benefit.
The prefix fingerprint is SHA256(system) + SHA256(tools wire array) with order load-bearing (crates/tui/src/prefix_cache.rs), and tools sit at the front of the rendered prompt on every provider. Any admission/eviction mutates the tools array, so system + tools + the entire conversation history re-prefills at miss price. docs/CACHE.md calls this "an expected, one-request miss" (lines 39-40) — wording written when cache writes were free:
| Provider/API | Cached-input read | Write premium | Tool-change invalidation |
|---|---|---|---|
| OpenAI Chat/Responses | 0.1x | 1.25x (GPT-5.6+) | prefix from that point (tools are first) |
| Anthropic Messages | 0.1x | 1.25x (5m) / 2x (1h) | any tool change invalidates the entire cache (tools are block 1) |
| DeepSeek | 1/30 – 1/50 | none | prefix-based |
| Moonshot Kimi k3 | 0.1x | 1x (5m) / 2x (1h) | prefix from that point |
Break-even for evicting one ~400-token schema from a session with H tokens of history: pay ≈ 0.9–2.25 × H once, save ≈ 0.9 × 400 per later request → needs ~75 further tool-free requests at H = 30K. Real conversations run 6–40 requests. Admission at least unlocks a needed capability; eviction only pays when the session is about to end anyway. LRU position alone is the eviction trigger today — no hysteresis, no cost term, and two alternating overflowing searches would re-prefill the full prefix every round.
2. The eviction→recovery cycle multiplies the loss (relevant to the in-flight reporting work).
Reporting evictions (as the Pinvou fork's open PR #101 does) is necessary for correctness — silent eviction caused a live incident where a model fabricated tool results for tools that had vanished from its <functions>. But the recovery contract ("re-search to re-activate") completes a cycle whose full cost stack, for one eviction of 4 browser tools (~460 tokens of schema) in a 30K-token history with 20 requests remaining (0.1x reads, 1.25x writes):
| # | Cost item | Tok-eq |
|---|---|---|
| 1 | Eviction invalidation (re-prefill + write premium) | ~34,500 |
| 2 | Eviction note in history (≈72–147 tokens/event; rides the result forever; 1 + 19×0.1x) | ~427 |
| 3 | Recovery re-search round trip | ~3,050 |
| 4 | Search-result payload becomes history | ~1,160 |
| 5 | Re-activation invalidation (tools array changes again — second re-prefill + write) | ~34,500 |
| Total vs never evicting | ≈ 73,600 (~2.5× the entire history) |
Two structural gaps in the reporting design: (a) notes are never retracted — ride-time netting (render_toolbox_eviction_notes) only trims the note at the moment it attaches to a result; once in history it is permanent, and the only "compaction" that removes it is generic conversation compaction; (b) re-activation carries no supersession marker — the re-admitting search result is byte-identical to a first-time hit (tool_references list only), so after recovery the model's history permanently contains "these tools are no longer callable" next to a tools array that has them again. (Compare: the direct-call hydration path does emit an explicit "was deferred and has now been loaded" receipt.)
3. The headline saving is first-turn-only, and the budget does not measure wire bytes.
The historical justification ("reduce catalog tokens 73%") measures a first-request, native-tools-only, bytes/4-estimated ratio — no provider, no tokenizer, no MCP, caching never modeled. Under caching, both an eager stable header and the deferred header are hit-priced after request 1; the durable steady-state saving is ~0.1x-scale. Separately, the 16 KiB toolbox budget is measured in engine-side serde bytes (ToolActivationCache::serialized_bytes), while the same tool serializes differently per wire family (Chat Completions wrapper ~69 B/tool, Anthropic ~40 B, Responses ~71 B) and per-provider sanitizers (Moonshot MFJS, root-level oneOf/anyOf flattening) change schemas further — so the budget is a proxy that matches no provider's actual token accounting.
4. In practice the mechanism survives by accident.
Across five real sessions on k3/DeepSeek (93–94% hit ratios with heavy tool_search churn vs 92.5% baseline without), activation happened at request 1 (short history) and the surface then stayed stable within turns. The measured re-prefill drivers were skill loading and engine restarts — not tool_search. The one live failure (mid-task silent eviction) is exactly the case with no offsetting saving.
Expected 期望
Surface-change decisions — especially LRU eviction — should be cost-aware or at least cost-attributed, and the eviction-recovery loop should not leave permanent stale statements in history:
- Hysteresis / eviction economics: a minimum activation lifetime, or an eviction cost term once current re-prefill size exceeds a few multiples of the evicted schema. Pure LRU position makes mid-task eviction the default worst case.
- Update
docs/CACHE.md's "expected, one-request miss" for the write-premium era, and reconcile observedcache_hit_tokens/cache_miss_tokensagainst declaredchange:<what>re-pins — today telemetry and decisions never meet (calculate_cache_savingsis#[cfg(test)]-only). - Supersession semantics for recovery: when a reported-evicted tool is re-activated, emit an explicit restored/re-admitted marker (hydration-receipt style) so the history's "no longer callable" note is not left as the last word; consider bounding note persistence (e.g. drop after the next successful activation of the named tools, or fold into compaction).
- Evaluate append-in-context injection for dynamic tools (Moonshot's documented pattern: keep the top-level
toolsarray stable, append dynamic declarations as a trailing system message insidemessages) — zero prefix invalidation by construction, which sidesteps items 1 and 5 of the cost stack entirely. - Measure the toolbox budget in wire-shaped bytes (or tokenizer tokens) per active wire format rather than engine serde bytes.
Steps to reproduce 复现步骤
Economic issue rather than a crash; the observable sequence:
- Start a conversation that activates ≥8 deferred tools (e.g. a browser task:
tool_searchfor browser tools, then a second search for page-management tools). - Observe the second search silently evicting earlier activations from the request tools array (no model-visible signal at HEAD
008d21d69—delta.evictedis consumed byremove_evicted_cache_activationsinexecute_tool_search_inner,tool_catalog.rs:1546-1551). - Watch
/cache stats+ provider usage: the next request reports a miss covering the full history; on Anthropic routes the write premium applies; re-searching to recover triggers a second full miss + write. - Cross-check the arithmetic against
calculate_turn_cost_from_usage_with_prision-billed totals for the session.
Codewhale version 版本
008d21d69 (workspace 0.10.2; latest tag v0.10.1) — verified at this HEAD on 2026-10-13; the mechanism is unchanged from v0.9.12 through HEAD. Observed live via the Pinvou downstream (fork pinned at v0.9.12 + maintenance), engine-level, platform-independent.
Where did it happen? 在哪里出现？
Engine — affects all surfaces (TUI / CLI / Desktop / Web) and all providers with prefix caching, which in 2026 is all of them.
Before you start 前置确认
- I searched existing issues and this is not a duplicate. 我已搜索过现有 issue。（recent 100 issues swept for tool/cache/deferred/eviction — no overlap）
Activity
chore(codewhale): bump gitlink for unbounded deferred toolbox
asto18089 commentedon Oct 13, 2026
Follow-up with the current activation mechanics and a feasibility analysis for direction 3 (append-in-context injection), plus a data point from the Pinvou fork.
How tools are added today
Activation mutates the request's top-level
toolsarray:active_tools_for_request, two-pass: eager head first, activated deferred appended in catalog order — the catalog-head stability invariant exists precisely to keep the array byte-stable across mode toggles).tool_search(or direct-call hydration) inserts names intoactive_tool_namesimmediately after the search executes; the next request's tools array grows by the new schemas at their catalog-order positions in the tail. The mutation is declared (pending_prefix_change_reason = "tool_surface") and the next request re-pins —docs/CACHE.md's "expected, one-request miss".Why the tail-append does not keep the cache warm
Tail-append keeps the array itself prefix-stable (earlier tools' bytes never move), but every provider renders the tools segment BEFORE the messages (Anthropic documents tools → system → messages explicitly; OpenAI/DeepSeek/Kimi equivalent), so any byte added to the tools array shifts the entire message history — the expensive part. Appending at the array's tail is therefore irrelevant to prefix cost; the only invalidation-free position for a tool schema is inside
messages.What direction 3 requires
Moving activation into a trailing system segment inside
messages(Moonshot's documented pattern) leaves the tools array byte-identical and the cache warm, but the top-leveltoolsarray is also the wire-level callability authority on strict providers: atool_callsentry naming a tool absent fromtoolsis rejected or degraded there. So the design needs, at minimum:Until that lands, every activation invalidation is functionally load-bearing (the hydration contract requires the schema to be visible in the very next request, and a search-referenced tool must be callable in the following round) — within the current wire contract, the only non-load-bearing mutations were eviction-driven, which is what motivated the fork data point below.
Fork data point
The Pinvou fork has cut the mechanism off pending this discussion:
TOOL_ACTIVATION_CACHE_MAX_NAMES/MAX_SCHEMA_BYTES→usize::MAX / 2(Pinvou#104), keeping admission bookkeeping and catalog-shrink revalidation intact. Result: evictions, LRU churn, and byte-growth drops disappear; the remaining invalidations are exactly "a needed schema becomes callable now".feat(tools): disable the deferred-toolbox size bounds (#104)
Hmbown commentedon Oct 17, 2026
🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.
Thank you @asto18089 for a careful cost analysis, with a write-premium table and a real-session cross-check. I checked each claim against current
main(f6583c3). As a dogfood run, Codewhale itself on DeepSeek Flash did the first pass, read-only. I then re-read the cited lines myself.Confirmed on
mainToolActivationCachedrops the oldest names until it is back under 8 names and 16 KiB (crates/tui/src/core/session.rs:84-92). There is no minimum lifetime and no cost term.tool_searchresult carries no eviction or re-admission signal. Its payload is onlytool_referencesandunavailable_tool_references(crates/tui/src/core/engine/tool_catalog.rs:1578-1582). By contrast, a direct call to a deferred tool emits an explicit "was deferred and has now been loaded" receipt (tool_catalog.rs:1314). So the incident you describe, a model calling tools that silently left its catalog, is structurally possible upstream.serialized_bytesisserde_json::to_vecof the full internalTool(session.rs:59-61), including fields such asdefer_loading,allowed_callersandcache_control(crates/protocol/src/request.rs:286-303). The Chat, Anthropic and Responses encoders each serialize tools differently, and no tokenizer is involved.docs/CACHE.mdis out of date here. It still describes a tool-surface change as an expected one-request miss, which does not account for write premiums.Not applicable upstream. The permanent eviction notes in history come from the fork's reporting PR.
mainhas no eviction note at all (render_toolbox_eviction_notesdoes not exist here). The retraction problem therefore does not arise, but the silence above does.Not verifiable from source. The cost table and the five-session hit ratios need billed usage, not code reading. If you can share the per-request
cache_hit_tokens/cache_miss_tokensfrom those sessions, even aggregated, they would let us size this properly.Where it stands. This is tracked internally. Two steps look cheapest and are needed for correctness whatever the economics:
tool_searchresult, as the hydration path already does.Hysteresis, and keeping the top-level
toolsarray stable with dynamic declarations appended inmessages, are design choices with cache trade-offs per provider, and the maintainers still need to decide them. Nothing has landed for this yet.— Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.
fix(tool-search): name the deferred tools a search activation evicted
jimy-r commentedon Oct 18, 2026
You can keep the eviction policy and drop most of its cost by evicting only when the prefix is about to be written again. Three events already force that. They are a gap longer than the cache TTL, a compaction and a model switch. An eviction scheduled at one of them adds little, because the history behind the tools is being written at that moment regardless. Between those events, let the activation cache run over its budget and queue the evictions. A few KB of extra tool schema read at the cached rate is cheap next to a full prefill of the history at the write premium.
On the data request, each request's usage fields carry it. On Anthropic they are cache_creation_input_tokens and cache_read_input_tokens. Log them with a marker for every change to the tools array, then attribute each write spike to the nearest preceding event. Spikes that land on a tools change are the eviction cost, measured instead of estimated. I run that attribution over my own Claude Code transcripts, and it splits the write tokens by idle gap, model switch, compaction and change to the tool list.
Drafted with Claude Code and reviewed by me before posting.
Hmbown commentedon Oct 18, 2026
🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.
Thanks @jimy-r. This is the most practical version of the hysteresis idea so far. The cache gets rewritten anyway after a gap longer than the TTL, at a compaction, and at a model switch. An eviction scheduled at one of those costs only the bytes of the tools it drops, not a re-prefill of the whole history at the write premium. Letting the activation cache run a few KB over budget in between, and queuing the evictions, trades cheap cached reads for avoided writes. That is the right direction for the cost @asto18089 measured.
Status on
main, so the thread is current:tool_searchresult now names the tools an activation evicted (evicted_tool_references, with a "search again before using it" reason). So the silent-eviction failure, a model calling a tool that left its catalog, now has a signal. My earlier reply predates this.serde_jsonover the internalTool, not in the active wire format.toolsarray stable with dynamic declarations appended inmessages, are the candidates. They trade off differently on Anthropic, where any tool change invalidates the whole cache, versus prefix-based providers. That choice is with the maintainers.Your measurement plan is the right way to settle it: log
cache_creation_input_tokens/cache_read_input_tokens(Anthropic) andcache_hit/cache_miss(DeepSeek), with a marker for every change to the tools array, and attribute each write spike to the nearest preceding event. If anyone has a few long real sessions with those fields, posting the per-request numbers here would let us size the policy instead of arguing it.— Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.