Skip to content

V2: keep the live rate during tool calls (tool-input deltas + opt-in llama-swap slot polling) - #2

Open
AfroPK wants to merge 1 commit into
jashjacob:mainfrom
AfroPK:fix/v2-live-rate-during-tool-calls
Open

AfroPK wants to merge 1 commit into
jashjacob:mainfrom
AfroPK:fix/v2-live-rate-during-tool-calls

Conversation

@AfroPK

@AfroPK AfroPK commented Nov 21, 2026

Copy link
Copy Markdown

Problem

On OpenCode V2 the live rate drops to tok/s - for most of an agent run, even while the model is generating:

  1. Tool-call arguments aren't counted. When the model writes a file or an edit, its output streams as session.tool.input.delta. wireV2 only listens to session.text.delta and session.reasoning.delta, so after idleTimeoutMs the rate falls to - during the "Preparing write…" phase.
  2. Local models (llama.cpp via llama-swap) emit nothing during tool calls. llama-server holds a tool call back until it is complete. I captured /api/event on OpenCode 2.3.2: session.tool.input.started → ended arrive together at the end, with no deltas in between, while llama-server's /slots showed n_decoded climbing at about 37 tok/s the whole time. No event-based meter can see those tokens.

Changes

  • src/v2.ts
    • Subscribe to session.tool.input.delta and record it like the other deltas.
    • New opt-in llama-swap source. For sessions whose provider ID is listed in llamaSwapProviders, poll GET {llamaSwapUrl}/upstream/<model>/slots every llamaSwapPollMs from session.step.started until the execution ends, and record the growth of the processing slot's n_decoded (exact token counts). The first reading only sets the baseline, and a new id_task (the next step of a tool loop) counts from zero. While llama-swap answers, its counter replaces event deltas so nothing is counted twice. If it fails or is unreachable, event deltas are used as before. Polling stops on execution end, on session.deleted and on dispose.
    • fetch/now can be injected (optional 4th parameter) so the poller is testable.
  • src/tui.tsx: the meter <text> gets wrapMode="none" and flexShrink={0}. A narrow prompt.footer.status used to wrap the line onto two rows (… · cache / 82%).
  • README: the new options, plus the V2 "How it works" section.

New options (V2 only, polling is off by default):

Option Default
llamaSwapUrl "http://127.0.0.1:8080"
llamaSwapProviders []
llamaSwapPollMs 250

Testing

  • npm test: 125 pass. New tests cover tool-input deltas, polling across an execution, slot-counter precedence over deltas, a new request on the slot, no polling when it isn't configured or for remote providers, and the llama-swap-down fallback plus stopping on dispose.
  • npm run typecheck: clean.
  • Manually tested on OpenCode 2.3.2 (Windows) with a llama-swap model while it was writing a file. Before: tok/s - the whole time. After: the live rate tracked llama-server's own decode speed (about 36–37 tok/s, graph and trend updating).
  • npm run smoke fails on my machine (npm 11 / Node 26) the same way on main without these changes, at the npm pack --json parsing in scripts/smoke.mjs. CI's Node 20/22 should be unaffected.

…ping

During an agent run most output is tool-call arguments (e.g. a file body for
write/edit). OpenCode V2 streams those as session.tool.input.delta, which the
meter ignored, so the live rate fell to "-" while the model was busy writing.
Count them like text and reasoning deltas.

Local models served by llama-swap are worse off: llama-server holds a tool
call back until it is complete, so OpenCode emits no events at all while one
is generated. With the new opt-in llamaSwapProviders option the meter polls
llama-server's per-slot n_decoded counter through llama-swap from step start
to execution end and records exact decoded tokens; event deltas remain the
fallback when llama-swap does not answer.

The meter text also no longer wraps onto a second row when the
prompt.footer.status slot gets narrow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant