Skip to content

fix(llm): populate usage for the Claude Code local-agent path - #140

Merged
jmagly merged 2 commits into
elder-plinius:mainfrom
N3thunt3r69:fix/local-agent-usage-tracking
Aug 1, 2026
Merged

fix(llm): populate usage for the Claude Code local-agent path#140
jmagly merged 2 commits into
elder-plinius:mainfrom
N3thunt3r69:fix/local-agent-usage-tracking

Conversation

@N3thunt3r69

Copy link
Copy Markdown
Contributor

Fixes #139.

LocalAgentAdapter.chat() never set usage on its LLMResponse for the Claude Code local-agent path, so AgentLoop's per-task token budget check always compared against zero and never fired. localAgentChat() now requests --output-format json for the claude agent id only; LocalAgentAdapter parses the real envelope into usage.promptTokens/completionTokens, with a character-based estimate as a fallback if parsing fails. Codex and Hermes are untouched.

Contribution Receipt

  • Change: Populate per-call usage for the Claude Code local-agent path (--output-format json) so AgentLoop's token budget check can fire.
  • Scope class: local_lab
  • Target authority: not_applicable (LLM backbone plumbing only; verification used a throwaway loopback fixture, not an authorized external target)
  • Network use: loopback (verification target was a local throwaway HTTP server; the Claude Code CLI itself makes outbound calls to Anthropic under the operator's own account, inherent to exercising this adapter, not target network use)
  • Run mode labels: local_agent, tool_backed
  • Model/harness labels: model=claude-sonnet-5, provider=Anthropic (via Claude Code CLI, no API key), agent_runtime=Claude Code, harness=manual-review (ad hoc verification script, not a committed bench harness), tool_access=local_only
  • Commands run:
    • npm run typecheck -> pass
    • npm test -> pass (675/675 vitest, 11/11 ops-preflight, 19/19 model-matrix)
    • npm run doctor -> pass (30/33, 3 warnings: semgrep/promptfoo missing, API health offline with the server not running — no blockers)
    • npm run verify-claims -> pass (27/27, no headline numbers changed)
  • Artifacts: none committed. Verification used an ad hoc script (deleted after use) against a throwaway loopback HTTP fixture, plus two direct claude -p --output-format json calls confirming the envelope shape.
  • Redaction: not_applicable (no secrets or evidence involved)
  • Claims changed: none
  • Abstentions/refusals: none
  • Residual risk: Codex and Hermes local-agent paths are untouched and unverified (this fix targets claude only). The character-based fallback estimate (used only if the JSON envelope fails to parse) is a rough approximation, not exact. agentFailureOutput's plain-text failure detection in local-agents.ts is not updated for JSON-wrapped error envelopes (pre-existing behavior, unchanged here).

LocalAgentAdapter.chat() never set `usage` on its LLMResponse, so
AgentLoop's token budget check (tokensUsed >= maxTokens) silently
never fired for Claude Code missions. maxIterations was the only
brake, and it bounds turn count, not actual spend.

Claude Code's --output-format json (instead of text) returns exact
per-call token counts, including prompt-cache creation/read tokens
that a text-length estimate has no way to see. localAgentChat now
requests that format for the claude agent id only. LocalAgentAdapter
parses the envelope into usage.promptTokens/completionTokens, with a
character-based estimate as a fallback if parsing fails, so usage
never goes back to undefined.

Verified directly against the CLI: a live run now reports
promptTokens 24458, completionTokens 1662 instead of undefined/0.
Full suite: 675/675, no regressions.

Fixes elder-plinius#139
@jmagly

jmagly commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Maintainer courtesy cleanup pushed as 11e3d4c.

I kept the change narrow:

  • preserve the parsed Claude result when the envelope has missing, empty, zero, malformed, or negative usage;
  • accept real usage only when token fields are finite, non-negative, and non-zero in aggregate;
  • otherwise invoke the existing character estimate instead of returning zero usage;
  • add behavioral coverage for real cache-token aggregation, every fallback shape, non-JSON output, and tool-call parsing from the envelope result.

Verification on a synthetic merge into current main (9020e5f): typecheck passed; focused suite 33/33; full suite 748/748 Vitest plus 11/11 ops preflight and 19/19 model matrix; lint 0 errors (145 existing warnings on the merged tree). I’m waiting for hosted CI on the new exact head before merge.

@jmagly jmagly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 11e3d4c884e05931e8728a427a6333888da86513.

No blocking findings after maintainer-courtesy cleanup. Real Claude JSON usage is aggregated correctly; absent or invalid usage retains the parsed result and fails over to a non-zero estimate; non-Claude paths remain unchanged. Behavioral regression coverage exercises the reported budget-accounting boundary and JSON-wrapped tool calls.

Verification: hosted CI passed all gates; synthetic merge into current main passed typecheck, focused 33/33, full 748/748 Vitest plus ops/model checks, and lint with zero errors.

@jmagly
jmagly merged commit d471da6 into elder-plinius:main Aug 1, 2026
1 check passed
@N3thunt3r69
N3thunt3r69 deleted the fix/local-agent-usage-tracking branch August 2, 2026 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claude Code local-agent responses carry no usage: AgentLoop's token budget check never fires

2 participants