Skip to content

fix: avoid double-counting context token overhead - #279

Merged
Eigenwise merged 1 commit into
Eigenwise:mainfrom
Excelius-Wang:fix/accurate-context-token-accounting
Aug 20, 2026
Merged

fix: avoid double-counting context token overhead#279
Eigenwise merged 1 commit into
Eigenwise:mainfrom
Excelius-Wang:fix/accurate-context-token-accounting

Conversation

@Excelius-Wang

Copy link
Copy Markdown
Contributor

Summary

  • Count system and history messages as a single request to avoid duplicating request-level framing tokens.
  • Measure tool schema overhead against the actual message list.
  • Preserve tools-only counting when no messages exist and cover both paths with regression tests.
  • Document the additive token attribution contract and its tools-only exception.

Why

LiteLLM token counts for separately counted message groups are not additive because each call includes request-level framing. The previous implementation added independently counted system and history values and measured tools against a synthetic empty message, so context utilization could be overestimated.

With LiteLLM 1.80.9, the regression fixture for gpt-4o-mini is counted as 72 tokens for the complete request. The previous calculation returned 79. This change returns 72 and keeps the breakdown additive.

The regression test compares count_context() against LiteLLM's complete-request count without hard-coding those values.

Validation

  • Token counter tests passed with LiteLLM 1.50.0, 1.80.9, and 1.97.0.
  • Full core test suite: 336 passed, 3 skipped.
  • Black, Flake8, and git diff --check passed.
  • Sphinx HTML build completed with no warnings beyond the existing baseline.

@Eigenwise
Eigenwise merged commit 903b532 into Eigenwise:main Aug 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants