Skip to content

fix: omit non-numeric usage fields from token usage report#1664

Open
rajkumar-prog wants to merge 3 commits into
openai:mainfrom
rajkumar-prog:main
Open

fix: omit non-numeric usage fields from token usage report#1664
rajkumar-prog wants to merge 3 commits into
openai:mainfrom
rajkumar-prog:main

Conversation

@rajkumar-prog
Copy link
Copy Markdown

@rajkumar-prog rajkumar-prog commented May 16, 2026

Problem

When running evals, the OpenAI API returns nested objects such as
CompletionTokensDetails and PromptTokensDetails alongside plain
integer fields in the usage response. The original code attempted to
sum all fields directly, causing:

TypeError: unsupported operand type(s) for +: 'int' and 'CompletionTokensDetails'

Fixes #1582

Root cause

dict(usage) produces a mix of plain integers (completion_tokens,
prompt_tokens, total_tokens) and nested Pydantic models
(completion_tokens_details, prompt_tokens_details). The summation
loop hit these nested objects and failed because they are not numeric.

Fix

Filter at the key level — only include keys where all events have
None or numeric values. Nested objects are excluded entirely from
the report rather than being reported as a misleading 0.

total_usage = {
    key: sum(int(u[key]) for u in usage_events if u.get(key) is not None)
    for key in usage_events[0]
    if all(u.get(key) is None or isinstance(u.get(key), (int, float)) for u in usage_events)
}

Changes

  • evals/cli/oaieval.py — filter non-numeric fields at the key level
  • evals/cli/oaieval_test.py — regression test verifying that nested detail objects are skipped while real token counts are summed correctly

Tests

  • venv/bin/python -m pytest evals/cli/oaieval_test.py — passed
  • Full venv/bin/python -m pytest could not complete locally; the venv is missing several project dependencies (mock, torch, nltk, anthropic, google-generativeai) due to conflicts with Python 3.9
  • GitHub reports no CI checks configured on this PR branch

CompletionTokensDetails and PromptTokensDetails are Pydantic models,
not integers, so int() raises TypeError on them. Use isinstance check
to only sum fields that are actually numeric, skipping nested objects.
@rajkumar-prog rajkumar-prog changed the title fix: convert token usage values to int before summing fix: omit non-numeric usage fields from token usage report May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TypeError: unsupported operand type(s) for +: 'int' and 'CompletionTokensDetail

1 participant