Skip to content

fix(token-meter): stop consuming abandoned streams - #509

Open
rioyu123 wants to merge 1 commit into
stripe:mainfrom
rioyu123:fix/token-meter-stream-cancellation
Open

fix(token-meter): stop consuming abandoned streams#509
rioyu123 wants to merge 1 commit into
stripe:mainfrom
rioyu123:fix/token-meter-stream-cancellation

Conversation

@rioyu123

Copy link
Copy Markdown

Summary

The OpenAI and Anthropic streaming meters currently split the SDK stream and drain a metering branch in an unawaited background task. The SDK tee() iterators do not propagate return(), so breaking out of the returned stream does not close the source request. The second OpenAI split also retains results for a peek branch that stops after its first chunk.

This change meters each provider inline through a single SDK Stream wrapper. Usage is accumulated as chunks pass to the caller and logged after normal completion. Early termination now closes the source iterator, while the returned value keeps the provider stream class and its stream methods. Provider SDKs remain optional runtime dependencies.

Early-cancelled streams are no longer drained solely to obtain final usage metadata. If the consumer stops before that metadata arrives, the source is closed and no complete usage event is emitted from unavailable final metadata.

Testing

  • Added real OpenAI and Anthropic stream tests covering lazy consumption and provider stream-class preservation.
  • Added controlled-generator coverage for early consumer termination, source finalization, and one-time error propagation.
  • Ran the full token-meter test suite and TypeScript build checks.

@cla-assistant

cla-assistant Bot commented Aug 29, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@cla-assistant

cla-assistant Bot commented Aug 29, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant