Skip to content

Add complete multi-round Messages usage accounting #315

Description

@franciscojavierarceo

Problem statement / motivation

Operators need the full inference cost of a Messages turn, including rounds used by built-in tools. Provide consistent, documented accounting for JSON and SSE clients without counting repeated usage snapshots more than once.

Proposed solution

  • Add a typed per-round usage accumulator and a turn aggregate shared by the JSON and streaming Messages paths.
  • Define input, output, cache-read, and cache-creation accounting for supported upstream fields. Distinguish missing usage from zero and retain documented provider semantics.
  • Specify how complete turn usage is exposed while preserving the native Messages streaming schema. If the wire protocol cannot represent every total, document a separate typed telemetry surface rather than inventing wire fields.
  • Separate per-round records from the aggregate so downstream metering counts each inference contribution once.

Acceptance criteria

  • Deterministic one-, two-, and multi-tool-round fixtures verify totals for JSON and SSE.
  • Tests cover initial/terminal usage snapshots, repeated cumulative counters, absent/partial usage, cache counters, and unknown supported extensions.
  • JSON and SSE expose equivalent complete-turn accounting through the documented surfaces.
  • Errors, cancellation, client-tool handoff, and round-limit exhaustion report available usage as partial when appropriate; they never fabricate a completed total.
  • Usage from hidden built-in-tool rounds is included without exposing their internal content or double-counting the public aggregate.
  • Document units, accounting boundaries, and examples for operators.

Additional context

Enterprise Readiness enhancement. Coordinate with #279 for OpenTelemetry export; this issue owns usage semantics and correctness, not a billing platform.

Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.

Enterprise Readiness tracker: #316

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestenterprise-readinessEnhancements for secure, reliable enterprise deployments

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions