Skip to content

Add bounded, validated Messages streaming and tool scheduling #313

Description

@franciscojavierarceo

Problem statement / motivation

Enterprise deployments need predictable memory use and explicit completion semantics for Messages requests that execute built-in tools. Extend the existing native Messages loop with a documented lifecycle and resource contract across all inference rounds.

Proposed solution

  • Validate message and content-block transitions through one Messages ingestion path, keeping transport framing and client delivery separate from semantic validation.
  • Bound retained text, thinking, signatures, tool arguments, tool call outputs, pending calls, and reconstructed history. Distinguish retained bytes from wire bytes so smaller deltas do not consume disproportionate response capacity.
  • Reuse bounded scheduling contracts for built-in tools, with defined timeout, overload, disconnect, cancellation, and task-cleanup behavior.
  • Preserve native Messages content, client-executed function tools, rebased indexes, and one public message lifecycle.

Acceptance criteria

  • Invalid ordering, duplicate indexes/completions, missing identifiers, malformed event data, and open blocks at termination have explicit tested outcomes.
  • Premature EOF cannot dispatch an incomplete tool call or become an apparently successful message. Document any intentional engine-specific tolerance; retain forward compatibility for harmless unknown events.
  • Limits cover accumulated content and pending work across rounds, with deterministic errors and bounded resource release.
  • Tests cover fragmented UTF-8/SSE, many small deltas, large arguments/outputs, slow readers, disconnects during inference/tools, and round-limit exhaustion.
  • Preserve the message_stop completion and forced-tool-choice behavior shipped in fix: finish Messages rounds at message_stop #290 and fix: let Messages answer after forced tool use #296.

Additional context

Enterprise Readiness enhancement. Coordinate with #241/#244 and #288 for shared boundary and budget contracts; this issue owns Messages behavior. No worker-placement change is required without measurements.

Source review at afa6c14, not a new runtime reproduction: Messages streaming and Messages JSON loop.

Enterprise Readiness tracker: #316

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestenterprise-readinessEnhancements for secure, reliable enterprise deployments

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions