Skip to content

[Bug]: stream retries re-send already-billed requests and retry deterministic 4xx #2677

Description

@1688mengdie

Problem

execute_sse_request retries with the same request_body in two cases that
should be terminal:

  1. A TTFT (time-to-first-token) timeout re-sends an already-sent request. The
    request body is sent to the server before the first-token wait begins, so
    the retry re-sends an already-billed request (double-billing for the same
    logical turn).
  2. A deterministic 4xx (for example 400) is retried. A deterministic client
    error fails identically on every attempt, so the retry only burns the
    request budget/credits and adds latency.

Root cause

The retry decision in execute_sse_request does not distinguish terminal from
transient outcomes. The TTFT branch continues using the same request body,
and the non-2xx branch retries every status, including 400. The
SEND_MESSAGE_STREAM_ATTEMPTS counter is amplified by these re-sends.

Repro

  1. For TTFT: mock a server that never completes a response, inject a small
    ttft_timeout, and observe execute_sse_request re-send the same body
    across attempts.
  2. For 4xx: mock a server that returns 400 and observe the fixture is called
    more than once (the old bad_requests_then_success fixture even reports
    success when a later attempt returns 200).

Impact

Billing is charged twice for the same logical turn on TTFT timeout, and
deterministic client errors waste credits/latency across up to
SEND_MESSAGE_STREAM_ATTEMPTS re-sends.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions