Align the request pipeline with the AI SDK

[cunninghamcard-bit]

Aligns the request pipeline with the Vercel AI SDK contract (reference baseline 63db193). The design spec ships with the change as docs/ai-sdk-request-pipeline.md, including a difference table for every deliberate deviation. History is organized as six commits in build order: core, provider-utils, providers, FFI, bindings, docs.

What this fixes relative to master

The two headline problems are the retry layering and the error domain; everything else falls out of them.

• Retry lived in the wrong layer. In master, retry was internal to the HTTP module — its own docs state the provider does not observe retries happening. That conflated one logical call with one HTTP exchange: errors carried no attempt history and no exhausted-vs-non-retryable classification, recordings could not distinguish an operation's Nth attempt from its Nth exchange, and no policy above the transport (idempotency, per-operation budgets) could see the retry boundary. Now Core owns operation retry (RetryError { maxRetriesExceeded | errorNotRetryable }, messages verbatim to the AI SDK, full attempt history) and provider-utils performs exactly one exchange per call — the AI SDK's own layering.
• The error domain was lossy. Five providers (ElevenLabs, AssemblyAI, Deepgram, fal, Hume) parsed failed responses through the OpenAI-shaped DEFAULT_ERROR_STRUCTURE (error.message / error.type), so their real error formats (detail.*, plain {error}, err_code/err_msg, FastAPI detail, top-level message/code) surfaced as empty messages; and a failing response's body was buffered in full with no size bound before parsing. Each mapper now matches its provider's documented format, and error bodies read best-effort under a 64 KiB public cap (post-decode) with a 1 MiB parse bound so oversize-but-valid error JSON still reaches the mapper.
• A retried video start could bill twice — a direct casualty of the old layering. The billable predictLongRunning POST went through the HTTP layer's silent retry with no idempotency key, so a retry after a lost response re-submitted a paid generation. One idempotency key is now minted per logical start, reused across retry attempts, and never sent on status polls (AI SDK generate-video parity).
• One malformed frame killed the whole stream. Core consumers, both FFI pumps, and the Node and Python drivers each latched on the first parse error and truncated the stream. A recoverable frame error now travels as data and the stream continues; only transport/Core errors are terminal.
• Stream termination depended on the provider closing the connection. With no timeout configured, a provider that sent the final chunk but held the connection open hung the consumer forever. The consumption wrapper is now an unconditional terminal fuse under plain default options.
• One process-wide HTTP client leaked dead connections across runtimes. Pooled connections are driven by the runtime that made the request; a finished runtime left dead connections checked in, and the HTTP-layer retry masked the resulting failures (roughly one run in six flaky on direct SPI calls). The client is now keyed per runtime with a bounded cache.

Core

• Operation retry moved into Core. The two-layer primitive mirrors retryWithExponentialBackoff(RespectingRetryHeaders): a generic loop that does not know ApiCallError, plus a wrapper that fills should_retry and get_delay_ms. RetryError { maxRetriesExceeded | errorNotRetryable } preserves the full attempt history; messages match the AI SDK fixtures verbatim. Full Jitter (RFC-0009) is the default get_delay_ms for the exponential branch; a server retry-after(-ms) hint is honored exactly (ms < 60s || ms < exponential).
• Timeouts and cancellation. AbortSignal is a thin CancellationToken wrapper carrying only caller cancellation; total/step/first-chunk/chunk deadlines live in the operation future (tokio::select!, nothing spawned). The first-chunk budget is armed before do_stream setup; chunk timers reset only on semantic output (isOutputChunk parity).
• Streams survive malformed frames, at every boundary — and always terminate. A bad JSON/schema frame is surfaced as an Err item and the stream continues; only transport/Core errors are terminal. Core alone was not enough: both FFI pumps, the Node napi driver, and the Python pyo3 driver each latched on the first Err and truncated the stream, so a recoverable frame error now travels as data — a StreamPart::Error frame on the plain path, skipped on the chunk-typed chat-completions path whose wire cannot carry it. The consumption wrapper doubles as the terminal fuse: Finish and non-recoverable errors end the stream under every configuration, including plain default options. StreamTextResult::text() / consume() keep consuming to the trailing Finish so usage and recording still land.
• A transport error on the first SSE event fails the attempt inside the retry boundary (the RFC-0016 peek is retained — a deliberate deviation, see the spec).
• VideoModel split into do_start / do_status. Core owns the poll loop and retries each phase independently, matching the AI SDK generate-video flow: do_start — billable — mints one idempotency key per logical start outside the retry closure (a caller-supplied key wins), and the poll budget paces the loop between status checks while a hung status GET is bounded by the per-exchange 30s response guard. VideoCallOptions defaults n and provider_options on deserialization — typed binding structs omit unset fields, and a strict parse rejected minimal options at the FFI boundary.

Provider utils / providers

• Single-exchange helpers (post_json_to_api, post_form_data_to_api, post_to_api, get_from_api) with typed response handlers replace send* / send_with_retry_raw / ErrorStructure; every helper call performs exactly one fetch attempt, records exactly one exchange, and non-streaming exchanges keep a per-exchange 30s whole-response guard.
• The error path keeps the provider's error. Error bodies read best-effort with a 64 KiB public cap (enforced after lossy UTF-8 decoding) and a 1 MiB parse bound so oversize-but-valid error JSON still reaches the provider's mapper; a handler failure on a 429/503 keeps the status's retryability; a mid-body transport failure reports no HTTP status. In-band stream errors are redacted and URL-sanitized at the shared stream_error_api_call entry — the raw request context providers hold never reaches the public error or recordings.
• Failed-response mappers match each provider's documented format (ElevenLabs detail.status/detail.message, AssemblyAI's plain {error: string}, Deepgram err_code/err_msg, fal's FastAPI detail, Hume's top-level message/code) instead of assuming OpenAI's shape, and string provider codes are no longer JSON-quoted.
• The shared HTTP client is keyed by runtime, and the cache is bounded. Pooled connections are driven by tasks on the runtime that made the request, so one process-wide client let a finished runtime leave dead connections checked in — and the OS can recycle their ports to fresh listeners, handing later requests a corpse. The old HTTP-layer retry masked this; with retry in Core, direct SPI calls (every wiremock provider test) hit it as flaky transport failures roughly one run in six. Runtimes leave no drop signal, so the per-runtime cache evicts wholesale at a small cap instead of growing without bound.
• All providers migrated to the handler API; list_models exchanges apply the Core retry primitive with the provider's configured settings. Recording distinguishes operation attempts from exchange indices, with step-labeled contexts for Router/MoA children.

Scope notes for reviewers

• Download/SSRF security is not in this PR. An earlier revision of the spec asked get_from_api to carry AI SDK's validateUrl / credentialedOrigin / trustedOrigin contract from day one, and the implementation grew a full inline guard — duplicating fix: validate and pin provider-supplied download URLs (SSRF) #163, which delivers exactly that. It has been removed here and the spec now hands the contract to fix: validate and pin provider-supplied download URLs (SSRF) #163, which rebases onto this pipeline and wires the provider call sites. Redirects follow reqwest's default policy in the meantime, recorded as one logical exchange.
• Three files overlap fix: validate and pin provider-supplied download URLs (SSRF) #163 for lint reasons only. trace/hash.rs, openai/embedding.rs, and ws.rs needed the new clippy 1.98 lints silenced. Both PRs do it with the same #[allow] attributes and the same wording (as_chunks::<N>() is stable only since Rust 1.88, while the workspace MSRV is 1.85), so whichever lands second is a no-op in those files.

Verification

Organized by what this PR could break, and the gate that would catch it:

Risk	Gate	Where
Retry, backoff, retry-after semantics	Ported AI SDK fixture tests; error messages match the AI SDK verbatim	aimux-core retry tests
Timeout semantics (total / first-chunk / chunk-idle)	Paused-time matrix tests for timer arming, reset-on-output, and cleanup	aimux-core/tests/stream_timeout_matrix_test.rs
Per-provider request/response behavior	43 test files ported 1:1 from the AI SDK's own provider test suites (each header cites its reference/ai/packages/.../*.test.ts source); wiremock exercises the full pipeline over real local HTTP	aimux-providers/tests
Error wire contract	Golden error-variant test with a compile-time exhaustive match, so adding a variant cannot bypass the wire-tag check	aimux-core/tests/error_value_golden_test.rs
C ABI drift	Header/exports agreement test pins all 115 exported symbols against aimux-ffi.h	aimux-ffi/tests/exports_smoke_test.rs
Typed TS surface drift	ts-rs regeneration gate	scripts/gen_ts_types.py --check
Binding regressions	Per-binding CI jobs: Node (6 targets), Python, Go (2 OS), Java (JDK 8/11/17/21), Kotlin (JDK 17/21), Swift, Flutter (android/ios), plus contract tests	CI matrix
Stream lifecycle across FFI	Contract tests pinning recoverable-frame continuation on both pump shapes	aimux-ffi/src/lib.rs tests

All 24 CI checks are green on the current head.

Real-provider smoke (DeepSeek, run at the current head)

A live end-to-end pass through the full pipeline (Core retry → single-exchange helpers → OpenAI-compat provider) against api.deepseek.com:

1. generate_text OK: text="pong" usage=Some(2) finish=Stop
2. stream_text OK: 11 deltas, finish seen, text="1\n2\n3\n4\n5"
3. 401 mapping OK: status=Some(401) retryable=false provider_message="Authentication Fails, Your api key: ****alid is invalid"
4. operation timeout OK: request timed out: Total timeout of 1ms exceeded
SMOKE PASSED

What each line proves: (1) a real success round-trip parses text/usage/finish; (2) a real SSE stream delivers deltas, Finish, and then terminates (the terminal fuse, under plain default options); (3) a real 401 surfaces the provider's own message, is classified non-retryable, and is not retried even with max_retries: 2; (4) a 1 ms budget trips the typed AiMuxError::Timeout, not a mislabeled transport error.

To reproduce with any OpenAI-compatible key (no repo changes needed):

let model = aimux_providers::provider("deepseek", Some(key), "deepseek-chat", None)?;
let r = aimux_core::generate::generate_text(model.as_ref(), "Reply with exactly: pong",
    GenerateTextOptions { max_output_tokens: Some(40), ..Default::default() }).await?;

Review guide

Where the diff concentrates, and why:

• aimux-provider-utils/src/http.rs (the largest change, mostly deletions): retry, error parsing, and recording bookkeeping moved out. What remains is transport — build the request, perform one exchange, record it.
• aimux-core/src/generate.rs (+710): the orchestration that used to hide inside send() is now visible at the operation layer — the retry closure around do_generate/do_stream, the timeout deadlines (tokio::select! in the operation future, nothing spawned), and the stream-consumption wrapper that doubles as the terminal fuse.
• aimux-core/src/recording.rs (+455) plus plumbing: with retry above the HTTP layer, the transport no longer knows which attempt it is serving, so a RecordingContext is built once per operation and travels with the request — a shared attempt allocator, per-context exchange counters, and child() contexts that label Router/MoA steps while keeping attempt numbers unique across the whole call. Streaming response recording holds this context instead of a clone of the prepared request.
• aimux-core/src/video_model.rs (+549): every video provider used to hand-roll its own start+poll loop; the do_start/do_status split moves that flow into Core once — poll pacing, per-phase retry, idempotency-key minting — ported from the AI SDK's executeStartStatusFlow. Roughly 40% of the file is its test module.

Six commits in build order (core → provider-utils → providers → ffi → bindings → docs), each message stating its rationale. Two external review rounds are folded in; every confirmed finding was either fixed or deliberately re-aligned with the AI SDK's reference behavior (the spec's §3 table records all deviations).

Related issues: advances #95 (error-domain alignment: retry error expression, provider error classification, stream error semantics, the C ABI error surface) and partially resolves #161 (video now has the user-facing operation layer — generate_video() over do_start/do_status).

Follow-up: #165 moves tool-call input parsing and validation to the Core boundary. It is based directly on master and independent of this PR; whichever lands second rebases over textual overlap only.

[eric8810]

Summary

The direction of this PR is valuable: centralizing retry/timeout behavior, introducing structured errors, making malformed stream frames recoverable, and sharing clients per runtime are all worthwhile changes. The implementation is also well documented and the current CI matrix is green.

However, this refactor changes several public behavioral contracts in ways that can return incorrect results or silently remove reliability guarantees. I do not think the current revision is safe to merge yet.

Major issues

1. Video generation ignores n and max_videos_per_call, so multi-video requests return only one result (and n = 0 still starts a request).
2. Stream first-chunk/chunk deadlines include time when the caller is not polling the returned stream. Valid buffered output can therefore be replaced by a timeout based on consumer scheduling rather than provider output latency.
3. File uploads no longer use the configured retry policy. The public retry configuration remains available, but OpenAI, Anthropic, and Google file operations now perform only one exchange.
4. Core retries wrap entire multi-stage transcription operations. A transient failure after a successful upload can replay the upload instead of retrying only the failed stage.
5. Video provider metadata from the start response is silently lost when the completion response uses the same provider key.
6. Successful JSON responses may buffer up to 2 GiB and are then represented in multiple in-memory forms, which turns the limit into a realistic OOM risk rather than a protective bound.

Requested changes

Please preserve the existing public contracts while moving ownership into Core: split video calls according to the requested count, measure stream timeouts at the producer boundary, define explicit retry/idempotency behavior for file and multi-stage operations, merge nested provider metadata without dropping start data, and use a conservative/configurable success-body cap or streaming deserialization. Regression tests should cover each boundary above.

The targeted Core retry/video and provider-utils test sets pass, but they do not cover these contract failures; one existing first-chunk test currently codifies the consumer-delay behavior described above.

[eric8810]

[P1] Honor the requested video count. This path performs exactly one do_start call and never reads options.n or max_videos_per_call. Consequently n > 1 still returns one video, while n = 0 still starts a provider request. Please validate the count, split it into provider-sized batches, execute all required calls, and aggregate their results/metadata.

[cunninghamcard-bit]

Fixed in 857f54c / f315b86: n = 0 is rejected before any request; n is split into max_videos_per_call batches run concurrently (AI SDK Promise.all parity), one idempotency key each, results concatenated. Test on ScriptedVideoModel asserts the batch count from a peak-in-flight counter.

[eric8810]

[P2] Do not measure consumer delay as provider timeout. This deadline continues to elapse while the returned stream is not being polled, and the biased select chooses it before an already-buffered chunk. A provider can therefore produce a valid first chunk on time, yet a caller that polls later receives only a timeout. The same issue affects chunk_ms while the consumer processes the previous item. Please drive/timestamp provider output independently of consumer polling.

[cunninghamcard-bit]

Fixed in 7a83301: a pump task now drives the provider stream and arms/resets the first-chunk and chunk deadlines on arrival; the returned stream only reads from a channel and aborts the pump on drop. The first-chunk test that codified the consumer-delay behaviour now asserts the opposite, and consumer_delay_does_not_count_against_the_chunk_timer covers chunk_ms under paused time. Spec §3/§8.1 and acceptance 21b/21c record the pump and the unconditional tokio-runtime requirement.

[eric8810]

[P2] File uploads have fallen out of the retry pipeline. This replacement changes a retrying send into a single exchange, while Files::upload_file has no Core wrapper and provider with_retry_config APIs are still exposed. The same regression exists in Anthropic and Google file implementations. Please define explicit retry/idempotency semantics for file uploads and either preserve the configured behavior or remove the misleading configuration surface.

[cunninghamcard-bit]

Fixed in 0baf004: OpenAI, Anthropic and Google file operations go through the Core retry primitive with the provider's configured retry settings, same as list_models. Uploads are not billable, so plain retry of the whole exchange is the documented policy. One 503-then-200 test per provider.

[eric8810]

[P2] Avoid replaying completed stages. If this transcript-creation request fails transiently, the error reaches the Core retry around the whole do_generate, which uploads the audio again before retrying this request. The old request-level retry retried only the failing exchange. Please preserve the successful upload reference across retries or split the workflow into explicit start/status stages. Gladia has the same shape.

[cunninghamcard-bit]

Fixed in c38bfd4: AssemblyAI and Gladia retry upload, submit and poll independently, so a transient submit failure reuses the existing upload_url. Exhaustion surfaces as AiMuxError::Retry, which the outer Core retry passes through rather than re-running the operation. Wiremock test asserts exactly one upload request. The other multi-stage providers (fal, luma, BFL, klingai, runwayml, revai, replicate) already retry their post-billing stages this way.

[eric8810]

[P2] Preserve metadata from both phases. or_insert keeps the completion object wholesale when both start and completion metadata use the same provider key, so all start-only fields under that provider are silently discarded. Please merge the nested provider objects (with a documented collision policy) instead of merging only at the top-level provider key.

[cunninghamcard-bit]

Fixed in 01da456 / f315b86: per provider key, object fields are unioned and the completion value wins on a field collision (documented on the merge function). Unit test with the same provider key in both phases.

[eric8810]

[P2] Use a protective success-body limit. A 2 GiB cap still allows a response to be buffered into a Vec, parsed into serde_json::Value, and then cloned/deserialized, so a single response can exhaust process memory well before the nominal limit. Please use a conservative/configurable bound and preferably deserialize without retaining multiple full representations.

[cunninghamcard-bit]

Fixed in ea9a33d: successful JSON bodies use a separate DEFAULT_MAX_JSON_RESPONSE_SIZE of 64 MiB, overridable per request via HttpRequest::max_json_response_bytes; typed handlers deserialize from the bytes once instead of via a Value clone. Binary downloads keep the 2 GiB bound. Test: an over-cap JSON success body fails with the size-limit error.

[cunninghamcard-bit]

@eric8810 All six findings are addressed (replies on each thread point to the commits), plus a few defects found while auditing the rest of the diff: Node/Python search() options overriding the explicit query, Node/Python transcribe/rerank/search rejecting any non-empty options, Java VideoCallOptions.fps typed as Double, and aimux-web fixtures writing schema 1. All 24 checks are green on 2d850db. Ready for another look.

Outdated: all six findings have fix commits (video batching/n validation, producer-side stream timers, file-upload retry, staged transcription retry, metadata deep-merge, JSON body cap). Maintainer requested merge.