feat(providers): ElevenLabs scribe_v2_realtime streaming transcription (RFC-0034 P2) - #184
Draft
eric8810 wants to merge 6 commits into
Draft
feat(providers): ElevenLabs scribe_v2_realtime streaming transcription (RFC-0034 P2)#184eric8810 wants to merge 6 commits into
eric8810 wants to merge 6 commits into
Conversation
…bs/Cartesia do_stream) P1 WS proxy tunnel honoring the global ProxyConfig (CONNECT + no_proxy, SOCKS errors loudly); P2 ElevenLabs scribe_v2_realtime do_stream; P3 Cartesia ink-2 do_stream. No unified session abstraction (D1), minimal parameter surface (D5). Closes #178 (research record attached there).
…proxy (RFC-0034 P1) ws_connect now consults the global ProxyConfig that already governs HTTP: wss uses https_url (all_url fallback), ws uses http_url (all_url fallback), no_proxy entries are honored with reqwest NoProxy semantics (comma-separated, suffix match, '*', port-specific entries), and matched targets keep the direct path unchanged. Tunneled connects: manual TCP to the proxy -> CONNECT (with Basic Proxy-Authorization from proxy-URL userinfo) -> 2xx validation -> WS handshake over the tunnel via client_async_tls_with_config (explicit ring provider + webpki-roots for wss targets, Plain for ws). SOCKS and https-scheme proxies fail loudly as UnsupportedFunctionality instead of silently bypassing the configured proxy; proxy CONNECT rejections surface as non-retryable ApiCall errors carrying the proxy status. The tunnel steps are bounded by first_chunk_ms and raced against abort by the existing ws_connect select. rustls/webpki-roots/base64 join the optional 'ws' feature only.
Independent review (PR #183) surfaced three defects and a test gap: - IPv6 proxy hosts never connected: url::Url::host_str returns bracketed literals and the socket resolver wants the bare address — strip them. - Proxy credentials leaked into error strings: all resolve_proxy error paths now mask userinfo (scheme://***@host), matching the existing request-URL sanitization. - CONNECT rejections were blanket non-retryable: classify by the shared is_retryable_status rule (407/403 stay terminal, 502/503/504 retry); EOF-before-response (status 0) is transient. - Missing promised tests added: abort during the CONNECT tunnel (against a black-hole proxy), unresponsive proxy tripping first_chunk_ms, 503 retryability, CONNECT wire shape (request line, Host, no stray Proxy-Authorization), IPv6 bracket stripping, credential masking. - no_proxy docstring now states the CIDR divergence; init_proxy documents that WS reads the config live. - RFC-0034: OQ2 verified and closed — stream_transcribe has no attempt retry and should not (audio input is consumed once; replayable sources would be new design). Status line and test list updated to match what landed.
…l cost Re-review after P1: the decisive difference vs P2 (ElevenLabs) is verifiability. ElevenLabs ships a public API reference, so mock tests pin documented facts; Cartesia's turns API is behind a login wall, its event schema is only inferable from the Python SDK, and the API is new enough to churn — on top of the D4 lesson that mock-only wire shapes can be wrong. Deferring loses nothing: ink-2 already fails honestly with UnsupportedFunctionality on both paths, and nothing depends on it. Restart triggers (any one): a user asks; the docs come out from behind the login wall; a key + smoke decision. Gate comment in cartesia.rs records the deferral. D6 added.
…n (RFC-0034 P2)
ElevenLabsTranscriptionModel gains do_stream behind the existing
'realtime' feature, gated on scribe_v2_realtime* (inverse gating in
do_generate rejects realtime IDs with a clear message instead of
posting them to the batch endpoint).
Wire shape per the public API reference (2026-09), pinned
field-by-field by local mock-server tests:
- config travels on the URL query (model_id, audio_format
pcm_{rate}/ulaw_{rate}, commit_strategy=manual, optional
language_code/include_timestamps from providerOptions.elevenlabs);
xi-api-key rides the WS handshake headers
- audio: base64 in input_audio_chunk JSON with sample_rate; a
hold-one-chunk pipeline puts commit:true on the LAST real chunk (an
empty stream commits a single empty chunk); audio waits for
session_started
- partial_transcript -> TranscriptPartial (replace semantics),
committed_transcript[_with_timestamps] -> TranscriptFinal + Finish
(segments from words, spacing entries dropped), then client
close(1000)
- 14 documented error event names classified: rate_limited /
queue_overflow / resource_exhausted retry, the rest terminal
- the server never closes after commit, so a settle window (chunk_ms,
5s default) bounds the wait with an empty Finish — the stream always
terminates
- non-pcm/ulaw input formats fail fast without connecting
Live-API smoke pending (no key at implementation time) — same posture
as RFC-0028 D4; the do_stream doc comment says so.
Also: cartesia.rs gate comment now records the P3 deferral (RFC-0034
D6); RFC status/plan rows updated.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements P2 of RFC-0034: ElevenLabs
scribe_v2_realtimestreaming transcription (do_streamonElevenLabsTranscriptionModel, behind the existingrealtimefeature).Stacked on #183 (P1, WS proxy) — until that merges, this diff includes P1's commits; it collapses to the P2 delta afterwards. P3 (Cartesia) is deferred with explicit restart triggers (RFC-0034 §4/D6); the gate comment in
cartesia.rsrecords it.What lands
scribe_v2_realtime*→do_stream; everything else →do_generate.do_generatenow rejects realtime IDs with a clear message instead of posting them to the batch REST endpoint.commit_strategy=manualfixed, D3;languageCode/includeTimestampsthe only provider options, D5),xi-api-keyhandshake header, base64input_audio_chunkframes withsample_rate,commit:trueriding the last real chunk (hold-one-chunk pipeline; empty stream commits an empty chunk),partial_transcript→TranscriptPartial(replace semantics),committed_transcript[_with_timestamps]→TranscriptFinal+Finish(word segments, spacing dropped), clientclose(1000)after the final.chunk_ms, 5 s default) bounds the wait with an emptyFinish— mirroring Align the request pipeline with the AI SDK #164's terminal-fuse principle.rate_limited/queue_overflow/resource_exhaustedretryable, the rest terminal; raw event preserved inresponse_body.Verification
aimux-providers/tests/elevenlabs_realtime_stream_test.rs(happy path with query/header/chunk-frame assertions, timestamps variant, empty-audio commit, 4-way error classification, silent-server settle, mid-session abort, symmetric gating, format rejection). Workspace clippy clean.do_streamdoc comment. Smoke through a proxy also closes P1's untested wss+rustls branch.Closes the P2 scope of #178.