Transcription streaming follow-ups (RFC-0028): remaining WS providers, proxy gap, callback mode

[eric8810]

RFC-0028 is marked 已实现 (2026-08-14, three phases landed at once), but the RFC itself records deferred items that nothing currently tracks. This issue picks them up.

1. Only OpenAI realtime has a WebSocket implementation. The skeletons for ElevenLabs, Cartesia, and other WS-based transcription providers were explicitly deferred ("骨架同构,后续按需加" — §-scoped in rfc/0028-transcription-streaming.md).
2. WebSocket connections bypass the configured proxy. tokio-tungstenite has no proxy parameter (Connector only selects TLS), so the global ProxyConfig used by the HTTP client cannot be reused and streaming transcription ignores the proxy. The RFC declares this honestly; no fix or workaround is tracked.
3. Pull-based FFI only. The C ABI exposes transcription streaming as pull (NextPart); the callback style was deferred as an optional enhancement.

The RFC's §10 also records deliberate implementation differences from the design text — worth folding into this tracking when touched.

[eric8810]

Research notes (2026-09-13)

Investigated all three deferred items against the code and the providers' current public APIs.

Current state in the tree

• Only OpenAI implements do_stream — openai/transcription.rs (~268 lines), monolithic: the generic skeleton (connect-before-stream, audio append/commit pump, biased select, client-side close on completion) and the OpenAI wire specifics are interleaved. There is no shared WS-session layer a second provider could call.
• Cartesia is the worst gap: is_streaming_transcription_model_id already gates ink-2* (cartesia.rs ~L618) and do_generate returns UnsupportedFunctionality for it ("WS-only endpoint") — but no do_stream exists. Calling ink-2 today fails on both paths.
• ElevenLabs ships only scribe_v1 batch POST; scribe_v2_realtime has zero code.
• xAI: no STT at all (chat only). RFC-0028 §1.1 lists "Grok STT" as WS-based in the AI SDK ecosystem, but there is no public xAI STT endpoint to target — propose dropping it from scope until that changes.
• Proxy gap confirmed deliberate: ProxyConfig (global OnceLock, exposed via FFI/Node/Python init_proxy) drives reqwest for all HTTP; ws.rs connects with bare connect_async and bypasses all of it (the file header documents this). Behind a corporate proxy every AI call is proxied except realtime transcription, which either fails or bypasses the boundary.
• Pull-based FFI is complete (session symbols, bounded-channel backpressure, bounded join); Go lifecycle was hardened in fix(go)!: eliminate lifecycle deadlocks #157 (RFC-0028 D6). No callback path exists, by design.
• No transport metadata in the registry/catalogue — nothing can statically declare "this provider supports WS realtime STT"; capability discovery fails only at runtime with UnsupportedFunctionality. Same gap Complete recording and replay (RFC-0023): protocol-generic replay, one record format, all bindings #167 hits when adding protocol to ProviderRecord.

Provider wire facts (docs + official SDKs, 2026-09)

• ElevenLabs Scribe v2 Realtime: wss://api.elevenlabs.io/v1/speech-to-text/realtime; auth via xi-api-key header (fine from Rust) or single-use token query. Single inbound message type input_audio_chunk (base64 JSON, commit flag, sample_rate); outbound session_started / partial_transcript / committed_transcript (+timestamps/entities variants) / warning / 14 typed error events. Termination via commit_strategy (manual/vad); no explicit terminate message — differs from OpenAI's client-side close.
• Cartesia Ink-2: turns WebSocket; auth Authorization: Bearer header; binary audio frames + JSON text commands (config, finalize in manual mode, close); auto-finalize and manual-finalize modes.
• tokio-tungstenite: proxy support is explicitly out of scope upstream (tungstenite-rs#177, maintainer's position); the standard route is manual TCP dial → HTTP CONNECT → validate 200 → client_async_tls_with_config with a self-supplied rustls ClientConfig — exactly what RFC-0028 Open Question 2 sketched. This will never land upstream; it's on us.

Proposed workstreams

• W1 — WS proxy tunnel (independent, broadest value): ws_connect consults the global ProxyConfig, self-implements no_proxy matching, and tunnels via CONNECT when applicable. Care points: rustls ClientConfig must match reqwest's (webpki roots); a SOCKS all_url is not tunnelable via CONNECT — must error loudly, not silently direct-connect.
• W2 — extract the shared skeleton first, then providers by protocol distance: (1) refactor OpenAI's do_stream into a shared WS-transcription-session layer (termination policy parameterized); (2) ElevenLabs next — base64-JSON audio + event stream is closest to the OpenAI shape; (3) Cartesia after — binary frames + finalize/close commands stress-test the abstraction, and it is the only provider with a gating stub but no implementation; (4) xAI dropped until an endpoint exists; (5) each provider needs one live-API smoke (RFC-0028 D4 records that OpenAI never got one).
• W3 — callback mode: propose closing as wontfix. Pull-based was a deliberate, sound choice for the C ABI (no re-entrancy/thread constraints); streaming sugar belongs as thin async-iterator wrappers in Node/Python bindings with zero wire changes.

Suggested order: W1 and W2-step-1 in parallel, then ElevenLabs → Cartesia serially. Scoping decision (what's worth doing) to follow in this issue.

[eric8810]

Final state of the three tracked items:

1. Remaining WS providers — ElevenLabs scribe_v2_realtime implemented with mock tests (feat(providers): ElevenLabs scribe_v2_realtime streaming transcription (RFC-0034 P2) #184); Cartesia ink-2 deferred with explicit restart triggers (RFC-0034 §4/D6: login-walled docs, young turns API, unverifiable wire, zero deferral cost — gate comment in cartesia.rs records it); xAI dropped (no public endpoint).
2. WS proxy gap — closed (feat(provider-utils): WS proxy tunnel + RFC-0034 (realtime STT follow-ups) #183): the global ProxyConfig now governs ws:///wss:// via a CONNECT tunnel (Basic proxy auth, reqwest-aligned no_proxy, loud SOCKS/https refusal, shared retryability classification). The wss+rustls path is verified by a manual handshake-level smoke against the real wss://api.elevenlabs.io through a local CONNECT proxy (live_wss_handshake_through_connect_proxy, #[ignore], ran green).
3. Callback mode — wontfix per the research notes above; pull-based stays, streaming sugar belongs in Node/Python binding wrappers.

The only deliberately open item: a full transcript round-trip smoke needs a provider key (D4 posture, flagged in code and RFC-0034 §3.5).