fix(react-native): Disconnecting voice conversations over WebSocket in RN

[brojd]

Demo: https://youtu.be/tax9cbgDYMM

Fixes #605

Problem

WebSocket voice conversations disconnect immediately on React Native while working fine on web. The root cause: the WebSocket voice path uses MediaDeviceInput and MediaDeviceOutput from @elevenlabs/client, which rely on web-only APIs (AudioContext, AudioWorklet, HTMLAudioElement) that don't exist in React Native.

The React Native setup strategy (reactNativeSessionSetup) delegated to webSessionSetup for all connection types. This worked for WebRTC (where LiveKit's native polyfills handle audio), but crashed immediately for WebSocket voice because the web audio pipeline tried to instantiate browser APIs.

Solution

Introduce React Native-specific input and output controllers for the WebSocket path using react-native-audio-api:

• Input (ReactNativeInputForWebSocket): Uses AudioRecorder connected to an AudioContext via a RecorderAdapterNode to capture microphone audio as float PCM, converts to 16-bit PCM, and dispatches to the WebSocket connection.

• Output (ReactNativeOutputForWebSocket): Uses AudioBufferQueueSourceNode for gapless streaming playback. PCM chunks from the server are converted to AudioBuffer objects and enqueued for continuous, gap-free native playback.

The setup strategy now branches on connectionType:

• "websocket": Uses the new RN-specific controllers with react-native-audio-api managing the audio session.
• Default (WebRTC): Unchanged — configures LiveKit AudioSession and delegates to webSessionSetup.

Why react-native-audio-api?

Few options have been evaluated:

Approach	Verdict
react-native-audio-api	Chosen. Provides AudioBufferQueueSourceNode for gapless output streaming and AudioRecorder with onAudioReady callback for real-time mic input. Maintained by Software Mansion, MIT license.
expo-av	Rejected for output — plays audio files, not streams. Each PCM chunk required writing a WAV file to disk, loading, playing, and cleaning up — causing choppy audio with gaps between chunks.
Custom native module	Rejected — would require maintaining iOS (AVAudioEngine) and Android (AudioTrack) native code. react-native-audio-api provides the same capability from JS.
LiveKit MediaRecorder shim	Rejected for input — the @livekit/react-native MediaRecorder shim only works with tracks from active WebRTC peer connections, not standalone getUserMedia streams.

Changes

@elevenlabs/client (patch):

• VoiceConversation.ts: Wrap preliminary getUserMedia() in try/catch (non-fatal on RN). Guard document.addEventListener / document.removeEventListener with typeof document checks.
• ConnectionFactory.ts: Add function overloads so createConnection({ connectionType: "websocket" }) returns Promise<WebSocketConnection> without casts.
• internal.ts: Export attachInputToConnection, attachConnectionToOutput, createConnection, and related types for the RN package.

@elevenlabs/react-native (patch):

• New ReactNativeInputForWebSocket — mic capture via react-native-audio-api
• New ReactNativeOutputForWebSocket — gapless audio playback via react-native-audio-api
• Updated index.react-native.ts — branch setup strategy on connectionType
• Added react-native-audio-api as optional peer dependency

Example app:

• Added react-native-audio-api dependency
• Updated metro.config.js for monorepo resolution
• Native project updates from expo prebuild (adding react-native-audio-api native module)

---

Note

Medium Risk
Introduces a new React Native audio capture/playback pipeline for WebSocket voice using react-native-audio-api and changes session setup branching, which can affect microphone permissions, audio session behavior, and streaming stability on mobile.

Overview
Fixes React Native WebSocket voice sessions disconnecting by adding RN-specific input/output controllers (mic capture via AudioRecorder and gapless playback via AudioBufferQueueSourceNode) and routing the RN setup strategy to use them when connectionType: "websocket".

@elevenlabs/client is updated to be more RN-tolerant (non-fatal preliminary getUserMedia permission warmup, typed createConnection overloads) and to export additional internal helpers (createConnection, attachInputToConnection, attachConnectionToOutput, and related types) consumed by the RN package.

The Expo RN example is updated to include react-native-audio-api and Metro singleton resolution, and removes committed native ios/ and android/ project artifacts (with .gitignore adjusted accordingly).

^{Written by Cursor Bugbot for commit ee24197. This will update automatically on new commits. Configure here.}

[kraenhansen]

Thanks a lot for taking a look at this! On a general note, I think we should rely on Expo CNG and add the android and ios directories to the .gitignore of the repo.

I've added a few questions / comments below.

[kraenhansen]

Not a huge fan of silencing errors in general 🤔 At the very least we should warn here. It seems we could easily be moved up above the first try { to prevent a try in a try.

[brojd]

Makes sense!

[kraenhansen]

I'd like to avoid runtime checks like these in these common files, instead this should be a part of the client/src/platform/ abstraction and injected by the platform-specific SDK.

[brojd]

Actually now that I'm looking at those checks, they are unneeded bcs wakeLock is null in RN, I'll revert them. Thanks for pointing out!

[kraenhansen]

Same as above.

[kraenhansen]

I was expecting that we'd be able to reuse the LiveKit AudioSession utility / MediaRecorder shim to start and stop sound capturing even for WebSocket - was that something you experimented with initially or did you go straight for a "react-native-audio-api" based solution?

I see your PR description says the LiveKit MediaRecorder shim was rejected, but have you actually tried experimented with an approach based off that? I'd like for us to avoid taking on another native dependency, if at all possible.

[brojd]

I tried but it appeared that they are designed to work with WebRTC peer connection which is undefined in case of WebSocket and audio processing didn't work and I failed there :( Then I tried expo-av but the sound was breaking up. The last 2 options were writing native code for iOS and Android or use this react-native-audio-api lib. I decided to go with the last to not introduce Swift/Kotlin code there.

[brojd]

On a general note, I think we should rely on Expo CNG and add the android and ios directories to the .gitignore of the repo

Agree! I was hesitant bcs android/ios dirs were already committed. Let's remove them and add to .gitignore 👍

[kraenhansen]

Once #624 merge I'll update next and you can rebase 👍 I will likely want to try the LiveKit MediaRecorder myself first before merging this PR though.

[cursor]

Silenced getUserMedia changes web error flow

Medium Severity

Wrapping the preliminary getUserMedia in a separate try/catch that only logs a warning changes the error flow on web. Previously, a denied mic permission would fail immediately before establishing a server connection. Now, the error is silently swallowed, a server connection is established (consuming server resources), and the session only fails later when MediaDeviceInput.create() tries getUserMedia again — producing a less clear error path and wasting a connection slot.

 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

[cursor]

Resource leak on partial failure during WebSocket setup

Medium Severity

In reactNativeSessionSetup, the WebSocket path creates connection, input, and output sequentially with no try/catch around the group. If ReactNativeOutputForWebSocket.create throws after input was successfully created, neither input.close() nor connection.close() is called, and the audio session remains active (AudioManager.setAudioSessionActivity is never set to false). Similarly, if input creation fails, the connection is leaked. The web version avoids this partially via Promise.all, but here sequential creation requires explicit cleanup of earlier resources on later failures.