Skip to content

Fix WebRTC alignment and RN volume on LiveKit voice sessions - #813

Open
nathanpogue6 wants to merge 5 commits into
mainfrom
nathan/webrtc-audio-alignment
Open

Fix WebRTC alignment and RN volume on LiveKit voice sessions#813
nathanpogue6 wants to merge 5 commits into
mainfrom
nathan/webrtc-audio-alignment

Conversation

@nathanpogue6

@nathanpogue6 nathanpogue6 commented May 25, 2026

Copy link
Copy Markdown
Contributor

Fixes WebRTC/LiveKit voice sessions dropping audio_event.alignment on the data channel (playback correctly uses LiveKit tracks, but timing data is needed for karaoke highlighting). Alignment-only audio events are now forwarded to onAudioAlignment without playback payload. Stale alignment after barge-in is ignored using the same interrupt event_id guard as onAudio. React Native setVolume now applies to LiveKit remote audio tracks when no web audio adapter is present. WebSocket voice is unchanged; tests cover LiveKit, WebSocket, and interrupt behavior.


Note

Medium Risk
Touches real-time voice event handling and WebRTC data-channel filtering; behavior is well covered by new tests but affects karaoke/alignment UX and RN volume on LiveKit sessions.

Overview
Fixes LiveKit/WebRTC voice sessions dropping character alignment on the data channel while playback still uses remote audio tracks. WebRTCConnection now forwards alignment-only audio events (empty audio_base_64) and ignores audio data messages without alignment.

VoiceConversation.handleAudio applies the same interrupt event_id guard to onAudioAlignment as to playback callbacks, so stale alignment after barge-in is ignored. Alignment-only events no longer trigger onAudio, mode changes, or feedback updates.

On platforms without a web audio adapter (e.g. React Native), output volume is applied via LiveKit RemoteAudioTrack.setVolume, with remote tracks tracked through subscribe/unsubscribe lifecycle.

New tests cover WebRTC data channel behavior, WebSocket alignment forwarding, interrupt staleness, and LiveKit volume fallback.

Reviewed by Cursor Bugbot for commit 158e536. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 304c2ef. Configure here.

Comment thread packages/client/src/VoiceConversation.ts
nathanpogue6 and others added 4 commits May 25, 2026 14:25
Reorder guards so isValidSocketEvent narrows before accessing audio_event, fixing client tsc build in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nathanpogue6
nathanpogue6 requested a review from kraenhansen May 25, 2026 19:17
@kraenhansen

Copy link
Copy Markdown
Member

@nathanpogue6 do you know if this is a regression or if it just never worked?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants