Implement audio support, fixes #59#60
Open
ThomasWaldmann wants to merge 9 commits into
Open
Conversation
- Enable audio capture in ScreenCaptureKit and exclude current process audio. - Add SBAudioConverter to dynamically convert float32 non-interleaved PCM to S16 interleaved stereo PCM at 48000Hz. - Implement explicit SDL audio callback and 1-second circular ring buffer on the receiver to bypass OS buffer delays. - Add smooth-discard sliding-window resync that ceiling caps audio backlog at 80ms without causing silent gaps or clicks. - Restore hardware buffer size to 1024 samples to protect playout against CPU scheduling underflow. - Add comprehensive architecture documentation in docs/audio.md.
…apture - Migrate virtual display capture in Extended Display Mode from legacy video-only CGDisplayStream to modern ScreenCaptureKit (SCStream). - Expose virtual extended desktop as a standard SCDisplay inside SCShareableContent. - Enable capturesAudio = true on the virtual display stream to capture and stream system audio in real-time. - Unify the sender-side capture pipeline for both Mirror and Extended Display sessions. - Update docs/audio.md to document the unified ScreenCaptureKit pipeline.
- Introduce session-specific 'Stream Mac audio' toggles in TBDisplaySenderContentView.swift. - Bind per-session toggles to individual session audioEnabled states for instant, live muting. - Decouple the global audioEnabled setting in TBDisplaySenderManager.swift so it solely defines the default for new sessions. - Update docs/audio.md to document the per-session audio control and live cutoff logic.
…ove dead code - Extract ring_read() helper using memcpy with wrap-around handling, replacing per-byte iteration in audio_callback. - Use memcpy with wrap-around for the ring buffer write path in the TB_PKT_AUDIO_FRAME handler. - Remove unused audio_started field from struct app.
- Guard AVAudioConverter input callback with inputConsumed flag to prevent duplicate buffer delivery if the converter re-invokes the closure (e.g. during sample rate conversion). - Replace unsafe OpaquePointer cast in dead interleaved branch with assertionFailure + early return, since ScreenCaptureKit always delivers float32 non-interleaved audio.
Contributor
Author
|
Gemini FTW! Works now, please review. Before merge, guess I want to squash some commits. |
ThomasWaldmann
changed the title
Closed
- Modify drain_socket to track total bytes read. - Yield 1ms (SDL_Delay) in event loop when socket is idle to eliminate 100% CPU busy-spinning, preventing real-time thread starvation. - Raise audio backlog latency ceiling from 80ms to 150ms (28800 bytes) to cushion against ScreenCaptureKit variable delivery chunks and network jitter. - Update build number.
- Update docs/audio.md to reflect the increased 150ms (28800 bytes) latency safety ceiling. - Add documentation detailing how receiver busy-spinning CPU usage was eliminated, resolving real-time audio thread starvation.
Owner
|
this code it was very important but i had to some breakchanges. Thank's a lot. |
Contributor
Author
Owner
|
Thanks again for this work. The audio relay pieces have been folded into the 3.0 release branch and are part of the new release PR to main. Really appreciate the contribution here. |
Contributor
Author
|
@swellweb If you have all you need from here, feel free to close. |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements low-latency raw PCM audio streaming between the sender and the receiver, unified capture using ScreenCaptureKit for both Mirror and Extended Display sessions, and session-specific live audio controls.
Summary of Changes
Low-Latency Raw PCM Audio Streaming & Playout:
ScreenCaptureKitand converts thefloat32non-interleaved PCM toS16interleaved stereo PCM at48000HzusingSBAudioConverter.SDL_Delay(1)) in the event loop when the network socket is idle (0 bytes read), avoiding thread starvation and ensuring real-time scheduling priority for the SDL audio thread.Unified ScreenCaptureKit & Extended Display Mode Audio:
CGDisplayStreamto modern ScreenCaptureKit (SCStream).capturesAudio = truefor all active streams.Per-Session Mute & Audio Controls:
TBDisplaySenderContentView.swift).Hardening & Performance Refinements:
memcpyand wrap-around handling in the ring buffer write and read paths.SBAudioConverterinput callback with aninputConsumedflag to safeguard against duplicate buffer delivery edge cases.docs/audio.md.