fix(preview): adaptive streaming to fix blur, bandwidth, and latency#422
Merged
Conversation
Rework the browser preview pipeline into a Chrome-Remote-Desktop-style adaptive stream: - Capture at native viewport resolution; display fit-scale is CSS-only. Client renders frames 1:1 (canvas backing store follows frame size), removing the upscaling that made the preview permanently blurry. - Prefer VP9 in quantizer mode, negotiated with the client's decode support at stream start (VP8 bitrate-mode fallback): cheap per-frame quality during motion, raised further under congestion. - Static top-off: when the screencast goes quiet for 300ms, push one lossless PNG screenshot through the encoder near-losslessly so text sharpens right after motion stops; idle pages cost ~0 bandwidth. Deferred while the channel is congested and abandoned if motion resumes, so large refreshes never add latency. - Reliable ordered DataChannel (VP8/VP9 delta chains cannot tolerate loss/reorder); latency is bounded by source-side frame dropping with a small 64KB backpressure window instead — queued bytes are display lag, so the stream degrades to coarser-but-current, never stale. - Fragment frames over 64KB (packet type 2) to stay under SCTP message-size limits; packet header now carries a codec byte. - Keyframes on demand only: channel open, encoder reconfigure, or the new preview:browser-stream-keyframe request (PLI equivalent) that also works on still pages; no periodic keyframe bursts. - Rate-limit encoding to the configured framerate at the CDP feeder with a trailing edge — the last frame of a damage burst is always encoded, keeping input feedback immediate while capping CPU and bandwidth (screencast fires at compositor rate, up to 60fps). - Add a 5s grace period for transient ICE 'disconnected' states on flaky networks (mobile/tunnel) — fixes the endless loading/reconnect loop when the preview is accessed remotely. - Drop audio chunks during congestion; the client's playback scheduler handles gaps cleanly.
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rework the browser preview pipeline into a Chrome-Remote-Desktop-style adaptive stream: