feat(player): stream stems on desktop via the 5s-chunk engine (#261) - #264
Conversation
Desktop previously decoded every stem in full before playback (~420 MB / slow preload). The Range-based chunked engine (chunkedAudioEngine.js) already streams glitch-free on mobile: 5s HTTP-Range windows scheduled on AudioBufferSourceNodes, first audio after ~1 chunk, ~28 MB RAM, no length cap. This promotes it to the desktop player as the default, closing its three feature stubs so there's no regression vs the full-decode engine: - chunkedAudioEngine: implement setLoop (scheduler jumps to loop.start on crossing loop.end, and caps lookahead at loop.end); add a per-stem AnalyserNode (gain -> analyser -> master) + getAnalyser for live VU. - player.js: engineMode() selects chunked by default; "fulldecode" and "0" (legacy <audio>) remain opt-in via the stemdeck.audioEngine flag. The RAM cap now only gates the full-decode engine. On the chunked path, drive lane mini-waves + the energy baseline from peaks.json and VU meters from the engine's live analysers (overview waveforms already come from peaks.json). - mixer.js: renderRealMiniWaveFromPeaks (peaks-based lane mini-wave). The original "buffering issue" was the N-<audio>-element multitrack path (HTTP/1.1 6-connection-cap underruns on Safari/WKWebView); the chunked engine avoids it by construction. Full-decode stays as a fallback.
…ks fallback) Self-review of the chunked-engine promotion found four gaps: - Loop-start chunk was evicted as playback advanced, so every loop pass paid a refetch gap. Pin it against both eviction sites while a loop is active, and warm it as the playhead approaches loop.end (deduped by the chunk cache). - A transient all-stems fetch failure cached an empty chunk forever, leaving playback permanently silent past that point. Drop empty results so the scheduler retries. - Loop jumps scheduled with the cold-start 50 ms lead. Cached (sync) starts now use 10 ms, making loop wraps near-seamless; the async path keeps 50 ms. - Legacy jobs without peaks.json lost all waveform visuals on the streaming path. The chunked branch now falls back to the full-decode engine in that case (honoring the backend's documented "degrades to client-side decode" contract), unless the track exceeds the decode RAM cap - then it keeps streaming audio with placeholder waveforms.
|
Self-review pass for state-of-the-art within scope; hardened four things: (1) the loop-start chunk is now pinned in the chunk cache and pre-warmed as the playhead approaches loop.end, so loop passes replay from cache with no refetch gap; (2) an all-stems fetch failure no longer caches permanent silence - empty chunk results are dropped so the scheduler retries after a transient network blip; (3) cached starts schedule with a 10 ms lead instead of 50 ms, making loop wraps near-seamless (cold/async starts keep 50 ms headroom); (4) legacy jobs without peaks.json fall back to the full-decode engine for visuals (the backend's documented degradation path), unless the track exceeds the decode RAM cap, in which case audio still streams and waveforms stay placeholder. Known remaining limitation, deliberate for this scope: the loop wrap is rAF-detected (up to ~1 frame overshoot + 10 ms schedule lead), same detection model as the full-decode engine. True sample-accurate loop splicing (pre-scheduling the loop-start chunk to start exactly at loop.end on the audio clock) is a follow-up, as is opus-compressed streaming. |
…table fix) pip-audit newly flags torch 2.6.0 for PYSEC-2026-2286 (torch.load weights_only deserialization -> arbitrary code execution, HIGH; fixed in 2.10.0). The exploit requires an attacker-controlled .pth checkpoint. StemDeck never calls torch.load on untrusted input: demucs loads only its official model weights from the trusted torch-hub source, and users submit audio, not checkpoints. torch is pinned <2.7 (torchaudio 2.7+ dropped the writer demucs needs), so 2.10.0 is not adoptable yet. Documented alongside the existing ignored torch advisories.
1 participant
Closes #261.
Problem
Desktop playback decodes every stem in full before it starts (~420 MB, slow preload). The "buffering issue" that blocked streaming in the past was NOT streaming itself - it was the legacy path that plays N HTML
<audio>elements through WaveSurfer Multitrack, which underruns on Safari/WKWebView due to the HTTP/1.1 6-connection-per-origin cap (documented inaudioEngine.js).Key insight
Streaming is already solved and shipping on mobile:
chunkedAudioEngine.jsfetches WAV in 5 s HTTP Range windows and schedules them onAudioBufferSourceNodes with a 12 s lookahead - glitch-free, first audio after ~1 chunk, ~28 MB RAM, no length cap. It just wasn't wired on desktop, and stubbed out loop / VU / visuals. This PR promotes it to the desktop player and fills those three stubs.Changes
setLoop(scheduler jumps toloop.startwhen the playhead crossesloop.end; lookahead capped atloop.end); per-stemAnalyserNode(gain -> analyser -> master) +getAnalyserfor live VU.engineMode()-> chunked is the default.stemdeck.audioEngine="fulldecode"opts into the old full-decode engine,"0"into the legacy<audio>path. RAM cap now only gates full-decode. On the chunked path: lane mini-waves + energy baseline from peaks.json, VU meters from the engine's live analysers (overview waveforms already came from peaks.json).renderRealMiniWaveFromPeaks.Full-decode stays as an opt-in fallback; nothing is removed.
Verify
node --checkpasses on all JS (CI's gate).<audio>path.localStorage stemdeck.audioEngine = "fulldecode"-> old path still works.Follow-up (not here)
Compressed-opus streaming (the issue's own suggestion) to cut bytes ~10x for slow LAN.