Phase 3 — Performance: memory & throughput

[thcp]

Tracking issue for Phase 3 of the health report roadmap: streamed peaks (constant memory), folding stem-presence into the same pass, SSE dirty-flag, optional shifts=2 quality setting.

Findings tracked here:

• Peaks computation loads entire WAVs into RAM #286 — HIGH
• Stem presence re-decodes every stem via ffmpeg #287 — HIGH
• Model weights reloaded per job #288 — MED (instrumented; real hardware data posted, closed with a decision, spun off Persistent demucs worker: eliminate per-job model reload on GPU (35-42% of separate stage) #309 (also closed))
• SSE polls at 5 Hz per connection #289 — MED
• Exports re-encode on every request #290 — LOW

[thcp]

Implementation plan — Phase 3: Performance

Findings covered: #286, #287, #288, #289 (+#285 from Phase 2), #290, plus the --shifts quality setting from the roadmap. 5 PRs (one explicitly a stretch). Depends on Phase 1's timing telemetry (#293) for the before/after numbers — measure first, then optimize.

---

PR-3A — SSE dirty-flag + tear-proof job serialization (#289, #285)

app/core/models.py: add version: int = 0 to Job (exclude from to_state() and from _JOB_FIELDS — internal like cancel_requested). _set() increments job.version after all field writes.

app/api/events.py stream loop: replace serialize-and-compare with:

last_v = -1
while loop.time() < deadline:
    v = job.version
    if v != last_v:
        snapshot = job.to_state()
        if job.version != v:      # torn read: writer ran mid-serialize
            continue               # loop immediately, re-serialize
        yield f"data: {json.dumps(snapshot)}\n\n"
        last_v = v
        ...

• Fixes Non-atomic multi-field job updates visible to SSE #285 for real: a snapshot taken while _set was mid-flight is detected (version moved) and discarded, so a torn progress/stage pair never reaches the client.
• Perf: idle connections now do one int compare per 0.2 s tick instead of a full to_state() + json.dumps — at the 200-connection cap that eliminates ~1,000 serializations/s.
• Keepalive cadence unchanged (count ticks, emit : keepalive every ~75).
• Terminal check moves to after a successful yield (status is in the snapshot), plus a cheap job.status in TERMINAL check on the no-change path so already-terminal jobs still close promptly.

Tests: extend the SSE tests: no duplicate frames when state is static; frame emitted on _set; a simulated concurrent _set during serialize (monkeypatch to_state to bump version mid-call) yields only consistent snapshots; terminal close still works.

---

PR-3B — Single-pass streamed peaks + presence (#286, #287)

New module app/pipeline/audio_stats.py:

def scan_stem(path: Path, buckets: int = 1500) -> tuple[list[list[float]], float]:
    """One streamed pass: ([min,max] per bucket, rms). Constant memory."""

• info = sf.info(path); blocksize = max(1, ceil(info.frames / buckets)); iterate sf.blocks(path, blocksize=blocksize, dtype="float32", always_2d=True); per block: [float(b[:,0].min()), float(b[:,0].max())] + accumulate np.sum(b[:,0]**2) and count → rms = sqrt(sumsq/n).
• Memory: one block ≈ frames/1500 × channels × 4 B ≈ 2–3 MB for a 20-min stereo stem (vs ~420 MB today).

app/pipeline/collect.py — compute_stem_peaks: keep the signature and peaks.json format identical; body delegates to scan_stem and now also returns {name: rms} collected during the same pass.

app/pipeline/runner.py — _run_common: drop the separate compute_stem_presence call (analyze.py version and its 6 ffmpeg spawns are deleted); after compute_stem_peaks returns the rms map, derive presence inline: normalize the 6 demucs-stem rms values to 0–100 relative to max (exact logic from analyze.compute_stem_presence lines 236–246). original/mix are scanned for peaks but excluded from presence, matching today.

• Ordering note: presence is currently computed before the mix step and peaks after; move the combined scan to the current peaks position (after mix creation) — presence lands a few seconds later within the same job, invisible to the UI since both are only consumed at done.
• Known, accepted delta: presence RMS is now full-rate channel-0 instead of 22.05 kHz ffmpeg-downmixed mono. Presence is normalized-relative (0–100 vs the loudest stem), so displayed values shift by ≲1–2 points. Document in the PR body; update any test fixtures.

Tests: golden test — generate multi-tone WAVs, compare new peaks output to the old implementation's output (copy old impl into the test as reference) within float tolerance; rms correctness vs np.sqrt(np.mean(x**2)) full-load reference; presence normalization unchanged for equal inputs; peaks.json schema untouched (frontend contract).

Measured claim for the PR body: run a 20-min track before/after with stage_timings (Phase 1) and report the post= delta + peak RSS if measurable.

---

PR-3C — Demucs startup cost: measure, publish, decide (#288)

Explicitly a measurement task — subprocess isolation (kill-on-cancel, crash containment) is a design feature we keep.

• separate.py: record time.monotonic() at Popen and at the first stderr byte that parses as a progress percentage → job.stage_timings["separate_startup"]. One-line addition to the existing read loop.
• After it ships, run 3 jobs on a representative machine (CPU and GPU), post the numbers to Model weights reloaded per job #288, and close it with a decision comment: keep subprocess (expected: startup is 5–15 s of a 1–15 min stage = not worth a persistent-worker's complexity), or open a follow-up with a concrete design if it's >20% of separate time on GPU.

---

PR-3D — Separation quality setting: --shifts (#274 roadmap item)

Follows the exact demucs_device settings pattern (the newest, most complete example of the runtime-settings plumbing):

• app/core/settings.py: get_separation_quality() / set_separation_quality(v) with choices ("standard", "best"), default standard, env seed STEMDECK_SEPARATION_QUALITY.
• app/main.py: add to _settings_payload() + POST handler (422 on bad value).
• app/pipeline/separate.py: in _demucs_cmd, when quality == "best" append ["--shifts", "2"]. Applies on any device (CPU users accept the 2× wait knowingly).
• Settings UI (static/js/catalog.js): General tab, next to Compute device: select "Separation quality" — Standard / Best (2× slower), wired identically to the video-height select. Desc: "Best runs the separator twice with randomized shifts and averages the result — cleaner stems, twice the time."
• Tests: settings round-trip + rejection; _demucs_cmd includes/omits --shifts 2.

---

PR-3E (stretch, do last, OK to defer) — Export render cache (#290)

Server-mode nicety; skip if time-boxed effort exceeds a day.

• Key: sha1(job_id, ext, stems, gains, start, end, export_sample_rate); store at CACHE_DIR/mixdown/<key>.<ext>.
• Tee-while-streaming: _stream_ffmpeg writes chunks to <key>.tmp as it yields; on clean returncode==0 finish, os.replace tmp→final; on any failure/disconnect, unlink tmp. Cache hit → FileResponse (first-byte latency for hits drops to ~0, misses unchanged).
• Bound: after each finalize, prune dir to 20 files / 500 MB by mtime.
• Invalidation: key includes every render input; stems are immutable per job id; export_sample_rate is in the key so a settings change misses cleanly.
• Tests: hit/miss, failed render leaves no cache entry, prune bound.

---

Sequencing & estimates

3A (0.5d) → 3B (1d, the big memory win) → 3C (0.25d) → 3D (0.5d) → 3E (stretch 1d).
Manual verification: 3B on the longest track in the library — confirm identical waveforms render in the studio (peaks contract) and presence bars look sane; 3A via devtools EventSource frames during a job.

[thcp]

Phase 3 status: 4 of 5 shipped, closing out for now

PR	Closes	Commit
#305	#289, #285 (SSE dirty-flag + tear-proof serialization)	c896b9c
#306	#286, #287 (single-pass streamed peaks + presence)	6eb1741
#307	#288 (startup-cost instrumentation)	5e8caeb
#308	--shifts separation quality setting (roadmap item)	68c449d

#288 stays open: PR-3C only instruments stage_timings["separate_startup"] -- closing it needs real timing numbers from a few jobs run on representative CPU/GPU hardware, which isn't something producible from a dev sandbox. Whoever has real hardware handy: run a few jobs, check separate_startup in the job's metadata.json or the completion log line, and post the numbers here with a keep-or-redesign decision.

#290 (export render cache) deferred: explicitly marked a stretch item in the implementation plan ("OK to defer" / "skip if time-boxed effort exceeds a day"). More substantial than the other four (tee-while-streaming cache, bounded prune, cache-key invalidation across stems/gains/region/sample-rate) -- left as a standalone backlog item rather than rushed into this phase.

Leaving this issue open until #288 and #290 land.

[thcp]

Update: #290 shipped too

PR #311 (`679eb78`) added the mixdown render cache -- all 5 originally-planned Phase 3 items are now done except #288, which stays open by design (it needs real hardware timing data to close with a decision, not more code -- see the comment on #288 itself and the #309 follow-up it produced).

Phase 3 final tally: #305, #306, #307, #308, #310 (bonus follow-up), #311. All merged.

[thcp]

All 5 findings + the #309 follow-up are closed. This tracking issue is done — closing.