Playful browser studio: pads, beats, voice, and stem split

THIRD_PARTY_NOTICES.md

@@ -83,6 +83,7 @@ this list is checked before adding any new dependency.
 | Splendid Grand Piano, Greg Sullivan Wurlitzer EP200, VCSL TX81Z FM piano, FluidR3_GM renderings | Public domain / CC-BY 3.0 / CC0 1.0 / MIT+CC-BY 3.0 | Same sample set as Browser Studio above (streamed via smplr, not re-fetched/re-verified) — reachable from the homepage's instrument browser via its "Real instrument sound" toggle (**on by default as of 2026-10-04** — see below), for the families with a credible sampled match (keyboards, guitars, bowed/plucked strings, brass, winds, mallets). Wurlitzer attribution lives in `website/studio/SOUND_LIBRARY.md`, same as before. |
 | **(2026-10-04)** VCSL Concert Harp, Marimba, Tubular Bells 1, Tenor Saxophone — Versilian | CC0 1.0 | Same `sgossner/VCSL` repo/license as the TX81Z piano above (CC0 covers the whole collection, not per-instrument — no new verification needed), reached via smplr's own `Versilian`/`loadVersilianInstrument` API instead of the `ElectricPiano`/`Soundfont` wrappers. Upgrades `harp`/`marimba`/`bell`/`saxophone` from a FluidR3_GM soundfont rendering to a real multi-sampled recording. Every sfz path was checked directly against the host's own `sfz_files.json` manifest and confirmed to resolve (sfz file + its `.ogg`/`.m4a` sample assets, all HTTP 200) before being added — not assumed from smplr's own example path, which points at a `Strings/Violin` instrument this particular mirror does not actually have (checked and confirmed absent, so bowed strings/brass/winds/choir/bass/guitars were deliberately left on their existing, already-working FluidR3 mapping rather than guessing). |
 | **(2026-10-04)** `synth_bass_1` (FluidR3_GM) for the homepage's `synthbass` family | MIT (SoundFont) / CC-BY 3.0 (renderings) | Same already-verified FluidR3_GM source and exact instrument id Browser Studio's own "Synth Bass" entry already uses — extends coverage to an already-approved source's existing catalogue, not a new dependency. |
+| Studio v2 effects, reverb impulse responses, Voice Lead synth, MPM pitch tracker | Original code (no third-party license) | Impulse responses are generated at runtime in `website/studio/js/fx.js`; v2 added no third-party samples or libraries |
 
 Full per-set provenance and confidence notes: [`website/studio/SOUND_LIBRARY.md`](website/studio/SOUND_LIBRARY.md).
 

docs/STUDIO_PLAN.md

@@ -1,6 +1,7 @@
 # MyDawsons Studio — redesign plan (browser DAW)
 
-Status: **Phase 1 MVP built on branch `feat/studio-redesign`, served at `/studio/`.
+Status: **v2 (Phase 2 essentials + DAW upgrade) built on branch `feat/studio-redesign`, served at `/studio/`.
+Preview: https://mydawsons-studio-preview.vercel.app/studio/ (throwaway Vercel project).
 Production (`mydawsons.com`, GitHub Pages from `main`) is untouched.**
 
 ## Where things live
@@ -30,10 +31,39 @@ Production (`mydawsons.com`, GitHub Pages from `main`) is untouched.**
 - **Timeline**: add Drums/Keys/Audio tracks; record pads/keys/voice into clips (count-in, metronome, optional 1/16 quantize); upload audio (file picker or drag-drop); CC0 loop library (preview, drag or +); clip move (also across tracks of the same kind), trim both edges, loop (drag right edge to repeat), reverse, tape speed 0.5–2×, stretch (keeps pitch, WSOLA), pitch ±12 st (keeps length), "Fit to tempo", volume, split at playhead, duplicate, delete, quantize/transpose note clips; track mute/solo/volume/pan; snap to 1/16; zoom.
 - **Export**: offline render of the whole mix to 16-bit 44.1 kHz stereo WAV.
 
-### Phase 2 — make it sticky
-- Autosave to IndexedDB + project list; undo/redo; save/load project files (JSON + audio blobs).
-- Piano-roll/step editor for note clips (edit notes, draw beats), velocity lanes.
-- Per-track effects (reverb, delay, EQ, compressor, filter) and a master section; send/return reverb.
+### v2 — built (October 2026)
+
+**Typing layout (default "Compact"; Settings ⚙ keeps "Classic" v1).** Matched by physical key (`KeyboardEvent.code`), so it works on AZERTY/QWERTZ too. Drums and piano play at the same time — no mode switch.
+
+```
+Drums, left hand (same shape as the 3x6 pads)     Piano, right hand
+  Q W E R T Y   toms / percussion                   U I . P [ ]     C# D# (gap) F# G# A#
+  A S D F G `   hats / cymbals                      H J K L ; ' \   C  D  E  F  G  A  B
+  Z X C V B N   kick, kick 2, snare, snare 2,      Octave: ,  down   .  up
+                clap, rim
+```
+Every pad and piano key shows its key. Shift = harder hit. Space play/stop, Enter record, Home, Delete, Ctrl/⌘+Z / Shift+Ctrl/⌘+Z (or Ctrl+Y) undo/redo, Esc, `?` help.
+
+**Import from anywhere.** Upload or drop files anywhere on the page; "🔗 From link" fetches an audio URL directly, and if the host blocks CORS retries via `website/api/fetch-audio.js` (Vercel function, preview project only — same-site callers, http(s) only, private IPs blocked after DNS + on every redirect, audio content types, 30 MB / 20 s caps). Streaming services (YouTube, Spotify, SoundCloud, Apple Music…) are refused in both client and proxy. "⏺ Record a tab" uses `getDisplayMedia` (tab/system audio) + MediaRecorder → new audio clip at the playhead. A one-line rights note is shown in the import bar and the footer.
+
+**Clip editing.** ⇆ reverse button on every audio clip (one click) + right-click / long-press clip menu (reverse, half/normal tape speed, fit to tempo, fades, convert voice → notes, loop, split, duplicate, track effects, delete). Plain edge drag = trim (or repeat if looping); **Alt/Option + edge drag = time-stretch** (pitch kept; dashed edge cursor while Alt is held, live "Stretch 1.25×" tooltip). Fades: drag the small top-corner handles or pick from the clip bar.
+
+**Effects** (`js/fx.js`, original Web Audio). Per track: EQ (3-band), filter (LP/HP/BP), distortion, compressor, chorus, delay (tempo-synced presets), convolution reverb with **self-generated impulse responses** (seeded decaying noise — no IR files to license). Each effect has presets, and one-click chain presets: Small room, Big hall, Telephone, Radio, Lo-fi, Vocal polish, Punchy drums, Dreamy, Dub echo. Master bus: reverb + limiter (ceiling) + volume. Export uses the identical chain.
+
+**Phase-2 essentials.** Undo/redo (80 steps, snapshot-based, buffers referenced by id), autosave to IndexedDB (project JSON + raw audio, restored on load with a "Start new" option), 7-step onboarding tour on first visit (re-run from `?`), shortcuts help (`?`), clip fades.
+
+**Extras chosen (high value, low risk).** Drum step sequencer (▦ Steps on any drum clip, 1/16 grid up to 64 steps), tap tempo, a tuner (note + cents needle) in the voice panel, "Voice Lead" built-in synth that follows pitch bends. Skipped for now to avoid bloat: piano-roll editor (next on the list), a separate metronome UI (Click already exists).
+
+**Voice → any instrument.** "Sing → instrument" with its own *Play as* menu (every keyboard sound, plus "Drums — beatbox mode"); if a Keys track is selected, the voice plays and records onto that track. Pitch tracking moved from YIN to the **McLeod Pitch Method** in the AudioWorklet with a 32 ms window and a 5.3 ms hop (was 43 ms / 10.7 ms). Note tracker with hysteresis on pitch (±0.7 st band, 3 frames to start, 4 to change, 6 to stop) and on level (gate opens at the sensitivity threshold, closes at 60 %), velocity from loudness, optional **Glide** (live pitch-bend on built-in synths; sampled instruments take the start offset). Press ● to record the notes. **Convert a vocal clip to instrument**: clip menu → offline MPM over the clip → notes on a new Keys track (uses the voice panel's instrument, scale snap and octave).
+CREPE-tiny was considered: the model is MIT-licensed, but it adds ~2 MB of weights + TF.js/ONNX runtime and a 64 ms+ analysis window, so it would *raise* latency. MPM is the lower-latency choice for live play; CREPE stays a Phase-3 option for offline conversion accuracy.
+
+Tests: headless Chrome with a fake mic (synthesised sung melody C-D-E-G-E-C): voice played 60 62 64 67 64 60; clip conversion produced 64 67 64 60 60 62 (first note clipped by the fake device start). See the v2 report for full results.
+
+### Phase 2 — remaining
+
+- Project list + save/load project files (JSON + audio blobs). (Autosave + undo/redo done in v2.)
+- Piano-roll editor for keys clips, velocity lanes. (Drum step grid done in v2.)
+- Send/return reverb buses; automation. (Per-track FX + master done in v2.)
 - Self-host (or R2/CDN-mirror) the streamed piano/soundfont samples for reliability and preload the default sounds; progress UI on first load.
 - Loop library v2: tempo/key metadata, auto-fit on drop, more CC0 genres (guitar/bass/keys/vocal chops); search.
 - Latency calibration wizard (tap-along) for mic takes and voice-to-instrument.
@@ -57,7 +87,13 @@ website/studio/
   js/instruments.js  keyboard sounds (smplr: Splendid piano, e-pianos, FluidR3 GM) + built-in synths
   js/clipfx.js       reverse, WSOLA time-stretch, pitch shift, waveform peaks
   js/voice.js        mic capture, note tracker, beatbox classifier + training
-  js/voice-worklet.js AudioWorklet: capture + YIN + band energies (audio thread)
+  js/voice-worklet.js AudioWorklet: capture + McLeod pitch (MPM) + band energies (audio thread)
+  js/keymap.js       compact (v2) and classic computer-keyboard layouts
+  js/fx.js           per-track effects, generated reverb IRs, presets, master bus
+  js/store.js        undo/redo snapshots + IndexedDB autosave
+  js/importers.js    URL import (+ proxy fallback) and tab/system-audio capture
+  js/pitch.js        MPM + note tracker (offline vocal -> notes)
+website/api/fetch-audio.js  Vercel function: CORS fallback for URL import (preview only)
   js/piano.js, pads.js, midi.js, loops.js, audio.js
   vendor/smplr-1.1.0.mjs (MIT)
   samples/           CC0 / PD one-shots + loops (5 MB, ogg + m4a for Safari)
@@ -74,9 +110,11 @@ playback and WAV export on exactly the same code path.
 - Samples: Splendid Grand (PD), Wurlitzer by Greg Sullivan (CC-BY 3.0, attributed), VCSL TX81Z (CC0), FluidR3_GM (MIT / CC-BY 3.0 renderings), Sonic Pi pack (CC0), TR-808 & LM-2 (PD per distributor).
 - Excluded: MusyngKite/FatBoy (CC-BY-SA), Hydrogen kits (GPL), Drum Abuse (no license found), amen/breakbeat loops (commercial-record provenance).
 
-## Known limits of the MVP
-- No undo, no autosave (a leave-page warning guards unsaved work).
+## Known limits
+- Autosave lives in this browser only (IndexedDB); no cloud sync or project files yet. Undo history is per tab and holds processed audio in memory.
 - Sampled instruments need network on first use; built-in synths/kits work offline.
-- Voice-to-instrument latency ~50–70 ms (analysis window + confirmation frames); monophonic; tuned on synthetic test signals only so far.
+- Voice-to-instrument latency ~35–50 ms (32 ms window + 3 × 5.3 ms confirmation); monophonic; tuned on synthetic test signals only so far.
+- Glide bends only the built-in synths live (smplr samplers can't be re-pitched after note-on).
+- Tab capture needs Chrome/Edge on desktop; Safari/Firefox have no tab-audio capture. The URL proxy only exists on the Vercel preview; on GitHub Pages, CORS-blocked links show a "download it and drag it in" message.
 - Stretch/pitch processing runs on the main thread (fine for loops; a few seconds for long songs).
 - Web MIDI isn't available in Safari/iOS.

services/ai-sidecar/app/api/routes_stems.py

@@ -0,0 +1,87 @@
+"""POST /stems — upload a song, split it with Demucs, fetch the WAVs.
+
+This is the local engine the browser studio calls. It is not part of the
+static website and it does not use a paid API.
+
+Run it yourself (first launch downloads the free htdemucs_6s weights):
+
+    cd services/ai-sidecar
+    pip install -r requirements.txt
+    uvicorn app.main:app --host 127.0.0.1 --port 8765
+
+Then paste http://127.0.0.1:8765 into Studio's "Local engine URL".
+The sidecar binds to localhost; CORS is open so the browser page can call it.
+"""
+
+from __future__ import annotations
+
+import re
+import threading
+import uuid
+from pathlib import Path
+
+from fastapi import APIRouter, File, HTTPException, UploadFile
+from fastapi.responses import FileResponse
+
+from app.config import detect_device
+from app.jobs import manager
+from app.jobs.models import JobStatus
+from app.storage.paths import job_dir
+
+router = APIRouter()
+
+_MAX_BYTES = 40 * 1024 * 1024
+_AUDIO_SUFFIXES = {".wav", ".mp3", ".flac", ".ogg", ".m4a", ".aiff", ".aif", ".mp4", ".webm", ".aac"}
+_STEM_NAME = re.compile(r"[a-z0-9_-]+")
+
+
+def _run(job_id: str, input_path: Path) -> None:
+    manager.update(job_id, status=JobStatus.RUNNING, stage="separating_stems", progress=0.05)
+    try:
+        from app.pipeline.stems import separate
+
+        def on_progress(stage: str, progress: float) -> None:
+            manager.update(job_id, stage=stage, progress=progress)
+
+        paths = separate(input_path, job_dir(job_id), device=detect_device(), on_progress=on_progress)
+        stems = {name: name for name in paths}
+        manager.update(job_id, status=JobStatus.DONE, stage=None, progress=1.0, result={"stems": stems, "engine": "demucs"})
+    except Exception as exc:  # noqa: BLE001 — surface the failure on the job
+        manager.update(job_id, status=JobStatus.FAILED, error=str(exc))
+
+
+@router.post("/stems", status_code=202)
+async def start_stems(file: UploadFile = File(...)) -> dict[str, str]:
+    raw = await file.read()
+    if not raw:
+        raise HTTPException(status_code=400, detail="empty file")
+    if len(raw) > _MAX_BYTES:
+        raise HTTPException(status_code=413, detail="file is over 40 MB")
+    suffix = Path(file.filename or "audio").suffix.lower()
+    if suffix not in _AUDIO_SUFFIXES:
+        suffix = ".wav"
+    job_id = str(uuid.uuid4())
+    dest = job_dir(job_id)
+    dest.mkdir(parents=True, exist_ok=True)
+    (dest / f"input{suffix}").write_bytes(raw)
+    manager.create(job_id)
+    threading.Thread(target=_run, args=(job_id, dest / f"input{suffix}"), daemon=True).start()
+    return {"job_id": job_id}
+
+
+@router.get("/stems/{job_id}")
+def get_stems(job_id: str):
+    record = manager.get(job_id)
+    if record is None:
+        raise HTTPException(status_code=404, detail="job not found")
+    return record
+
+
+@router.get("/stems/{job_id}/{name}")
+def get_stem_file(job_id: str, name: str):
+    if _STEM_NAME.fullmatch(name) is None:
+        raise HTTPException(status_code=400, detail="bad stem name")
+    path = job_dir(job_id) / f"{name}.wav"
+    if not path.is_file():
+        raise HTTPException(status_code=404, detail="stem not ready")
+    return FileResponse(path, media_type="audio/wav", filename=f"{name}.wav")

services/ai-sidecar/app/main.py

@@ -10,6 +10,7 @@
 import sys
 
 from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
 
 from app.api import (
     routes_admin,
@@ -18,11 +19,22 @@
     routes_generate,
     routes_health,
     routes_pitch,
+    routes_stems,
 )
 
 app = FastAPI(title="dawsons-ai-sidecar")
 
+# The browser studio calls this from another origin (the static site).
+# The process still binds to localhost; this only lets that page read it.
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
 app.include_router(routes_health.router)
+app.include_router(routes_stems.router)
 app.include_router(routes_analyze.router)
 app.include_router(routes_classify.router)
 app.include_router(routes_generate.router)

website/api/fetch-audio.js

@@ -0,0 +1,89 @@
+// Vercel serverless function (preview project only — GitHub Pages ignores
+// it). Fetches a public audio file for the Studio's "Import from URL" when
+// the hosting site doesn't send CORS headers.
+// Guard rails: same-site callers only, http(s) only, no private/loopback
+// addresses (checked after DNS and on every redirect), audio content
+// types only, 30 MB cap, 20 s timeout, and streaming services refused.
+const dns = require("dns").promises;
+const net = require("net");
+
+const MAX_BYTES = 30 * 1024 * 1024;
+const BLOCKED = /(^|\.)(youtube\.com|youtu\.be|googlevideo\.com|spotify\.com|scdn\.co|soundcloud\.com|sndcdn\.com|music\.apple\.com|tidal\.com|deezer\.com|netflix\.com|music\.amazon\.)/i;
+const OK_TYPES = /^(audio\/|video\/(ogg|webm|mp4)|application\/(ogg|octet-stream))/i;
+
+function privateIp(ip) {
+  if (net.isIPv4(ip)) {
+    const [a, b] = ip.split(".").map(Number);
+    return a === 10 || a === 127 || a === 0 || (a === 169 && b === 254) || (a === 172 && b >= 16 && b <= 31) ||
+      (a === 192 && b === 168) || (a === 100 && b >= 64 && b <= 127) || a >= 224;
+  }
+  const v = ip.toLowerCase();
+  return v === "::1" || v === "::" || v.startsWith("fc") || v.startsWith("fd") || v.startsWith("fe80") ||
+    (v.startsWith("::ffff:") && privateIp(v.slice(7)));
+}
+
+async function checkUrl(u) {
+  if (!/^https?:$/.test(u.protocol)) throw new Error("Only http(s) links are allowed.");
+  if (BLOCKED.test(u.hostname)) throw new Error("Streaming services can't be fetched. Use tab capture for audio you have rights to.");
+  const host = u.hostname.replace(/^\[|\]$/g, "");
+  const addrs = net.isIP(host) ? [{ address: host }] : await dns.lookup(host, { all: true });
+  if (!addrs.length || addrs.some((a) => privateIp(a.address))) throw new Error("That address isn't reachable from here.");
+}
+
+function sameSite(req) {
+  const site = req.headers["sec-fetch-site"];
+  if (site === "same-origin") return true;
+  const origin = req.headers.origin || "";
+  return /^https:\/\/([a-z0-9-]+\.)*(mydawsons\.com|vercel\.app)$/i.test(origin) && /mydawsons/i.test(origin);
+}
+
+module.exports = async (req, res) => {
+  const fail = (code, error) => {
+    res.statusCode = code;
+    res.setHeader("content-type", "application/json");
+    res.setHeader("cache-control", "no-store");
+    res.end(JSON.stringify({ error }));
+  };
+  if (req.method !== "GET") return fail(405, "GET only");
+  if (!sameSite(req)) return fail(403, "This helper only serves the MyDawsons Studio.");
+  let target;
+  try {
+    target = new URL(String(new URL(req.url, "http://x").searchParams.get("url") || ""));
+  } catch {
+    return fail(400, "Missing or invalid url.");
+  }
+  const ctrl = new AbortController();
+  const timer = setTimeout(() => ctrl.abort(), 20000);
+  try {
+    let r;
+    for (let hop = 0; hop < 4; hop++) {
+      await checkUrl(target);
+      r = await fetch(target.href, { redirect: "manual", signal: ctrl.signal, headers: { "user-agent": "MyDawsonsStudio/2 (+https://mydawsons.com/studio/)" } });
+      if (r.status >= 300 && r.status < 400 && r.headers.get("location")) {
+        target = new URL(r.headers.get("location"), target);
+        continue;
+      }
+      break;
+    }
+    if (!r || !r.ok) return fail(502, `The site answered ${r ? r.status : "nothing"}.`);
+    const type = r.headers.get("content-type") || "application/octet-stream";
+    if (!OK_TYPES.test(type)) return fail(415, "That link isn't an audio file (it looks like a web page). Link straight to the .mp3/.wav/.ogg file.");
+    const len = Number(r.headers.get("content-length") || 0);
+    if (len > MAX_BYTES) return fail(413, "That file is over 30 MB.");
+    const chunks = [];
+    let total = 0;
+    for await (const chunk of r.body) {
+      total += chunk.length;
+      if (total > MAX_BYTES) return fail(413, "That file is over 30 MB.");
+      chunks.push(chunk);
+    }
+    res.statusCode = 200;
+    res.setHeader("content-type", type);
+    res.setHeader("cache-control", "private, max-age=600");
+    res.end(Buffer.concat(chunks.map((c) => Buffer.from(c))));
+  } catch (e) {
+    return fail(e.name === "AbortError" ? 504 : 400, e.name === "AbortError" ? "The site took too long." : e.message || "Fetch failed.");
+  } finally {
+    clearTimeout(timer);
+  }
+};