From c9251c2691294be51cfb0e290bb54aaafa6cef93 Mon Sep 17 00:00:00 2001 From: Astryks Date: Thu, 15 Oct 2026 13:02:43 +1100 Subject: [PATCH 1/2] Studio v2: compact two-hand key layout, effects, import from anywhere, voice to any instrument - Compact typing layout (drums Q-Y/A-G`/Z-N left hand, piano H-' + U I P [ ] right hand, octave , .) playable together; classic layout kept in Settings; key labels on every pad/key - Per-track FX (EQ, filter, distortion, compressor, chorus, delay, convolution reverb with generated IRs) + presets (Small room, Big hall, Telephone...), master reverb + limiter - Import: audio URL (CORS-aware, Vercel proxy fallback), tab/system audio capture, drop anywhere; streaming services refused; rights note - Clip: one-click reverse, right-click/long-press menu, Alt-drag edge to time-stretch, fades - Undo/redo, IndexedDB autosave, onboarding tour, ? shortcuts, step sequencer, tap tempo, tuner - Voice: McLeod pitch tracking (32 ms window, 5 ms hop), hysteresis, glide, any instrument, convert vocal clip to notes Co-authored-by: Astryks --- THIRD_PARTY_NOTICES.md | 1 + docs/STUDIO_PLAN.md | 56 ++- website/api/fetch-audio.js | 89 ++++ website/studio/SOUND_LIBRARY.md | 22 +- website/studio/index.html | 91 +++- website/studio/js/audio.js | 25 +- website/studio/js/fx.js | 320 +++++++++++++ website/studio/js/importers.js | 133 ++++++ website/studio/js/instruments.js | 15 +- website/studio/js/keymap.js | 72 +++ website/studio/js/main.js | 724 +++++++++++++++++++++++++++-- website/studio/js/pads.js | 7 +- website/studio/js/piano.js | 15 +- website/studio/js/pitch.js | 162 +++++++ website/studio/js/project.js | 175 ++++++- website/studio/js/store.js | 158 +++++++ website/studio/js/timeline.js | 136 +++++- website/studio/js/voice-worklet.js | 77 +-- website/studio/js/voice.js | 67 ++- website/studio/studio.css | 97 ++++ 20 files changed, 2266 insertions(+), 176 deletions(-) create mode 100644 website/api/fetch-audio.js create mode 100644 website/studio/js/fx.js create mode 100644 website/studio/js/importers.js create mode 100644 website/studio/js/keymap.js create mode 100644 website/studio/js/pitch.js create mode 100644 website/studio/js/store.js diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index d9f8131..9736451 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -83,6 +83,7 @@ this list is checked before adding any new dependency. | Splendid Grand Piano, Greg Sullivan Wurlitzer EP200, VCSL TX81Z FM piano, FluidR3_GM renderings | Public domain / CC-BY 3.0 / CC0 1.0 / MIT+CC-BY 3.0 | Same sample set as Browser Studio above (streamed via smplr, not re-fetched/re-verified) — reachable from the homepage's instrument browser via its "Real instrument sound" toggle (**on by default as of 2026-10-04** — see below), for the families with a credible sampled match (keyboards, guitars, bowed/plucked strings, brass, winds, mallets). Wurlitzer attribution lives in `website/studio/SOUND_LIBRARY.md`, same as before. | | **(2026-10-04)** VCSL Concert Harp, Marimba, Tubular Bells 1, Tenor Saxophone — Versilian | CC0 1.0 | Same `sgossner/VCSL` repo/license as the TX81Z piano above (CC0 covers the whole collection, not per-instrument — no new verification needed), reached via smplr's own `Versilian`/`loadVersilianInstrument` API instead of the `ElectricPiano`/`Soundfont` wrappers. Upgrades `harp`/`marimba`/`bell`/`saxophone` from a FluidR3_GM soundfont rendering to a real multi-sampled recording. Every sfz path was checked directly against the host's own `sfz_files.json` manifest and confirmed to resolve (sfz file + its `.ogg`/`.m4a` sample assets, all HTTP 200) before being added — not assumed from smplr's own example path, which points at a `Strings/Violin` instrument this particular mirror does not actually have (checked and confirmed absent, so bowed strings/brass/winds/choir/bass/guitars were deliberately left on their existing, already-working FluidR3 mapping rather than guessing). | | **(2026-10-04)** `synth_bass_1` (FluidR3_GM) for the homepage's `synthbass` family | MIT (SoundFont) / CC-BY 3.0 (renderings) | Same already-verified FluidR3_GM source and exact instrument id Browser Studio's own "Synth Bass" entry already uses — extends coverage to an already-approved source's existing catalogue, not a new dependency. | +| Studio v2 effects, reverb impulse responses, Voice Lead synth, MPM pitch tracker | Original code (no third-party license) | Impulse responses are generated at runtime in `website/studio/js/fx.js`; v2 added no third-party samples or libraries | Full per-set provenance and confidence notes: [`website/studio/SOUND_LIBRARY.md`](website/studio/SOUND_LIBRARY.md). diff --git a/docs/STUDIO_PLAN.md b/docs/STUDIO_PLAN.md index 4a30689..dbd9d8e 100644 --- a/docs/STUDIO_PLAN.md +++ b/docs/STUDIO_PLAN.md @@ -1,6 +1,7 @@ # MyDawsons Studio — redesign plan (browser DAW) -Status: **Phase 1 MVP built on branch `feat/studio-redesign`, served at `/studio/`. +Status: **v2 (Phase 2 essentials + DAW upgrade) built on branch `feat/studio-redesign`, served at `/studio/`. +Preview: https://mydawsons-studio-preview.vercel.app/studio/ (throwaway Vercel project). Production (`mydawsons.com`, GitHub Pages from `main`) is untouched.** ## Where things live @@ -30,10 +31,39 @@ Production (`mydawsons.com`, GitHub Pages from `main`) is untouched.** - **Timeline**: add Drums/Keys/Audio tracks; record pads/keys/voice into clips (count-in, metronome, optional 1/16 quantize); upload audio (file picker or drag-drop); CC0 loop library (preview, drag or +); clip move (also across tracks of the same kind), trim both edges, loop (drag right edge to repeat), reverse, tape speed 0.5–2×, stretch (keeps pitch, WSOLA), pitch ±12 st (keeps length), "Fit to tempo", volume, split at playhead, duplicate, delete, quantize/transpose note clips; track mute/solo/volume/pan; snap to 1/16; zoom. - **Export**: offline render of the whole mix to 16-bit 44.1 kHz stereo WAV. -### Phase 2 — make it sticky -- Autosave to IndexedDB + project list; undo/redo; save/load project files (JSON + audio blobs). -- Piano-roll/step editor for note clips (edit notes, draw beats), velocity lanes. -- Per-track effects (reverb, delay, EQ, compressor, filter) and a master section; send/return reverb. +### v2 — built (October 2026) + +**Typing layout (default "Compact"; Settings ⚙ keeps "Classic" v1).** Matched by physical key (`KeyboardEvent.code`), so it works on AZERTY/QWERTZ too. Drums and piano play at the same time — no mode switch. + +``` +Drums, left hand (same shape as the 3x6 pads) Piano, right hand + Q W E R T Y toms / percussion U I . P [ ] C# D# (gap) F# G# A# + A S D F G ` hats / cymbals H J K L ; ' \ C D E F G A B + Z X C V B N kick, kick 2, snare, snare 2, Octave: , down . up + clap, rim +``` +Every pad and piano key shows its key. Shift = harder hit. Space play/stop, Enter record, Home, Delete, Ctrl/⌘+Z / Shift+Ctrl/⌘+Z (or Ctrl+Y) undo/redo, Esc, `?` help. + +**Import from anywhere.** Upload or drop files anywhere on the page; "🔗 From link" fetches an audio URL directly, and if the host blocks CORS retries via `website/api/fetch-audio.js` (Vercel function, preview project only — same-site callers, http(s) only, private IPs blocked after DNS + on every redirect, audio content types, 30 MB / 20 s caps). Streaming services (YouTube, Spotify, SoundCloud, Apple Music…) are refused in both client and proxy. "⏺ Record a tab" uses `getDisplayMedia` (tab/system audio) + MediaRecorder → new audio clip at the playhead. A one-line rights note is shown in the import bar and the footer. + +**Clip editing.** ⇆ reverse button on every audio clip (one click) + right-click / long-press clip menu (reverse, half/normal tape speed, fit to tempo, fades, convert voice → notes, loop, split, duplicate, track effects, delete). Plain edge drag = trim (or repeat if looping); **Alt/Option + edge drag = time-stretch** (pitch kept; dashed edge cursor while Alt is held, live "Stretch 1.25×" tooltip). Fades: drag the small top-corner handles or pick from the clip bar. + +**Effects** (`js/fx.js`, original Web Audio). Per track: EQ (3-band), filter (LP/HP/BP), distortion, compressor, chorus, delay (tempo-synced presets), convolution reverb with **self-generated impulse responses** (seeded decaying noise — no IR files to license). Each effect has presets, and one-click chain presets: Small room, Big hall, Telephone, Radio, Lo-fi, Vocal polish, Punchy drums, Dreamy, Dub echo. Master bus: reverb + limiter (ceiling) + volume. Export uses the identical chain. + +**Phase-2 essentials.** Undo/redo (80 steps, snapshot-based, buffers referenced by id), autosave to IndexedDB (project JSON + raw audio, restored on load with a "Start new" option), 7-step onboarding tour on first visit (re-run from `?`), shortcuts help (`?`), clip fades. + +**Extras chosen (high value, low risk).** Drum step sequencer (▦ Steps on any drum clip, 1/16 grid up to 64 steps), tap tempo, a tuner (note + cents needle) in the voice panel, "Voice Lead" built-in synth that follows pitch bends. Skipped for now to avoid bloat: piano-roll editor (next on the list), a separate metronome UI (Click already exists). + +**Voice → any instrument.** "Sing → instrument" with its own *Play as* menu (every keyboard sound, plus "Drums — beatbox mode"); if a Keys track is selected, the voice plays and records onto that track. Pitch tracking moved from YIN to the **McLeod Pitch Method** in the AudioWorklet with a 32 ms window and a 5.3 ms hop (was 43 ms / 10.7 ms). Note tracker with hysteresis on pitch (±0.7 st band, 3 frames to start, 4 to change, 6 to stop) and on level (gate opens at the sensitivity threshold, closes at 60 %), velocity from loudness, optional **Glide** (live pitch-bend on built-in synths; sampled instruments take the start offset). Press ● to record the notes. **Convert a vocal clip to instrument**: clip menu → offline MPM over the clip → notes on a new Keys track (uses the voice panel's instrument, scale snap and octave). +CREPE-tiny was considered: the model is MIT-licensed, but it adds ~2 MB of weights + TF.js/ONNX runtime and a 64 ms+ analysis window, so it would *raise* latency. MPM is the lower-latency choice for live play; CREPE stays a Phase-3 option for offline conversion accuracy. + +Tests: headless Chrome with a fake mic (synthesised sung melody C-D-E-G-E-C): voice played 60 62 64 67 64 60; clip conversion produced 64 67 64 60 60 62 (first note clipped by the fake device start). See the v2 report for full results. + +### Phase 2 — remaining + +- Project list + save/load project files (JSON + audio blobs). (Autosave + undo/redo done in v2.) +- Piano-roll editor for keys clips, velocity lanes. (Drum step grid done in v2.) +- Send/return reverb buses; automation. (Per-track FX + master done in v2.) - Self-host (or R2/CDN-mirror) the streamed piano/soundfont samples for reliability and preload the default sounds; progress UI on first load. - Loop library v2: tempo/key metadata, auto-fit on drop, more CC0 genres (guitar/bass/keys/vocal chops); search. - Latency calibration wizard (tap-along) for mic takes and voice-to-instrument. @@ -57,7 +87,13 @@ website/studio/ js/instruments.js keyboard sounds (smplr: Splendid piano, e-pianos, FluidR3 GM) + built-in synths js/clipfx.js reverse, WSOLA time-stretch, pitch shift, waveform peaks js/voice.js mic capture, note tracker, beatbox classifier + training - js/voice-worklet.js AudioWorklet: capture + YIN + band energies (audio thread) + js/voice-worklet.js AudioWorklet: capture + McLeod pitch (MPM) + band energies (audio thread) + js/keymap.js compact (v2) and classic computer-keyboard layouts + js/fx.js per-track effects, generated reverb IRs, presets, master bus + js/store.js undo/redo snapshots + IndexedDB autosave + js/importers.js URL import (+ proxy fallback) and tab/system-audio capture + js/pitch.js MPM + note tracker (offline vocal -> notes) +website/api/fetch-audio.js Vercel function: CORS fallback for URL import (preview only) js/piano.js, pads.js, midi.js, loops.js, audio.js vendor/smplr-1.1.0.mjs (MIT) samples/ CC0 / PD one-shots + loops (5 MB, ogg + m4a for Safari) @@ -74,9 +110,11 @@ playback and WAV export on exactly the same code path. - Samples: Splendid Grand (PD), Wurlitzer by Greg Sullivan (CC-BY 3.0, attributed), VCSL TX81Z (CC0), FluidR3_GM (MIT / CC-BY 3.0 renderings), Sonic Pi pack (CC0), TR-808 & LM-2 (PD per distributor). - Excluded: MusyngKite/FatBoy (CC-BY-SA), Hydrogen kits (GPL), Drum Abuse (no license found), amen/breakbeat loops (commercial-record provenance). -## Known limits of the MVP -- No undo, no autosave (a leave-page warning guards unsaved work). +## Known limits +- Autosave lives in this browser only (IndexedDB); no cloud sync or project files yet. Undo history is per tab and holds processed audio in memory. - Sampled instruments need network on first use; built-in synths/kits work offline. -- Voice-to-instrument latency ~50–70 ms (analysis window + confirmation frames); monophonic; tuned on synthetic test signals only so far. +- Voice-to-instrument latency ~35–50 ms (32 ms window + 3 × 5.3 ms confirmation); monophonic; tuned on synthetic test signals only so far. +- Glide bends only the built-in synths live (smplr samplers can't be re-pitched after note-on). +- Tab capture needs Chrome/Edge on desktop; Safari/Firefox have no tab-audio capture. The URL proxy only exists on the Vercel preview; on GitHub Pages, CORS-blocked links show a "download it and drag it in" message. - Stretch/pitch processing runs on the main thread (fine for loops; a few seconds for long songs). - Web MIDI isn't available in Safari/iOS. diff --git a/website/api/fetch-audio.js b/website/api/fetch-audio.js new file mode 100644 index 0000000..1b9b113 --- /dev/null +++ b/website/api/fetch-audio.js @@ -0,0 +1,89 @@ +// Vercel serverless function (preview project only — GitHub Pages ignores +// it). Fetches a public audio file for the Studio's "Import from URL" when +// the hosting site doesn't send CORS headers. +// Guard rails: same-site callers only, http(s) only, no private/loopback +// addresses (checked after DNS and on every redirect), audio content +// types only, 30 MB cap, 20 s timeout, and streaming services refused. +const dns = require("dns").promises; +const net = require("net"); + +const MAX_BYTES = 30 * 1024 * 1024; +const BLOCKED = /(^|\.)(youtube\.com|youtu\.be|googlevideo\.com|spotify\.com|scdn\.co|soundcloud\.com|sndcdn\.com|music\.apple\.com|tidal\.com|deezer\.com|netflix\.com|music\.amazon\.)/i; +const OK_TYPES = /^(audio\/|video\/(ogg|webm|mp4)|application\/(ogg|octet-stream))/i; + +function privateIp(ip) { + if (net.isIPv4(ip)) { + const [a, b] = ip.split(".").map(Number); + return a === 10 || a === 127 || a === 0 || (a === 169 && b === 254) || (a === 172 && b >= 16 && b <= 31) || + (a === 192 && b === 168) || (a === 100 && b >= 64 && b <= 127) || a >= 224; + } + const v = ip.toLowerCase(); + return v === "::1" || v === "::" || v.startsWith("fc") || v.startsWith("fd") || v.startsWith("fe80") || + (v.startsWith("::ffff:") && privateIp(v.slice(7))); +} + +async function checkUrl(u) { + if (!/^https?:$/.test(u.protocol)) throw new Error("Only http(s) links are allowed."); + if (BLOCKED.test(u.hostname)) throw new Error("Streaming services can't be fetched. Use tab capture for audio you have rights to."); + const host = u.hostname.replace(/^\[|\]$/g, ""); + const addrs = net.isIP(host) ? [{ address: host }] : await dns.lookup(host, { all: true }); + if (!addrs.length || addrs.some((a) => privateIp(a.address))) throw new Error("That address isn't reachable from here."); +} + +function sameSite(req) { + const site = req.headers["sec-fetch-site"]; + if (site === "same-origin") return true; + const origin = req.headers.origin || ""; + return /^https:\/\/([a-z0-9-]+\.)*(mydawsons\.com|vercel\.app)$/i.test(origin) && /mydawsons/i.test(origin); +} + +module.exports = async (req, res) => { + const fail = (code, error) => { + res.statusCode = code; + res.setHeader("content-type", "application/json"); + res.setHeader("cache-control", "no-store"); + res.end(JSON.stringify({ error })); + }; + if (req.method !== "GET") return fail(405, "GET only"); + if (!sameSite(req)) return fail(403, "This helper only serves the MyDawsons Studio."); + let target; + try { + target = new URL(String(new URL(req.url, "http://x").searchParams.get("url") || "")); + } catch { + return fail(400, "Missing or invalid url."); + } + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), 20000); + try { + let r; + for (let hop = 0; hop < 4; hop++) { + await checkUrl(target); + r = await fetch(target.href, { redirect: "manual", signal: ctrl.signal, headers: { "user-agent": "MyDawsonsStudio/2 (+https://mydawsons.com/studio/)" } }); + if (r.status >= 300 && r.status < 400 && r.headers.get("location")) { + target = new URL(r.headers.get("location"), target); + continue; + } + break; + } + if (!r || !r.ok) return fail(502, `The site answered ${r ? r.status : "nothing"}.`); + const type = r.headers.get("content-type") || "application/octet-stream"; + if (!OK_TYPES.test(type)) return fail(415, "That link isn't an audio file (it looks like a web page). Link straight to the .mp3/.wav/.ogg file."); + const len = Number(r.headers.get("content-length") || 0); + if (len > MAX_BYTES) return fail(413, "That file is over 30 MB."); + const chunks = []; + let total = 0; + for await (const chunk of r.body) { + total += chunk.length; + if (total > MAX_BYTES) return fail(413, "That file is over 30 MB."); + chunks.push(chunk); + } + res.statusCode = 200; + res.setHeader("content-type", type); + res.setHeader("cache-control", "private, max-age=600"); + res.end(Buffer.concat(chunks.map((c) => Buffer.from(c)))); + } catch (e) { + return fail(e.name === "AbortError" ? 504 : 400, e.name === "AbortError" ? "The site took too long." : e.message || "Fetch failed."); + } finally { + clearTimeout(timer); + } +}; diff --git a/website/studio/SOUND_LIBRARY.md b/website/studio/SOUND_LIBRARY.md index ca7b99e..c3794e0 100644 --- a/website/studio/SOUND_LIBRARY.md +++ b/website/studio/SOUND_LIBRARY.md @@ -41,6 +41,10 @@ the **Drum Abuse** packs (no license statement found), **Hydrogen** drum kits - **Lo-fi Dusty** processing (sample-rate reduction, bit-crush, low-pass, soft clip) applied to CC0/PD one-shots. - Built-in keyboard synths: Analog Synth, Sub Bass, Chiptune (`js/instruments.js`). - Metronome click. +- **Reverb impulse responses (v2)**: generated in the browser by `impulse()` in `js/fx.js` (seeded decaying filtered noise + early reflections). No third-party IR files are used, so there's nothing to license. +- **Voice Lead** built-in synth and every effect (EQ, filter, distortion, compressor, chorus, delay, master limiter) — original Web Audio code. + +v2 added **no new third-party samples or code**. ## Code dependencies @@ -50,12 +54,18 @@ the **Drum Abuse** packs (no license statement found), **Hydrogen** drum kits | Web Audio API, AudioWorklet, Web MIDI, MediaDevices | — | Browser platform APIs | — | No Tone.js is used in this MVP — raw Web Audio + smplr covers scheduling, -sampling and export with fewer moving parts. WSOLA time-stretch, YIN pitch -tracking and the beatbox classifier are original implementations of -well-known public techniques. +sampling and export with fewer moving parts. WSOLA time-stretch, McLeod +pitch tracking (MPM; YIN in v1) and the beatbox classifier are original +implementations of well-known published techniques. CREPE was evaluated +but not shipped (see docs/STUDIO_PLAN.md). ## User content -Files a user uploads or records stay in their browser tab. Nothing is -uploaded to a server. Users are responsible for rights to material they -import (e.g. samples from commercial records). +Files a user uploads, records or captures from a tab stay in their browser +(autosaved to IndexedDB on their device). Nothing is uploaded to a server. +"Import from link" downloads the file into the browser; when a site blocks +cross-origin requests, the preview's `/api/fetch-audio` helper fetches it +server-side and passes it straight through without storing it. Streaming +services are refused, and the UI tells users to only use sounds they have +the rights to. Users are responsible for rights to material they import +(e.g. samples from commercial records). diff --git a/website/studio/index.html b/website/studio/index.html index c1c314c..ce2da97 100644 --- a/website/studio/index.html +++ b/website/studio/index.html @@ -27,19 +27,29 @@
+ - + - +
-
+
+ + + +
+ + + + +
@@ -48,9 +58,9 @@
- + Keys: C4 - +
@@ -67,7 +77,7 @@

🥁 Drum pads

-

Tap the pads, or use keys 1–6 · Q–Y · A–H. Tap nearer the bottom of a pad to hit harder.

+

@@ -76,11 +86,12 @@

🥁 Drum pads

- +
+ + + + + + + +