Pattern-mediated synesthesia with Strudel, Hydra, and a camera pointed at the room.
Loud means big. Bass means blue. Beat means flash.
Every audio-reactive visual you've seen does the same three things. That's not synesthesia. That's a screensaver.
The sound already knows what it is. A chord. A rhythm. The third note in a sequence of five. The camera already sees the room. None of that reaches the screen.
We're wiring it together.
Strudel for the sound. Hydra for the visuals. A camera pointed at the room. One pattern driving all of it. What you hear shapes what you see. The way you move shapes what you hear.
The crowd becomes the instrument. The room becomes the screen.
Pick your entry path. Don't read all four.
| You are | Go to |
|---|---|
| Never live-coded before | Challenge 01 — Pattern Ears |
| Used Strudel / TidalCycles before | Challenge 02 — Pattern Eyes |
| Shader / creative coding background | Challenge 03 — Crowd as Texture |
| Want the hard problem | Challenge 04 — Closed Loop |
| Have your own project | Use lib/ and references/, build whatever |
No setup. Open strudel.cc, paste, hear it. Learn mini-notation, layer sounds, apply pattern transformations. Done when: you'd let your pattern play for 5 minutes untouched.
Still in the REPL. Drive Hydra visuals from Strudel's pattern structure — not FFT, not amplitude. One mini-pattern, two outputs (sound + visual). Done when: changing the musical structure visibly changes the visual.
Local setup. Webcam → ml5 body segmentation → Hydra source. The humans in the room become the visual medium, isolated and rhythmically transformed. Done when: people in the room are inside the visual.
Full feedback. Optical flow on the segmentation mask → Strudel control signal → pattern → audio + visual. Movement shapes sound. Sound shapes visual. Visual is the crowd. Done when: someone waves and both the music and the projection respond.
- Open https://strudel.cc
- Paste the snippets from the challenge README
- Hit
Ctrl+Enterto evaluate
Webcam APIs require localhost or HTTPS. They won't work from file://.
git clone https://github.com/vibecodingnights/synthesia.git
cd synthesia
npx serve .
# open http://localhost:3000/challenges/03-crowd-texture/Chrome. Allow webcam when prompted. First load downloads ~5MB of ml5 weights.
challenges/ ← the four challenges, each self-contained
lib/ ← optical-flow.js, seg-to-hydra.js — utilities for Ch 03/04
references/ ← strudel, hydra, ml5 cheatsheets; sharp-edges doc
projector/ ← the ambient visual running on the venue projector all night
- Strudel cheatsheet — mini-notation, pattern functions,
onDraw,queryArc - Hydra cheatsheet — functions,
H()/P(),.src(), blending - ml5 bodySegmentation — v1.0 API, mask types, gotchas
- Sharp edges — AudioContext, HTTPS, FFT, ml5 v1.0 breaks
External:
- Strudel REPL — https://strudel.cc
- Strudel docs — https://strudel.cc/learn/
- Hydra book — https://hydra-book.glitch.me/
- Flok (multiplayer live coding) — https://flok.cc (opt-in, session URL on the whiteboard)
Date: TBD · 6:00–10:00 PM · Frontier Tower SF
6:00–6:15 Arrival. Projector is running. Repo URL on the board.
6:15–6:20 5-minute intro. The whole pitch.
6:20–9:30 Build. No check-ins. Hosts float.
9:30–10:00 Opt-in demos. 2–3 minutes each. "I got stuck on X" counts.
Bring a laptop. Headphones on the tables. You don't need to know what Strudel is. You'll know by midnight.