An infinite whiteboard you control with your hands and your voice. Reach out and grab an idea, make a fist to drag the board, throw a note away to delete it, and just speak to write.
Overview · How it works · Interaction model · Screenshots · Architecture · Quickstart · Documentation
Whiteboards and digital canvases assume you sit at a desk with a mouse and keyboard, or that you can walk up and touch the surface. That breaks down when you are standing back from a screen, your hands are busy, or touching the display is impractical.
Mindboard is a Natural User Interface (NUI) answer to that: you stand in front of a webcam and a microphone and manipulate the ideas directly, with your hands in the air and your voice, so the computer disappears and the interaction becomes reach, grab, point, and speak. Everything runs in the browser, on the device; there is no server and nothing leaves the machine. A mouse works too, so the app is fully usable without a camera or microphone.
Hands do the spatial work; voice does the content work. They never fight for the same job.
Hand poses are classified from finger geometry, how many fingers are extended, which stays stable regardless of how the hand is turned. The trained MediaPipe classifier is used only for thumbs up / down, because a thumb reads as zero extended fingers and geometry alone cannot see it. The cursor tracks the palm centre for every pose, so it never jumps when a finger curls.
| Pose | Action |
|---|---|
| Point (index only) | laser pointer, becomes the pen in draw mode |
| Two fingers on a note | pick it up and move it (a fast fling deletes it, undoable) |
| Fist | pan the board |
| Open hand | release |
| Two fists | zoom the board |
| Two "victory" hands | resize the note between them |
| Thumbs up / down | confirm / cancel dictation |
| Say | Action |
|---|---|
| "record" | create a note at the cursor and start dictating |
| "edit note" | dictate into the selected note |
| "save note" | finish dictating |
| "grab" / "drop" | attach the selected note to your hand, then drop it |
| "draw" / "stop drawing" | enter / leave draw mode |
Command words are stripped from the dictated text, and a short toast confirms each command the moment it is heard. Speech is written incrementally onto the note, so you can type and dictate on the same note without one overwriting the other, and a word you delete mid-dictation does not come back.
Move your hand (or drag the mouse) to draw a smoothed line. An open hand lifts the pen, a held fist erases. Hold thumbs-up for two seconds to save and leave, thumbs-down to discard. With the mouse, click a stroke, or drag a box around several, and press Delete.
Double-click empty canvas to create a note (dictation starts automatically), double-click a note to edit it, drag to move, drag a box to select, and drop an image file onto the canvas to import it. Trackpad and touch pinch-zoom and two-finger pan are supported.
Every input funnels through one small state machine, and each intent has a single distinct gesture or command, so modes can never collide.
Application states and the gestures / commands that move between them.
How a detected hand maps to an action.
The infinite canvas, sticky notes, imported images, minimap, three themes.
Hand tracking on: the palm-centre cursor and the live camera preview with the tracked hand.
Mindboard is a 100% client-side single-page app, no backend, no database, no cloud. The webcam and microphone streams are processed in the browser and never leave the device.
Webcam ─▶ MediaPipe GestureRecognizer ─▶ pose engine (finger geometry + palm cursor)
Mic ─▶ Web Speech API ─▶ command parser + dictation state machine
Mouse ─▶ pointer handlers │
▼
Zustand store ⇄ React Flow canvas ⇄ SVG ink layer
│
└─▶ localStorage (persist)
The pose engine (src/hands/handPose.ts) and the voice-command parser (src/voice/voiceCommands.ts) are pure and covered by unit tests.
The build followed the course's design pipeline: sketches and a low-fidelity wireframe, an interaction storyboard, then implementation and evaluation.
Low-fidelity wireframe, layout and interaction intent.
Storyboard, the hands-free path from idea to arranged note.
The "grab" / "drop" voice-and-hand interaction is a modern take on Bolt's 1980 "Put-That-There": the voice supplies the verb, the hand supplies the location.
- Vite + React + TypeScript, strict, no runtime framework beyond React
- React Flow (
@xyflow/react), the infinite canvas - Zustand, state, persisted to
localStorage - Web Speech API, voice commands and dictation (Chromium)
- MediaPipe Tasks Vision, in-browser hand landmarks and gesture classification
- lucide-react, icons
- Vitest, unit tests for the pose and voice-command logic
Requires Node 18+ and a Chromium-based browser (Chrome / Edge), voice uses the Web Speech API, which only ships in Chromium. The webcam and microphone are optional; the app falls back to mouse and keyboard.
npm install
npm run dev # start the dev server (http://localhost:5173)
npm run build # type-check and build for production
npm run preview # preview the production build
npm test # run the unit testssrc/
canvas/ infinite canvas, ink layer, node types
hands/ webcam tracking, pose engine, gesture-to-action mapping
voice/ speech recognition, command parsing, dictation
nodes/ note and image node components
store/ Zustand board store (notes, strokes, viewport, undo)
components/ top bar, status bar, help overlay, toasts
styles/ design tokens, themes, React Flow theme
DOCUMENTATION.md, technical documentation