Course: Natural User Interfaces (NUI) University: HTW Berlin Project: Mindboard, a gesture and voice driven infinite whiteboard
Mindboard is an infinite whiteboard. You create sticky notes, move them around freely, zoom in and out, draw on the board, and dictate text into notes. The goal is to do all of this without menus, toolbars, or keyboard shortcuts, using your hands (via webcam hand tracking), touch gestures, and your voice.
The idea behind it is that the interface should get out of the way. There are no buttons to hunt for and no menus to navigate. You point, grab, speak, and draw. A mouse and keyboard still work, so the app is usable even without a camera or microphone.
| Layer | Technology |
|---|---|
| Framework | React 18 + TypeScript |
| Build tool | Vite 5 |
| Infinite canvas | React Flow v12 (@xyflow/react) |
| State management | Zustand |
| Hand tracking | MediaPipe Tasks Vision (@mediapipe/tasks-vision) |
| Voice input | Web Speech API |
| Icons | lucide-react |
| Styling | CSS custom properties (design tokens) |
| Tests | Vitest |
I chose React Flow because it handles the low level canvas work (coordinate transforms, hit testing, the minimap, pan and zoom), so I could focus on the gesture and voice layer on top instead of rebuilding node rendering and viewport math.
MediaPipe runs entirely in the browser through WebAssembly and WebGL. There is no server and no cloud API. The model downloads once and then detects hands in real time.
Zustand keeps state simple: one store, no boilerplate, and it saves to localStorage, so a board survives a page reload.
src/
main.tsx entry point, loads the theme, mounts App
App.tsx top level: voice, hand tracking, canvas, bars
canvas/
Canvas.tsx infinite canvas (React Flow + touch gestures + mouse drawing)
DrawLayer.tsx SVG ink layer for freehand strokes
nodeTypes.ts node registry ("note" and "image")
nodes/
NoteNode/ editable text card (dictation, resize)
ImageNode/ draggable and resizable image node
imageUtils.ts image file to downscaled data URL
hands/ hand tracking
HandTracker.ts MediaPipe lifecycle (landmarks + thumb labels)
handPose.ts finger-geometry pose engine + cursor anchor + latch
useHandTracking.ts hook: webcam to screen-space cursors
useHandActions.ts maps poses to canvas actions
HandActions.tsx thumbs hold-to-confirm for dictation and draw
HandOverlay.tsx camera preview + hand skeleton + cursor
handCursorShared.ts shared cursor / draw-mode / voice-grab state
gestures/ touch and trackpad
useGestures.ts two-finger pinch-zoom and pan
useLongPress.ts press and hold to create a note
GestureHint.tsx zoom percentage flash
voice/
useVoiceControl.ts Web Speech command machine + dictation
voiceCommands.ts pure command parsing + dictation reconciliation
VoiceContext.ts React context for voice state
store/
boardStore.ts Zustand store: notes, strokes, viewport, theme, undo
components/
TopBar.tsx brand, skins, hands, voice, draw, fit, reset, theme
StatusBar.tsx zoom, note count, hand/voice status
HintOverlay.tsx first-run gesture and voice guide
UndoToast.tsx undo-delete toast
styles/
tokens.css design tokens (light/dark)
skins.css the three visual skins
reactflow-theme.css React Flow overrides
There are three input paths: hands (webcam), voice (microphone), and mouse or touch. Hands do the spatial work (move, pan, zoom, resize, draw). Voice does the content work (create a note, dictate, edit). Mouse and touch can do everything as a fallback.
Turn it on with the "hands" button in the top bar. The first time, it downloads the MediaPipe model and asks for camera permission, then runs in real time.
While it is on you see two things. A small camera preview in the corner shows the webcam feed with the tracked hand drawn on top (hidden by default, toggled with the camera button). A cursor on the board follows your hand. The cursor position is the centre of the palm for every pose, so it stays steady when you change pose. An earlier version tracked the fingertip for some poses and the palm for others, which made the cursor jump; using one fixed point fixed that.
Poses and their actions:
| Pose | Fingers | Action |
|---|---|---|
| Point | index only | laser pointer (the pen in draw mode) |
| Grab (victory) | index + middle | pick up and move the note under the cursor |
| Fist | none extended | pan the board |
| Open hand | three or more | release |
| Two fists | both hands | zoom the board |
| Two victory hands | both hands | resize the note between them |
| Thumbs up / down | trained label | confirm / cancel dictation (or save/discard in draw mode) |
A few rules keep these from interfering:
- Panning the board and moving a note are different gestures, so one never triggers the other.
- The two-hand modes only fire when both hands show the same pose, so zoom and resize can never be confused.
- Single-hand actions only fire when exactly one hand is visible.
- Point does nothing on the board on its own; it is only the laser pointer, and the pen in draw mode. This is deliberate: while a hand changes from pointing to the victory pose it briefly passes through the point shape, and giving point a board action made the cursor twitch during that transition.
- To delete a note, grab it and release while flinging it quickly (over 42 pixels per frame). This is undoable.
Notes are never created or edited by a hand pose. That is done by voice (section 4.3). Early versions created notes with a point-and-hold dwell and stopped dictation with a held open palm. Both misfired in testing, so hand-driven create and edit were removed and hands kept the continuous spatial work.
How poses are detected (handPose.ts):
Poses come from counting extended fingers, not from a trained gesture label. For each finger the engine measures the joint angle and decides extended or curled (EXTEND_COS = 0.45), which is stable no matter which way the hand is turned. Zero extended fingers is a fist, index only is a point, index and middle is grab, three or more is an open hand. The trained MediaPipe classifier is used only for thumbs up and thumbs down, because a thumb reads as zero extended fingers and the geometry cannot see it.
Earlier versions drove the whole app from MediaPipe's trained gesture labels. On a live webcam those labels flickered from frame to frame (the same steady hand would read point, then fist, then grab), and the labels are orientation trained, so a point aimed downward would drop out. Switching to finger counting removed that instability, because the number of extended fingers does not change frame to frame.
A per-pose latch filters out single-frame noise. It is asymmetric: entering a pose from rest takes two frames (responsive), but switching from one committed pose to another takes six (sticky), so a gesture holds instead of flipping mid-action.
Why the pinch pose was removed: an early version used a pinch (thumb near index) as the grab. On a webcam a real pinch curls the other fingers, so its shape is close to a fist, and the pinch detector kept firing before the fist was recognised, which broke panning and erasing. The fix was a different vocabulary, not another threshold. The victory pose (two extended fingers) cannot be confused with a fist (zero) or an open hand (three or more).
How the cursor works (useHandTracking.ts):
- MediaPipe returns 21 landmarks per hand, normalised to the range 0 to 1, for up to two hands, plus the thumb label and its confidence.
- The cursor is the palm centre (the wrist plus the four knuckle joints, averaged) for every pose.
- The X coordinate is mirrored, so moving your hand right moves the cursor right.
- The coordinates are scaled to screen pixels.
- A light exponential moving average smooths the cursor so it does not vibrate when the hand is held still.
How actions work (useHandActions.ts):
- One victory hand grabs the note under the cursor and moves it; a fast release throws it away.
- One fist pans the board.
- Two fists zoom the board around the midpoint between the hands.
- Two victory hands resize the note under the midpoint. The resize is smoothed and tolerates a few dropped frames so tracking noise does not shake it.
- Point and open hand do nothing on the board (point is the laser, open is release).
- A short grace of a few frames keeps a grab or pan alive across the one or two "no pose" frames that happen mid-transition.
The pipeline:
webcam frame
-> MediaPipe GestureRecognizer.recognizeForVideo()
-> 21 landmarks (+ thumb label) for up to 2 hands
-> handPose.ts: finger-count pose + palm-centre cursor + asymmetric latch
-> useHandTracking: screen-space cursors with pose
-> useHandActions: move / pan / zoom / resize / draw / delete
The canvas is infinite. You move around it with several inputs.
Trackpad and mouse: two-finger scroll pans, trackpad pinch or Ctrl/Cmd with the wheel zooms, and a middle-click or right-click drag pans.
Touch: a two-finger drag pans, and a pinch zooms. The zoom is anchored to the midpoint between the fingers, so it zooms into whatever is between them.
Touch handling lives in useGestures.ts. It attaches raw touchstart, touchmove, and touchend listeners to the canvas wrapper. With two fingers down it tracks the distance between them (for zoom) and their midpoint (for pan), recomputes the zoom as startZoom * (currentDistance / startDistance), and shifts the viewport so the point under the midpoint stays fixed. The listeners are not passive, so it can call preventDefault() and block the browser's own pinch-zoom. All of the gesture state is kept in refs, not React state, so a gesture never causes a re-render. Zoom is limited to 25 to 400 percent.
Voice uses the Web Speech API, driven by a small command state machine in useVoiceControl.ts. Turn it on with the "voice" button. One recogniser then runs continuously.
To create a note hands-free: say "record" (or "new note"). A note appears at the hand cursor if a hand is visible, otherwise at the centre of the view, and dictation starts. Speak the content and it streams into the note. Say "save note" (or "stop recording") to finish.
To edit a note: select it (click it, or grab it with a hand), say "edit note", and the dictation appends to its text. "save note" finishes.
Other commands: "grab" attaches the selected note to your hand so it follows the cursor, and "drop" writes back its position. "draw" and "stop drawing" enter and leave draw mode.
Command words never end up in the note text. The recogniser has two modes. In idle mode it scans for a trigger ("record", "new note") or the edit command and, on a match, strips the command word and keeps any words spoken after it. In dictating mode it routes speech into the active note and scans for a stop word. Because a command only acts in the matching mode, saying "record" inside note content keeps it as text. The command patterns live at the top of voiceCommands.ts and are easy to change.
Two details make voice feel responsive and safe to combine with the keyboard. Commands act on interim (partial) results rather than waiting for the end of speech, which removes the delay that made commands feel sluggish. And dictation is written incrementally onto the note's current text (see reconcileSpeech in voiceCommands.ts): it only ever manages the trailing word it is still hearing, so you can type or delete on the same note without the recogniser overwriting your edit, and a word you delete mid-dictation does not come back when the recogniser finalises.
Feedback: the moment a command is recognised a small toast flashes at the top ("recording", "note saved", and so on), and the note being dictated into shows a recording dot. Voice state is shared through VoiceContext.ts so the note, status bar, and top bar can read it. The hand cursor reaches the voice controller through a small shared module (handCursorShared.ts), which is how "record" drops the note where your hand is.
Moving: click and drag a note (mouse), single-finger drag (touch), or grab it with the victory pose (hand). During a drag React Flow keeps a local copy of the position, and the final position is written to the store and saved on release.
Editing: double-click a note to edit it (this also starts dictation if voice is available). Enter saves, Shift+Enter adds a newline, Escape saves and stops recording, and clicking away saves. The textarea carries a nodrag class so typing and selecting text do not start a drag.
Resizing: each note has a small handle in the bottom-right corner. Drag it with the mouse, or show the victory pose with both hands over a note and move them apart or together.
Enter draw mode with the "draw" button or the voice command "draw". In draw mode the hand is taken over by the pen, and pan and grab are disabled so the pen owns the input.
Move your hand, or drag the mouse, to draw. The pen is down by default; an open hand lifts it so you can reposition without a line, and a held fist erases strokes you touch. Hold thumbs up for about two seconds to save and leave draw mode, or thumbs down to discard the strokes drawn in this session and leave. With the mouse you can also click a stroke, or drag a box around several, and press Delete.
Strokes are stored in board coordinates, so they pan and zoom with the notes and are saved with the board. They are drawn as a smoothed curve (a Catmull-Rom spline rendered as a Bezier path) rather than straight segments, and each stroke's path is cached, so only the stroke being drawn is recomputed while you draw. This is in DrawLayer.tsx.
Throw a grabbed note to delete it (grab with the victory pose, drag, release while flinging fast). Right-click a note for a delete menu. Or select a note and press Backspace or Delete.
Every deletion is reversible, because a hands-free interface can misfire. A toast appears for five seconds and Cmd/Ctrl+Z restores the last deleted note with its id, position, size, and text. The undo stack (the last 50 deletions) lives in the store and is cleared when the tab closes. This is in UndoToast.tsx and boardStore.ts.
Images are normal nodes on the canvas: draggable, resizable, and saved like notes. Import them by dragging image files onto the canvas, or with the "image" button in the top bar. On import each image is downscaled before it is stored as a data URL, so the saved board stays within the localStorage budget. This is in imageUtils.ts and nodes/ImageNode/.
- Command toast: flashes under the top bar when a voice command is recognised.
- Hand cursor: follows the palm, and changes shape with the mode (a laser dot, a pen nib in draw mode).
- Camera preview: a small feed with the hand skeleton, so you can see what the model sees.
- Status bar: zoom, note count, voice state, and the current hand pose.
- Zoom flash: shows the zoom percentage briefly during a touch pinch.
Everything lives in one Zustand store (boardStore.ts):
notes[] { id, x, y, text, width?, height?, kind ('note'|'image'), src? }
strokes[] freehand ink, each { id, points[], color, size }
viewport { x, y, zoom }
theme 'light' | 'dark'
skin 'krnl0' | 'minimal' | 'bold'
selectedId selected note
editingId note in edit mode
selectedStrokeIds strokes selected with the mouse
deletedStack last 50 deleted notes (for undo)
The store saves to localStorage on a 250 millisecond debounce after any change, and restores from it on load. The theme and skin are stored separately so they can be applied before the first render and avoid a flash of the wrong style.
There is a light and a dark theme, toggled from the top bar. On top of that there are three skins, chosen from a segmented control: Terminal (a green, terminal look), Minimal (neutral, indigo accent), and Bold (violet and cyan). Each skin is a set of CSS custom-property overrides in skins.css, applied through a data-skin attribute, so every component that already uses the tokens restyles for free. The accent colour is --acid (#c9f158 in the default skin) and is used for the selection ring, the recording indicator, and the hand cursor. Both the theme and the skin are saved.
- Direct manipulation. Notes are objects you grab and move, not rows in a list. The canvas is a space, not a document.
- Gesture navigation. Pinch to zoom and two-finger pan feel like handling a physical map, with no sliders or scroll bars.
- Hand tracking. You point at things, grab a note with two fingers, and make a fist to slide the board. The vision model runs locally in the browser.
- Voice for content. Saying "record" and speaking is faster than typing, and command words are stripped from the text.
- Minimal chrome. There are no floating panels; the top bar is a brand and a few small buttons.
- Division of labour. Hands do the spatial work and voice does the content work, and they combine: point at empty space, say "record", and the note appears under your finger and fills with your words.
- Little instruction needed. Each action maps to a simple hand shape and a physical metaphor (two fingers to pick up, fist to grab the board, open hand to release, speak to write), and a one-screen first-run guide covers the rest.
npm install
npm run dev # start the dev server, then open the printed URL (usually http://localhost:5173)
npm run build # type-check and build for production
npm test # run the unit tests (pose engine + voice parsing)For full functionality you need a Chromium-based browser (Chrome or Edge) for the Web Speech API and MediaPipe, a webcam for hand tracking, and a microphone for voice. None of the three are required for mouse and keyboard use. The first load fetches the MediaPipe runtime and model from a CDN and then caches them.
The pose engine went through a few generations. The first used a thumb-to-index distance for pinch, which was too noisy on a real webcam. A later one drove everything from MediaPipe's trained gesture labels, but those flickered frame to frame and are orientation trained, so a steady hand kept changing pose and a downward point dropped out. The current engine classifies poses by counting extended fingers, which is stable and orientation independent, and keeps the trained model only for thumbs up and down, which finger counting cannot see. Combined with the asymmetric latch, the pose no longer flips mid-action.
Several hand-driven ways to create a note were tried and all failed in testing: a double pinch (unreliable), a point-and-hold with a charging ring (fired by accident and blocked the cursor), and a single "grab" gesture for everything (it mixed moving a note with panning the board and broke zoom). The takeaway was that webcam gestures are good at continuous spatial manipulation but bad at discrete commands. So the discrete intents (create, edit, save) moved to voice, where a spoken word is unambiguous, and hands kept the continuous work. That split is the main interaction decision in the project.
A fingertip sits in a different place for each pose and jitters as the finger curls, so anchoring the cursor to it made the cursor jump on every pose change. The palm centre barely moves as the fingers open and close, so using it for every pose keeps the cursor steady.
I considered running MediaPipe in a Web Worker to keep the main thread free, but recognizeForVideo() needs the video element and its GPU context, which are not available in a worker. Detection runs on the main thread through requestAnimationFrame. Each frame takes only a few milliseconds on the GPU, which is fast enough to avoid jank.
- Voice needs a Chromium browser; the Web Speech API is not in Firefox or Safari.
- Hand tracking needs WebGL and runs poorly without it.
- The first load downloads the MediaPipe model from a CDN, then caches it.
- Undo covers deletion only; full undo and redo of moves and edits would need a command history.
- There is no multi-user collaboration; everything is local to the browser.
| File | What it does |
|---|---|
main.tsx |
Loads the saved theme and skin, mounts the app |
App.tsx |
Wires up voice, hand tracking, the canvas, and the bars |
Canvas.tsx |
Configures React Flow, touch gestures, mouse drawing, box-select, context menu |
DrawLayer.tsx |
Renders the freehand ink as smoothed, cached paths |
nodeTypes.ts |
Maps "note" and "image" to their components |
NoteNode.tsx |
A note: display and edit modes, dictation, resize |
ImageNode.tsx |
A draggable, resizable image node |
imageUtils.ts |
Reads an image file, downscales it, returns a data URL and size |
HandTracker.ts |
MediaPipe lifecycle: load, start and stop the webcam, per-frame landmarks and thumb label |
handPose.ts |
Finger-geometry pose engine, palm-centre cursor, asymmetric latch |
useHandTracking.ts |
Hook: webcam to screen-space cursors with pose |
useHandActions.ts |
Maps poses to canvas actions (move, pan, zoom, resize, draw, delete) |
HandActions.tsx |
Thumbs hold-to-confirm for dictation and draw mode |
HandOverlay.tsx |
Camera preview, hand skeleton, on-screen cursor |
handCursorShared.ts |
Shared cursor, draw-mode and voice-grab state |
useGestures.ts |
Touch and trackpad pinch-zoom and pan |
useLongPress.ts |
Press and hold to create a note |
GestureHint.tsx |
Zoom percentage flash during a pinch |
useVoiceControl.ts |
Web Speech command machine and dictation |
voiceCommands.ts |
Pure command parsing and dictation reconciliation (unit tested) |
VoiceContext.ts |
React context for voice state |
boardStore.ts |
Zustand store: notes, strokes, viewport, theme, skin, undo, persistence |
TopBar.tsx |
Brand, skins, hands, image, voice, draw, fit, reset, help, theme |
StatusBar.tsx |
Zoom, note count, voice and hand status, hints |
HintOverlay.tsx |
First-run gesture and voice guide |
UndoToast.tsx |
Undo-delete toast |
tokens.css |
Design tokens for the light and dark themes |
skins.css |
The three visual skins |
reactflow-theme.css |
React Flow styling overrides |