╔══════════════════════════╗
        ║  ┌──────────────────┐   ║
        ║  │  ◈  HOLOLINK  ◈  │   ║
        ║  └──────────────────┘   ║
        ║    ╱|  ╱╲  ╱╲  |\╲     ║
        ║   ╱ | ╱  ╲╱  ╲ | ╲╲    ║
        ║  ●──●──●──●──●──●──●   ║
        ╚══════════════════════════╝

Hololink

AR-gesture control surface for a 6-DOF robotic arm

Demo  •  Features  •  Quick Start  •  Architecture  •  AR Commands  •  Configuration  •  Development  •  Tech Stack

---

Demo

Hololink overlays a live Three.js robot arm on the camera feed — pinch gestures and ArUco markers drive real-time 6-DOF inverse kinematics

---

Features

Feature	Description
AR scene overlay	Three.js 6-link robot arm rendered transparently over the live video feed via react-three-fiber
Gesture control	Pinch, spread, and point gestures detected from webcam feed drive IK target position
ArUco marker registry	Register fiducial markers (IDs 0–255) and bind each to a named arm command
Inverse kinematics	Damped least-squares IK solver (UR5-style DH params) with joint-limit clamping
30 Hz telemetry stream	WebSocket ticker pushes joint state at 30 Hz; frontend renders live sparkline charts
Calibration wizard	3-step wizard saves a 4×4 camera↔world homogeneous transform to the backend
Session history	AR sessions are persisted in SQLite with frame + command counts
One-command setup	docker compose up --build boots the full stack

---

Quick Start

Prerequisites

• Docker Desktop or Docker Engine + Compose v2
• A webcam (optional — the app runs in synthetic vision mode without one)

Run

git clone <your-remote-url> hololink
cd hololink
cp .env.example .env
docker compose up --build

Open http://localhost:3000 — the AR interface loads immediately.
API docs at http://localhost:8000/docs.

Without a webcam the video feed shows a placeholder and the backend generates synthetic marker positions and fingertip poses automatically.

---

Architecture

Browser                          Backend (FastAPI)
┌─────────────────────────┐      ┌──────────────────────────────┐
│  VideoFeed               │      │  /ws/control  (WebSocket)    │
│  ─ getUserMedia()        │ ──── │  vision.detect()             │
│  ─ 100ms frame tick  ────┼──WS─▶│  command_mapper.map()        │
│                          │      │  arm_state.apply_command()   │
│  ARScene (Three.js)      │◀──WS─│  arm_state.tick() @ 30 Hz   │
│  ─ RobotArm (6-DOF)      │      │  telemetry.push()            │
│  ─ OrbitControls         │      └──────────────────────────────┘
│                          │
│  MarkerOverlay (canvas)  │      ┌──────────────────────────────┐
│  CommandPalette          │ HTTP │  REST routes                 │
│  JointPanel              │─────▶│  /sessions /markers          │
│  TelemetryChart          │      │  /robot /calibration         │
│  SessionList             │      └──────────────────────────────┘
│  CalibrationWizard       │
└─────────────────────────┘      SQLite  (/data/hololink.db)

Data flow:

1. VideoFeed sends a {type:"frame", ts} message every 100 ms over the WebSocket
2. The backend calls vision.detect() — in synthetic mode this produces animated marker positions + fingertip keypoints
3. command_mapper classifies the gesture (pinch/spread/point) and any bound marker action
4. If a command fires, arm_state.apply_command() queues a waypoint trajectory
5. A 30 Hz async ticker steps through waypoints and broadcasts {type:"joint"} messages
6. The frontend store receives joint angles → RobotArm re-renders in Three.js

---

AR Commands

Gesture mapping

Gesture	Detected by	Command
Pinch (thumb + index close)	Fingertip distance < threshold	grip
Open hand (spread > 80 px)	Thumb–pinky spread	wave
Pinch release	Fingertip distance > release threshold	release

Marker binding

Register any ArUco marker ID with the REST API and bind it to a command:

curl -X POST http://localhost:8000/markers/ \
  -H 'Content-Type: application/json' \
  -d '{"marker_id":7, "label":"Extend", "action":"extend", "size_mm":60}'

Preset commands

Command	Description
home	Return all joints to zero position
grip	Close gripper fully
release	Open gripper fully
wave	Execute wave motion sequence
extend	Extend arm to maximum reach
retract	Retract arm to compact position
tuck	Tuck arm to side-safe position

---

Configuration

Variable	Default	Description
DATABASE_URL	sqlite:////data/hololink.db	SQLite path (mounted volume)
CORS_ORIGINS	http://localhost:3000	Allowed CORS origins (comma-separated)
ARM_DOF	6	Degrees of freedom
IK_MAX_ITER	150	IK solver maximum iterations
IK_DAMPING	0.05	Damped least-squares damping factor
IK_TOLERANCE	0.001	IK convergence tolerance (m)
TELEMETRY_BUFFER_SIZE	500	Rolling joint-state buffer length
SYNTHETIC_VISION	true	Use synthetic marker/fingertip data when no real frame is provided
NEXT_PUBLIC_API_BASE	http://localhost:8000	Backend REST base URL
NEXT_PUBLIC_WS_BASE	ws://localhost:8000	Backend WebSocket base URL

---

Development

Backend

cd backend
pip install -e ".[dev]"
# run
uvicorn app.main:app --reload
# test
pytest

Frontend

cd frontend
npm install
cp .env.local.example .env.local
npm run dev

Generate a fresh placeholder GIF

convert -size 1280x720 \
  -delay 40 xc:'#020617' \
  -delay 40 xc:'#0f172a' \
  -delay 40 xc:'#1e293b' \
  -loop 0 docs/demo/hololink-demo.gif

---

Tech Stack

Layer	Technology
Frontend framework	Next.js 14 (App Router)
Language	TypeScript 5
Styling	Tailwind CSS 3
3D / AR	Three.js 0.167 + react-three-fiber + drei
State	Zustand 4
Animation	Framer Motion 11
Backend framework	FastAPI 0.115
ORM / DB	SQLModel + SQLite
Numerics	NumPy (IK solver)
Runtime	Python 3.12 / Node 20
Container	Docker + Compose v2

---

License

MIT © 2026 LeDucDiLac