AI Gesture-Controlled Racing — Steer With Your Hands, No Controller Needed
Games · Features · How It Works · Architecture · Tech Stack · Getting Started
GestureKart AI Racing is a browser racing experience with a twist: your hands are the steering wheel. Two games in one — a 3D night-drive racing sim controlled entirely by webcam hand gestures, plus a classic arcade Kart Racing mode. Built with Three.js for 3D rendering and MediaPipe Hands for real-time hand tracking.
Race through a neon-lit cyberpunk tunnel at 60 FPS — or fall back to keyboard, touch, or even phone gyroscope controls on any device.
| Mode | Description | Controls |
|---|---|---|
| Virtual Steering | 3D first-person racer through a neon tunnel with HUD, speed gauge, gearbox, and obstacle traffic | Hands (webcam) · Keyboard · Touch |
| Kart Racing | Fast-paced arcade kart racer with its own menu, tracks, and gameplay loop | Keyboard · Touch |
- Drive with both hands — car accelerates when both hands are detected; palms mapped to steering angle
- Smooth tracking — exponential landmark smoothing + dead zone + non-linear steering curve for natural feel
- Live camera panel — see your hand skeleton overlay while you play
- Keyboard —
Wgas ·A/Dsteer ·Uauto-accelerate - Touch controls — on-screen buttons for mobile play
- Gyroscope mode — tilt your phone to steer
- Auto-accelerate — toggle for hands-free gas
- 3D cockpit — dashboard, gauges, steering wheel with real-time rotation, mirrors, windshield
- Obstacle traffic — AI cars to dodge; collisions end the race with screen shake
- Race timer & score — 90-second race, lap tracking, position, and final results screen
- Procedural engine audio — Web Audio API synth engine sound scaled to speed
- Speed lines & vignette — dynamic juice effects at high velocity
- Sensitivity calibration — adjustable steering sensitivity and smoothing
flowchart LR
A[Webcam] --> B[MediaPipe Hands]
B --> C[Palm Center Extraction]
C --> D[Smoothing Filter]
D --> E[Steering Mapping]
E --> F[3D Camera & Car]
F --> G[Collision / Score / HUD]
H[Keyboard] --> D
I[Touch] --> D
J[Gyroscope] --> D
- MediaPipe extracts 21 hand landmarks per frame (up to 2 hands)
- The palm center (wrist + index + middle MCP) is mapped to a 0–1 steering axis
- Exponential smoothing removes jitter; a dead zone prevents drift
- The steering axis drives the 3D camera, cockpit, and headlights in Three.js
src/
├── main.ts # Entry point: UI, menu flow, game loop
├── game/
│ └── Game.ts # Three.js scene, physics, collisions, HUD state
├── input/
│ ├── HandTracker.ts # MediaPipe hand pipeline + steering mapping
│ └── Keyboard.ts # Keyboard state handler
├── utils/
│ └── smoothing.ts # Exponential moving-average filter
└── style.css # Full UI styling (HUD, panels, overlays)
public/
└── kart-racing/ # Kart Racing arcade game (standalone page)
Design principles: the hand tracker is input-agnostic — keyboard, touch, and gyro overrides plug into the same steering pipeline.
| Layer | Technology |
|---|---|
| Language | TypeScript |
| 3D Rendering | Three.js · WebGL |
| Vision AI | MediaPipe Hands |
| Build | Vite |
| Audio | Web Audio API (procedural) |
| Deployment | Vercel |
- Node.js 18+
- A webcam (for hand tracking)
# 1. Clone & install
git clone https://github.com/Manthan-13521/GestureKart-AI-Racing.git
cd GestureKart-AI-Racing
npm install
# 2. Run the dev server
npm run dev
# 3. Open the game
# http://localhost:5173| Script | Description |
|---|---|
npm run dev |
Start the Vite dev server |
npm run build |
Type-check + production build |
npm run preview |
Preview the production build |
- Multi-track selection & lap racing
- Leaderboards (local + online)
- Gesture calibration screen
- Two-hand asymmetric steering (gas + steer)
- Mobile PWA install support
Contributions, issues, and feature requests are welcome. Fork the repo, make your change, and open a pull request.
All rights reserved.