Gesture Control

Control your computer with hand gestures — no mouse or keyboard needed.

---

Real-time webcam-based hand tracking that lets you move the cursor, click, scroll, and adjust system brightness & volume using intuitive hand gestures.

---

Table of Contents

• Features
• Gesture Reference
• Tech Stack
• Prerequisites
• Installation
• Usage
• API Endpoints
• Project Structure
• How It Works

---

Features

	Feature	Description
Cursor	Mouse Control	Move and click the mouse using hand gestures
Drag	Drag & Drop	Grab and drag items with a fist gesture
Scroll	Smart Scrolling	Vertical and horizontal scrolling via pinch
System	Brightness & Volume	Adjust system settings with pinch movements
Hands	Multi-hand Support	Detects both hands with configurable dominant hand
Stable	Smooth Tracking	Kalman filtering and frame stabilization reduce jitter

---

Gesture Reference

Dominant hand = your configured primary hand  |  Non-dominant hand = the other

Gesture	Hand	Action
V Sign	Dominant	Activate cursor mode
Fist	Dominant	Drag / grab
Middle finger	Dominant (in cursor mode)	Left click
Index finger	Dominant (in cursor mode)	Right click
Two fingers closed	Dominant (in cursor mode)	Double click
Pinch	Non-dominant	Scroll — Y-axis vertical, X-axis horizontal
Pinch	Dominant	System — X-axis brightness, Y-axis volume

---

Tech Stack

Backend	Python • FastAPI • MediaPipe • OpenCV • PyAutoGUI • pycaw • screen-brightness-control
Frontend	React 19 • Vite • Tailwind CSS • Axios

---

Prerequisites

• Python 3.8+
• Node.js 18+
• Webcam connected and accessible
• Windows OS (uses Windows-specific APIs for audio/brightness)

---

Installation

1. Clone the repository

git clone https://github.com/your-username/gesture-control.git
cd gesture-control

2. Backend setup

Gesture Control

Control your computer with hand gestures using real-time webcam-based hand tracking. Move the cursor, click, scroll, and adjust system brightness and volume — all without touching your mouse or keyboard.

Built with MediaPipe for hand detection, FastAPI for the backend, and React for the frontend dashboard.

Features

• Cursor Control — Move and click the mouse with hand gestures
• Drag & Drop — Grab and drag with a fist gesture
• Scroll — Vertical and horizontal scrolling via pinch gestures
• Brightness & Volume — Adjust system brightness and volume with pinch movements
• Multi-hand Support — Detects both hands with configurable dominant hand
• Stabilized Tracking — Kalman filtering and frame stabilization reduce jitter

Gesture Reference

Gesture	Action
V Sign (dominant hand)	Activate cursor mode
Fist (dominant hand)	Drag / grab
Middle finger (in cursor mode)	Left click
Index finger (in cursor mode)	Right click
Two fingers closed (in cursor mode)	Double click
Pinch (non-dominant hand)	Scroll (Y-axis) and horizontal scroll (X-axis)
Pinch (dominant hand)	Brightness (X-axis) and volume (Y-axis)

Tech Stack

Backend: Python, FastAPI, MediaPipe, OpenCV, PyAutoGUI, pycaw, screen-brightness-control

Frontend: React 19, Vite, Tailwind CSS, Axios

Prerequisites

• Python 3.8+
• Node.js 18+
• A webcam
• Windows OS (uses Windows-specific APIs for audio/brightness control)

Installation

Backend

cd backend
pip install -r requirements.txt

3. Frontend setup

Frontend

cd frontend
npm install

---

Usage

Open two terminals and start both services:

# Terminal 1 — Start the backend (http://localhost:8000)
cd backend
python main.py

# Terminal 2 — Start the frontend (http://localhost:5173)
## Usage

Start both the backend and frontend:

```bash
# Terminal 1 — Backend (runs on http://localhost:8000)
cd backend
python main.py

# Terminal 2 — Frontend (runs on http://localhost:5173)
cd frontend
npm run dev

Then open the frontend in your browser to configure the dominant hand and start/stop gesture recognition.

---

API Endpoints

Method	Endpoint	Description	Body
POST	/start	Start gesture recognition	—
POST	/stop	Stop gesture recognition	—
GET	/status	Get system status	—
POST	/set_dominant_hand	Set dominant hand	{"dominant_hand": "left"}

Example responses

// GET /status
{ "running": true }

// POST /start
{ "status": "Gesture Controller started" }

// POST /set_dominant_hand
{ "status": "Dominant hand set to left" }

---

Open the frontend in your browser to configure the dominant hand and start/stop gesture recognition.

API Endpoints

Method	Endpoint	Description
POST	/start	Start gesture recognition
POST	/stop	Stop gesture recognition
GET	/status	Get system status ({"running": true/false})
POST	/set_dominant_hand	Set dominant hand ({"dominant_hand": "left"})

Project Structure

gesture_control/
│
├── backend/
│   ├── main.py                  # FastAPI server & API routes
│   ├── gesture_controller.py    # Main controller orchestrating the system
│   ├── gesture_recognition.py   # Hand landmark detection & gesture classification
│   ├── controller.py            # Mouse, brightness & volume control
│   └── requirements.txt         # Python dependencies
│
├── frontend/
│   ├── src/
│   │   ├── App.jsx              # Root React component
│   │   └── components/
│   │       └── GestureControl.jsx   # Dashboard UI
│   ├── package.json             # Node dependencies
│   └── vite.config.js           # Vite configuration
│
└── gesture.py                   # Standalone gesture module (reference)

---

How It Works

Webcam  ──>  MediaPipe  ──>  Gesture Classifier  ──>  System Actions
  │              │                   │                       │
  │         Detects up to       5-bit binary            Cursor move
  │         2 hands with        finger encoding         Click / Drag
  │         21 landmarks        + 4-frame               Scroll
  │         per hand            stabilization           Brightness
  │                                                     Volume

1. Capture — Webcam frames are processed in real time via OpenCV
2. Detect — MediaPipe identifies up to 2 hands and extracts 21 landmarks each
3. Classify — Finger states are encoded as a 5-bit binary value; 4 consecutive frames of the same gesture are required to confirm it
4. Execute — Recognized gestures map to system actions with Kalman-filtered smoothing for stable cursor movement

---

Built for educational and personal use.

├── backend/ │ ├── main.py # FastAPI server │ ├── gesture_controller.py # Main controller orchestrating the system │ ├── gesture_recognition.py # Hand landmark detection and gesture classification │ ├── controller.py # Mouse, brightness, and volume control │ └── requirements.txt ├── frontend/ │ ├── src/ │ │ ├── App.jsx │ │ └── components/ │ │ └── GestureControl.jsx # Dashboard UI │ ├── package.json │ └── vite.config.js └── gesture.py # Standalone gesture module (reference) ```

How It Works

1. Capture — Webcam frames are processed in real time
2. Detect — MediaPipe identifies up to 2 hands and extracts 21 landmarks per hand
3. Classify — Finger states are encoded as a 5-bit binary value to identify gestures, with 4-frame stabilization to filter noise
4. Execute — Recognized gestures are mapped to system actions (cursor movement, clicks, brightness/volume adjustments)

License

This project is for educational and personal use.