Control your PC's mouse cursor entirely with hand gestures via your webcam. No extra hardware needed. Built with Python, OpenCV, cvzone, and MediaPipe's 21-point hand skeleton model.
- How It Works
- Python Version Warning
- Requirements
- Installation — Linux
- Installation — Windows
- Running the App
- Gesture Reference
- Understanding the Camera Window
- Configuration (config.json)
- Project Structure
- How the Code Works
- Troubleshooting
- Adding New Gestures
- Dependencies
Each webcam frame is passed through MediaPipe's hand landmark model, which detects 21 key points on your hand (fingertips, knuckles, wrist) and returns their X/Y pixel coordinates in real time.
Cursor movement uses relative positioning (like a trackpad). When you raise your index finger, the cursor continues from wherever it already was and moves based on how much your hand moves — not based on the absolute position of your hand in the camera frame. This means no sudden jumps when you start moving.
Clicks fire at the exact position the cursor was at before you pinched — a freeze window stops all cursor movement the moment a pinch gesture is detected, so the cursor stays put while the click fires.
Python 3.11 is required. Python 3.12 and 3.13 will NOT work.
MediaPipe dropped support for the mp.solutions API in Python 3.12+. Running on the wrong version gives this immediately on launch:
AttributeError: module 'mediapipe' has no attribute 'solutions'
The only fix is Python 3.11. Always use a virtual environment (venv) so your system Python doesn't matter.
- Python 3.11 (not 3.12, not 3.13)
- A working webcam (built-in or USB)
- Decent lighting — MediaPipe struggles in dim or backlit conditions
- A plain background helps detection accuracy
- Linux or Windows
Works on Fedora, Nobara, Ubuntu, Debian, and most other distros.
python3.11 --versionIf not installed:
Fedora / Nobara:
sudo dnf install python3.11 python3.11-develUbuntu / Debian:
sudo apt update && sudo apt install python3.11 python3.11-venv python3.11-dev
python3.11-devel/python3.11-devis required for building theevdevC extension thatpynputdepends on.
cd "AR Hand Tracking"python3.11 -m venv venv
source venv/bin/activateYour prompt will show (venv) when active.
pip install -r requirements.txtpython -c "import mediapipe; import cvzone; import pynput; print('All good')"Download from: https://www.python.org/downloads/release/python-3119/
Scroll to Files → download Windows installer (64-bit).
During install, check ✅ Add Python to PATH.
Verify:
py -3.11 --versioncd "C:\Users\YourName\Documents\AR Hand Tracking"py -3.11 -m venv venv
venv\Scripts\activatePowerShell: if activation is blocked, run once:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
pip install -r requirements.txtpython -c "import mediapipe; import cvzone; import pynput; print('All good')"Always activate the venv first.
source venv/bin/activate
python main.pyvenv\Scripts\activate
python main.py.\venv\Scripts\Activate.ps1
python main.pyA window opens showing your webcam feed with hand landmarks. Raise your index finger to start moving the cursor — it continues from wherever your cursor already was.
- Press Q while the camera window is focused
- Or close the terminal running the script
| Gesture | Hand pose | Action |
|---|---|---|
| Move cursor | Index finger up. Thumb state doesn't matter. Middle/ring/pinky folded. | Cursor moves relative to hand movement |
| Left click | Pinch — bring thumb tip close to index tip | Single left click |
| Scroll up | Index + middle fingers up, index higher than middle | Scrolls up |
| Scroll down | Index + middle fingers up, middle higher than index | Scrolls down |
| Right click | Thumb + pinky up, index/middle/ring folded | Single right click |
| Idle | Any other pose, or no hand detected | Nothing — cursor stays put |
- Good lighting from the front is the single biggest factor in detection quality
- Face your palm toward the camera — don't angle it too far sideways
- Use Idle deliberately to reposition your hand without moving the cursor — just fold all fingers or show a flat palm, reposition, then raise your index again
- The pinch proximity ring (circle between thumb and index in the camera window) turns green as you approach the click threshold — use it to calibrate your pinch distance
- Clicks have a cooldown (default 400ms) to prevent double-firing
| Element | What it is |
|---|---|
| Coloured dots + lines | 21 MediaPipe landmarks — the hand skeleton |
| Grey rectangle | Active zone reference (informational only in relative mode) |
| Top-left label | Currently detected gesture, colour-coded |
| Top-right number | Live FPS counter |
| Circle between thumb and index | Pinch proximity ring — grey → green as you approach click |
| Bottom-left | Q to quit reminder |
{
"cam_index": 0,
"cam_w": 640,
"cam_h": 480,
"detection_con": 0.8,
"active_zone": [0.2, 0.2, 0.8, 0.8],
"smooth_frames": 8,
"click_cooldown_ms": 400,
"scroll_speed": 3,
"scroll_min_diff": 15,
"pinch_threshold": 40,
"pinch_scale_to_hand": false,
"ref_hand_size": 100,
"gesture_debounce_frames": 2,
"move_sensitivity": 3.5
}| Key | Default | What it does | Tuning advice |
|---|---|---|---|
cam_index |
0 | Which camera to open | Change to 1 or 2 if you have multiple cameras |
cam_w / cam_h |
640 / 480 | Capture resolution | 640×480 is the sweet spot. Higher = heavier CPU |
detection_con |
0.8 | MediaPipe detection confidence (0.0–1.0) | Lower to 0.6 if hand isn't detected. Raise to 0.9 to reduce false positives |
active_zone |
[0.2,0.2,0.8,0.8] | Reference rectangle drawn on screen. Informational in relative mode. | Leave as-is |
smooth_frames |
8 | Frames to average cursor position over | Raise to 10–12 for smoother but slightly laggy cursor. Lower to 4–5 for snappier |
click_cooldown_ms |
400 | Minimum ms between clicks | Raise if double-clicking accidentally |
scroll_speed |
3 | Scroll lines per gesture frame | Raise for faster scroll |
scroll_min_diff |
15 | Min pixel gap between index/middle tips to confirm scroll direction | Raise if scroll flickers |
pinch_threshold |
40 | Pixel distance for pinch detection | Raise (50–60) if clicking accidentally. Lower (25–30) if pinch doesn't register |
gesture_debounce_frames |
2 | Gesture must hold N frames before firing | Raise to 3–4 if random false clicks occur |
move_sensitivity |
3.5 | How many screen pixels to move per camera pixel of hand movement | Raise for faster cursor. Lower for more precision |
AR Hand Tracking/
│
├── main.py Entry point. Webcam loop, HUD drawing, graceful shutdown.
├── config.json All runtime settings — the only file to edit for tuning.
├── requirements.txt pip dependencies.
├── README.md This file.
│
├── core/
│ ├── detector.py Wraps cvzone HandDetector. flipType=False (mirror handled
│ │ in main.py). Exposes process(), fingers_up(), hand_size().
│ ├── gesture.py fingers[] + lm[] → gesture string. MOVE ignores thumb
│ │ state for robustness against label inconsistencies.
│ └── smoother.py Rolling average deque. flush() clears buffer after clicks.
│
├── control/
│ ├── actions.py Gesture → action. Relative cursor movement (trackpad style).
│ │ Click freeze window prevents cursor drift during pinch.
│ └── mouse.py pynput wrapper. Uses XTest — reliable under Wayland/XWayland.
│
└── utils/
├── config.py Loads config.json once, exposes CFG dict to all modules.
├── mapping.py Camera→screen coordinate mapping (kept for reference).
└── screen.py Gets monitor resolution at startup.
Webcam frame
↓
cv2.flip(frame, 1) mirror — natural left/right feel
↓
Detector.process(frame)
├── findHands(flipType=False) cvzone + MediaPipe, draws skeleton
└── lmList → [[id,x,y], ...] 21 landmarks
↓
Detector.fingers_up() [thumb, index, middle, ring, pinky]
Detector.hand_size() wrist-to-MCP9 distance
↓
gesture.recognize(fingers, lm)
├── pinch dist < threshold → CLICK
├── index up, mid/ring/pinky down → MOVE (thumb ignored)
├── index+middle up, deadzone check → SCROLL UP/DOWN
└── thumb+pinky up → RIGHT_CLICK
↓
actions.dispatch(gesture, lm, sw, sh)
├── MOVE
│ ├── if freeze window active → skip (cursor stays frozen)
│ ├── if _prev_hand is None → read OS cursor via pynput.position()
│ │ set anchor, return (no movement this frame)
│ └── else: dx/dy delta from prev_hand → scale by sensitivity
│ → smoother.smooth() → pynput.move()
├── CLICK/RIGHT_CLICK
│ ├── extend freeze window (cursor not touched during pinch)
│ ├── reset _prev_hand (next MOVE re-anchors to OS cursor)
│ └── if confirmed + cooldown: pynput.click() at current cursor pos
└── SCROLL → pynput.scroll()
↓
draw_hud() → cv2.imshow()
Wrong Python version. Recreate venv with 3.11:
rm -rf venv && python3.11 -m venv venv && source venv/bin/activate && pip install -r requirements.txtMissing Python dev headers:
# Fedora/Nobara:
sudo dnf install python3.11-devel
# Ubuntu/Debian:
sudo apt install python3.11-devThen retry pip install -r requirements.txt.
Make sure you have the latest actions.py which uses relative (delta) movement. Old versions used absolute coordinate mapping.
Make sure you have the latest gesture.py. The MOVE check now ignores thumb state — only index up + middle/ring/pinky down is required.
Increase move_sensitivity in config.json (try 4.0 or 5.0).
Lower move_sensitivity (try 2.5). Also increase smooth_frames to 10–12.
In core/detector.py, confirm findHands is called with flipType=False. Do NOT add any manual label swap.
Make sure pynput is installed: pip install pynput. The mouse.py must use pynput, not pyautogui.
Increase smooth_frames to 10–12 in config.json.
Raise scroll_min_diff to 25–30.
- Improve front lighting
- Lower
detection_conto 0.6 - Hold hand 30–60cm from camera
- Use a plain background
Change cam_index in config.json from 0 to 1 or 2.
Harmless Qt scaling warning. Suppress with:
QT_FONT_DPI=96 QT_AUTO_SCREEN_SCALE_FACTOR=0 python main.pyOr add to top of main.py before imports:
import os
os.environ.setdefault('QT_FONT_DPI', '96')
os.environ.setdefault('QT_AUTO_SCREEN_SCALE_FACTOR', '0')DOUBLE_CLICK = "double_click"
# Inside recognize(), before the final return IDLE:
if fingers[1] == 1 and fingers[2] == 1 and fingers[3] == 1 and fingers[4] == 0:
return DOUBLE_CLICKfrom core.gesture import MOVE, CLICK, RIGHT_CLICK, SCROLL_UP, SCROLL_DOWN, DOUBLE_CLICK
elif gesture == DOUBLE_CLICK and confirmed:
_freeze_until = now + _FREEZE_S
_prev_hand = None
if now - _last_click > _COOLDOWN:
from control.mouse import double_click
double_click()
_last_click = now
_smoother.flush()_HINTS["double_click"] = "Double Click!"
_COLOR["double_click"] = (0, 255, 180)| Package | Purpose |
|---|---|
opencv-python |
Webcam capture, frame display, drawing |
cvzone |
HandDetector wrapper around MediaPipe |
mediapipe |
21-point hand landmark model — Python 3.11 only |
pynput |
Mouse control via XTest — reliable under Wayland/XWayland |
pyautogui |
Screen size fallback only |
numpy |
Coordinate math |
screeninfo |
Primary monitor resolution detection |