✋🤖 Hand-Tracking Control

Control robots in real time with nothing but your hand and a webcam.

A computer-vision + robotics project that turns your bare hand into a controller for two physically simulated robots — a dexterous Shadow Hand and a Crazyflie 2 quadrotor — using MediaPipe hand tracking and the MuJoCo physics engine.

---

✨ What it does

Point a webcam at your hand and the system:

1. detects 21 hand landmarks per frame with MediaPipe,
2. interprets them into a structured, handedness-aware hand state (which fingers are extended, where your palm is in the frame), and
3. maps that state onto the actuators of a MuJoCo robot, stepping and rendering the physics live.

Demo	What your hand does	Robot
🖐️ Shadow Hand teleop	Curl a finger → the robot curls the matching finger	24-DoF Shadow Hand
🚁 Drone flight	Raise/lower your hand → throttle · move left/right → roll	Crazyflie 2 quadrotor
🔢 Finger counter	—	none (pipeline sanity check)

Shadow Hand teleoperation

Gesture drone flight

---

🏗️ Architecture

flowchart LR
    A[📷 Webcam<br/>OpenCV] --> B[HandTracker<br/>MediaPipe]
    B --> C[HandObservation<br/>landmarks · handedness<br/>fingers open/closed · palm pos]
    C --> D{Control mapping}
    D -->|finger curl| E[Shadow Hand<br/>actuators]
    D -->|palm position| F[Crazyflie<br/>thrust + moments]
    E --> G[MuJoCo step + render<br/>MujocoViewer]
    F --> G

Loading

The codebase is split into a small reusable core and thin demo scripts:

src/handctrl/
├── hand_tracker.py     # HandTracker + HandObservation — all MediaPipe logic
├── mujoco_viewer.py    # MujocoViewer — GLFW window, camera, rendering
├── config.py           # paths & constants (resolved relative to the repo)
├── shadow_hand_demo.py # finger curl → Shadow Hand actuators
├── drone_demo.py       # palm position → drone thrust/roll
├── finger_counter.py   # webcam-only finger detector
└── inspect_model.py    # utility: dump a model's joints/actuators/bodies

---

🚀 Quickstart

Requirements: Python 3.10–3.12, a webcam, and a GPU is not required.

# 1. Clone
git clone https://github.com/Manas-arumalla/Hand-Tracking-Control.git
cd Hand-Tracking-Control

# 2. Create an environment and install
python -m venv .venv
# Windows:        .venv\Scripts\activate
# macOS / Linux:  source .venv/bin/activate
pip install -r requirements.txt

# 3. Run a demo (from the src/ directory, or `pip install -e .` first)
cd src
python -m handctrl.finger_counter      # quick sanity check — no simulator
python -m handctrl.shadow_hand_demo    # teleoperate the Shadow Hand
python -m handctrl.drone_demo          # fly the Crazyflie

Or install the package and use the console entry points from anywhere:

pip install -e .
shadow-hand-demo
drone-demo
finger-counter

---

🎮 Controls

Shadow Hand — curl any finger in front of the camera and the robot mirrors it. Camera: drag with the mouse to orbit, scroll to zoom, arrow keys to rotate, +/- to zoom.

Drone — proportional, smooth control:

Your hand	Drone response
Raise ⬆️ / lower ⬇️	More / less thrust (climb / descend)
Move left ⬅️ / right ➡️	Roll left / right
No hand	Hover at neutral thrust

Press q or Esc to quit any demo. Every demo accepts --camera, --scene, --smoothing and other flags — run with --help to see them.

---

🔬 How the hand is interpreted

• Finger extension (Index–Pinky) is decided by comparing each fingertip to its PIP joint on the vertical axis.
• The thumb is handled separately and is handedness-aware — MediaPipe's Left/Right classification flips the comparison so the thumb is read correctly for either hand (the naive single-axis test only works for one hand orientation).
• The webcam image is mirrored so the on-screen view behaves like a real mirror.
• Control targets are exponentially smoothed and clamped to each actuator's real ctrlrange, so the robots move smoothly and commands are never silently saturated.

---

📦 Tech stack

Concern	Tool
Hand landmark detection	MediaPipe Hands
Camera I/O & overlay	OpenCV
Physics & rendering	MuJoCo + GLFW
Numerics	NumPy
Robot models	MuJoCo Menagerie (Shadow Hand, Crazyflie 2)

---

🙏 Credits & licenses

This project's own code is released under the MIT License.

The robot models in models/ are from Google DeepMind's MuJoCo Menagerie and retain their original licenses (Apache-2.0 for the Shadow Hand, BSD-3-Clause for the Crazyflie). See the LICENSE file inside each model directory.

Built by Manas Reddy Arumalla.