Bionic Hand

A vision-controlled bionic hand that maps human hand gestures to servo motion in real time. A webcam feed is processed with OpenCV and MediaPipe to track hand landmarks, the distance between key landmarks (fingertip to mid-joint) is calculated per finger and mapped to a servo angle, and the angle is sent over serial communication to an ESP32, which drives the servos to mimic the gesture on the physical hand.

How it works

Webcam Feed
    │
    ▼
MediaPipe Hand Landmark Detection (21 landmarks per hand)
    │
    ▼
Distance Calculation (tip landmark to mid landmark, per finger)
    │
    ▼
Map Distance to Servo Angle (0°–180°)
    │
    ▼
Serial Communication (USB) to ESP32
    │
    ▼
ESP32 drives servos, fingers move

1. Landmark detection: MediaPipe Hands returns 21 (x, y, z) landmarks per detected hand from the webcam feed.

2. Distance mapping: For each finger, the distance between its tip landmark and mid landmark is measured:

   finger_tips = [4, 8, 12, 16, 20]
   finger_mids = [3, 6, 10, 14, 18]

3. Angle conversion: Each distance value is linearly mapped to a servo angle range using numpy.interp (or equivalent), so finger curl/extension maps to servo rotation.

4. Serial transmission: Angles are packed and sent over serial (USB) to the ESP32 at a fixed baud rate.

5. Actuation: The ESP32 parses the incoming values and moves each servo to the target angle, replicating the hand gesture mechanically.

Hardware

• ESP32 development board
• MG90S micro servo motors (one per finger)
• 5V BEC (Battery Eliminator Circuit) for stable servo power
• 3S LiPo battery (servo power supply)
• USB webcam
• SolidWorks-designed hand structure (see CAD files/)

Servo to GPIO pin mapping

Finger	Servo GPIO Pin
Thumb	18
Index	19
Middle	21
Ring	22
Pinky	23

Software & dependencies

• Python 3.x
• OpenCV (opencv-python)
• MediaPipe
• PySerial
• NumPy
• Arduino IDE (or PlatformIO) for ESP32 firmware

Install Python dependencies:

pip install opencv-python mediapipe pyserial numpy

Repository structure

bionic-hand/
├── README.md
├── .gitignore
├── vision/              # OpenCV + MediaPipe landmark detection & servo angle mapping
│   └── hand_tracker.py
├── firmware/            # ESP32 sketch, receives serial data, drives servos
│   └── servo_control.ino
└── CAD files/           # SolidWorks hand structure design files

Setup & usage

1. Flash the ESP32

   • Open firmware/servo_control.ino in Arduino IDE
   • Select your ESP32 board and correct COM port
   • Upload

2. Run the vision script

   cd vision
   python hand_tracker.py

   • Update the serial port name in the script to match your ESP32's port (e.g. /dev/ttyUSB0 on Linux, COM3 on Windows)

3. Wire it up

   • Connect servos to GPIO 18, 19, 21, 22, 23 on the ESP32 (thumb through pinky respectively)
   • Power servos via the 5V BEC from the 3S LiPo. Do not power servos directly from the ESP32 or USB, as this can draw more current than the board can safely supply

4. Point the webcam at your hand. The physical hand should mimic your finger movements in real time.

CAD

Hand structure design files are in CAD files/, designed in SolidWorks.

Future improvements

• Add per-finger calibration for different hand sizes
• Wireless (Bluetooth/WiFi) control instead of wired serial
• Closed-loop feedback using servo position sensing
• Support for full 3D hand pose instead of 2D landmark distance

Author

Parth Keshari Robolution, BIT Mesra github.com/parthkeshari