Hand Gesture Recognition using MobileNetV2

A deep learning computer vision project that recognizes 10 hand gestures from grayscale images using MobileNetV2 transfer learning.

---

📋 Table of Contents

• Overview
• Project Objective
• Technologies Used
• Model Architecture
• Dataset
• Project Structure
• Requirements
• Setup
• How to Run
• Model Performance
• Generated Outputs
• Learning Outcomes
• Future Improvements
• Author
• Internship Details

---

🎯 Overview

This project is part of the SkillCraft Technology Machine Learning Internship. It classifies images from the LeapGestRecog dataset into 10 hand gesture categories.

Workflow:

• Inspect and organize the gesture dataset
• Create subject-based training, validation, and testing splits
• Convert grayscale images to RGB and resize them to 128 × 128 pixels
• Use MobileNetV2 pretrained on ImageNet as a feature extractor
• Train and evaluate a 10-class classifier
• Generate training graphs and a confusion matrix
• Perform real-time recognition with a webcam

---

🎯 Project Objective

Build a multi-class CNN system that accurately recognizes hand gestures while keeping people in the test set separate from people in the training set.

---

🛠 Technologies Used

• Python 3.10+
• TensorFlow / Keras
• MobileNetV2
• NumPy
• Scikit-learn
• Matplotlib
• Seaborn
• OpenCV
• Transfer learning

---

🏗 Model Architecture

The pretrained MobileNetV2 base is frozen and connected to a custom classification head.

Input Image (128 x 128 grayscale)
            |
            v
     Convert to RGB
            |
            v
 MobileNetV2 (ImageNet)
            |
            v
 Global Average Pooling
            |
            v
       Dropout (0.2)
            |
            v
 Dense Layer (10 units, Softmax)
            |
            v
     Gesture Prediction

Parameter	Value
Input size	128 × 128 × 3
Batch size	4
Epochs	3
Optimizer	Adam
Loss	Sparse categorical crossentropy
Random seed	42
Classes	10

---

📊 Dataset

The project uses the LeapGestRecog dataset:

• 10 subjects
• 10 gesture classes
• 200 images per class for each subject
• 20,000 images in total

Subject Split

Split	Subjects	Images
Training	00-06	14,000
Validation	07	2,000
Testing	08-09	4,000

Gesture Classes

palm, l, fist, fist_moved, thumb, index, ok, palm_moved, c, down

---

📁 Project Structure

SCT_ML_Task4/
├── README.md
├── requirements.txt
├── dataset/
│   └── leapGestRecog/
│       ├── 00/
│       ├── 01/
│       ├── ...
│       └── 09/
├── models/
│   └── gesture_model.keras
├── results/
│   ├── confusion_matrix.png
│   ├── training_accuracy.png
│   └── training_loss.png
├── src/
│   ├── inspect_dataset.py
│   ├── train.py
│   └── evaluate.py
└── demo/
    └── webcam.py

The virtual environment and temporary editor files are intentionally excluded from this structure.

---

📦 Requirements

Python 3.10+ and a webcam are recommended. Install the project dependencies with:

pip install -r requirements.txt

---

⚙️ Setup

1. Create and activate a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

Windows Command Prompt:

python -m venv .venv
.venv\Scripts\activate.bat

2. Install dependencies

pip install -r requirements.txt

3. Prepare the dataset

Place the LeapGestRecog subject folders inside:

dataset/leapGestRecog/

The folder must contain subjects 00 through 09, each with the 10 gesture-class folders.

4. Configure the dataset path

Before training or evaluation, set DATASET_PATH in src/train.py and src/evaluate.py to the location of your local leapGestRecog folder.

---

▶️ How to Run

Inspect the dataset

python src/inspect_dataset.py

Train the model

python src/train.py

The trained model is saved to models/gesture_model.keras and training graphs are saved to results/.

Evaluate the model

python src/evaluate.py

This generates the confusion matrix and classification metrics for the test subjects.

Run webcam recognition

python demo/webcam.py

Press q in the webcam window to quit.

---

📈 Model Performance

Metric	Value
Test Accuracy	94.15%
Test Loss	0.1526
Test Images	4,000

The test set contains 400 images for each gesture class.

---

📂 Generated Outputs

File	Description
models/gesture_model.keras	Saved trained model
results/training_accuracy.png	Training and validation accuracy
results/training_loss.png	Training and validation loss
results/confusion_matrix.png	Class-level prediction performance

---

🎓 Learning Outcomes

• Image classification with CNNs
• Transfer learning with MobileNetV2
• Subject-based dataset splitting
• Grayscale image preprocessing and RGB conversion
• Multi-class classification with softmax output
• Evaluation using accuracy, precision, recall, F1-score, and confusion matrices
• Real-time webcam-based computer vision

---

🚀 Future Improvements

• Fine-tune selected MobileNetV2 layers
• Apply additional image augmentation
• Improve webcam prediction stability
• Add gesture tracking
• Compare other CNN architectures
• Deploy the model as a desktop or web application

---

👨‍💻 Author

Devarakonda Vignesh Varsha

---

📌 Internship Details

Detail	Information
Internship	SkillCraft Technology
Domain	Machine Learning
Task	Task 04 - Hand Gesture Recognition
Model	MobileNetV2 Transfer Learning
Dataset	LeapGestRecog
Test Accuracy	94.15%

---

Built with ❤️ during SkillCraft Technology ML Internship