Real-Time Sign Language to Speech Conversion
Because communication shouldn’t need a Poké Flute.
People using sign language or gestures face communication barriers with non-signers. This project bridges that gap by translating hand gestures into real-time spoken language, enabling inclusive and accessible communication.
- Classify hand gestures from images
- Convert predicted gestures into text
- Speak the text aloud using Text-to-Speech
Input Image / Frame ↓ CNN Gesture Classifier (PyTorch) ↓ Predicted Gesture Label ↓ Text-to-Speech (pyttsx3) ↓ Audio Output
- Language: Python 3.10 / 3.11
- Deep Learning: PyTorch
- Dataset: ASL Alphabet Dataset
- Image Processing: torchvision, OpenCV
- Text-to-Speech: pyttsx3
Dataset link:
https://www.kaggle.com/datasets/grassknoted/asl-alphabet
Create Virtual Environment python -m venv venv venv\Scripts\activate # Windows
pip install --upgrade pip pip install torch torchvision numpy opencv-python pyttsx3
Python 3.14 is NOT supported. Use Python 3.10 or 3.11.
Training the Model python train.py
Loads ASL dataset using ImageFolder
Applies data augmentation
Trains CNN classifier
Saves trained model (model.pth)
Gesture Prediction + Speech python infer.py
Loads trained model
Predicts gesture label
Converts prediction into speech using pyttsx3
Text-to-Speech
Offline TTS
Low latency
Platform independent
High accuracy on static ASL images
Real-time text and audio output
Performance depends on lighting and background
Static gestures only
No continuous sentence recognition
Sensitive to background noise and lighting
Real-time webcam gesture detection
MediaPipe hand landmark integration
Continuous sign language recognition
Transformer-based gesture modeling
Multilingual speech output