Signify ✋🗣️

Real-time hand gesture and speech-to-text recognition for accessible communication.

Built at Hack The Ridge 2025, Signify is a computer-vision and speech-recognition application that bridges spoken and signed communication. It watches your webcam for hand gestures and listens through your microphone at the same time, converting both into live, on-screen captions, so a conversation between a hearing person and a non-verbal or Deaf/Hard-of-Hearing person can flow through one shared visual feed.

How it works

Signify runs two recognition pipelines in parallel:

• Hand gesture recognition — Uses MediaPipe Hands to track 21 landmarks per hand in real time. A custom rules-based classifier inspects finger counts, joint angles, and inter-finger distances to map hand shapes to words and short phrases (e.g. open palm facing up → "Hello / Hi", thumb up → "Yes / Good", fist with palm down → "Stop / Wait"). A stability filter requires a gesture to be held for several consecutive frames before it's accepted, which cuts down on flickery, false-positive detections.
• Speech-to-text — Uses the speech_recognition library (Google Speech API backend) on a background thread to transcribe live microphone audio without blocking the video loop. Energy and pause thresholds are tuned for low-latency, natural conversational pauses.

Both streams write into a shared rolling subtitle buffer that's overlaid directly on the camera feed, so gestures and speech appear together as a single caption history.

Bonus: Six-Seven detector 🎶

As a fun easter egg, Signify also tracks vertical motion of both hands simultaneously. If it detects a synchronized up-and-down "six-seven" motion, it flashes an on-screen alert and plays a sound effect.

Features

• 🖐️ Real-time two-hand tracking via MediaPipe
• 🔤 30+ recognized gestures mapped to common words/phrases
• 🎙️ Live, low-latency speech-to-text transcription on a separate thread
• 💬 Combined, scrolling subtitle overlay for both gesture and speech output
• 🔊 Gesture-triggered sound effects
• 🛡️ Robust camera fallback handling (auto-tries multiple camera indices)

Tech stack

Component	Library
Video capture & rendering	OpenCV (opencv-python)
Hand landmark detection	MediaPipe
Speech-to-text	SpeechRecognition
Audio playback	pygame (pygame.mixer)
Numerical processing	NumPy

Getting started

Prerequisites

• Python 3.8+
• A webcam and microphone
• An internet connection (speech recognition calls the Google Speech API)

Installation

git clone https://github.com/<your-username>/Signify.git
cd Signify
pip install opencv-python numpy mediapipe pygame SpeechRecognition PyAudio

Note: PyAudio can require extra system-level setup depending on your OS:

• Windows: pip install pyaudio usually works directly, or use pipwin install pyaudio if it fails.
• macOS: brew install portaudio before pip install pyaudio.
• Linux: sudo apt-get install portaudio19-dev before pip install pyaudio.

Running it

python run.py

• Allow camera and microphone access when prompted.
• Show a hand gesture to the camera or speak naturally — captions appear at the bottom of the window.
• Press ESC to quit, or say "stop listening" to end speech recognition.

Gesture reference (examples)

Gesture	Meaning
Open palm, facing up	Hello / Hi
Thumbs up	Yes / Good / Agree
Closed fist, palm down	Stop / Wait
Index finger pointing	You / Point
Index + middle together	Peace / Victory
Thumb + pinky out	Hang loose / Call me
Thumb, index, pinky out	I Love You (ASL)
All five fingers, palm down raised	Thank You

(Full gesture-to-meaning logic lives in recognize_gesture() in run.py.)

Project background

Signify was built in 8 hours at Hack The Ridge 2025 with the goal of making everyday communication more accessible, letting sign language and spoken language meet in the same visual space without requiring either party to learn a new tool or workflow.

Future improvements

• Replace the rules-based gesture classifier with a trained ML model for higher accuracy and a larger vocabulary
• Support full ASL fingerspelling and dynamic (motion-based) signs
• Add multi-language speech recognition
• Sentence-level gesture sequencing instead of single-word mapping
• Package as a standalone desktop app

Contributors

• Moeen Shahid
• Arnav Govil
• Zohaib Mansoor
• Ryan Mampilly

License

This project is licensed under the MIT License.