Real-Time ASL Interpreter (Hand-Motion Transformer)

An end-to-end assistive platform translating American Sign Language (ASL) gestures in real time using a standard webcam, powered by MediaPipe, a Hand-Motion Transformer, FastAPI, and Angular[cite: 1].

---

🚀 Key Features

• 2,731 ASL Classes: Trained on 83,399 video samples from the ASL Citizen dataset[cite: 1].
• Spatio-Temporal Feature Representation: Extracts 21 landmarks per hand with wrist-centered normalization and frame-to-frame velocity differences, creating an optimized $30 \times 252$ feature vector[cite: 1].
• High-Performance Transformer: Achieved 65.49% Top-1 and 89.56% Top-5 accuracy on 9,921 test samples, outperforming LSTM, BiLSTM, and GRU baselines[cite: 1].
• Full-Stack Architecture: Real-time FastAPI backend handling inference and prediction logging, paired with an Angular UI supporting sentence stabilization and browser-based Text-to-Speech (TTS)[cite: 1].

---

📊 Benchmark Comparison

Model Architecture	Top-1 Accuracy	Top-5 Accuracy	Macro F1	Weighted F1	Latency
Hand-Motion Transformer (Ours)	65.49%	89.56%	0.6389	0.6411	0.687 ms/sample
Attention-BiLSTM	64.38%	88.10%	0.6304	0.6329	0.655 ms/sample
BiLSTM	62.59%	86.85%	0.6127	0.6141	0.444 ms/sample
BiGRU	62.32%	86.42%	0.6068	0.6090	0.647 ms/sample
CNN-LSTM	59.47%	85.87%	0.5716	0.5753	0.496 ms/sample
GRU	56.89%	83.88%	0.5460	0.5485	0.368 ms/sample
LSTM	55.35%	83.70%	0.5322	0.5345	0.383 ms/sample

---

📁 Repository Structure

• Fyp backend/: FastAPI inference service, feature normalization, and prediction endpoints.
• Fyp frontend/: Angular client handling webcam capture, MediaPipe tracking, and TTS controls.
• FYP model/: Training, evaluation, and benchmark pipelines for sequence models.

---

🛠️ Local Setup

1. Backend Service

cd "Fyp backend"
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
python run_backend_local.py

2. Frontend Interface

cd "Fyp frontend"
npm install
npm start