🤟 SignNet -- Sign Language Recognition with Attention Mechanisms

[ [ [ [

A comprehensive deep learning project for Indian Sign Language (ASL) alphabet recognition, featuring three state-of-the-art CNN architectures with attention mechanisms for improved accuracy and generalization.

📋 Table of Contents

Overview
Models
Features
Installation
Dataset Structure
Usage
Model Performance
Results
Project Structure
Contributing
License
Acknowledgments

🎯 Overview

This project implements three progressively advanced CNN architectures for sign language alphabet recognition (A-Z), demonstrating the impact of attention mechanisms on model performance:

Base CNN: Standard convolutional neural network with batch normalization
CNN + SE Blocks: Enhanced with Squeeze-and-Excitation blocks for channel attention
CNN + SE + CBAM: Advanced model combining SE blocks with Convolutional Block Attention Module

Key Highlights

✅ Three model architectures with increasing complexity
✅ Complete training and inference pipelines
✅ Real-time webcam prediction support
✅ Batch processing capabilities
✅ Interactive visualizations
✅ Comprehensive performance analysis tools

🏗️ Model

1. Base CNN Model

Architecture: 4 convolutional blocks with batch normalization and max pooling

Conv(32) → BN → Pool → Conv(64) → BN → Pool → 
Conv(128) → BN → Pool → Conv(256) → BN → Pool → 
FC(512) → Dropout → FC(256) → Dropout → FC(26)

Features:

Standard CNN architecture
Batch normalization for stable training
Dropout regularization (0.2)
Baseline performance reference

Files: BaseCNNModel.py, Base_inference.py, Base_tester.py

2. CNN with SE Blocks

Architecture: Base CNN + Squeeze-and-Excitation blocks after each conv layer

[Conv(32) → BN → SE] → Pool → [Conv(64) → BN → SE] → Pool → 
[Conv(128) → BN → SE] → Pool → [Conv(256) → BN → SE] → Pool → 
FC(512) → Dropout → FC(256) → Dropout → FC(26)

SE Block Components:

Global Average Pooling (Squeeze)
Two FC layers with reduction ratio 16 (Excitation)
Sigmoid activation for channel weighting

Improvements over Base CNN:

📈 +2-3% validation accuracy
🎯 Better feature channel prioritization
💪 Reduced overfitting
🔍 Improved discrimination of similar signs

Files: Base_SEBlock_Model.py, SE_inference.py, BaseSE_tester.py

3. CNN with SE + CBAM

Architecture: SE blocks in first 3 layers + CBAM in final layer

[Conv(32) → BN → SE] → Pool → [Conv(64) → BN → SE] → Pool → 
[Conv(128) → BN → SE] → Pool → [Conv(256) → BN → CBAM] → Pool → 
FC(512) → Dropout → FC(256) → Dropout → FC(26)

CBAM (Convolutional Block Attention Module):

Channel Attention: Both average and max pooling features
Spatial Attention: Focuses on important spatial locations
Sequential refinement: Channel → Spatial attention

Improvements over CNN + SE:

📈 Additional 1-2% accuracy gain
🗺️ Spatial attention for better localization
🎯 Enhanced robustness to hand position variations
⚡ Dual attention mechanism (channel + spatial)

Files: Base_SE_CBAM_Model.py, SE_CBAM_inference.py, Base_SECbam_Tester.py

✨ Features

Training Features

✅ Data augmentation (rotation, flipping, color jittering, random erasing)
✅ Label smoothing for better generalization
✅ Learning rate scheduling (Cosine Annealing with Warm Restarts)
✅ Early stopping with patience
✅ Gradient clipping
✅ Model checkpointing (saves best model)
✅ Training history visualization

Inference Features

🖼️ Single image prediction with top-K results
📁 Batch processing with automatic accuracy calculation
🎥 Real-time webcam prediction with FPS counter
📊 Confusion matrix generation
📈 Performance analysis and comparison
💾 Results export to CSV
🎨 Beautiful visualizations

📦 Installation

Prerequisites

Python 3.8 or higher
CUDA-capable GPU (recommended) or CPU
Webcam (optional, for real-time prediction)

Setup

Clone the repository

git clone https://github.com/AdityaMohanty374/SignNet.git
cd SignNet

Create virtual environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies

pip install -r requirements.txt

Requirements

torch>=2.0.0
torchvision>=0.15.0
numpy>=1.24.0
matplotlib>=3.7.0
seaborn>=0.12.0
scikit-learn>=1.3.0
opencv-python>=4.8.0
Pillow>=10.0.0
tqdm>=4.65.0
pandas>=2.0.0

📂 Dataset Structure

Organize your sign language dataset as follows:

dataset/
├── A/
│   ├── image_001.jpg
│   ├── image_002.jpg
│   └── ...
├── B/
│   ├── image_001.jpg
│   └── ...
├── C/
│   └── ...
...
└── Z/
    └── ...

Dataset Requirements:

26 folders (A-Z)
Images in JPG/PNG format
Recommended: 200+ images per letter
Image size: Any (will be resized to 224×224)

Popular Datasets:

ISL Alphabet Dataset (Kaggle)
Sign Language MNIST

🚀 Usage

Training Models

1. Train Base CNN Model

python BaseCNNModel.py

2. Train CNN + SE Model

python Base_SEBlock_Model.py

3. Train CNN + SE + CBAM Model

python Base_SE_CBAM_Model.py

Training Configuration: Edit the hyperparameters in the respective Base_*.py files:

NUM_CLASSES = 26
BATCH_SIZE = 32
NUM_EPOCHS = 100
LEARNING_RATE = 0.001
WEIGHT_DECAY = 5e-4
DROPOUT_RATE = 0.3

Training Output:

best_sign_language_model.pth (Base CNN)
best_se_sign_language_model.pth (SE model)
best_se_cbam_sign_language_model.pth (SE+CBAM model)
Training history plots
Confusion matrices

Inference

Single Image Prediction

Base CNN:

from Base_inferece import SignLanguagePredictor

predictor = SignLanguagePredictor('best_sign_language_model.pth')
result = predictor.predict_image('test_image.jpg')
print(f"Predicted: {result['top_prediction']['letter']}")
print(f"Confidence: {result['top_prediction']['confidence']:.2f}%")

# With visualization
predictor.visualize_prediction('test_image.jpg', save_path='result.png')

SE Model:

from SE_inference import SESignLanguagePredictor

predictor = SESignLanguagePredictor('best_se_sign_language_model.pth')
predictor.visualize_prediction('test_image.jpg')

SE+CBAM Model:

from SE_CBAM_inference.py import SECBAMSignLanguagePredictor

predictor = SECBAMSignLanguagePredictor('best_se_cbam_sign_language_model.pth')
predictor.visualize_prediction('test_image.jpg')

Batch Processing

# Process entire folder
results = predictor.batch_predict('test_images/', output_csv='results.csv')

# Automatic accuracy calculation (if images named as: A_001.jpg, B_045.jpg, etc.)
# Output includes: filename, predicted_letter, confidence, true_label, correct

Real-time Webcam Prediction

# Start webcam prediction
predictor.predict_webcam(mirror=True, show_fps=True)

# Controls:
# - Press 'q' to quit
# - Press 's' to save current frame
# - Press 'r' to reset prediction

Performance Analysis

# Compare model performance on test set
analysis = predictor.compare_with_baseline('test_images/')
print(f"Accuracy: {analysis['accuracy']:.2f}%")
print(f"Average Confidence: {analysis['avg_confidence']:.2f}%")

📊 Model Performance

Benchmark Results

Model	Parameters	Val Accuracy	Training Time	Inference Speed
Base CNN	~50M	87-90%	~2 hours	30 FPS
CNN + SE	~52M (+4%)	89-93%	~2.5 hours	28 FPS
CNN + SE + CBAM	~53M (+6%)	91-95%	~3 hours	26 FPS

Tested on NVIDIA Tesla T4, batch size 32, 100 epochs

Key Improvements

CNN + SE vs Base CNN:

✅ +2-3% accuracy improvement
✅ Better generalization (smaller train-val gap)
✅ Improved feature channel selection
✅ More robust to lighting variations

CNN + SE + CBAM vs CNN + SE:

✅ +1-2% additional accuracy
✅ Better spatial feature localization
✅ Enhanced robustness to hand position
✅ Improved discrimination of similar signs (M/N, K/V)

Attention Mechanism Benefits

Benefit	SE Blocks	CBAM
Channel Attention	✅	✅
Spatial Attention	❌	✅
Parameter Overhead	+2%	+3%
Accuracy Gain	+2-3%	+3-5%
Overfitting Reduction	✅	✅✅

📈 Results

Confusion Matrix Examples

The models show excellent performance across all letters, with occasional confusion between similar hand shapes:

Common Confusions:

M ↔ N (similar finger positions)
K ↔ V (similar hand orientations)
A ↔ S (closed fist variations)

CBAM Improvements: The spatial attention in CBAM significantly reduces these confusions by focusing on discriminative spatial features.

Training Curves

All models show:

Smooth convergence with cosine annealing
Reduced overfitting with attention mechanisms
Stable validation performance

📁 Project Structure

sign-language-recognition/
├── README.md
├── requirements.txt
├── LICENSE
│
├── Models/
│   ├── BaseCNNModel.py                  # Base CNN model
│   ├── Base_SEBlock_Model.py            # CNN + SE model
│   └── Base_SE_CBAM_Model.py            # CNN + SE + CBAM model
│
├── Inference/
│   ├── Base_inference.py           # Base CNN inference
│   ├── SE_inference.py             # SE model inference
│   |── SE_CBAM_inference.py        # SE+CBAM inference
|   └── Testers/
|        ├── BaseSE_tester.py
|        ├── Base_SECbam_Tester.py
|        └── Base_tester.py
│
├── checkpoints/
│   ├── best_sign_language_model.pth
│   ├── best_se_sign_language_model.pth
│   └── best_se_cbam_sign_language_model.pth
│
└── results/
    ├── training_history/
    ├── confusion_matrices/
    └── predictions/

🔬 Technical Details

SE Block (Squeeze-and-Excitation)

class SEBlock(nn.Module):
    def __init__(self, channels, reduction=16):
        super(SEBlock, self).__init__()
        self.squeeze = nn.AdaptiveAvgPool2d(1)
        self.excitation = nn.Sequential(
            nn.Linear(channels, channels // reduction, bias=False),
            nn.ReLU(inplace=True),
            nn.Linear(channels // reduction, channels, bias=False),
            nn.Sigmoid()
        )
    
    def forward(self, x):
        b, c, _, _ = x.size()
        # Squeeze: Global information embedding
        y = self.squeeze(x).view(b, c)
        # Excitation: Adaptive recalibration
        y = self.excitation(y).view(b, c, 1, 1)
        # Scale: Channel-wise multiplication
        return x * y.expand_as(x)

CBAM Block

class CBAM(nn.Module):
    def __init__(self, channels, reduction=16, kernel_size=7):
        super(CBAM, self).__init__()
        
        # Channel Attention
        self.mlp = nn.Sequential(
            nn.Linear(channels, channels // reduction, bias=False),
            nn.ReLU(inplace=True),
            nn.Linear(channels // reduction, channels, bias=False)
        )
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.max_pool = nn.AdaptiveMaxPool2d(1)
        self.sigmoid_channel = nn.Sigmoid()
        
        # Spatial Attention
        self.conv_spatial = nn.Conv2d(2, 1, kernel_size, stride=1,
                                      padding=kernel_size // 2, bias=False)
        self.sigmoid_spatial = nn.Sigmoid()

    def forward(self, x):
        b, c, _, _ = x.size()
        
        # ---- Channel Attention ----
        avg_out = self.mlp(self.avg_pool(x).view(b, c))
        max_out = self.mlp(self.max_pool(x).view(b, c))
        channel_att = self.sigmoid_channel(avg_out + max_out).view(b, c, 1, 1)
        x = x * channel_att.expand_as(x)
        
        # ---- Spatial Attention ----
        avg_out = torch.mean(x, dim=1, keepdim=True)
        max_out, _ = torch.max(x, dim=1, keepdim=True)
        spatial_att = self.sigmoid_spatial(self.conv_spatial(torch.cat([avg_out, max_out], dim=1)))
        x = x * spatial_att.expand_as(x)
        
        return x

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

Areas for Contribution

🆕 New attention mechanisms (ECA, CBAM variants)
📊 Additional datasets and benchmarks
🚀 Model optimization (pruning, quantization)
🌐 Web/mobile deployment
📱 Mobile-optimized models
🎥 Video sequence recognition

Development Setup

# Clone repo
git clone https://github.com/AdityaMohanty374/SignNet.git

# Create branch
git checkout -b feature/your-feature

# Make changes and commit
git commit -m "Add your feature"

# Push and create PR
git push origin feature/your-feature

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Papers:
- Squeeze-and-Excitation Networks (Hu et al., 2018)
- CBAM: Convolutional Block Attention Module (Woo et al., 2018)
Datasets:
- Indian Sign Language Alphabet Dataset (Kaggle)
- Sign Language MNIST
Frameworks:
- PyTorch
- OpenCV
- Scikit-learn

📞 Contact

Author: Aditya Mohanty
Email: mohantyaditya589@gmail.com
GitHub: @AdityaMohanty374

🌟 Star History

If you find this project helpful, please consider giving it a star ⭐!

📚 Citation

If you use this code in your research, please cite:

@software{sign_language_recognition_2024,
  author = {Aditya Mohanty},
  title = {SignNet-Sign Language Recognition with Attention Mechanisms},
  year = {2024},
  url = {https://github.com/AdityaMohanty374/SignNet}
}

Made with ❤️ for the deaf and hard-of-hearing community

⭐ Star this repo if you find it helpful!

Latest commit

History

Folders and files

Repository files navigation