Irish Sign Language (ISL) Translation App

A comprehensive Irish Sign Language translation application for iOS and Web, featuring grammar-aware translation, real-time ISL-to-speech and speech-to-ISL translation, a crowdsourced dictionary, and continuous learning capabilities.

🌟 Features

Core Features

• ISL Dictionary: Browse and search Irish Sign Language signs with video demonstrations
• Grammar-Aware Translation: Authentic ISL grammar (Topic-Comment structure, not word-for-word)
• ISL to Speech: Real-time translation of sign language to spoken words using camera
• Speech to ISL: Convert spoken language to ISL gloss with non-manual markers
• Crowdsourcing: Community-driven sign submissions and validation
• Learning Mode: Structured courses and practice with AI feedback
• Offline Support: Download dictionary and models for offline use

🆕 Translation Engine Features

• Topic-Comment Word Order: Proper ISL sentence structure
• WH-Question Formation: WH-words at end with eyebrow markers
• Non-Manual Markers: Facial expressions, head movements, body position
• Classifiers: Entity, handling, and surface classifiers
• Time-First Grammar: Temporal markers at sentence start
• LLM-Powered: Uses OpenAI GPT-4 with ISL grammar knowledge

iOS App Features (v1.1)

• Enhanced Translation View: Glass-morphism UI with smooth transitions between Sign→Speech and Speech→Sign modes
• Hand Landmark Overlay: Real-time visualization of detected hand landmarks with customizable display
• Camera Controls: Improved camera handling with front-camera mirroring support
• Contribution Flow: Step-by-step contribution wizard with consent management, recording, and review
• Sign Playback: Word-by-word ISL translation playback with fingerspelling support
• Modern UI: Material-design inspired components with haptic feedback

Web App Features

• ISL Fingerspelling Alphabet: Interactive learning module for all 26 letters with descriptions and progress tracking
• Numbers Learning: Dedicated page for learning ISL number signs
• ISL Fundamentals: Core concepts and basic phrases learning module
• Grammar Lessons: Learn ISL grammar rules, classifiers, and sentence patterns
• Contribution Tracking: View your contributed signs and their validation status
• User Profiles: Track your learning progress and contributions

📁 Project Structure

IrishSignLanguage/
├── apps/
│   ├── ios/                    # Native SwiftUI iOS app
│   │   └── ISLTranslator/
│   │       ├── Models/         # Data models
│   │       ├── Services/       # API, Auth, Settings managers
│   │       ├── Views/          # SwiftUI views
│   │       │   ├── Contribute/ # Contribution flow with camera recording
│   │       │   ├── Dictionary/ # Sign dictionary browser
│   │       │   ├── Learn/      # Learning modules
│   │       │   ├── Profile/    # User profile
│   │       │   └── Translate/  # Real-time translation
│   │       └── Extensions/     # Swift extensions
│   └── web/                    # Next.js web application
│       └── src/
│           ├── app/            # App Router pages
│           │   ├── api/        # API routes
│           │   ├── auth/       # Authentication pages
│           │   ├── contribute/ # Contribution interface
│           │   ├── dictionary/ # Sign dictionary
│           │   ├── learn/      # Learning modules
│           │   └── translate/  # Web translation
│           ├── components/     # Reusable components
│           └── lib/            # Utilities and context
├── packages/
│   ├── api/                    # Node.js/Express backend API
│   │   └── src/
│   │       ├── middleware/     # Auth and error handling
│   │       ├── routes/         # API endpoints
│   │       └── services/       # Translation engine, R2 storage
│   ├── database/               # Prisma schema and migrations
│   │   └── prisma/             # Database schema and seeds
│   ├── data-scraper/           # 🆕 ISL data scraping tools
│   │   └── src/
│   │       ├── scrapers/       # IDS dictionary, linguistics scrapers
│   │       ├── processors/     # Video download, R2 upload
│   │       └── cli/            # Command-line tools
│   └── ml-models/              # ML model definitions and training
│       └── training/           # Model training scripts, Modal.com GPU training
├── infrastructure/
│   └── docker/                 # Container definitions
├── scripts/                    # Setup and utility scripts
├── supabase/                   # Supabase configuration
└── docs/                       # Documentation

🚀 Getting Started

Prerequisites

• Node.js 18+
• npm 10+
• Xcode 15+ (for iOS development)
• Python 3.10+ (for ML training)

Required Services

1. Supabase - PostgreSQL database and auth
2. Firebase - User authentication
3. Cloudflare R2 - Video storage (with zero egress fees)
4. OpenAI API - Translation engine
5. Vercel - Web app hosting (optional)
6. Modal.com - Cloud GPU training (optional)

Installation

1. Clone the repository:

git clone https://github.com/your-org/irish-sign-language.git
cd irish-sign-language

2. Install dependencies:

npm install

3. Set up environment variables:

cp .env.example .env.local
# Edit .env.local with your configuration

Required environment variables:

# Database
DATABASE_URL=postgresql://...
DIRECT_URL=postgresql://...

# Firebase Auth
FIREBASE_API_KEY=...
FIREBASE_AUTH_DOMAIN=...
FIREBASE_PROJECT_ID=...

# OpenAI (for translation engine)
OPENAI_API_KEY=sk-...

# Cloudflare R2 (for video storage)
CLOUDFLARE_ACCOUNT_ID=...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=isl-videos
R2_PUBLIC_URL=https://...

4. Set up the database:

# Generate Prisma client
npm run db:generate

# Push schema to database
npm run db:push

# Seed with grammar rules and sample data
cd packages/database
npm run seed:all

5. Start development servers:

# Start all services
npm run dev

# Or start individually
npm run api:dev    # Backend API (port 3001)
npm run web:dev    # Web frontend (port 3000)

iOS Development

1. Open the iOS project in Xcode:

open apps/ios/ISLTranslator.xcodeproj

2. Select your target device/simulator and run

Web Development

1. Navigate to the web app:

cd apps/web

2. Start the development server:

npm run dev

3. Open http://localhost:3000

🔧 Technology Stack

iOS App

• SwiftUI with iOS 17+ features
• Vision Framework (hand pose detection)
• Core ML (on-device inference)
• AVFoundation (camera/video recording)
• Speech Framework (TTS/STT)
• Combine for reactive updates

Web App

• Next.js 14 (App Router)
• TensorFlow.js + MediaPipe
• Tailwind CSS
• TypeScript
• Framer Motion (animations)
• Lucide React (icons)

Backend

• Node.js + Express
• PostgreSQL + Prisma ORM
• Firebase Auth
• OpenAI GPT-4 (translation engine)
• Cloudflare R2 (video storage)

ML Pipeline

• TensorFlow/Keras
• MediaPipe (hand pose)
• Modal.com (cloud GPU training)
• Core ML export for iOS
• TFLite export for mobile

Data Pipeline

• Puppeteer (web scraping)
• Cloudflare R2 (storage)
• PostgreSQL (metadata)

📡 API Endpoints

Authentication

• POST /api/auth/register - User registration
• GET /api/auth/me - Get current user

Signs & Dictionary

• GET /api/signs - List all signs
• GET /api/signs/:id - Get sign details
• GET /api/signs/search/:query - Search signs
• GET /api/categories - List categories

Translation (🆕)

• POST /api/translate - Translate English to ISL gloss
• POST /api/translate/batch - Batch translation
• POST /api/translate/feedback/correct - Mark translation correct
• POST /api/translate/feedback/correction - Submit correction
• GET /api/translate/grammar-rules - Get ISL grammar rules
• GET /api/translate/classifiers - Get ISL classifiers
• GET /api/translate/non-manual-markers - Get facial/body markers
• GET /api/translate/sentence-patterns - Get sentence patterns

Contributions

• GET /api/contributions - List contributions
• POST /api/contributions - Submit new contribution
• GET /api/contributions/my - Get user's contributions

Upload (R2)

• POST /api/upload/sign-video - Upload sign video to R2
• POST /api/upload/contribution-video - Upload contribution video
• GET /api/upload/:key - Get signed URL for R2 object

🗄️ Database Schema

Core Models

• User - User accounts with roles and consent
• Sign - Dictionary entries with translations
• Category - Sign categories and subcategories
• SignMedia - Videos and images for signs

Grammar Models (🆕)

• GrammarRule - ISL grammar rules with examples
• Classifier - Handshape classifiers (CL:1, CL:3, etc.)
• NonManualMarker - Facial expressions, head movements
• SentencePattern - ISL sentence templates

Translation Models (🆕)

• TranslationPair - English ↔ ISL gloss pairs
• TranslationSession - Translation history for learning
• ScrapedSource - Data source tracking
• ScrapedEntry - Scraped dictionary entries

ML Models

• HandPoseData - Training data from videos
• MLModel - Model versions and metrics
• TranslationFeedback - User feedback for improvement

🔄 Continuous Learning Pipeline

1. Data Collection: User interactions generate training data
2. Feedback Loop: Users can correct translations
3. Export: python training/export_training_data.py
4. Train: modal run training/modal_train.py --alphabet
5. Deploy: Models uploaded to R2, served to apps

📊 Data Scraping

Scrape ISL resources from official sources:

cd packages/data-scraper

# Scrape IDS dictionary
npm run scrape:ids

# Download and process videos to R2
npm run process:videos --limit 50

Sources:

• Irish Deaf Society Dictionary
• ISL Linguistics Videos (Vimeo)
• DCU STEM Glossary
• Oireachtas ISL Glossary

☁️ Cloud Infrastructure

Recommended Architecture

┌──────────────────┐     ┌──────────────────┐
│   Vercel         │     │   Cloudflare R2  │
│   (Web App)      │     │   (Video/Media)  │
└────────┬─────────┘     └────────┬─────────┘
         │                        │
         │    ┌───────────┐       │
         └────┤ Supabase  ├───────┘
              │ (Database)│
              └───────────┘

• Vercel: Web app hosting (Next.js)
• Cloudflare R2: Video storage (zero egress fees!)
• Supabase: PostgreSQL database
• Firebase: User authentication
• Modal.com: Cloud GPU training

Why Cloudflare R2?

Video streaming is bandwidth-intensive. R2 has zero egress fees, making it ideal for serving sign language videos:

• 1TB video served = $0 with R2 vs ~$90 with AWS S3
• Automatically cached at edge
• S3-compatible API

🤝 Contributing

We welcome contributions from the community! Please see CONTRIBUTING.md for guidelines.

Crowdsourcing

Help expand the ISL dictionary by contributing sign videos through the app. All contributions go through community validation to ensure accuracy.

Development Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📅 Recent Updates

January 2026

• Translation Engine: Grammar-aware English→ISL translation with GPT-4
• ISL Grammar Database: Rules, classifiers, non-manual markers, sentence patterns
• Data Scraping: Automated scraping from Irish Deaf Society
• Cloudflare R2 Integration: Zero-egress video storage
• Cloud GPU Training: Modal.com integration for model training
• Continuous Learning: User feedback pipeline for model improvement
• Enhanced iOS translation UI with glass-morphism design
• Added hand landmark visualization overlay
• ISL fingerspelling alphabet learning module for web

📜 License

This project is licensed under the MIT License - see LICENSE for details.

🙏 Acknowledgments

• Irish Deaf Society for linguistic resources
• ISL researchers at UCD and DCU
• The Irish deaf community
• All our community contributors

📧 Contact

For questions or partnerships, contact: [your-email@example.com]

---

Note: This project uses ISL (Irish Sign Language), which is distinct from BSL (British Sign Language) and ASL (American Sign Language). Each sign language has its own grammar and vocabulary.