GLOSSA - YOUR ASL Intepreter

🧠 ASL Fingerspelling Recognition and Word Resolution System

This project implements a real-time ASL (American Sign Language) fingerspelling recognition pipeline on AWS, paired with a MongoDB Atlas Search–powered lexicon for correction and disambiguation.

It supports both online decoding (low-latency letter aggregation → word resolution) and offline ingestion (document uploads → Textract OCR → Strand Agent–driven alias generation), leveraging Atlas fuzzy search (and optional autocomplete) over surface and aliases to map noisy sequences (e.g., AW6 → AWS).

---

🚀 System Overview

🩵 Online Path — Real-Time Recognition

1. Client → API Gateway (WebSocket) → Lambda

   • The client streams landmark frames from the camera.
   • The inbound Lambda authenticates sessions and publishes landmarks to a Kinesis stream (asl-landmarks-stream).

2. Word Resolver Service

   • Consumes letter predictions, aggregates them per session, and applies stability logic (≈200 ms sliding window).
   • Uses Redis for session state (windows + committed letters).
   • Once a stable letter sequence is formed, it queries the RAG service (MongoDB Atlas Search) to correct or expand partial words (e.g., "AW6T" → "AWS").

3. Outbound Lambda → API Gateway Management API

   • Pushes finalized word results back to the client over the same WebSocket connection.
   • Supports per-session acknowledgements and latency tracking.

---

📘 Offline Path — Knowledge Base Construction

1. User Uploads Document

   • PDFs are uploaded to kb_uploads (S3).
   • The SubmitLambda starts an Amazon Textract job and records job metadata in DynamoDB (kb_jobs).

2. SNS → SQS → Ingest Lambda

   • When Textract finishes, it posts to SNS → SQS.
   • The ingest Lambda retrieves text, removes stopwords/URLs, deduplicates tokens, and writes clean words to kb_raw.

3. Alias Builder Lambda (Strand Agent)

   • Processes the cleaned tokens in batches (100–200).
   • Invokes an LLM-backed Strand Agent with domain-specific confusion rules (e.g., W↔6, O↔0, N↔M).
   • The agent generates realistic spelling-level alias variants such as:

     {"surface": "AWS", "aliases": ["AW6", "A-W-S", "A W S"]}

   • Confidence-weighted filtering ensures only high-quality aliases (≥ 0.5 score) are retained.
   • Aliases are written to kb_aliases (S3) and bulk-upserted into MongoDB Atlas.

---

🧩 MongoDB Atlas Search — Lexicon & RAG Layer

Each document in the lexicon collection represents a known token (or phrase) for a specific user/tenant.

{
  "surface": "16-SERVICE",
  "user_id": "thongnguyen",
  "aliases": ["16 SERVICE", "16-SER2ICE", "16-SER7ICE"],
  "confidence_scores": { "16 SERVICE": 1.0, "16-SER2ICE": 0.93 },
  "alias_count": 3,
  "type": "unigram"
}