CaptionAid

CaptionAid is a Chrome/Edge extension and companion website backed by Flask. Both surfaces prepare a complete timed YouTube transcript, then the shared backend converts it to ASL gloss and matches clips from word_to_url.json. Captions and sign clips appear beside the playing video in either surface, and both write to the same durable history. The AssemblyAI audio adapter remains available when a video has no usable YouTube caption track.

Prerequisites

• Python 3.11 or newer is recommended.
• Node.js 22+ for the React companion website.
• AWS credentials with access to the configured S3 bucket.
• AssemblyAI API key.
• Gemini API key is optional. Without it, the backend uses its deterministic fallback glossing logic.

Captioned YouTube videos use the fast transcript-first path. Videos without a usable caption track fall back to audio transcription, which is slower and can be affected by YouTube download restrictions in cloud environments.

For the deployed companion website, keep the unpacked CaptionAid extension loaded. YouTube blocks transcript and media requests from many data-center IPs, including Vercel. The extension reads the public timed caption track from the user's browser and sends only the transcript and timestamps to the shared backend. The website stays on the same page; glossing, clip matching, playback, and durable history still happen through the deployed app.

One-Time Setup

Pick one environment (WSL, macOS, or Windows) and use it for both Python and Node. Do not reuse venv or frontend/node_modules between WSL and Windows; their native packages are different.

Create one virtual environment at the repository root and install the backend:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

On Windows PowerShell, activate with:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Install the companion website from that same environment:

cd frontend
npm ci --include=optional
cd ..

Confirm Node is visible:

node --version

Environment

Create .env in the repository root from .env.example. These names are exact; boto3 and the backend will not read aliases such as AWS_ACCESS_KEY or ASSEMBLY_AI.

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-2
S3_BUCKET=...
ASSEMBLYAI_API_KEY=...
GEMINI_API_KEY=...

Do not commit .env.

Run Every Day

Start the backend from the repository root:

source .venv/bin/activate
python3 app.py

On Windows PowerShell, activate with .\.venv\Scripts\Activate.ps1 and run python app.py instead.

The backend runs at http://127.0.0.1:5001. Verify it in another terminal:

curl http://127.0.0.1:5001/health

The response should be {"ok":true}.

Run The Companion Website

The React companion website and browser extension use the same Flask API and caption records. Keep the backend running, then use a second terminal:

cd frontend
npm run dev

Open http://127.0.0.1:5173. Vite proxies /api requests to the Flask backend at http://127.0.0.1:5001.

The website stays focused on two real workflows:

• / accepts a YouTube URL through Prepare video, prepares it without opening another tab, and reviews the shared caption history, ASL gloss, and synchronized sign clips.
• /signs searches and plays entries from the complete word_to_url.json vocabulary.

Deleting a History item removes its transcript and matched clips. In the deployed S3-backed app it also removes that video's captured audio and job files.

The History page refreshes automatically while the extension processes a video.

For a production-style local build, compile the frontend first and then start Flask. Flask serves the built site and API from the same port:

cd frontend
npm ci --include=optional
npm run build
cd ..
python3 app.py

Then open http://127.0.0.1:5001.

Deploy To Vercel

The repository includes Dockerfile.vercel, so the companion website and Flask API deploy together as one Vercel project and share one URL. The image builds the React site, runs Python 3.12, and serves the production build through Gunicorn.

Production does not use SQLite for extension captions. Vercel instances are temporary, so every AssemblyAI chunk job and caption result is stored in the existing S3 bucket under captionaid/v2/. No additional database is required.

1. Push this branch to the repository hosted on GitHub.
2. In Vercel, choose Add New > Project and import the repository.
3. In Settings > Build and Deployment, set the Framework Preset to Services. Keep the project root set to the repository root. Do not set a custom build command or output directory. vercel.json explicitly routes the deployment to Dockerfile.vercel.
4. Add these environment variables for Production and Preview:

CAPTION_AWS_ACCESS_KEY_ID=...
CAPTION_AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-2
S3_BUCKET=...
ASSEMBLYAI_API_KEY=...
GEMINI_API_KEY=...

The CAPTION_AWS_* values are the same IAM access key ID and secret normally stored locally as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The aliases avoid collisions with platform-managed AWS variables inside Vercel containers; the standard names remain supported for local development.

GEMINI_API_KEY is optional. The AWS identity needs s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket access to the configured bucket. Do not add CAPTION_STORE; the Vercel image sets it automatically.

5. Press Deploy. After the deployment is ready, open /api/health on its URL. A correctly configured deployment responds with {"ok":true} after verifying real access to the S3 bucket.
6. Reload the unpacked extension. Test preparation once from the deployed website and once from a YouTube tab to confirm both surfaces populate the same History page.

After reloading the extension, refresh any website or YouTube tabs that were already open so the updated bridge/content scripts are injected.

If /api/health returns 503, its missing list names the environment variables that still need to be added. After changing variables in Vercel, redeploy so the new values reach the running app.

Load The Extension

1. Open edge://extensions or chrome://extensions.
2. Enable Developer mode.
3. Choose Load unpacked and select the repository's extension folder.
4. After any extension code change, press Reload on the extension card and refresh the YouTube tab.
5. Open a public YouTube video that has English captions.
6. Open CaptionAid and press Prepare captions. CaptionAid pauses the video, resolves the complete timed transcript, and builds all gloss/sign matches.
7. Wait for the popup to say the captions are ready. The overlay is shown automatically; use Show captions and Hide captions to control it, then press play to keep the captions and sign clips synchronized.

The extension uses https://seo-final-project.vercel.app by default so its captions always appear in the deployed companion website. For local extension development, set captionAidBackend in chrome.storage.local to http://localhost:5001; clear that override before the deployed demo.

Verify

Run the local pipeline verification without making AssemblyAI or YouTube calls:

python -m unittest discover -s tests -v
python backend/verify_pipeline.py
cd frontend && npm run build

For a full caption test, use the extension and then inspect the result:

curl http://127.0.0.1:5001/captions/video/VIDEO_ID
curl http://127.0.0.1:5001/api/sessions

A successful caption chunk has status: "ready", transcript text, non-empty gloss, and non-empty clips when vocabulary words match. Prepared YouTube caption tracks do not include reliable speaker labels; the AssemblyAI audio adapter is the path that supplies diarization. CaptionAid prefers the small timed transcript track over downloading an entire video, which keeps prepared mode faster and avoids the media-download failure seen on Vercel.