VidMind

Fully local, AI-powered video intelligence platform.
Upload a video or paste a YouTube URL → get a structured document, transcript, summary, and downloadable files. Zero cloud. All local.

---

System Requirements

Hardware

Component	Minimum	Recommended
RAM	16 GB	32 GB
GPU VRAM	8 GB	24 GB+
Disk	20 GB free	100 GB+ free
CPU	4 cores	8+ cores

GPU is optional but strongly recommended. Without GPU, Whisper and LLM inference will be significantly slower.

Software

• Docker + Docker Compose (v2.22+)
• NVIDIA Container Toolkit (for GPU passthrough, optional)
• Ports: 3000 (frontend), 8000 (backend), 9000 (Whisper), 11434 (Ollama), 6379 (Redis)

---

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        User Browser (:3000)                       │
└───────────────────────────┬──────────────────────────────────────┘
                            │ HTTP / WebSocket
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│  nginx (frontend)  ──────proxy──▶  FastAPI Backend (:8000)       │
│  React SPA          ◀──────/api────  Uvicorn                      │
└───────────────────────────────────┬──────────────────────────────┘
          │                 │                   │
          ▼                 ▼                   ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐
│  Ollama      │  │  Whisper     │  │  Redis (:6379)       │
│  (:11434)    │  │  ASR (:9000) │  │  (Celery broker)     │
│  LLMs        │  │  faster-     │  └──────────┬───────────┘
│  Embeddings  │  │  whisper     │             │
└──────────────┘  └──────────────┘             ▼
                                      ┌──────────────────────┐
                                      │  Celery Worker       │
                                      │  (solo, concurrency=1)│
                                      │  Pipeline execution  │
                                      └──────────────────────┘

Service Communication

From	To	Protocol	Purpose
Browser	nginx	HTTP/WS	SPA + live transcription
nginx	FastAPI	HTTP	Reverse proxy /api
FastAPI	Ollama	HTTP REST	LLM inference + embeddings
FastAPI	Whisper	HTTP REST	Speech-to-text
FastAPI	Redis	Redis protocol	Celery task broker
Celery Worker	Ollama	HTTP REST	LLM agents (clean/structure/summarize)
Celery Worker	Whisper	HTTP REST	Transcription
Celery Worker	yt-dlp/ffmpeg	Subprocess	Video download + audio extraction
Celery Worker	Disk	File I/O	Session storage (JSON + media)

Docker Compose Services

Service	Image	Port	Memory Limit	GPU
ollama	ollama/ollama:latest	11434	6 GB	Yes (reserved)
whisper	fedirz/faster-whisper-server:latest-cuda	9000	6 GB	Yes (reserved)
redis	redis:7-alpine	6379	256 MB	No
backend	Custom (backend/Dockerfile)	8000	1 GB	No
frontend	Custom (frontend/Dockerfile, nginx)	80 → 3000	128 MB	No
worker	Custom (backend/Dockerfile)	—	2 GB	No

All services share a bridge network vidmind_net.

---

Tools & How They Connect

AI / ML Stack

Tool	Role	How It Connects
Ollama	Local LLM server	FastAPI and Celery worker call POST /api/generate and /api/chat via httpx. Used for transcript cleaning, structuring, summarization, and embedding generation.
faster-whisper-server	ASR inference	FastAPI and Celery worker call POST /v1/audio/transcriptions with WAV file. Model: Byne/whisper-large-v3-arabic. Chunks audio > 600s into overlapping segments.
pyannote.audio	Speaker diarization	Optional. Returns speaker segments that get merged into transcript timestamps. Requires HuggingFace token.
diarize	Speaker diarization	Optional. Open-source alternative to pyannote, no token required.
Demucs (torchaudio)	Audio preprocessing	Optional. Separates speech from background noise/music before Whisper transcription.
deepmultilingualpunctuation	Punctuation restoration	Optional. Post-processes raw Whisper output via HuggingFace pipeline.
nomic-embed-text (Ollama)	Text embeddings	Called via Ollama API for semantic search. Combined with keyword matching (rapidfuzz) for hybrid search scoring.

Video / Audio Stack

Tool	Role	How It Connects
yt-dlp	Video download	Subprocess call from Celery worker. Downloads from YouTube, Twitter, Vimeo, etc.
ffmpeg	Audio extraction	Subprocess call. Extracts 16 kHz mono WAV from downloaded video. Handles chunking.

Document Generation Stack

Tool	Role
python-docx	Generate .docx files with structured sections
reportlab	Generate .pdf files
fpdf2	Alternative PDF generation
markdown2	Convert markdown content to HTML
arabic-reshaper + python-bidi	Arabic text rendering support

Frontend Stack

Tool	Role
React 18	UI framework
Vite 5	Build tool + dev server
Tailwind CSS 3	Utility-first styling with custom dark theme
React Router 6	Client-side routing
Axios	HTTP client → FastAPI
lucide-react	Icon library

---

How LLMs Are Used

VidMind uses Ollama to run local LLMs. No cloud API calls.

LLM Service (backend/app/services/llm_service.py)

• Endpoint: POST /api/generate or /api/chat to OLLAMA_BASE_URL (default http://ollama:11434)
• Model selection: Uses DEFAULT_LLM_MODEL env var; falls back to first available model if not found
• Retry: 3 attempts with exponential backoff (tenacity)
• Timeout: 900s configurable read timeout
• Memory management: Can unload LLM from GPU before Whisper transcription to free VRAM
• Keep-alive: Models stay loaded between requests via keep_alive parameter

LLM Agents (backend/app/agents/pipeline_agents.py)

The pipeline runs 4 sequential LLM agents, each with a specialized system prompt:

Agent	System Prompt	Input	Output	Purpose
1. Cleaner	CLEAN_SYSTEM	Raw transcript	Cleaned transcript	Fix grammar, punctuation, filler words ("umm", "uh"), preserve technical terms. Never translates Arabic.
2. Structurer	STRUCTURE_SYSTEM	Cleaned transcript	JSON sections [{title, content}]	Break into 3–10 logical sections with descriptive titles.
3. Summarizer	SUMMARIZE_SYSTEM	Structured sections	[{title, summary, key_points}]	Concisely summarize each section with 3–5 bullet key points.
4. Executive	EXECUTIVE_SYSTEM	All prior outputs	Executive summary + key takeaways	High-level overview. If ENABLE_MULTI_SUMMARY=true, also generates short, detailed, bullet-point, and Q&A format summaries.

Embeddings

• Semantic search: Uses nomic-embed-text via Ollama to generate embeddings
• Storage: In-memory numpy array per session
• Scoring: Combined keyword (rapidfuzz) + semantic (cosine similarity) scores

Recommended Models

Model	Size	Speed	Quality	Best For
llama3.1:70b	40 GB	Slow	Best	Production with GPU
llama3.1:8b	5 GB	Fast	Great	Most users
llama3.2:3b	2 GB	Very Fast	Good	Quick testing
qwen2.5:72b	45 GB	Slow	Best	Arabic content
aya:35b	20 GB	Medium	Great	Multilingual content
qwen2.5:7b	4.5 GB	Fast	Great	Arabic + general

---

Workflows

Main Pipeline (9 steps)

The pipeline is orchestrated by Celery and defined in backend/app/services/pipeline_plan.py. Steps are tracked in session.json with completed/failed status.

Input: Video URL / Upload / Audio / Transcript
  │
1. download ───────────────── yt-dlp (or file copy for uploads)
  │                            ✗ Skipped if audio/transcript input
  ▼
2. extract_audio ──────────── ffmpeg → 16 kHz mono WAV
  │                            ✗ Skipped if audio/transcript input
  ▼
3. transcribe ─────────────── Whisper ASR (faster-whisper-server)
  │                            Chunked overlap for audio > 600s
  │                            ✗ Skipped if transcript input
  ▼
4. diarize ────────────────── Heuristic or ML diarization (optional)
  │                            Assigns Speaker N: labels to timestamps
  ▼
5. clean ──────────────────── Ollama Agent 1: Clean transcript
  ▼
6. structure ──────────────── Ollama Agent 2: JSON sections
  ▼
7. summarize ──────────────── Ollama Agent 3 + 4
  │                            Executive summary + key takeaways
  │                            Optional: Multi-style summaries
  ▼
8. islamic ────────────────── Fuzzy matching against Quran/Hadith datasets
  │                            Not LLM-based; uses rapidfuzz sliding window
  │                            Threshold: 72% (Quran), 68% (Hadith)
  ▼
9. generate_docs ──────────── TXT + DOCX + PDF generation
  │                            Also generates SRT + VTT
  ▼
Output: sessions/<topic>/<date>/<session-id>/

Pipeline Control

Feature	Mechanism
Pause	_check_pause_stop() between steps checks session.json flag
Resume	Restarts from the failed step, rebuilding context
Stop	Sets a stop flag; pipeline exits at next check
Retry	User can retry from any failed step via RetryPanel UI

Input Type Handling

Input Type	Steps Skipped	Starts At
Video upload	—	Step 1
YouTube URL	—	Step 1
Audio upload	1, 2	Step 3
Transcript upload (.txt/.srt/.vtt)	1–3	Step 5

Recording Multi-Track

Each session supports multiple recordings (video/audio files). Tracks are independently processed and stored under sessions/<session-id>/recordings/<recording-id>/.

Live Transcription

Browser Mic ──WebSocket──▶ FastAPI ──HTTP──▶ Whisper ASR
                         Real-time streaming transcription
                         POST /api/live/start
                         WS  /api/live/ws/{session_id}

Watch Folder Auto-Transcription

Filesystem ──watchdog──▶ WatchFolderService ──▶ Celery Task
                       Monitors directory for new media files
                       Automatically creates sessions and starts pipeline

YouTube Caption Fetch

User input URL ──▶ YouTubeTranscript API ──▶ YouTube captions
                 Standalone endpoint (bypasses Whisper ASR)
                 GET /api/transcript/youtube?url=...

Search

Search Query
  ├── Keyword: rapidfuzz fuzzy matching against section titles + content
  └── Semantic: nomic-embed-text embeddings → cosine similarity
      ↓
 Combined score → ranked results across all sessions

Document Export

Session Data
  ├── TXT: Plain text with chapter/section structure
  ├── DOCX: Word document with styled headers + structure
  ├── PDF: Reportlab-generated print-ready document
  ├── SRT: Timestamped subtitle format
  ├── VTT: Web subtitle format
  └── ZIP: Full archive of all session files

---

Quick Start

Prerequisites

• Docker + Docker Compose
• NVIDIA Container Toolkit (GPU recommended)
• 40 GB+ disk (for 70B model) — or use llama3.1:8b (5 GB)

Start

chmod +x setup.sh
./setup.sh

Open http://localhost:3000

Manual Start

docker compose up -d --build

# Pull your preferred model
docker exec vidmind_ollama ollama pull llama3.1:8b

---

Configuration

Variable	Default	Description
DEFAULT_LLM_MODEL	llama3.1:8b	Ollama model for pipeline agents
OLLAMA_BASE_URL	http://ollama:11434	Ollama server URL
OLLAMA_READ_TIMEOUT	900	LLM request timeout (seconds)
WHISPER_MODEL	Byne/whisper-large-v3-arabic	Whisper model ID
WHISPER_CHUNK_SECONDS	600	Max seconds per audio chunk
CHUNK_OVERLAP_SECONDS	60	Overlap between audio chunks
SESSIONS_DIR	/app/sessions	Session storage path
DATA_DIR	/app/data	Reference data path
REDIS_URL	redis://redis:6379	Celery broker URL
ENABLE_PUNCTUATION_RESTORATION	false	Enable punctuation restoration
ENABLE_SPEAKER_DIARIZATION	false	Enable ML-based speaker diarization
DIARIZATION_BACKEND	pyannote	pyannote or diarize
ENABLE_MULTI_SUMMARY	false	Generate multiple summary formats
ENABLE_AUDIO_PREPROCESSING	false	Enable Demucs enhancement
WATCH_FOLDER_PATH	—	Directory to watch for auto-transcription
WATCH_FOLDER_RECURSIVE	true	Watch subdirectories recursively
HUGGING_FACE_TOKEN	—	Required for pyannote diarization

---

Output Structure

sessions/
└── machine-learning-lecture/          ← topic slug
    └── 2025-01-15/                    ← date
        └── <session-id>/              ← unique session folder
            ├── session.json               ← metadata + step status
            ├── video.mp4                  ← original video
            ├── video_audio.wav            ← extracted audio
            ├── transcript_raw.txt         ← raw Whisper output
            ├── transcript_timestamped.txt ← with [HH:MM:SS] Speaker N:
            ├── transcript_clean.txt       ← AI-cleaned transcript
            ├── transcript.srt             ← subtitle format
            ├── transcript.vtt             ← web subtitle format
            ├── document.txt               ← full structured document
            ├── document.docx              ← Word document
            └── document.pdf               ← PDF document

---

API Endpoints

Method	Path	Purpose
GET/POST/DELETE	/api/sessions	Session CRUD
POST	/api/sessions/{id}/pipeline/start	Start pipeline
POST	/api/sessions/{id}/pipeline/pause	Pause pipeline
POST	/api/sessions/{id}/pipeline/resume	Resume pipeline
POST	/api/sessions/{id}/pipeline/stop	Stop pipeline
GET	/api/sessions/{id}/pipeline/status	Pipeline status
POST	/api/sessions/{id}/pipeline/retry	Retry failed step
GET	/api/export/session/{id}/transcript	Export transcript (txt/srt/vtt)
GET	/api/export/session/{id}/zip	Export ZIP archive
POST	/api/export/bulk	Bulk ZIP export
GET/PUT	/api/editor/{id}	Markdown editor
GET/POST	/api/tags	Tag management
GET	/api/search	Full-text + semantic search
GET	/api/islamic/references	Islamic reference lookup
POST	/api/elevenlabs/transcribe	ElevenLabs STT
POST	/api/live/start	Start live transcription
WS	/api/live/ws/{session_id}	Live transcription WebSocket
POST	/api/transcript/youtube	Fetch YouTube captions
GET	/api/models	List Ollama models
POST	/api/models/pull	Pull new Ollama model

---

Logs & Debugging

# All services
docker compose logs -f

# Individual services
docker compose logs -f worker
docker compose logs -f backend
docker compose logs -f ollama

# Check Ollama models
docker exec vidmind_ollama ollama list

# Restart
docker compose down
docker compose up -d --build

# Full reset (wipe volumes)
docker compose down -v

---

Known Issues

See FIXES.txt for 6 documented bugs and patch instructions, including:

• Duplicate volumes: key in docker-compose.yml
• dir() name collision in tasks.py
• Missing import arrow in sessions.py
• Language "auto" forcing Arabic incorrectly
• .env not in .gitignore (exposed API key)
• React useState evaluation timing in Dashboard.jsx

---

Tech Stack

• Frontend: React 18 + Vite 5 + Tailwind CSS 3 + React Router 6
• Backend: Python 3.11 + FastAPI + Celery + Redis
• AI: Ollama (LLMs) + faster-whisper-server (ASR) + pyannote.audio (diarization) + Demucs (audio enhancement) + deepmultilingualpunctuation
• Video: ffmpeg + yt-dlp
• Documents: python-docx + reportlab + fpdf2
• Search: rapidfuzz + nomic-embed-text (semantic)
• Live: WebSocket real-time microphone transcription
• Islamic References: Local Quran + Hadith JSON datasets (fuzzy matching)
• Containerization: Docker Compose (6 services, bridge network)
• License: MIT