Semantic search over your saved Instagram Reels — running entirely on your own hardware.
You saved 2999 Reels. You remember one of them had a purple car in it. Instagram gives you an endless grid and no way to ask.
ReelScope watches each video with a vision-language model, listens to it with Whisper, reads the text on screen, and indexes the result so you can just ask for what you remember — by object, action, mood, spoken advice, or on-screen text. No cloud API, no third-party service: every model runs locally on vLLM (NVIDIA) or MLX / LM Studio (Apple Silicon).
Real queries answered by the Telegram bot, against an index built from 3,395 saved posts — the corpus this project was developed and tested on.
“exercises in the gym for shoulders” Finds the movement, not the hashtag. |
“series of motivational quotes” Matches text rendered inside the video. |
“purple or pink car with fish design” The video from the top of this page — found from memory alone. |
“neon streets of Japan at night” Place, time of day, and mood in one query. |
flowchart TD
A[Saved collection<br/>Playwright] --> B[(URL registry · SQLite)]
B --> C[Download<br/>gallery-dl · yt-dlp]
C --> D[Adaptive frame sampling<br/>ffmpeg]
C --> E[Speech-to-text<br/>Whisper]
D --> F[Vision-language model<br/>Qwen3.5]
E --> F
F --> G[Structured description<br/>JSON]
G --> H[(Dense vectors<br/>sqlite-vec)]
G --> I[(Lexical index<br/>FTS5)]
flowchart TD
Q([Search query]) --> H[Dense retrieval<br/>Qwen3-Embedding]
Q --> I[Lexical retrieval<br/>BM25 · FTS5]
H --> J[Reciprocal rank fusion]
I --> J
J --> K[Cross-encoder rerank<br/>Qwen3-Reranker]
K --> L([Ranked results<br/>CLI · Telegram bot])
Retrieval is a two-stage pipeline, not a single vector lookup:
- Dense retrieval — the VLM description is embedded with
Qwen3-Embedding-0.6B(1024-d, stored insqlite-vec). The retrieval instruction is applied to the query only; documents are embedded bare. - Lexical retrieval — transcript, caption, hashtags, and on-screen text go into SQLite FTS5 with independently tunable BM25 field weights.
- Fusion — the two rankings are merged with reciprocal rank fusion, so neither branch can dominate on raw score scale.
- Reranking —
Qwen3-Reranker-0.6Breads each query/description pair and scores the probability that they match. Because those are absolute probabilities rather than ranks, a threshold can return fewer results instead of padding the list with noise.
- Multimodal understanding — frames, speech, and on-screen text are fused into one description per post.
- Structured output — category, mood, topics, objects, actions, setting, and language, not just a blob of prose.
- Hybrid search — dense + BM25 + RRF + cross-encoder reranking, all tunable per request.
- Multilingual — queries and descriptions share one embedding space, so the query language does not have to match the video's.
- Fully local — vLLM on CUDA, MLX / LM Studio on Apple Silicon. Weights and data never leave the machine.
- Telegram bot — the whole pipeline (export, ingest, search, status, settings, cookie rotation) from your phone.
- Resumable by design — every URL carries independent processing and embedding lifecycle state, so an interrupted run picks up exactly where it stopped.
Every model runs on both platforms — only the runtime underneath changes.
| Layer | Model | Linux · NVIDIA | macOS · Apple Silicon |
|---|---|---|---|
| Vision-language | Qwen3.5 | vLLM server | LM Studio server |
| Embeddings | Qwen3-Embedding-0.6B · 1024-d | vLLM server | MLX in-process, 4-bit |
| Reranking | Qwen3-Reranker-0.6B | vLLM server | MLX in-process |
| Speech | Whisper large-v3-turbo |
faster-whisper · CUDA | mlx-whisper |
| Storage | SQLite + sqlite-vec + FTS5 |
| Media | ffmpeg, gallery-dl, yt-dlp, Playwright |
| Interface | argparse CLI, aiogram 3 Telegram bot |
| Tooling | uv, ruff, pytest, Docker Compose, GitHub Actions |
The models are never all resident at once. Each stage loads what it needs, does its work, and releases the memory before the next one starts — which is what makes a 16 GB laptop a viable target rather than a compromise.
| Requirement | |
|---|---|
| Linux · NVIDIA | 24 GB VRAM for the default Qwen3.5-9B at bf16 with a long context. Smaller context or a quantized model fits less. |
| macOS · Apple Silicon | 16 GB unified memory. Developed and tested on a MacBook Air M4 (16 GB) running the 9B model. |
On Apple Silicon the memory-hungry stages are staged rather than stacked.
Whisper transcribes videos in batches of 25 and its MLX weights are explicitly
released — ModelHolder.model = None, gc.collect(), mx.clear_cache() —
before the vision-language model runs inference on that same batch. LM Studio
serves the VLM for the duration of the describe stage and is shut down when it
ends; only then does the embed stage load the MLX embedding model in-process.
On Linux the same discipline is automated through Docker Compose: the ingest and search commands start the container they need, wait for the model, and stop it when the stage finishes.
With 8 GB, drop to a smaller vision-language model — the 9B will not leave room for the rest of the pipeline.
git clone https://github.com/matarseks/ReelScope.git
cd ReelScope
uv sync.env is created from the matching platform profile on first run. Bring up the
inference backend for your platform first — Docker Compose on Linux, or
LM Studio on macOS:
docker compose -f infra/vllm/docker-compose.yml up -d # Linux / NVIDIAThe bot walks you through the two things that need credentials, so there is no file to place by hand.
uv sync --extra telegram --extra browser
uv run playwright install chromium
# set TELEGRAM_BOT_TOKEN in .env
uv run reelscope botSend /start. The first user to do so claims the bot and is written to the
allowlist. Onboarding then asks for two things in order:
- Instagram cookies — export a Netscape
cookies.txtwhile logged in (the bot links the browser extension it expects) and send the file into the chat. They are stored locally atdata/assets/www.instagram.com_cookies.txt. - Your Instagram username — saved as
INSTAGRAM_SAVED_URLsoExport savedknows which collection to read.
After that, Export saved, Ingest, and Search are buttons. Cookies grant
access to your Instagram session — treat them exactly like a password, and
rotate them with /update_cookies when Instagram invalidates them.
Same credentials, placed manually. export-saved drives a real browser, so it
needs the browser extra and a cookie file — it exits immediately if the file
is missing.
uv sync --extra browser
uv run playwright install chromium
# Export cookies.txt while logged into Instagram, then:
cp ~/Downloads/www.instagram.com_cookies.txt data/assets/
uv run reelscope export-saved --url "https://www.instagram.com/USERNAME/saved/all-posts/"
uv run reelscope ingest-content
uv run reelscope search "morning coffee tutorial"Override the cookie location with INSTAGRAM_COOKIES_FILE or --cookies.
Full walkthrough: Getting started.
Nothing in the description stage is tied to Instagram, and this path needs no
cookies or browser. ingest-content takes a local video, a folder of videos,
an ordered image carousel, or a text file of paths. For a folder, each video is
stored and embedded separately in data/local_dirs.sqlite3:
uv run reelscope ingest-content ./my-videos
uv run reelscope search "morning workout" --database data/local_dirs.sqlite3Use describe-content when you only want the structured JSON export.
Each described post becomes a structured record. Fields are populated only when the source actually supplied them.
{
"summary": "A person performs lateral raises with dumbbells in a gym, then demonstrates a seated overhead press. Text overlays name each movement and the target muscle head.",
"category": "fitness",
"topics": ["shoulder training", "lateral raise", "overhead press", "dumbbell workout"],
"objects": ["dumbbells", "weight bench", "gym mirror"],
"actions": ["lateral raise", "seated overhead press", "rest between sets"],
"setting": "indoor gym",
"text_on_screen": "SIDE DELTS · 3x12",
"mood": "energetic",
"language": "en"
}Search returns a versioned envelope containing the effective request parameters and the ranked hits, so a result is reproducible from its own payload:
{
"schema_version": 2,
"request": {
"query": "exercises in the gym for shoulders",
"mode": "hybrid",
"top_k": 5,
"rerank": true,
"rerank_threshold": 0.0
},
"hits": [
{
"rank": 1,
"score": 0.94,
"url": "https://www.instagram.com/reel/EXAMPLE123/",
"post_type": "reel",
"summary": "A person performs lateral raises with dumbbells in a gym…",
"category": "fitness",
"mood": "energetic",
"actions": ["lateral raise", "seated overhead press"],
"objects": ["dumbbells", "weight bench"],
"topics": ["shoulder training", "dumbbell workout"],
"model": "Qwen/Qwen3-Embedding-0.6B",
"dimension": 1024,
"vector_rank": 2,
"lexical_rank": 1,
"vector_score": 0.71,
"fts_score": 8.42,
"rerank_score": 0.94
}
]
}vector_rank, lexical_rank, and rerank_score are kept in the output on
purpose: you can see which retrieval branch found a result and how much the
cross-encoder moved it.
src/reelscope/
├── ingest/ Saved-collection export, downloads, URL registry
├── describe/ Frame sampling, VLM prompting, Whisper, result assembly
├── embed/ Embeddings, BM25, RRF fusion, reranking, search CLI
├── storage/ SQLite schema, migrations, sqlite-vec and FTS5 backends
├── bot/ aiogram routers, FSM onboarding, formatting
└── config/ Layered YAML + env configuration
- Getting started — installation, commands, options
- macOS setup — Apple Silicon, LM Studio, MLX
- SQLite cookbook — read-only queries against the index
- vLLM infrastructure · LM Studio lifecycle
uv run ruff format --check .
uv run ruff check .
uv run pytestCI runs formatting, linting, and the full test suite on every push.
MIT — see LICENSE.
ReelScope is a personal research tool for content you already saved. You are responsible for how you use it and for complying with the terms of any platform you point it at.