Search 7,000 memes by meaning, not filenames — monday morning returns the right meme in ~30 ms.
Stack: CLIP (ViT-B/32) · FAISS · PyTorch · EasyOCR · FastAPI · MongoDB · Streamlit · Chrome Manifest V3 · pytest · ruff · Docker
Memes are hard to search because the joke lives in two places at once: in the picture and in the caption burned into it. Filename search finds neither. This indexes both — CLIP embeddings of the image and of its OCR'd text, in two FAISS indexes — and blends the two rankings at query time.
Who it's for: anyone sitting on a folder of thousands of untagged memes — or any image set with text in it — who wants to find one by describing it. As a reference implementation, it's for engineers who want a complete, working example of multimodal retrieval: CLIP embeddings, a vector index, rank fusion and a serving layer, small enough to read end to end in an afternoon.
Try it without the full setup:
demo/builds a 400-meme subset that runs as a single Streamlit process — no API, no MongoDB, no 700 MB download. See demo/DEPLOY.md, which also covers putting it on a free Hugging Face Space. The main API is not currently hosted, so the Chrome extension needs your own server running.
| Mode | What it matches | Wins on |
|---|---|---|
text |
CLIP embedding of the OCR-extracted caption | "I remember the words" — one does not simply |
image |
CLIP embedding of the image itself | "I remember the picture" — surprised cat, memes with stylised text |
hybrid (default) |
Rank fusion of both | General queries, where agreement between the two is a useful tiebreaker |
Because CLIP maps text and images into one shared vector space, a text query can be
compared directly against image embeddings. That's what makes image mode work without
uploading a picture.
flowchart LR
subgraph offline["Offline — once, minutes"]
A[6,992 images] --> B[EasyOCR] --> C[CLIP ViT-B/32]
C --> D[(text.index)]
C --> E[(image.index)]
end
subgraph online["Online — per query, ~29 ms"]
Q["query: exam stress"] --> R[CLIP text encoder] --> S[search both indexes]
S --> T[rank fusion] --> U[MongoDB hydrate] --> V[ranked JSON]
end
D -.-> S
E -.-> S
Full data model, request lifecycle and the reasoning behind each component choice are in ARCHITECTURE.md.
| Memes indexed | 6,989 · CLIP ViT-B/32 · two 13.6 MB FAISS indexes |
| Recall@10 / MRR | 0.599 / 0.407 (1,000 known-item queries) |
| — vs. OCR-text index alone | 0.351 / 0.262 — the original behaviour |
| Query latency (hybrid) | p50 ~29 ms, of which ~28 ms is CLIP encoding and 0.30 ms is FAISS |
| Test suite | 87 tests, ~0.3 s, no model or database needed |
Switching hybrid fusion from weighted cosine sums to rank fusion lifted Recall@10 from 0.397 to 0.599 — CLIP's modality gap makes the two indexes' scores ~3× apart in magnitude, so a weighted sum is dominated by the text index at almost any weight. That story is the most interesting thing here: how it works.
Open-ended relevance was also judged by hand (nDCG@5 0.896), but a second annotator agreed at only Cohen's kappa 0.140 — poor, so those figures are indicative and the known-item numbers above are the ones to trust. Full method, weight sweeps and caveats in EVALUATION.md; LIMITATIONS.md is the honest list.
git clone https://github.com/SreenityaThatikunta/meme-search.git && cd meme-searchpython3.12 -m venv memenv && source memenv/bin/activate && pip install "setuptools<81" wheelpip install --no-build-isolation -r requirements.txtWhy the extra setuptools step
OpenAI's CLIP package installs from git and its setup.py still imports pkg_resources,
which setuptools removed in v81. On a fresh Python 3.12+ environment a plain
pip install -r requirements.txt fails with
ModuleNotFoundError: No module named 'pkg_resources'. Pinning setuptools below 81 and
disabling build isolation makes the install reproducible.
Add the dataset. The images aren't in the repo (700 MB) — place them in
data/images/images/. The FAISS indexes, id_map.json and memes_final.jsonl are
committed, so you can skip building them entirely. Pipeline intermediates
(memes.jsonl, memes_ocr.jsonl) and the dataset's own labels.csv are not: the first
two are regenerated by the pipeline, and labels.csv is only needed to rebuild the eval
query set, whose output is committed.
Load metadata and run (needs MongoDB up; idempotent, upserts on meme_id):
python scripts/pipeline/store_metadata.pyuvicorn main:app --port 8000streamlit run app.pyAPI at http://localhost:8000 (interactive docs at /docs), UI at
http://localhost:8501. GET /health reports which indexes loaded and therefore which
modes are available.
Rebuilding the indexes from scratch, Docker, and the Chrome extension
Rebuild the indexes (optional — prebuilt ones are committed):
python scripts/pipeline/scan_dataset.py && python scripts/pipeline/run_ocr_fast.py && python scripts/pipeline/embed_memes_fast.pyDocker brings up the API and MongoDB together, mounting ./data read-only rather than
baking a 700 MB dataset into the image. You still need to load metadata against the
containerised Mongo:
docker compose up --buildThe Chrome extension needs the local API running first — the backend isn't deployed:
uvicorn main:app --port 8000- Open
chrome://extensions/, enable Developer mode - Load unpacked → select
meme-extension/ - Click the toolbar icon, pick a mode, search
Read in this order — it's about 400 lines and it's the whole product:
- docs/OVERVIEW.md — how it works, in plain language, in five minutes
services/search.py— the query path end to endservices/fusion.py— the ranking maths, dependency-free
scripts/eval/ and services/metrics.py exist only to produce the numbers above.
Skip them unless you care how those were measured.
meme-search/
├── config.py # Env-driven configuration, single source of truth
├── main.py # FastAPI: POST /search/text, GET /health
├── app.py # Streamlit UI
├── services/ # search.py (query path), fusion.py (ranking),
│ # metadata.py (Mongo or JSONL), metrics.py (eval)
├── scripts/
│ ├── pipeline/ # Builds the product: scan → OCR → embed → store
│ ├── eval/ # Produces the numbers only; nothing here runs at query time
│ └── demo/ # Carves out the subset the hosted demo serves
├── demo/ # Hugging Face Space: entry point, deps, deploy guide
├── meme-extension/ # Chrome extension (Manifest V3)
├── tests/ # 87 tests, no model or DB required
├── docs/ # See below
└── data/ # Indexes + metadata committed; images are not
Each scripts/ subfolder has its own README explaining what to run and in what order.
| OVERVIEW.md | How it works, in five minutes — start here |
| ARCHITECTURE.md | Data model, two-phase design, component choices vs. alternatives |
| EVALUATION.md | Method, results, latency, and what the numbers don't show |
| LIMITATIONS.md | Known limitations and the roadmap, ordered by measured impact |
| API.md | Endpoint reference (or just open /docs on the running server) |
| CONFIGURATION.md | Every environment variable |
pip install -r requirements-dev.txt && pytest87 tests, ~0.3 seconds, no model download and no database. services/search.py defers its
torch / clip / faiss / pymongo imports to call time, which lets the suite substitute
fake indexes and an in-memory collection (tests/conftest.py) and
still cover the real ranking, id-mapping, hydration and HTTP error paths. It's also why CI
finishes in under a minute without installing PyTorch.
MIT — see LICENSE.
