feat: add opt-in TwelveLabs (Marengo/Pegasus) embedding_lib for video-aware RAG#65
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Conversation
…-aware RAG Add 'twelvelabs' as a new embedding_lib for the langchain text-RAG backend. MarengoEmbeddings implements the LangChain Embeddings interface (512-dim multimodal vectors) so it is a drop-in alternative to HuggingFaceEmbeddings in the existing Chroma vector store. A Pegasus analyze_video() helper turns videos (URL/asset/video_id) into text that can be indexed like any document. Opt-in and non-breaking: defaults are unchanged, the SDK is imported lazily, and the key is read from TWELVELABS_API_KEY (never logged). Includes a no-network unit test and a TWELVELABS_API_KEY-gated live test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR extends Colette's multimodal RAG with video by adding an opt-in
twelvelabsembedding library to the langchain text-RAG backend.What it adds
MarengoEmbeddings(src/colette/backends/langchain/rag/twelvelabs_embeddings.py): TwelveLabs' Marengo model exposed through the LangChainEmbeddingsinterface (512-dim multimodal vectors shared across text/image/audio/video). It's a drop-in alternative toHuggingFaceEmbeddingsin the existing Chroma vector store.embedding_lib: "twelvelabs"wired intoRAGObj(apidata.py) and theRAGTxt.initdispatch, alongside the existinghuggingface/colbert/vllmoptions.embedding_modelis optional and defaults tomarengo3.0.analyze_video()Pegasus helper: turns a video (public URL, uploaded asset id, or indexed video id) into text you can chunk and index like any other document — so video content becomes searchable next to your PDFs. Because Marengo embeds text and video into the same space, retrieval stays coherent.Why it helps Colette
Colette is already a multimodal/V-RAG system; this lets users bring video into the same RAG pipeline without leaving the existing config/registry conventions, using a hosted API (no extra GPU/VRAM needed for the embedder).
Opt-in / non-breaking
Defaults are unchanged. The
twelvelabsSDK is imported lazily (only when selected), and the API key is read fromTWELVELABS_API_KEYand never logged. Existinghuggingface/colbert/vllmpaths are untouched.How it was tested
ruff format+ruff checkclean on the new files (pre-existing repo lint untouched).@pytest.mark.smoke) verifying the LangChainEmbeddingswiring with a mocked client, plus argument validation foranalyze_video.TWELVELABS_API_KEY-gated live integration test (follows the repo'sCOLETTE_RUN_INTEGRATION=1convention) that confirms Marengo returns a real 512-dim text embedding. I ran it against the live API and it passes.The dependency is added to the
trag(text-RAG) optional extra inpyproject.toml, anddocs/source/users/configuration.mddocuments the new option.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.