Add LiteLLM embedding model support to RAG retriever by arash77 · Pull Request #1920 · bgruening/galaxytools

arash77 · 2026-07-17T10:41:14Z

This PR adds support for LiteLLM-hosted embedding models to the RAG Retriever tool, reusing the genai_models configuration from LLM Hub. The version suffix has also been bumped to 2.

bgruening · 2026-07-27T09:31:19Z

Still draft?

bgruening

@anuprulez ?

arash77 · 2026-07-27T09:35:58Z

Still draft?

Yes, I need test it more before making it ready.

anuprulez · 2026-07-28T08:19:22Z

I will have a look at it this week. thanks @arash77

arash77 · 2026-07-28T10:30:44Z

For testing it you can use Ollama locally and for example using this model:
ollama pull nomic-embed-text
then you can use this config files

tool-data/genai_models.loc:

nomic-embed-text-ollama	nomic-embed-text	Nomic Embed Text (Ollama Local)	embedding	ollama-local	Free

litellm_config.yaml:

servers:
  ollama-local:
    LITELLM_API_KEY: ollama
    LITELLM_BASE_URL: http://127.0.0.1:11434/v1

And then serve it with planemo:

planemo serve \
  --biocontainers \
  --docker_run_extra_arguments "--network host -e LITELLM_CONFIG_FILE='$PWD/litellm_config.yaml'" \
  --tool_data_table "$PWD/tool_data_table_conf.xml.sample" \
  --tool_data_path "$PWD/tool-data" \
  rag_retriever.xml

IvoLeist · 2026-07-28T14:18:21Z

@arash77 Anup asked me to test it and go through the changes as well:
I can confirm that testing it locally works 🚀

One thing I was stumbling about, since llm-hub just recently got the support for per-user quotas (#1922) maybe it would be worth to "equip" the RAG retriever tool with that functionality as well (?) 🤔

arash77 · 2026-07-28T14:21:16Z

@arash77 Anup asked me to test it and go through the changes as well: I can confirm that testing it locally works 🚀

One thing I was stumbling about, since llm-hub just recently got the support for per-user quotas (#1922) maybe it would be worth to "equip" the RAG retriever tool with that functionality as well (?) 🤔

Thank you for testing it. Actually, yes, that is worth adding here, as they are using the same thing. I will work on it.

anuprulez

thanks @arash77 I have added my comments inline.

Can I run models coming from both tables at the same time locally? I just wanted to test models from litellm and huggingface table to compare runtime, quality etc.

arash77 · 2026-07-29T13:13:16Z

Can I run models coming from both tables at the same time locally? I just wanted to test models from litellm and huggingface table to compare runtime, quality etc.

This should be already possible if you add both datatables.

The preinstalled (huggingface) path filters on free_tag=vector-rag and version=1; it does not use the domain column. Fix the documented/example domain value from 'text' to 'embedding' so the huggingface data table is consistent with the genai_models table used by the LiteLLM path. Addresses review feedback in bgruening#1920: the domain column should mark text embedding models as 'embedding' in both tables rather than 'text'. Also expand the sample header into a per-column reference matching the tabpfn huggingface.loc.sample, documenting the shared table and that free_tag is the per-tool selection key.

Mirror llm_hub's per-user request attribution (PR bgruening#1922): pass $__user_id__ and $__galaxy_url__ from the wrapper and forward them to the LiteLLM proxy as the standard OpenAI user field via OpenAIEmbedding's additional_kwargs (a SHA-256 of galaxy_url|user_id, namespaced per-instance and hashed so no personal data leaves for the provider). Anonymous sessions fall back to a per-instance shared "anonymous" bucket. The hash formula and "Anonymous" normalization are identical to llm_hub.py so both tools map one Galaxy user to one proxy identity, enabling per-user metering and budgets/rate limits on a shared proxy. Scoped to the LiteLLM path; local HuggingFace embeddings are unchanged (no proxy call). Addresses bgruening#1920 (comment)

The RAG Retriever's preinstalled path selects models via free_tag=vector-rag and version=1; it does not filter on the domain column. Set domain to 'embedding' for the two RAG embedding rows so the column is consistent with the genai_models.loc table used by the LiteLLM path. Also add 'tabular' and 'embedding' to the domain column's documented example values in the header (tabular was already in use by the tabpfn rows). This is a documentation/consistency change only — the tool's selection logic is unaffected since it filters on free_tag, not domain. Companion to bgruening/galaxytools#1920.

anuprulez · 2026-07-30T09:52:40Z

This looks good to me if no further changes are required :)

Remove the space around the slice colon in texts[i : i + batch] to satisfy flake8/PEP 8.

- Comment out example rows in genai_models.loc.sample (samples must not contain uncommented rows) - Drop LiteLLM proxy / genai_models / huggingface data-table details from the user-facing help and the no_options validator message - Mention hosted embedding models in .shed.yml long_description

Replace the custom LiteLLMEmbedding subclass with llama_index's built-in OpenAIEmbedding, passing the proxy model id via model_name= (the class's escape hatch around the hardcoded OpenAI model enum). This reuses the framework's batching, retry and client handling instead of hand-rolling an OpenAI client inside a BaseEmbedding subclass. Collapse the seven positional argv arguments (embed source, path, value, provider, top_k) into a single EMBED JSON blob built in the wrapper, so the Python side reads one typed config object instead of five loose strings. Restore explicit validation of the model/provider/path fields the old positional layout checked. Inline the litellm_client config resolution (load_litellm_config / resolve_server) into rag_retriever.py so the RAG tool is self-contained, and guard against an empty/blank LiteLLM config file. Make the request timeout and SDK retry count configurable via environment variables, read inline at the call site. Verified locally with planemo test (3/3 passing).

Co-authored-by: Anup Kumar, PhD <anup.rulez@gmail.com>

The preinstalled (huggingface) path filters on free_tag=vector-rag and version=1; it does not use the domain column. Fix the documented/example domain value from 'text' to 'embedding' so the huggingface data table is consistent with the genai_models table used by the LiteLLM path. Addresses review feedback in bgruening#1920: the domain column should mark text embedding models as 'embedding' in both tables rather than 'text'. Also expand the sample header into a per-column reference matching the tabpfn huggingface.loc.sample, documenting the shared table and that free_tag is the per-tool selection key.

Mirror llm_hub's per-user request attribution (PR bgruening#1922): pass $__user_id__ and $__galaxy_url__ from the wrapper and forward them to the LiteLLM proxy as the standard OpenAI user field via OpenAIEmbedding's additional_kwargs (a SHA-256 of galaxy_url|user_id, namespaced per-instance and hashed so no personal data leaves for the provider). Anonymous sessions fall back to a per-instance shared "anonymous" bucket. The hash formula and "Anonymous" normalization are identical to llm_hub.py so both tools map one Galaxy user to one proxy identity, enabling per-user metering and budgets/rate limits on a shared proxy. Scoped to the LiteLLM path; local HuggingFace embeddings are unchanged (no proxy call). Addresses bgruening#1920 (comment)

arash77 · 2026-09-04T13:36:51Z

Is anything blocking this, or can it be merged?

bgruening · 2026-09-04T14:00:47Z

Thanks everyone!

bgruening reviewed Jul 26, 2026

View reviewed changes

Comment thread tools/rag/tool-data/genai_models.loc.sample Outdated

bgruening reviewed Jul 26, 2026

View reviewed changes

Comment thread tools/rag/rag_retriever.xml Outdated

bgruening reviewed Jul 26, 2026

View reviewed changes

Comment thread tools/rag/rag_retriever.xml Outdated

bgruening approved these changes Jul 27, 2026

View reviewed changes

arash77 marked this pull request as ready for review July 27, 2026 18:21

anuprulez reviewed Jul 28, 2026

View reviewed changes

Comment thread tools/rag/tool-data/genai_models.loc.sample

Comment thread tools/rag/rag_retriever.xml Outdated

arash77 marked this pull request as draft July 29, 2026 13:37

arash77 mentioned this pull request Jul 29, 2026

Set domain=embedding for RAG embedding models in huggingface.loc usegalaxy-eu/infrastructure-playbook#2231

Merged

arash77 marked this pull request as ready for review July 29, 2026 15:03

arash77 and others added 7 commits August 6, 2026 13:29

Add LiteLLM embedding model support to RAG retriever

df221a1

Fix flake8 E203 whitespace before ':' in rag_retriever

976ac5f

Remove the space around the slice colon in texts[i : i + batch] to satisfy flake8/PEP 8.

Update tools/rag/rag_retriever.xml

a63ad16

Co-authored-by: Anup Kumar, PhD <anup.rulez@gmail.com>

arash77 force-pushed the rag-litellm-embeddings branch from ae1c84e to f1e9c8c Compare August 6, 2026 11:29

bgruening merged commit 48d8cfd into bgruening:master Sep 4, 2026
10 checks passed

Conversation

arash77 commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bgruening commented Jul 27, 2026

Uh oh!

bgruening left a comment

Choose a reason for hiding this comment

Uh oh!

arash77 commented Jul 27, 2026

Uh oh!

anuprulez commented Jul 28, 2026

Uh oh!

arash77 commented Jul 28, 2026

Uh oh!

IvoLeist commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arash77 commented Jul 28, 2026

Uh oh!

anuprulez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arash77 commented Jul 29, 2026

Uh oh!

anuprulez commented Jul 30, 2026

Uh oh!

arash77 commented Sep 4, 2026

Uh oh!

Uh oh!

bgruening commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IvoLeist commented Jul 28, 2026 •

edited

Loading