Skip to content

docs: aimlapi.com via LiteLLM β€” LLM pipeline and embeddings vectors backend - #1

Open
Lookoff-AIMLAPI wants to merge 2 commits into
masterfrom
docs/aimlapi-litellm
Open

docs: aimlapi.com via LiteLLM β€” LLM pipeline and embeddings vectors backend#1
Lookoff-AIMLAPI wants to merge 2 commits into
masterfrom
docs/aimlapi-litellm

Conversation

@Lookoff-AIMLAPI

Copy link
Copy Markdown
Member

What this is

txtai has no provider registry. GenerationFactory (src/python/txtai/pipeline/llm/factory.py)
dispatches to five backends β€” litellm, litert, llama.cpp, opencode, transformers β€” and
every hosted API goes through LiteLLM. The vectors side is the same shape: VectorsFactory
routes hosted embeddings APIs to src/python/txtai/vectors/dense/litellm.py.

So aimlapi.com needs no code here; it already works. What it needed was documentation, because
two things about it are not discoverable and each costs a user a failed run.

Docs only. No Python changed.

The two things

1. The LLM pipeline reads AIML_API_KEY. The existing docs say "must also set API key via
environment variable" without naming any variable. LiteLLM's aiml provider reads AIML_API_KEY
(litellm/llms/aiml/chat/transformation.py), which differs from the AIMLAPI_API_KEY used in the
provider's own documentation. There is no way to guess it.

2. aiml/ does not work for embeddings, and the docs now say so. LiteLLM lists aiml in
openai_compatible_providers (litellm/constants.py:843), but only completion() and speech()
consult that list. embedding() hardcodes its openai-compatible branch as
openai | together_ai | nvidia_nim | litellm_proxy (litellm/main.py:6220-6226), so aiml/ on
the embeddings route raises:

litellm.BadRequestError: Unmapped LLM provider for this endpoint.
You passed model=openai/text-embedding-3-small, custom_llm_provider=aiml.

The route that does work is the OpenAI-compatible one with api_base set β€” the same pattern
docs/pipeline/llm/llm.md already documents for a local OpenAI-compatible endpoint. That is what
the new ### litellm example in docs/embeddings/configuration/vectors.md shows.

The same section warns against putting api_key inside vectors. That dict is part of the index
configuration and is serialised verbatim into config.json on save() β€” verified: a live key
appeared in plain text in the saved index directory. api_base alone is safe to persist.

Verified live

Base URL https://api.aimlapi.com/v1, real key via env, through txtai's own classes β€” not curl,
not a mock.

The LLM example, exactly as documented (only AIML_API_KEY set, no OPENAI_API_KEY):

LLM framework  : LiteLLM
LLM("aiml/openai/gpt-5-5")("Answer with one word: what colour is a clear midday sky?")
  -> 'Blue'

Also called on aiml/anthropic/claude-sonnet-4.5, aiml/google/gemini-2.5-flash and
aiml/deepseek/deepseek-chat β€” all returned completions.

The embeddings example, the YAML block from the docs loaded verbatim into Embeddings(config):

vectors backend: LiteLLM
dimensions     : 3072
  feel good story        -> "Maine man wins $1M from $25 lottery ticket"
  climate change         -> "Canada's last fully intact ice shelf has suddenly collapsed"
  public health story    -> "US tops 5 million confirmed virus cases"
  war                    -> "Beijing mobilises invasion craft along coast ..."
  wildlife               -> "The National Park Service warns against sacrificing slower friends ..."

Real 3072-dimension vectors, five out of five nearest neighbours correct. Sibling embedding ids
called the same way: text-embedding-3-small 1536, qwen-text-embedding-v4 1024,
voyage-code-2 1536.

aiml/openai/text-embedding-3-small was also called and failed as described above β€” the docs
statement is tested in both directions.

Build and tests

baseline (1d45e88) after
mkdocs build exit 0, 0 errors, 381 warnings exit 0, 0 errors, 381 warnings
unittest -k testvectors 58 run, 42 errors, 1 skipped 58 run, 42 errors, 1 skipped
unittest -k testpipeline.testllm 58 run, 28 errors, 1 skipped 58 run, 28 errors, 1 skipped
MINIMAL=1 pip install . && python -c "import txtai" OK OK

Identical before and after, which is what a markdown-only diff should produce. The errors in both
columns are missing optional extras in the local environment (fastapi, staticvectors,
sentence-transformers, PIL, soundfile, timm, smolagents), not failures β€” the same tests
error on a pristine checkout. testvectors.testdense.testlitellm and
testpipeline.testllm.testlitellm, the two modules covering the documented code paths, pass in
both columns.

Not verified: the full pip install .[all,dev] + make data coverage matrix that CI runs. It
needs Java/Tika, llama-cpp-python built from source and the audio stack, none of which this diff
can reach β€” it touches two markdown files and no Python.

Rendered links in the built site were checked: ../../../pipeline/llm/llm and
../../methods/#txtai.embeddings.Embeddings.save both resolve against generated pages and anchors.

Two side observations, not fixed here

  • The method enum in docs/embeddings/configuration/vectors.md:15 lists
    transformers|sentence-transformers|llama.cpp|litellm|model2vec|external|words but
    VectorsFactory also accepts litert and onnx.
  • docs/embeddings/configuration/cloud.md:5 and docs/embeddings/indexing.md:41,61 link to
    #txtai.embeddings.base.Embeddings.<method> anchors. mkdocstrings emits
    #txtai.embeddings.Embeddings.<method> (no .base.), so those anchors do not resolve. The new
    link in this PR uses the working form.

Left alone to keep the diff minimal.

Commits

Two, deliberately separable:

  1. docs: cover aimlapi.com models on the LiteLLM LLM and vectors backends β€” the content above.
  2. chore(aimlapi): fork-only placement β€” do not send upstream β€” moves the aimlapi.com line to the
    top of the hand-ordered "LLM APIs" example block. Presentation only. Drop it before this goes
    anywhere upstream.

txtai delegates every hosted API to LiteLLM, so aimlapi.com already works
without any code here. Two things about it are not discoverable from the
existing docs and cost a user a failed run each:

- The LLM pipeline reads AIML_API_KEY, not the AIMLAPI_API_KEY that the
  provider's own docs use. The general "set the API key via environment
  variable" comment does not say which variable, and there is no way to
  guess it.
- LiteLLM registers `aiml` as an openai-compatible provider for chat
  completions only. Its embedding() dispatch does not consult that list,
  so an `aiml/` path in a vectors config fails with "Unmapped LLM provider
  for this endpoint". The working route is the openai-compatible one with
  api_base set, which is the same pattern already documented for a local
  OpenAI-compatible endpoint.

The vectors note also warns against putting api_key in `vectors`. That
dict is part of the index configuration and is serialised verbatim into
config.json on save, which writes a live key to disk next to the index.

Verified live against https://api.aimlapi.com/v1: LLM("aiml/openai/gpt-5-5")
returns a completion, and the documented embeddings config returns 3072
dimensions with correct nearest neighbours over a five-row index.
Moves the aimlapi.com example to the top of the hand-ordered "LLM APIs"
block in docs/pipeline/llm/llm.md. This is presentation, not information,
and it is the only ordering change available in this repository: txtai has
no provider registry, dropdown or badge mechanism, so there is nothing else
to place or feature. Drop this commit before the change goes upstream.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant