Skip to content

feat(llm): OpenAI-compatible remote backend via openai: model URIs - #954

Open
saifulapm wants to merge 1 commit into
tobi:mainfrom
saifulapm:remote-models
Open

saifulapm wants to merge 1 commit into
tobi:mainfrom
saifulapm:remote-models

Conversation

@saifulapm

Copy link
Copy Markdown

Closes #620. Relates to #692 (program consumers to the LLM interface).

Problem

qmd only runs local GGUF models through node-llama-cpp. On a machine without
a usable GPU (here: Fedora Asahi on an M2 Mac mini, no Vulkan prebuild for
linux-arm64) that means about one second per embedded chunk and a reranked
qmd query that takes minutes, while an OpenAI-compatible server that
already embeds and reranks in milliseconds sits idle next to it.

The LLM interface existed but nothing could be substituted for LlamaCpp:
the store, the session manager, the CLI and createStore all held the
concrete class.

What this does

  • OpenAiLLM in src/llm.ts: /embeddings (batched, 10 per request,
    4 in flight), /rerank (the Jina/Voyage/Cohere/llama-server shape),
    /chat/completions for generate, and expandQuery on top of it with the
    lex:/vec:/hyde: line format the local model was fine-tuned for. No
    tokenizer, nothing to pull, unload or dispose.
  • openai:<model id> URIs select the backend, the way hf: selects
    node-llama-cpp. createLlm() builds one backend per index from the three
    URIs and refuses a mix. Vectors are stored and fingerprinted under the
    openai: URI, so a backend switch is an ordinary model change
    (qmd embed -f), and remote embedding text is sent raw instead of with the
    EmbeddingGemma prefix.
  • Config: models.openai_base_url, models.openai_api_key (or
    QMD_OPENAI_BASE_URL / QMD_OPENAI_API_KEY, the key env wins so it can
    stay out of a shared index.yml), and models.openai_generate_params, a
    map merged into every chat request. The last one exists because reasoning
    models return an empty content after spending the whole budget thinking
    and the field that turns it off differs per server (reasoning_effort,
    think, …).
  • Consumers typed to LLM: Store.llm, getLlm, the session manager,
    withLLMSessionForLlm, searchVec/expandQuery/rerank overrides and the
    default singleton. tokenize/detokenize are optional on the interface;
    without them chunkDocumentByTokensWithLlm keeps the character-space
    chunks with estimated token counts (what Pluggable LLM backend — program consumers to the LLM interface #692 proposed). embedBatch and
    the model-name getters are declared on the interface because the store
    already used them.
  • CLI: qmd pull says the models are served remotely and does nothing;
    qmd doctor probes GET /models instead of the GPU and never prints the
    CPU warning for a remote backend; qmd status shows the endpoint; the
    bearer token is masked in the environment-override list.
  • Trust: an openai: model in a project-local .qmd/index.yml is a
    custom model and gated as before; openai_base_url joins the digest so
    changing the endpoint under approved models re-arms the gate. It decides
    where every indexed document is sent.

Default behaviour is unchanged: with hf: URIs createLlm returns the same
LlamaCpp as before, and LlamaCpp.expandQuery now calls the shared
parseExpansionLines with the exact logic it had inline.

Prior art

#517, #619, #629, #705, #720, #761 and #769 all attempt this; none applies to
current main and the closest in shape (#619, one provider switch with server
side model aliases) is 233 commits behind. This is a fresh, smaller take on
the same idea against main, following the review note on #517 (include
query expansion, unit-test the remote calls with mocked HTTP). Per-role
mixing of local and remote (#705's HybridLLM) is deliberately out of scope:
one backend per index keeps sessions, unloading and doctor simple.

Tests

test/llm-openai.test.ts stubs global fetch and checks request shapes
(paths, method, bearer header, bare model id, batching of 23 texts into
10/10/3 with vectors landing in order), error surfacing with the response
body, key precedence, chat body merge, expansion parsing including <think>
stripping and the error fallback, rerank de-duplication and index mapping,
listModels, the raw embedding format for openai: URIs, backend selection
and the mixed-URI refusal.

tsc --noEmit, oxlint and the vitest suite (51 files, 1257 passed, 79
skipped) are green under Node 24. The Bun leg of npm test was not run
(bun is not installed on the development machine).

Verified against a live proxy (pxy → DashScope text-embedding-v4, Jina
reranker, an Ollama-hosted chat model) on the GPU-less machine above:

operation local GGUF on CPU openai: via proxy
embed 23 docs / 209 chunks 4 m 19 s 7 s
qmd query with expansion + rerank 7 m 36 s 2.6 s (expand 1.4 s, embed 0.2 s, rerank 1.0 s)

Add OpenAiLLM, an LLM implementation over /embeddings, /rerank and
/chat/completions, selected by `openai:<model id>` URIs in models.embed,
.rerank and .generate the way `hf:` selects node-llama-cpp. The endpoint
and key come from models.openai_base_url / openai_api_key or
QMD_OPENAI_BASE_URL / QMD_OPENAI_API_KEY; openai_generate_params carries
server-specific chat fields such as reasoning_effort.

The store, session manager, CLI and createStore are typed against LLM
instead of LlamaCpp; tokenize/detokenize are optional and chunking falls
back to character estimates without them. pull is a no-op for remote
models, doctor probes GET /models instead of the GPU, and openai_base_url
joins the trust digest of project-local configs.

hf: URIs and the defaults are unchanged.
@saifulapm
saifulapm force-pushed the remote-models branch 2 times, most recently from 00cb763 to 3a10bec Compare September 14, 2026 14:45
saifulapm added a commit to saifulapm/dotfiles that referenced this pull request Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support OpenAI-compatible backends for generation, embeddings, and reranking

1 participant