Conversation
Add OpenAiLLM, an LLM implementation over /embeddings, /rerank and /chat/completions, selected by `openai:<model id>` URIs in models.embed, .rerank and .generate the way `hf:` selects node-llama-cpp. The endpoint and key come from models.openai_base_url / openai_api_key or QMD_OPENAI_BASE_URL / QMD_OPENAI_API_KEY; openai_generate_params carries server-specific chat fields such as reasoning_effort. The store, session manager, CLI and createStore are typed against LLM instead of LlamaCpp; tokenize/detokenize are optional and chunking falls back to character estimates without them. pull is a no-op for remote models, doctor probes GET /models instead of the GPU, and openai_base_url joins the trust digest of project-local configs. hf: URIs and the defaults are unchanged.
saifulapm
force-pushed
the
remote-models
branch
2 times, most recently
from
September 14, 2026 14:45
00cb763 to
3a10bec
Compare
saifulapm
added a commit
to saifulapm/dotfiles
that referenced
this pull request
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #620. Relates to #692 (program consumers to the
LLMinterface).Problem
qmd only runs local GGUF models through node-llama-cpp. On a machine without
a usable GPU (here: Fedora Asahi on an M2 Mac mini, no Vulkan prebuild for
linux-arm64) that means about one second per embedded chunk and a reranked
qmd querythat takes minutes, while an OpenAI-compatible server thatalready embeds and reranks in milliseconds sits idle next to it.
The
LLMinterface existed but nothing could be substituted forLlamaCpp:the store, the session manager, the CLI and
createStoreall held theconcrete class.
What this does
OpenAiLLMinsrc/llm.ts:/embeddings(batched, 10 per request,4 in flight),
/rerank(the Jina/Voyage/Cohere/llama-server shape),/chat/completionsforgenerate, andexpandQueryon top of it with thelex:/vec:/hyde:line format the local model was fine-tuned for. Notokenizer, nothing to pull, unload or dispose.
openai:<model id>URIs select the backend, the wayhf:selectsnode-llama-cpp.
createLlm()builds one backend per index from the threeURIs and refuses a mix. Vectors are stored and fingerprinted under the
openai:URI, so a backend switch is an ordinary model change(
qmd embed -f), and remote embedding text is sent raw instead of with theEmbeddingGemma prefix.
models.openai_base_url,models.openai_api_key(orQMD_OPENAI_BASE_URL/QMD_OPENAI_API_KEY, the key env wins so it canstay out of a shared
index.yml), andmodels.openai_generate_params, amap merged into every chat request. The last one exists because reasoning
models return an empty
contentafter spending the whole budget thinkingand the field that turns it off differs per server (
reasoning_effort,think, …).LLM:Store.llm,getLlm, the session manager,withLLMSessionForLlm,searchVec/expandQuery/rerankoverrides and thedefault singleton.
tokenize/detokenizeare optional on the interface;without them
chunkDocumentByTokensWithLlmkeeps the character-spacechunks with estimated token counts (what Pluggable LLM backend — program consumers to the LLM interface #692 proposed).
embedBatchandthe model-name getters are declared on the interface because the store
already used them.
qmd pullsays the models are served remotely and does nothing;qmd doctorprobesGET /modelsinstead of the GPU and never prints theCPU warning for a remote backend;
qmd statusshows the endpoint; thebearer token is masked in the environment-override list.
openai:model in a project-local.qmd/index.ymlis acustom model and gated as before;
openai_base_urljoins the digest sochanging the endpoint under approved models re-arms the gate. It decides
where every indexed document is sent.
Default behaviour is unchanged: with
hf:URIscreateLlmreturns the sameLlamaCppas before, andLlamaCpp.expandQuerynow calls the sharedparseExpansionLineswith the exact logic it had inline.Prior art
#517, #619, #629, #705, #720, #761 and #769 all attempt this; none applies to
current
mainand the closest in shape (#619, one provider switch with serverside model aliases) is 233 commits behind. This is a fresh, smaller take on
the same idea against
main, following the review note on #517 (includequery expansion, unit-test the remote calls with mocked HTTP). Per-role
mixing of local and remote (#705's
HybridLLM) is deliberately out of scope:one backend per index keeps sessions, unloading and doctor simple.
Tests
test/llm-openai.test.tsstubs globalfetchand checks request shapes(paths, method, bearer header, bare model id, batching of 23 texts into
10/10/3 with vectors landing in order), error surfacing with the response
body, key precedence, chat body merge, expansion parsing including
<think>stripping and the error fallback, rerank de-duplication and index mapping,
listModels, the raw embedding format foropenai:URIs, backend selectionand the mixed-URI refusal.
tsc --noEmit, oxlint and the vitest suite (51 files, 1257 passed, 79skipped) are green under Node 24. The Bun leg of
npm testwas not run(bun is not installed on the development machine).
Verified against a live proxy (pxy → DashScope
text-embedding-v4, Jinareranker, an Ollama-hosted chat model) on the GPU-less machine above:
openai:via proxyqmd querywith expansion + rerank