feat(llm): OpenAI-compatible remote embedding, reranking & query expansion (RemoteLLM + HybridLLM)

[Kaspre]

Problem

QMD's BM25 + vector + query-expansion + LLM-rerank pipeline has a hard dependency on local GGUF models via node-llama-cpp. Devices with inadequate or no GPU at all fall back to CPU embedding of 300M–0.6B models, which is unacceptably slow. Node-llama-cpp serialises inference through a single context, so concurrent workers queue and time out. Moving embedding/reranking/expansion to a dedicated server (vLLM, TEI, Ollama, llama.cpp, LiteLLM, OpenAI, …) removes both problems.

This builds directly on @georgelichen's #629, which contributed the core RemoteLLM/HybridLLM architecture. This PR rebases that scaffold onto current main and adds five refinements (A–E below) needed to make it production-correct against real OpenAI-compatible servers.

What this adds

A backend that talks to any OpenAI-compatible HTTP API, composed with the existing local backend so each operation can be routed independently. Fully opt-in — with no *_api_url configured, behaviour is byte-for-byte the current local-only path.

• RemoteLLM — /v1/embeddings, /v1/rerank, /v1/chat/completions; per-endpoint circuit breakers (an embed outage doesn't take down rerank/chat); bearer auth; char-based token approximation so chunking works without a local tokenizer.
• HybridLLM — routes embed/embedBatch/rerank/expandQuery to remote, generate/tokenize/detokenize to local LlamaCpp, with per-operation fallback.
• Widened LLM interface (embedBatch, tokenize/detokenize/countTokens, isRemote, embedModelName, rerankModelName?, generateModelName?, usesRemoteEmbedding?, supportsRerank/supportsExpand) and getDefaultLLM/setDefaultLLM alongside the existing getDefaultLlamaCpp/setDefaultLlamaCpp (no breaking change).

The five refinements on top of #629

• A — Sigmoid rerank normalization. llama.cpp's /v1/rerank (e.g. bge-reranker-v2-m3) and most cross-encoders return log-odds (~−10…+10), not 0–1. QMD's score blend and the --min-score 0.3 default assume probabilities, so without σ(x)=1/(1+e⁻ˣ) every blended score goes negative and every query returns "No results found". No-op for backends already emitting 0–1.
• B — RemoteLLM.expandQuery. Query expansion via /v1/chat/completions, emitting the same lex:/vec:/hyde: line format as the local model; falls back to local on error.
• C — Pre-flight embed probe. When usesRemoteEmbedding, embed one token at vectorIndex() startup so a bad URL / wrong model / auth failure surfaces immediately instead of silently falling back mid-batch.
• D — HybridLLM.rerank local fallback. Symmetric with the expandQuery fallback already in Add remote embedding, reranking, and query expansion support #629: if the remote backend doesn't support rerank, fall back to local LlamaCpp.rerank rather than blindly POSTing /v1/rerank at an embed-only server. Lets you mix "remote embed + local rerank" freely.
• E — Rerank oversized-payload recovery. When the server rejects a rerank request as too large (HTTP 413 / "too large to process" / context-length), RemoteLLM.rerank recursively bisects the batch and halve-truncates a single oversized document (down to a 32-char floor), remapping response indices to the originals and re-sorting by normalized score. Non-oversized errors still propagate to the circuit breaker / local fallback. Adapted from the recovery in @loopyd's #619.

Architecture

flowchart TD
    Q["qmd: index / query"] --> H{"HybridLLM<br/>remote config present?"}
    H -->|no| L0["LlamaCpp only — unchanged default"]
    H -->|yes| E["embed / embedBatch"] & RK["rerank"] & EX["expandQuery"] & G["generate"] & TK["tokenize / detokenize"]

    E --> R1["RemoteLLM → POST /v1/embeddings<br/>＋ pre-flight 1-token probe ◆C"]
    RK --> RKQ{"remote supportsRerank?"}
    RKQ -->|yes| R2["RemoteLLM → POST /v1/rerank<br/>＋ σ: log-odds→0‥1 ◆A<br/>＋ oversized split/truncate recovery ◆E"]
    RKQ -->|no| L1["LlamaCpp local rerank ◆D"]
    EX --> EXQ{"remote supportsExpand?"}
    EXQ -->|yes| R3["RemoteLLM → POST /v1/chat/completions ◆B"]
    EXQ -->|no| L2["LlamaCpp local expansion"]
    G --> L3["LlamaCpp local"]
    TK --> L4["LlamaCpp local (char-approx if usesRemoteEmbedding)"]

Loading

Configuration

Any OpenAI-compatible server works — point each operation wherever you like. Example (~/.config/qmd/index.yml):

models:
  embed_api_url:   http://embed-host:8081/v1   # vLLM / TEI / llama.cpp / OpenAI / Ollama
  embed_api_model: nomic-embed-text
  # rerank/expand are optional and independent — omit to keep them local:
  rerank_api_url:   http://rerank-host:8082/v1
  rerank_api_model: bge-reranker-v2-m3
  expand_api_url:   https://your-chat-endpoint/v1
  expand_api_model: <chat-model>
  # *_api_key optional (bearer)

Each *_api_url is independent: set only embed_api_url and rerank/expand stay local (fix D); set all three to go fully remote. Leave them all unset and nothing changes.

Backward compatibility

No breaking changes. All new behaviour is opt-in via config; local-only setups are unaffected; getDefaultLlamaCpp/setDefaultLlamaCpp retained.

Testing

Build (node scripts/build.mjs) and tsc --noEmit are clean. Under the project's CI test mode (CI=true, matching upstream CI), this branch shows no new failures vs main — the deterministic failing set is identical on both (the only failures are pre-existing WSL/Linux Git-Bash-path and skill-bundling tests, unrelated to this change). The new test/remote-llm.test.ts suite passes in full: embed/rerank routing, per-endpoint circuit breakers + independence, sigmoid normalization, oversized-rerank split/truncate recovery, auth, timeouts, expandQuery parsing + fallback, and the HybridLLM rerank fallback — all driven by in-process HTTP servers, no mocks.

This configuration runs in production today: embedding + reranking against two llama.cpp server instances and query expansion against a cloud chat endpoint — which is where refinements A, B, C (and the need for E) came from.

Related work & credit

This PR aims to consolidate the OpenAI-compatible-backend effort into one production-ready change, building on and crediting the prior attempts:

• @georgelichen — #629: the RemoteLLM/HybridLLM scaffold this PR is built on. Refinements A & D were discussed on the Add remote embedding, reranking, and query expansion support #629 / feat(llm): add remote embedding/reranking via OpenAI-compatible endpoints #575 threads.
• @loopyd — #619: an independent OpenAI-compatible backend (tested against llama-swap); refinement E is adapted from its rerank recovery. feat(Add OpenAI-compatible backend support) #619 uses a whole-backend provider switch (one base URL + server-side model aliases); this PR's HybridLLM generalizes that to independent per-operation endpoints with local fallback — the single-server case is just embed/rerank/expand_api_url all pointing at one server.
• @jonesj38 — #689: an OpenAI-specific backend; this PR's generic path covers the OpenAI case purely by configuration (embed_api_url: https://api.openai.com/v1) without hardcoding a provider.

Follow-ups (not in this PR)

• Byte-based embed batching (maxBatchBytes + chunk strategies + --max-batch-mb / --chunk-strategy flags), as in feat(Add OpenAI-compatible backend support) #619. This PR keeps the existing count-based batching (maxBatchSize); byte-aware batching is a natural follow-on.

[Kaspre]

Follow-up hardening has been pushed in db5a32e.

Latest changes:

• Preserve the model identity that actually produced rerank scores when writing rerank cache entries, so local fallback scores are not cached under a remote reranker key.
• Reject SDK store.embed({ model }) overrides when remote embedding is configured and the requested model differs from the configured remote embed model.
• Thread the per-store LLM through SDK searchVector() query embedding so remote SDK stores do not accidentally fall back to the global/default local LLM.
• Added focused regressions for rerank fallback cache identity, remote embed model override handling, and SDK searchVector() per-store LLM routing.

Validation run:

• CI=true node ./node_modules/vitest/vitest.mjs run --reporter=verbose --testTimeout 60000 test/store.test.ts test/sdk.test.ts passed: 289 passed, 19 skipped.
• test/remote-llm.test.ts passed outside the sandbox: 51 passed. The sandboxed run hit the known local listen EPERM restriction while starting its in-process HTTP server.
• pnpm build passed.
• node ./node_modules/typescript/bin/tsc -p tsconfig.build.json --noEmit passed.
• git diff --check passed.
• A multi-lane read-only review pass was run; it found the searchVector() per-store LLM issue above, which is now fixed, and the follow-up review came back clean.

Maintainers: this should be ready for review again.

[p0pfan]

I really hope this PR gets merged soon. I truly need this feature to use qmd on machines
without a GPU.

[udo76]

Running this on a Mac Mini M4 gateway with a dedicated GPU server (AMD Strix Halo, 128GB) hosting Qwen3-Embedding-4B and Qwen3-Reranker-4B on OpenAI-compatible endpoints (llama.cpp server).
Currently stuck with local 4B GGUF as workaround - works but ties up M4 resources.

The key use case for us: serving embedding and reranking to multiple machines on the network that are too weak to run meaningful models locally - thin clients, ARM64 SBCs, VPS nodes. A single GPU box providing high-quality retrieval for the whole fleet.
PR #705's HybridLLM per-operation fallback design matches this perfectly.

Happy to test any of the PRs against our endpoints.

[tobi]

Thank you (and all the others who worked on PRs related to this)

[georgelichen]

This is a strong PR overall. I really like the direction here: the RemoteLLM/HybridLLM split is thoughtful, the production hardening is substantial, and a lot of the follow-up work from #629 is integrated in a pragmatic way. Thanks for pushing this forward.

I went through the diff pretty carefully and found three things that I think are worth fixing before merge:

1. I think the publish workflow will fail to authenticate to npm as written.
   In .github/workflows/publish.yml, npm publish --provenance --access public no longer passes NODE_AUTH_TOKEN. actions/setup-node writes the registry config, but it still expects the token to be present in the environment. Without that, the release job looks correct but will fail at publish time.

2. I think the new launcher mis-detects installed packages on Windows.
   In bin/qmd, the installed-package heuristic uses pkgDir.split("/") to look for a node_modules segment. On Windows paths that never matches, so an installed package can be mistaken for a source checkout and incorrectly routed into source mode again. That seems to reintroduce the same class of Windows install/runtime issues this launcher rewrite is trying to fix.

3. I think --index path normalization is still incomplete on Windows.
   In src/cli/qmd.ts and src/collections.ts, the “treat this like a path and normalize it into an index name” branch only runs when the argument contains /. On Windows, --index C:\tmp\demo skips that path-normalization path, which can send the DB path and config path to unexpected locations. I reproduced that locally with Node path resolution.

None of these feel far from fixable, and I still think the overall PR is heading in a very good direction. I just wanted to flag these before merge so they do not turn into release-time or Windows-specific regressions.

[brettdavies]

Hi @Kaspre - #663 (qmd serve) and this PR both landed a class named RemoteLLM. I've already renamed mine to RemoteQMD (src/llm-remote.ts → src/remote-qmd.ts) to free the name up for yours, and nothing's blocked on your end. Wanted to introduce the two anyway, since they're adjacent and compose cleanly rather than collide.

The two sit on orthogonal axes:

• **feat(llm): OpenAI-compatible remote embedding, reranking & query expansion (RemoteLLM + HybridLLM) #705 - model-backend axis (where inference runs) ** Your RemoteLLM + HybridLLM enables a single qmd process offload embed / rerank / expand to an OpenAI-compatible provider (vLLM, Ollama, OpenAI) while keeping local generation.
• feat(serve): qmd serve shared model server + RemoteQMD client (supersedes #511) #663 - client axis (where qmd runs). Its RemoteQMD is a thin client that talks to a remote qmd serve over HTTP and loads no models locally.

flowchart LR
    client["thin qmd client<br/>RemoteQMD (PR 663)"]
    serve["remote qmd serve<br/>(owns index + corpus)"]
    local["local node-llama-cpp<br/>--low-vram, PR 663"]
    openai["OpenAI-compatible provider<br/>RemoteLLM + HybridLLM, PR 705"]

    client -->|"client axis<br/>(PR 663): where qmd runs"| serve 
    serve -->|"backend axis<br/>(PR 705): where inference runs"| local
    serve --> openai 

Loading

The two meet at the getDefaultLLM / setDefaultLLM seam that both PRs build on. A thin RemoteQMD client points at a remote qnd serve, and that server picks its model source: local node-llama-cpp, ollama-compat, or - once #705 lands - your OpenAI-compatible backend via HybridLLM. So #705 isn't a competing client; it's a backend option for the qma serve that #663 introduces.

RemoteLLM / HybridLLM stay yours. If you'd rather split the Remote* namespace differently, or you see a cleaner shape, say the word. The rename is cheap on our side and easy to redo.

[dfein38347g]

I ran into this same issue testing remote embedding against a server with --ctx-size 2048. The root cause: RemoteLLM and HybridLLM send text to the server without any token-count truncation. When chunks exceed the server's context window, the server returns 400 errors. The circuit breaker in RemoteLLM opens after 3 failures, cascading to all remaining chunks — in my case, only 145/864 chunks succeeded.

The local LlamaCpp path already handles this via truncateToContextSize() using the model tokenizer and QMD_EMBED_CONTEXT_SIZE (default 2048). So I exported the shared resolveEmbedContextSize() helper and added truncateForEmbed() to HybridLLM that uses the local tokenizer to truncate before remote delegation. Both embed() and embedBatch() are guarded.

After the fix:

qmd embed -f
# 876/876 chunks embedded, 0 errors

Full PR with 9 tests covering truncation, boundary cases, and the env var: Kaspre#1

Happy to adapt the approach however you'd prefer for this PR.

[soundadam]

I tested this head with Node 22 / pnpm 10: TypeScript passes and the focused suite passes (340 passed, 49 skipped). The current tests do not cover two correctness blockers:

1. Remote embedding bypasses the local context-window guard. RemoteLLM.embedBatchRequest() sends the input text directly, while the remote document path uses a fixed 3-chars/token approximation. This is unsafe for CJK/mixed text and for endpoints with a smaller context window; oversized requests can open the circuit breaker and fail most of a corpus. Please add an explicit remote embedding context limit and truncate/token-count both embed() and embedBatch() before the request, with CJK and mixed-language tests.

2. The OpenAI-compatible batch response is sorted by index and then returned positionally, but cardinality, unique indexes, complete 0..N-1 coverage, per-vector dimensions, and finite values are not validated. A response for three inputs containing only indexes [0, 2] will silently align index 2's vector with input 1 at the caller. Please construct the result array by validated index and fail the whole batch on missing, duplicate, out-of-range, non-finite, or dimension-mismatched entries.

The branch also currently conflicts with main, but the two issues above should be fixed independently of the rebase.

[mpetruc]

Thank you @tobi for creating and sharing this tool, and thank you @Kaspre @dfein38347g for these PRs. Remote embeddings endpoints are going to be a game changer and am eagerly awaiting for them to get merged. Fwiw - in the meantime i'd be more than glad to help with anything needed.