feat: add token-aware truncation to remote embedding path - #1
Open
dfein38347g wants to merge 11 commits into
Open
dfein38347g wants to merge 11 commits into
dfein38347g wants to merge 11 commits into
Conversation
…nsion Add a RemoteLLM backend that talks to any OpenAI-compatible HTTP API (vLLM, TEI, Ollama, llama.cpp --server, LiteLLM, OpenAI, ...), composed with the local LlamaCpp via a HybridLLM that routes each operation independently: - RemoteLLM: /v1/embeddings, /v1/rerank, /v1/chat/completions; per-endpoint circuit breakers; bearer auth; char-based token approximation so chunking works without a local tokenizer. - HybridLLM: embed/embedBatch/rerank/expandQuery -> remote, generate/tokenize/detokenize -> local, with per-operation local fallback. - Widened LLM interface (embedBatch, tokenize/detokenize/countTokens, isRemote, embedModelName, rerankModelName, generateModelName, usesRemoteEmbedding, supportsRerank/supportsExpand) plus getDefaultLLM/setDefaultLLM alongside the existing getDefaultLlamaCpp/setDefaultLlamaCpp (no breaking change). - Sigmoid normalization of log-odds rerank scores; RemoteLLM.expandQuery via chat completions; startup pre-flight embed probe; HybridLLM.rerank local fallback (symmetric with the expandQuery fallback). Opt-in via models.*_api_url / QMD_*_API_* env vars; with nothing configured the local-only path is byte-for-byte unchanged. Builds on @georgelichen's remote-LLM work in tobi#629. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ncating When a remote /v1/rerank request is rejected as too large (HTTP 413 / "too large to process" / context length), RemoteLLM.rerank recursively bisects the batch and halve-truncates a single oversized document down to a 32-char floor, remapping response indices to the originals and re-sorting by score. Non-oversized errors still propagate so the circuit breaker / HybridLLM local fallback can react. Adapted from the rerank recovery in tobi#619 (@loopyd). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- remoteConfigFromEnv: throw on a half-configured remote backend (embed_api_url set without embed_api_model, or vice-versa) instead of silently falling back to the local backend and skipping the remote pre-flight probe. - RemoteLLM.rerank: normalize scores once over the full (possibly recovery-split) result set, applying sigmoid only when logit-range values are present (any score < 0 or > 1). Rerankers that already return [0,1] probabilities (Cohere/Voyage-style) pass through unchanged, avoiding distortion of the blend and --min-score filtering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This personal checkout's native deps (better-sqlite3, sqlite-vec) are kept built for hermes Node 22 (ABI 127); production consumers (remnic/RMO, qmd-mcp) resolve qmd through this launcher via PATH. 2026-09-12 incident: an earlier npm install under Nix node 26 rebuilt better-sqlite3 for ABI 147, and every consumer using Node 22 died in process.dlopen with ERR_DLOPEN_FAILED (RMO warm-up looped 1800+ times; qmd-mcp crash-looped) — masked because the MCP-over-HTTP surface stayed up. - Pin the launcher shebang to /home/nathan/.hermes/node/bin/node so the launcher itself never depends on PATH node resolution (portable installs keep #!/usr/bin/env node — see tobi#381). - Add an ABI tripwire: if the resolved runner is Node and it reports any ABI other than 127, fail at spawn time with a message naming the rebuild command, instead of dying inside dlopen with a stack trace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Remote embedding (RemoteLLM/HybridLLM) sends text to the server without any truncation. When chunks exceed the server's
--ctx-size(e.g. 2048 tokens), the server returns 400 errors. RemoteLLM's circuit breaker opens after 3 failures, cascading to all remaining chunks — for a 864-chunk corpus, only 145 succeed.The local LlamaCpp path has
truncateToContextSize()using the model tokenizer andQMD_EMBED_CONTEXT_SIZE. The remote path has no equivalent guard.Solution
resolveEmbedContextSize()fromsrc/llm.ts(replaces the inline IIFE inLlamaCpp.EMBED_CONTEXT_SIZE)truncateForEmbed()toHybridLLM(src/hybrid-llm.ts) that uses the local tokenizer to tokenize, truncate toQMD_EMBED_CONTEXT_SIZE - 4(safety margin), and detokenize before delegating to the remoteembed()andembedBatch()QMD_EMBED_CONTEXT_SIZEenv varVerification
Related
Fixes the root cause discussed in tobi#705 — this is the same issue affecting the upstream PR.