feat: low-vram engine + qmd serve combined (production shape of #662 + #663) - #927
Draft
brettdavies wants to merge 20 commits into
Draft
brettdavies wants to merge 20 commits into
brettdavies wants to merge 20 commits into
Conversation
Adds `lowVram` to `LlamaCppConfig` (also `--low-vram` CLI flag and `QMD_LOW_VRAM=1` env). When enabled, the heavy generate (~2 GB) and rerank (~2.3 GB) models are disposed immediately after each use, while the tiny embed model (~320 MB) stays resident. Peak VRAM drops from ~5.4 GB to ~2.6 GB at the cost of per-stage load latency (~3 s → ~5.6 s on a typical GPU). This makes qmd usable on GPUs where loading all three models at once exhausts free VRAM — for example, sharing a 24 GB GPU with a ~20 GB Ollama instance. Addresses the failure mode tracked in tobi#275 across all entry points that construct an LlamaCpp instance: `qmd query`, `qmd mcp`, and the upcoming `qmd serve`. The pipeline stages (expand → embed → search → rerank) are inherently sequential, so disposing between them only costs reload time, not correctness. Concurrency: when lowVram is on, expandQuery and rerank calls serialize through per-method promise chains so a dispose can never race with another caller's in-flight use of the same model. embed and embedBatch remain parallel. The two chains are independent — expand and rerank against their separate heavy models can run in parallel against each other. The flag is global (`--low-vram` works on any subcommand that constructs an LlamaCpp), so qmd query, qmd mcp, and other one-shot commands all benefit — not just long-lived daemons. Naming follows the existing engine-knob pattern (QMD_FORCE_CPU, QMD_LLAMA_GPU).
In lowVram mode, wrap LlamaCpp.embed and embedBatch with a catch-and-retry path: when the underlying ensureEmbedContext(s) fails because node-llama-cpp pre-allocation refused for lack of VRAM, drain the generate and rerank chains (so any in-flight expand/rerank completes its own finally-dispose), defensively dispose both heavy models, and retry the operation once. Outside lowVram mode this is a pass-through, so default-path callers see zero overhead. isInsufficientVramError matches node-llama-cpp's 'too large for the available VRAM' string, our own 'Failed to create any (embedding|rerank) context' wrapper messages, and the bare 'out of memory' / 'insufficient VRAM' variants. Conservative on purpose: too broad and we'd evict on unrelated failures, too narrow and we'd miss the case the feature targets. The chain-drain via Promise.allSettled is what makes the dispose race-free against in-flight expand/rerank callers: by the time their chain settles, each caller has already run its own finally-dispose, so our subsequent disposeGenerateModel / disposeRerankModel calls land on null models and are no-ops. The retry then succeeds because the embed context can allocate into the freed VRAM. 5 new tests in test/llm-low-vram.test.ts cover: embed and embedBatch recover from VRAM error and return correct results, non-VRAM errors do not trigger eviction, lowVram=false skips the reclaim path entirely, and reclaim awaits in-flight generate/rerank chains before disposing.
Generalize embedWithReclaim into a single withReclaim(needs, fn) helper, then wire it into the expandQuery and rerank lowVram chain handlers. The 'needs' argument names which model the operation requires (so it's never disposed) and identifies which OTHER chain to drain. expandQuery uses needs='generate' (drains rerankChain, disposes rerank); rerank uses needs='rerank' (drains generateChain, disposes generate); embed continues to use needs='embed' (drains both, disposes both). The needs discrimination also prevents a self-await deadlock, because a caller inside generateChain cannot await generateChain without blocking on its own current link. By restricting drains/disposes to the OTHER chains, the reclaim path is always safe. Three new tests in test/llm-low-vram.test.ts: expandQuery retries after VRAM error and disposes rerank (not generate); rerank retries after VRAM error and disposes generate (not rerank); expand reclaim awaits an in-flight rerank chain before disposing rerank. Existing 11 lowVram tests still pass. Practical impact: a qmd query that previously failed with 'A context size of 2048 is too large for the available VRAM' on the expand stage now recovers if the only thing standing between it and its budget is a resident rerank model (or vice versa). External VRAM contention (e.g., a co-resident Ollama eating the budget) is still unrecoverable: that's the next layer the maintainers may want.
…ror strings Adds a tripwire test suite for `isInsufficientVramError`, the function that gates the two-pass reclaim path in `withReclaim`. The regex catalog inside the function has to match the real error strings that node-llama-cpp emits under VRAM pressure or the reclaim never fires when it should, and the user just gets a raw `Failed to create any rerank context` instead of a recovered query. Exports the function (was previously module-local) and pins it against two curated lists of strings. Positives that must match: | Error string | Source | | --- | --- | | `A context size of N is too large for the available VRAM` | `node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:30` single-sequence `InsufficientMemoryError` template | | `A context size of N with M sequences is too large for the available VRAM` | Same template, multi-sequence form | | `Failed to create any (rerank\|embedding) context` | Synthetic strings this codebase throws when the inner allocation loops give up | | `CUDA error: out of memory` | Generic CUDA OOM | | `ggml_cuda_compute_forward: out of memory` | GGML OOM under load | | `ran out of memory while allocating buffer` | Lowercase variant | | `Insufficient VRAM to load the model` | Safety-check variant | Negatives that must NOT match. False positives would chew through retries on errors the reclaim cannot fix: | Error string | Why it must not match | | --- | --- | | `model file not found` | IO error, not VRAM | | `Computing rankings is not supported for this model.` | Wrong architecture for the call | | `fetch failed: connect ECONNREFUSED ...` | Network error during model download | | `Invalid GGUF magic bytes` | File corruption | | `Something else went wrong` | Generic runtime error with no memory-related keywords | | `Loaded model into memory successfully` | Mentions "memory" but is not OOM (red-team: substring greediness) | | `VRAM usage: 12.3 GB / 24 GB` | Mentions "VRAM" but is a status line (red-team) | Input-shape tests round out the suite. The function must handle null, undefined, empty string, a bare string with a positive message, an arbitrary object that string-coerces to a positive message, and case-insensitivity across the catalog. Adding a positive case here that fails before updating the regex is the preferred way to fix a "reclaim didn't fire when it should have" report. The failing test pins the behavior, the regex update makes it pass, and future regressions are caught. Twenty-one tests pass on the new file. ## Changelog ### Added - `isInsufficientVramError` is now exported from `src/llm.ts` so the reclaim trigger function can be unit-tested directly against curated error strings. - `test/llm-vram-error-pinning.test.ts` pins the reclaim trigger against a sourced list of real node-llama-cpp error messages plus codebase-internal synthetic strings, and against a red-team list of similar-looking errors that must not trigger.
Hardens lowVram mode's `withReclaim` for cards where shared workloads (Ollama, gbrain, etc.) leave qmd only a few GB of headroom. The previous single-pass reclaim only freed generate and rerank. Neither helped when the rerank context allocation itself was the failing call and the embed model was the next-largest qmd-side consumer. The new flow on a non-embed reclaim: 1. First pass: drain the other chains, dispose generate model, dispose rerank model, dispose embed *contexts* (~143 MB × N). Retry once. 2. Second pass: if the retry also hits a VRAM error, drop the embed *model* (~568 MB). Retry once more. 3. Embed reclaim short-circuits the second pass because it cannot make progress without the model. Each dispose runs best-effort: a failure is logged with the dispose label and the remaining disposes still run, so a single broken dispose does not poison the retry. The original VRAM error is more useful than a downstream dispose error, and a retry against partial reclaim is still more likely to succeed than no retry at all. Also removes a misleading "retry without flash attention" branch in `ensureRerankContexts`: both `createRankingContext` calls passed identical options, and `LlamaRankingContextOptions` does not accept `flashAttention` (the option is silently dropped at `LlamaRankingContext.js:160-168`). The synthetic `Failed to create any rerank context` message is preserved so `isInsufficientVramError` continues to recognize it. Worst-case attempt count is capped at 3 (initial + first-pass retry + second-pass retry). Test coverage (`test/llm-low-vram.test.ts`): - First-pass reclaim symmetry for `embed`, `expandQuery`, `rerank` (existing tests updated for new dispose:embed-contexts event). - Second-pass escalation for `rerank` and `expandQuery` (3-attempt success path). - Second-pass cap for `rerank` and `expandQuery` (3-attempt give-up path). - `embed` reclaim never disposes embed contexts or model under any retry pattern. - Reclaim continues retry when a dispose helper itself throws (best-effort, warning logged). - Reclaim ignores non-VRAM errors (no dispose, no retry). - Reclaim treats a post-first-pass non-VRAM error as terminal (no escalation). 23 tests pass on the file. Wider suite delta is unchanged (the remaining 19 wider failures are pre-existing GPU-contention integration tests; addressed separately). - Low-VRAM reclaim now escalates to disposing the embed model when an initial dispose pass plus retry still hit a VRAM error, with all dispose calls running best-effort so a single failing dispose does not abort the retry. - Removed a no-op "retry without flash attention" fallback in `ensureRerankContexts` whose retry passed identical options to the failing call and whose `flashAttention` flag is silently dropped by `node-llama-cpp`'s `createRankingContext`.
Picks the rerank context size based on free VRAM at allocation time and budgets `computeParallelism` against the actual chosen size. The previous code allocated a fixed 4096-token context (~1146 MB with flash attention) and budgeted parallelism at a hardcoded 1000 MB/context, which both over-allocated on contested cards and skewed the parallelism estimate. Resolution order in `resolveRerankContextSize`: 1. `QMD_RERANK_CONTEXT_SIZE` env override (read at call time, not class-load) always wins. Falls back to adaptive logic on missing / non-positive / non-numeric values. 2. CPU mode (offload forced or no GPU) keeps the 4096 default. CPU RAM is plentiful and the per-token cost model does not apply. 3. GPU probe via `getVramState`. Free VRAM below 1500 MB shrinks to 2048; at or above the threshold stays at 4096. A probe failure conservatively falls back to 4096, where the caller's existing retry path will catch a too-tight allocation. Threshold derivation: the 4096-token rerank context with flash attention costs roughly 1.15 GB. The 1500 MB threshold reserves the context plus ~350 MB margin for the rerank model and concurrent allocations. The per-token cost model (0.28 MB/token) was measured against real loads: 4096 ≈ 1146 MB, 2048 ≈ 568 MB. The chosen size is stored on the instance as `rerankContextSize` so `rerankImpl` truncates documents against the same size that was actually allocated, never a static guess that would under-truncate when the context was shrunk. The field resets to `null` in `disposeRerankModel`, `unloadIdleResources`, and `dispose`, so a re-allocation re-probes VRAM. `computeParallelism` now receives `Math.ceil(contextSize * 0.28)` as the per-context MB estimate. At the 4096 default this is roughly 1146 MB (was 1000 MB hardcoded); at the 2048 fallback it is roughly 573 MB. Parallelism scales accordingly. Test coverage (`test/llm-adaptive-rerank.test.ts`): - Env override returns user-supplied size regardless of VRAM, including when VRAM is plentiful. - Invalid env override (0, negative, non-numeric, empty) falls back to adaptive logic. - CPU mode and CPU-offload-forced skip the VRAM probe entirely. - Plentiful VRAM returns 4096; exactly at threshold returns 4096; just below returns 2048. - Ollama-hogged GPU scenario (~1.4 GB free) returns 2048. - `getVramState` throwing falls back to 4096. - `ensureRerankContexts` stores the resolved size on `rerankContextSize`. - `computeParallelism` budget reflects the actual chosen size, not the old 1000 MB hardcode. - `disposeRerankModel` and `dispose` clear `rerankContextSize` so the next call re-probes. 18 tests pass on the new file; pre-existing lowVram concurrency tests on this branch are unaffected. - Rerank context size now adapts to free VRAM at allocation time, picking 2048 instead of the 4096 default when free VRAM drops below ~1.5 GB. The `QMD_RERANK_CONTEXT_SIZE` env override still wins when set. - Parallelism for rerank context creation is now budgeted against the actual chosen context size instead of a fixed 1000 MB-per-context estimate that assumed the older 2048 default. - Long-document truncation in `rerankImpl` now uses the actually allocated context size, not a static constant, so documents are never under-truncated when the context was shrunk under VRAM pressure.
…path Adds an end-to-end test that drives the two-pass reclaim against a real `node-llama-cpp` runtime instead of monkey-patched fakes. The synthetic tests in `llm-low-vram.test.ts` prove the orchestration of `withReclaim`. This file fills the gap they leave: - The real dispose helpers actually run against a live model and contexts. - The actual `InsufficientMemoryError` thrown by `node-llama-cpp` exercises the live `isInsufficientVramError` regex (B6 pins it string-by-string, this throws it). - The three-attempt cap holds against a real failing allocation, not just a synthetic throw. The recovery path (reclaim disposes other models and the retry succeeds) needs precise VRAM ballast and tear-down that is hard to make deterministic on a shared GPU, so it stays covered by the synthetic suite. Gated behind `QMD_RECLAIM_INTEGRATION=1`. The tests are skipped by default because: - They download the rerank model on first run (one-off, can take minutes). - They exercise real GPU allocation, so CI and remote-server-routed runs cannot meaningfully execute them. - They pair naturally with B3's VRAM precondition: the suite does not care about free VRAM, it only needs enough headroom to load the rerank model (~400 MB). Two cases: 1. **Oversized rerank context → reclaim gives up cleanly after 3 attempts.** Reflectively swap the static `RERANK_CONTEXT_SIZE` to 1 048 576 tokens (~290 GB context). The static is captured at class load, so `process.env` mutation alone would have no effect; the reflective swap is the only honest way to force a real allocation that no GPU can satisfy. Spy on `rerankImpl` to count attempts (wrap, do not replace, so the real implementation runs). Verify the call rejects, the spy ran exactly three times, and the surfaced error is recognized by `isInsufficientVramError`. If that last assertion ever fails, the lib reworded its error and the regex needs a new alternative — add a positive case to `llm-vram-error-pinning.test.ts` first, then update the regex. 2. **Non-VRAM rerank failure does not retry (missing model file).** Build a `LlamaCpp` pointed at `/nonexistent/path/to/no-such-model.gguf`, attempt rerank, verify exactly one attempt was made and the error is NOT classified as VRAM. Catches the inverse failure: if `isInsufficientVramError` ever started matching non-VRAM errors, the reclaim would chew through retries for nothing and bury the real diagnostic. Each test owns a fresh `LlamaCpp` and disposes it in `finally` so a failure cannot leak GPU resources into the next test. Verified locally: ``` QMD_RECLAIM_INTEGRATION=1 bun test --preload ./src/test-preload.ts test/llm-reclaim-integration.test.ts 2 pass / 0 fail / 6 expect() calls in 794ms ``` (Models pre-cached on the dev box; first-run download would dominate the timing.) ## Changelog ### Added - `test/llm-reclaim-integration.test.ts` real-GPU tests for the two-pass reclaim path, gated behind `QMD_RECLAIM_INTEGRATION=1` so they do not run in CI or remote-LLM-routed environments. Covers the give-up branch (oversized context forces real allocation failure, asserts the 3-attempt cap and `isInsufficientVramError` classification) and the no-retry branch (missing model file must not be misclassified as VRAM).
Adds qmd serve — HTTP server for embedding, reranking, query expansion. Supports local (node-llama-cpp) and rkllama (RK3588 NPU) backends. RemoteLLM client auto-activates via QMD_SERVER env var. Includes: - Batch embedding (single HTTP call for all chunks) - NPU timeout/retry tuning for ARM SBCs - rkllama rerank via logit-based scoring - Index endpoints: /search, /browse, /collections, /status - Security: default bind 127.0.0.1, 50MB body limit, type validation - Updated README and CHANGELOG (cherry picked from commit 108559d) (cherry picked from commit d7b340d)
Embeds the query via the configured backend (rkllama/local), then runs sqlite-vec nearest-neighbour search against stored vectors. Returns ranked results with cosine similarity scores. Enables TinyAgentOS to offer semantic memory search over HTTP. (cherry picked from commit cc225b1) (cherry picked from commit a33cf0f)
…I naming - Backend type: 'rkllama' → 'ollama' (Ollama-compatible, works with rkllama/ollama/etc) - CLI: --backend-url replaces --rkllama-url (old flag kept as deprecated alias) - Class: RKLlamaBackend → OllamaCompatBackend - Default URL: localhost:11434 (standard Ollama port) - All internal comments genericised - --rkllama-url and RKLLAMA_URL env var still work for backwards compat (cherry picked from commit b6c1019) (cherry picked from commit 6c21a7b)
…warning The `--low-vram` flag from the engine layer applies automatically to `qmd serve` because it constructs an `LlamaCpp` instance via the local backend. This commit adds three small UX touches so the feature is discoverable in the serve context: - Serve help text mentions `qmd serve --low-vram` alongside the other serve modes. - Startup log shows "Backend: local low-vram (one heavy model at a time)" and tags each model line with (resident)/(on demand) so an operator can see at a glance which models will reload between pipeline stages. - Warn (instead of silently ignoring) when `--low-vram` is combined with `--backend ollama` — ollama is a separate process whose model lifecycle qmd can't control, so the flag has no effect; operators should configure keep-alive on the upstream server instead. The underlying mechanism lives in `LlamaCpp` and is shared with `qmd query`, `qmd mcp`, and any other entry point that constructs a local engine.
…L reaches RemoteLLM
The serve / RemoteLLM commits earlier in this PR wire `setDefaultLLM(new RemoteLLM(...))` at the CLI entrypoint, but the store layer hardcoded `getDefaultLlamaCpp()` at every embed / expandQuery / rerank call site. The polymorphic accessor was set correctly and then bypassed one layer deeper. `qmd query` continued to load the local `LlamaCpp` for all three pipeline stages, allocating ~5.4 GB of VRAM even when a healthy `qmd serve` daemon was reachable.
On a VRAM-constrained box (e.g. shared with a co-resident Ollama), this silently OOMs the rerank stage with `Failed to create any rerank context`; with more headroom, the bug is invisible but the user is paying for two LLM backends.
Changes:
- `src/store.ts:84` (`getLlm`), `:3562` (embed), `:3737` (expandQuery), `:3783` (rerank) now use `getDefaultLLM()` and accept a widened `LLM` override instead of `LlamaCpp`. `Store.llm` widens to `LLM` to match the documented "Can be LlamaCpp or RemoteLLM" comment.
- `src/cli/qmd.ts:getStore` skips `setDefaultLlamaCpp(new LlamaCpp(...))` when `QMD_REMOTE_URL` is set, so the CLI no longer eagerly allocates VRAM the user delegated to `qmd serve`. The `--remote-url` flag mirrors into `process.env.QMD_REMOTE_URL` so one source of truth gates the guard.
- `src/llm.ts`: `LLM` interface gains optional `embedModelName` / `generateModelName` / `rerankModelName` readonly accessors so existing consumers can keep their `?? DEFAULT_*` fallbacks without casting. Also drops the buggy CommonJS `require("./llm-remote.js")` lazy import in `getDefaultLLM` (broke under vitest ESM resolution); the static import does not introduce a cycle because `llm-remote.ts` only type-imports from `llm.ts`.
- Indexing paths that genuinely need the local engine (`generateEmbeddings`, embedding fingerprint adoption) go through a new `requireLlamaCpp(llm, op)` helper. It accepts a real LlamaCpp unconditionally, refuses with a clear message when the active LLM is a RemoteLLM, and casts through for duck-typed test fixtures so existing mocks keep working. Querying (embed, expand, rerank, vec) works end-to-end against `RemoteLLM`.
Test changes:
- New `test/store-remote-llm.test.ts` (6 tests): covers both `llmOverride` and `getDefaultLLM()` routing for `expandQuery` and `rerank`, plus the `Store` adapter routing through `Store.llm`. All pass under both Node/vitest and Bun.
- `LlamaCpp Integration` describe block in `test/store.test.ts` now opportunistically routes through `qmd serve` (probes `http://127.0.0.1:7832/health` in `beforeAll`, calls `setDefaultLLM(new RemoteLLM(...))` when reachable). This turns the integration suite into an end-to-end regression check for the remote path AND lets the rerank tests pass on VRAM-constrained dev boxes where the local rerank model can't load alongside a co-resident model. Falls back to local LlamaCpp when the daemon isn't running.
- `rerank deduplicates identical chunks across files` spy updated from `getDefaultLlamaCpp` to `getDefaultLLM` to match the new routing.
- Test environment hermetics: `vitest.config.ts` and `src/test-preload.ts` both unset `QMD_REMOTE_URL` so a developer's shell-set env var doesn't accidentally route 200+ unrelated tests through a real daemon (or fail them with `ECONNREFUSED` when it isn't running).
Drop store.ts's requireLlamaCpp gate that refused RemoteLLM for generateEmbeddings and embedding-fingerprint adoption. Both paths use only LLM-interface methods (session.embed, chunkDocumentByTokens), so the LlamaCpp narrowing was overly defensive: it blocked qmd embed when QMD_REMOTE_URL was set without a real technical reason. Widen the supporting machinery to match: LLMSessionManager, LLMSession, withLLMSessionForLlm, and getSessionManager now operate on LLM, not LlamaCpp. embedBatch on the LLM interface accepts EmbedOptions for parity with embed. Add an optional ready() hook so backends with async metadata (RemoteLLM fetching model names from /health) can warm up before callers read embedModelName. chunkDocumentByTokens switches from getDefaultLlamaCpp() to getDefaultLLM(); the detokenize fallback in the pathological-single-line branch now char-truncates when llm.detokenize is absent (RemoteLLM never had it), keeping correctness without requiring a tokens-to-text round trip the dummy remote tokens could not service.
RemoteLLM now fetches /health in the background at construction time so embedModelName, generateModelName, and rerankModelName are populated by the time the embed flow reads them. A ready() method awaits the in-flight warm-up; generateEmbeddings calls it before deriving the active model URI so vectors are tagged with the server's real embedding model, not DEFAULT_EMBED_MODEL. RemoteLLM.embedBatch now forwards options to the server, and the /embed-batch handler on serve threads them to the local LlamaCpp backend (parity with the existing /embed path). ModelBackend.embedBatch widens to accept options across LocalBackend and OllamaCompatBackend. Tests cover both halves: a real RemoteLLM against a tiny inline /health stub drives generateEmbeddings to a clean early return (proving the gate is gone), and a TokenOnlyLLM mock exercises the new char-truncation fallback path in chunkDocumentByTokens for backends without detokenize.
vitest no longer scrubs QMD_REMOTE_URL from the test process: if it's set in the shell, a new setupFile (src/test-setup-remote.ts) registers a RemoteLLM as the default LLM so withLLMSession-driven tests share serve's resident models instead of allocating their own LlamaCpp on the GPU. Unset (CI default) keeps the original hermetic behavior. The LlamaCpp Integration describe block in test/llm.test.ts now also skips when QMD_REMOTE_URL is set, since those tests bind directly to getDefaultLlamaCpp() and would still try to load local models even though serve owns the VRAM. Net effect on this box: the three VRAM-bound failures (mcp hybridQuery reranks, mcp hybridQuery full pipeline, llm session rerank) pass cleanly instead of failing 'context size 2048 too large for the available VRAM'.
When the index was built with a different embedding model than the server is currently using, the query vector's dimension doesn't match the stored vectors and sqlite-vec throws a raw "Dimension mismatch" error that surfaced as an opaque 500. Catch that case and return a 409 with the query model + dimension and a rebuild hint, so the operator immediately knows the index needs rebuilding (or the wrong model is configured) instead of debugging a 500. (cherry picked from commit 01cb340)
…ontracts (cherry picked from commit 81d2e29)
The low-vram tests monkey-patch the *Impl methods to exercise the reclaim and serialization wrappers without loading real models. Two ambient env vars distort them: CI trips LlamaCpp's _ciMode guard, which throws before the wrapper runs, and a leaked QMD_LOW_VRAM=1 flips the lowVram=false cases to true. Neutralize both in beforeAll and restore in afterAll so the suite is deterministic regardless of environment.
Open PR tobi#705 (OpenAI-compatible remote embedding/reranking) introduces its own RemoteLLM plus HybridLLM for the model-backend tier (qmd talking to a model provider). Our class is a different thing at a different tier: an LLM-shaped client that talks to a remote qmd serve instance over HTTP. Two classes named RemoteLLM would clash on merge and read as the same concept. Rename the class RemoteLLM to RemoteQMD and RemoteLLMConfig to RemoteQMDConfig, and the file src/llm-remote.ts to src/remote-qmd.ts (a near-homophone of tobi#705's src/remote-llm.ts). RemoteQMD also matches the repo's all-caps QMD type convention (QMDStore) and leaves room to grow search/vsearch methods if the client later pulls the corpus, not just models. The getDefaultLLM/setDefaultLLM seam both PRs share is left unchanged. The --remote-url flag and QMD_REMOTE_URL env var are unchanged.
This was referenced Aug 25, 2026
The oxlint fence on main (anti-slop/no-chained-type-assertions) rejects the `x as unknown as T` chains the low-vram suites used to reach private members. A widen() helper moves the unknown-widening to the test boundary, so each site narrows with one assertion and keeps its precise structural type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The combined production shape of the low-vram engine (#662) and
qmd serve(#663): both stacks on one branch, rebased onto currentmain, plus the two pieces that only make sense when both are present: the serve-side low-vram surface (qmd serve --low-vramhelp text, startup log tagging models resident/on-demand, the warning when combined with--backend ollama) and the low-vram test-fixture alignment against the widenedEmbeddingResult/RerankResultcontracts.Filed as its own PR so the combined shape has a stable branch: #662 and #663 are now single-concern (engine-only and serve-only, independently mergeable in either order), and anyone running the combination in production (@jaylfc) can track this branch instead of one I rewrite during the split. Merge either this PR or #662 + #663; not both. If the two single-concern PRs land, this one closes as redundant.
Net diff vs
mainis the union of the two PRs plus the integration commits. See #662 for the engine design (one heavy model at a time, two-pass VRAM reclaim with embed-model escalation, adaptive rerank context size) and #663 for the serve design (qmd serveHTTP server,RemoteQMDclient, ollama-compat backend, store-layer routing throughgetDefaultLLM, embed-over-remote).Changelog
Added
--low-vrammode (flag orQMD_LOW_VRAM=1): loads one heavy model at a time and reclaims VRAM on embed pressure, so qmd coexists with another large model (e.g. Ollama) on the same GPU.qmd servesubcommand: long-running HTTP server exposing embed, rerank, expand, tokenize, vsearch, and read-only index endpoints, withlocal(node-llama-cpp) andollama(Ollama-compatible REST) backends.RemoteQMDclient implementing theLLMinterface against aqmd serveinstance, auto-activated byQMD_REMOTE_URLor--remote-url <url>.qmd serve --low-vramsurfaces the engine flag in serve help and the startup log (models tagged resident/on-demand), and warns when combined with--backend ollama, which owns its own model lifecycle.Changed
getDefaultLLM()soQMD_REMOTE_URLreaches the remote client at every call site.Type of Change
feat: New feature (non-breaking change which adds functionality)Related Issues/Stories
Testing
Test Summary:
QMD_RECLAIM_INTEGRATION=1).tsc --noEmitclean.Files Modified
Modified:
src/serve.ts,src/cli/qmd.ts: the serve↔low-vram surface (help text, startup log, ollama warning) present only in this combined shape.test/llm-low-vram.test.ts: mock fixtures aligned with the widenedEmbeddingResult/RerankResultcontracts from the serve stack.Created:
Renamed:
Deleted:
Breaking Changes
Deployment Notes