test: VRAM precondition gate for integration suites with opt-in Ollama auto-evacuate

[brettdavies]

VRAM precondition gate for GPU-dependent integration suites

Why

The GPU-dependent integration sub-suites in llm.test.ts, mcp.test.ts, store.test.ts, eval.test.ts, and sdk.test.ts load real GGUF models and exercise embed, rerank, and expandQuery against a real node-llama-cpp runtime. When another workload is holding most of the GPU (Ollama parking a large model, a sibling daemon mid-job, a stale qmd serve), those allocations fail and the suites produce a cascade of Failed to create any rerank context failures that look like code regressions but are not.

The existing describe.skipIf(!!process.env.CI) gates handle CI correctly but leave a dev box exposed: the suites run, fail loudly, and obscure the genuine regressions the suite is supposed to catch. Asking developers to chase ghost failures every time a co-resident model is loaded is the wrong default.

This PR adds a one-shot VRAM probe at module load and gates ten integration sub-suites that perform real allocations on whether the GPU has enough headroom. Suites that do not need a real LLM (BM25 search, MCP HTTP transport, path utilities, store creation, chunking metadata) continue to run unconditionally.

What this PR adds

A helper at test/_helpers/vram-precondition.ts, wired into the existing describe.skipIf cascades:

File	Suite	Why gated
test/llm.test.ts	LlamaCpp Integration	Real embed / rerank / expandQuery
test/llm.test.ts	LLM Session Management	Lifecycle test using real LLM
test/mcp.test.ts	searchVec (vector similarity)	Real embedding for the query
test/mcp.test.ts	hybridQuery (expansion + reranking)	Real expandQuery and rerank
test/mcp.test.ts	MCP HTTP Transport	End-to-end suite including reranking
test/store.test.ts	Token-based Chunking	Real tokenizer load
test/store.test.ts	LlamaCpp Integration	Real embed + vector search
test/eval.test.ts	Vector Search	Real embedding for the eval corpus
test/eval.test.ts	Hybrid Search (RRF)	Real embed + rerank for the eval corpus
test/sdk.test.ts	with LLM query expansion	Real expandQuery

The helper decides whether to skip:

1. Cached result if present. Suites that import the helper a second time hit the cache.
2. QMD_SKIP_GPU_INTEGRATION=1 short-circuits to false without probing. Useful for fast unit-loop iteration on a healthy box when you just want the non-GPU tests.
3. First VRAM probe via getLlama().getVramState(). Threshold is 4 GB free, which covers embed (~0.7 GB) + rerank (~1.5 GB) + generate (~1.5 GB) + slack. CPU mode (no GPU) returns true without probing.
4. If the first probe is insufficient AND QMD_TEST_EVACUATE_VRAM=1 is set, the helper asks Ollama to unload its resident models via POST /api/generate {keep_alive: 0} for each model listed by GET /api/ps, then re-probes once. Off by default: test infrastructure silently terminating another process's loaded models would be astonishing without explicit consent. OLLAMA_HOST overrides the default http://127.0.0.1:11434 endpoint.
5. Probe failure fails open (returns true) so a genuine code failure surfaces in the test rather than being silently masked by a skip.

The eviction step is Ollama-only. Other VRAM consumers (qmd serve, sibling daemons) are deliberately out of scope: killing a user-facing daemon is too astonishing even behind the opt-in flag. All steps run best-effort with explicit timeouts (2 s for /api/ps, 5 s per unload). Connection failures, non-OK responses, and per-model unload errors are logged but do not abort the re-probe.

Dependencies on node-llama-cpp and the eviction HTTP calls are injected through options.getLlama and options.evacuate so the unit tests at test/vram-precondition.test.ts drive the gate without making real GPU calls or HTTP requests.

Behavior on the merged result

Environment	Probe outcome	Suites
CI (CI=1)	not run	skipped (unchanged)
Healthy local box (≥ 4 GB free GPU VRAM)	sufficient	run
CPU-only or node-llama-cpp not built	probe fails open	run (existing fail-loud behavior preserved)
QMD_SKIP_GPU_INTEGRATION=1	short-circuits	skipped
Contested local box (< 4 GB free), no opt-in	insufficient	skipped with operator-readable warning
Contested local box, QMD_TEST_EVACUATE_VRAM=1, eviction frees enough	sufficient on re-probe	run
Contested local box, QMD_TEST_EVACUATE_VRAM=1, eviction insufficient	still insufficient	skipped with post-eviction message

Measured locally

Same contested GPU for both runs (RTX 3090 Ti, Ollama holding ~20 GB of gemma4:26b, ~3.1 GB free).

Default (skip path, no opt-in)

The probe sees 2.81 GB free, prints the operator-readable warning, and skips the gated suites.

Before this PR: 389 pass /  0 skip /  8 fail
After this PR:  354 pass / 62 skip /  0 fail

The eight integration failures previously surfaced as Failed to create any rerank context cascades are now clean skips with a single warning line:

⚠ Integration tests skipped: only 2.81 GB free GPU VRAM (need ≥ 4 GB).
  Set QMD_TEST_EVACUATE_VRAM=1 to attempt eviction first,
  or QMD_SKIP_GPU_INTEGRATION=1 to silence this probe.

Evacuate path (QMD_TEST_EVACUATE_VRAM=1)

The probe sees 2.81 GB free, hits the eviction branch, logs the eviction attempt, and asks Ollama to unload its loaded model. Re-probe shows headroom restored and the gated suites then run.

⚠ Insufficient VRAM (2.81 GB free, need ≥ 4 GB);
  QMD_TEST_EVACUATE_VRAM=1, attempting to evict known consumers...
⚠ Eviction: asking Ollama to unload 1 model(s): gemma4:26b-a4b-it-q4_K_M
✓ Eviction succeeded; 22.31 GB free, integration tests will run.

Whole suite (29 files): 876 pass / 80 skip / 0 fail against the gated suites.

Two caveats worth knowing:

1. The integration tests that run after eviction can still hit individual rerank context allocation failures on a bare tobi:main checkout because the vanilla single-pass reclaim path is the weak link, not the headroom this gate restores. This is the same failure mode that feat(llm): --low-vram mode with on-demand reclaim on embed VRAM pressure #662 (lowVram two-pass reclaim) is designed to fix. Stacking feat(llm): --low-vram mode with on-demand reclaim on embed VRAM pressure #662 on top of this PR turns the post-eviction integration suite green; this PR alone restores the headroom but does not change reclaim behavior.
2. Ollama may re-load a model between the eviction call and the actual test allocations if another client (an MCP agent, another ollama run, a residual keep_alive from a recent request) re-requests the same model during the suite run. The gate is a probe-once mechanism; it does not pin Ollama into a quiet state for the duration of the suite.

The eviction path is most useful on developer machines where the operator opts in deliberately and expects the test session to take priority over background Ollama state for a few minutes.

Test coverage

19 unit tests on the helper (test/vram-precondition.test.ts), all driven through injected getLlama and evacuate so no real GPU or HTTP request runs in this suite:

• CPU mode and CPU-offload-forced skip the VRAM probe entirely.
• Plentiful VRAM (20 GB), exactly at threshold (4 GB), just below threshold (boundary behavior).
• Ollama-hogged scenario (~1.4 GB free) returns false and emits the operator-readable warning.
• QMD_SKIP_GPU_INTEGRATION=1 forces a skip without probing.
• getLlama and getVramState throwing both fail open.
• Result is cached so repeated calls do not re-probe.
• Skip warning logs at most once per process.
• Eviction does not run when QMD_TEST_EVACUATE_VRAM is unset.
• Eviction runs when QMD_TEST_EVACUATE_VRAM=1 and first probe insufficient, then re-probes.
• Eviction runs but does not free enough → returns false with post-eviction message.
• Eviction throws → still re-probes (best-effort), the throw is logged.
• First probe already sufficient → eviction is not called even when env=1.
• QMD_SKIP_GPU_INTEGRATION=1 precedence wins over QMD_TEST_EVACUATE_VRAM=1.
• Red-team: env values other than the exact string "1" ("0", "false", "true", "yes", "") do not trigger skip or eviction.
• Default eviction path against an unreachable Ollama logs the connection failure and the re-probe still runs.

The helper test harness explicitly snapshots and clears QMD_SKIP_GPU_INTEGRATION, QMD_TEST_EVACUATE_VRAM, and OLLAMA_HOST in beforeEach / afterEach, so the helper suite passes identically under each of (no env vars), QMD_TEST_EVACUATE_VRAM=1, and QMD_SKIP_GPU_INTEGRATION=1. Without that isolation the tests would inherit the operator's chosen env at the surrounding process level and false-fail.

tsc --noEmit clean on the helper and its tests.

Backwards compatibility

• Purely additive. Suites that already had describe.skipIf(!!process.env.CI) keep the CI short-circuit and add the VRAM precondition via boolean-OR.
• Default behavior on a healthy local box is unchanged: the probe returns true, the suites run.
• Default behavior in CI is unchanged: CI=1 short-circuits before the probe matters.
• The two new env vars (QMD_SKIP_GPU_INTEGRATION, QMD_TEST_EVACUATE_VRAM) are opt-in and read precisely (only the string "1" triggers). OLLAMA_HOST is read only when eviction is opted into.

Changelog

Added

• VRAM precondition helper at test/_helpers/vram-precondition.ts that probes free GPU VRAM once per process and reports whether the integration suites can run meaningfully. Gates ten GPU-dependent sub-suites across test/llm.test.ts, test/mcp.test.ts, test/store.test.ts, test/eval.test.ts, and test/sdk.test.ts.
• QMD_SKIP_GPU_INTEGRATION=1 env var forces the integration suites to skip without probing the GPU.
• QMD_TEST_EVACUATE_VRAM=1 env var opts into asking Ollama to unload its resident models when the initial VRAM probe shows insufficient headroom, then re-probing. Off by default.
• OLLAMA_HOST env var overrides the default http://127.0.0.1:11434 endpoint used by the eviction step.

Type of Change

• test: Adding or updating tests
• feat: New feature
• BREAKING CHANGE: Breaking API change

Related Issues/Stories

• Story: n/a
• Issue: n/a
• Architecture: n/a
• Related PRs: independent of feat(llm): --low-vram mode with on-demand reclaim on embed VRAM pressure #662 (lowVram engine) and feat(serve): qmd serve shared model server + RemoteQMD client (supersedes #511) #663 (qmd serve). Addresses the same underlying contested-GPU problem but at the test-infrastructure layer rather than the runtime layer. The evacuate path's "still red after eviction" caveat above is exactly what feat(llm): --low-vram mode with on-demand reclaim on embed VRAM pressure #662 fixes at the engine level; stacking the two turns the post-eviction integration suite green.

Testing

• Unit tests added (test/vram-precondition.test.ts, 19 cases)
• Integration tests added (the gated suites themselves now skip cleanly when GPU is contested)
• Manual testing completed (verified against an Ollama-hogged GPU, both skip and evacuate paths)
• All tests passing

Files Modified

Modified:

• test/llm.test.ts: VRAM precondition import + probe + skipIf extension on two integration sub-suites.
• test/mcp.test.ts: same on three integration sub-suites.
• test/store.test.ts: same on two integration sub-suites.
• test/eval.test.ts: same on two integration sub-suites.
• test/sdk.test.ts: same on one integration sub-suite.

Created:

• test/_helpers/vram-precondition.ts: the helper itself.
• test/vram-precondition.test.ts: unit tests for the helper (19 cases including red-team).

Renamed:

• None.

Deleted:

• None.

Breaking Changes

• No breaking changes.

Deployment Notes

• No special deployment steps required. The new env vars are opt-in; default behavior on a healthy local box and in CI is unchanged.