Measure the 7B's VRAM peak instead of bracketing it

[UniverseScripts]

What this changes

MIN_FREE_VRAM_MIB was inferred from two data points — a segfault at 6561 MiB free and a success at 6721 — and its own comment admitted the value chosen inside that band was "slightly optimistic". Nobody had watched the card during a load, so the requirement had never been observed, only guessed from whether the process survived.

pipeline/tools/vram_probe.py samples nvidia-smi at 5 Hz across a load and reports peak = free_before − min(free) — the same source and unit the gate compares against. A peak in torch's units would need converting through a bookkeeping layer this repo already documents as wrong by gigabytes on WDDM.

Six loads, 6/6 succeeded:

peak (MiB)	6056	6059	6104	6161	6171	6239

Max 6239, spread 183. LLM_MIN_FREE_VRAM_MIB = 6420 (max peak + spread) now lives in config.py and is read by both s19_generate and the demo service's explanation.py.

The two gates had drifted 1455 MiB apart

s19_generate still asserted vram_free_gb > 5.5 (≈5245 MiB). That was written when the figure came from torch.cuda.mem_get_info() and was optimistic by gigabytes; once vram_status() started telling the truth, 5.5 GB sat below a level already known to segfault, so the assertion protected nothing. One definition, two readers.

The sampler runs in the parent

A load that does not fit segfaults — no exception, no unwinding. A sampler started by the child is orphaned by exactly the outcome most worth recording. Measured: three leaked nvidia-smi processes across three failed runs, before this was fixed. A crash is now a measurement, with its trace preserved.

PM_LLM_EMBED_DEVICE, defaulting to cpu

embed_tokens and lm_head are 545 M parameters each over a 152,064-token vocabulary, and bitsandbytes quantises neither — an nn.Embedding is not an nn.Linear, and an untied lm_head is what get_keys_to_not_convert returns. That is 2080 MiB of unquantised 16-bit on an 8 GB card.

Moving the embedding to the host frees 1039 MiB in torch's books and returns 130 to the driver. The rest is a hole inside a partially-used segment that empty_cache() cannot return, because expandable_segments is a no-op on Windows. Reading there says "not worth it" — and that is the wrong place to read. The 909 MiB stays as reusable arena, so generation allocates inside it instead of asking the driver for new segments:

	free after generation	generation
embed=cuda	119 / 97 / 117 MiB	25.6 / 17.8 / 21.6 s
embed=cpu	770 / 692 / 678 MiB	14.4 / 13.2 / 13.3 s

~600 MiB more headroom and generation 37% faster. One sha256 over all six runs, so the placement is output-neutral rather than a quality trade. ⚠️ It does not move the load peak and cannot lower the gate.

Two traps the placement sets, both handled

• Tied embeddings share one storage, so moving the input embedding drags lm_head to the CPU silently. Every Qwen2.5 below 7B ties, and config's documented fallback is "a 3B model" — a reachable path, so it refuses rather than degrades. Verified firing.
• model.device reports the first parameter's device, which is now the host. Anything routing inputs through it sends them to the CPU, cache_position and position_ids inherit that device, and the rotary embedding dies on mat2 is on cpu. The load succeeded and the first forward failed. Generator.device carries the compute device, read before the move makes the question ambiguous.

Verification

reports/tool_vram_probe.json carries both arms. It holds MiB figures, timings, provenance and hashes only — no prose, since generated text quotes per-patient MIMIC values and reports/ is tracked.

Downstream in pulsemind_demo (UniverseScripts/pulsemind_demo#, branch feat/live-model-integration): check_node 18/18, check_conventions 16/16, check_service 35/35, check_llm 10/10, floor_margin clean, secrets_gate passed.

⚠️ 6420 is the lowest-that-loads setting, chosen deliberately. It sits below the 6561 that segfaulted once, so it will permit a configuration that has failed. Warm the explainer before a demo, when a crash costs a restart rather than an audience.

🤖 Generated with Claude Code