Read free VRAM from the driver, not from CUDA's bookkeeping - #8
Open
UniverseScripts wants to merge 3 commits into
Open
Read free VRAM from the driver, not from CUDA's bookkeeping#8UniverseScripts wants to merge 3 commits into
UniverseScripts wants to merge 3 commits into
Conversation
`torch.cuda.mem_get_info()` reports what the CUDA runtime believes a new
allocation could obtain. On this WDDM setup that is not what the device has
free, and it errs optimistic. Measured 2026-08-19 with the 7B resident:
nvidia-smi 324 MiB free of 8151
torch 6677 MiB free of 8151
`s19_generate` gates on `vram_free_gb > 5.5`. Torch would have cleared that
gate with a third of a gigabyte actually available, which is the mechanism
behind the load segfaulting after a preflight said there was room.
`vram_status()` asks nvidia-smi, falls back to torch when the driver is not
reachable, and names which one answered so a report cannot quietly carry the
optimistic figure. `capability_check` keeps its `vram_free_gb` / `vram_total_gb`
keys and their units because s19 reads them -- what changed is that they are
now true -- and gains MiB values and the source alongside.
Totals were never in dispute: 8151 MiB is 7.96 GiB is 8.55 decimal GB. The
card is an 8 GB card, and the "8.5 GB" and "8.15 GB" figures in the notes were
that same measurement with its units mangled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZbJV3yRcxhRHBzEBannhP
`edge/__pycache__/edge_server.cpython-312.pyc` was the only bytecode file in the index, against a `.gitignore` that has carried `__pycache__/` since line 32. Ignored patterns do not apply to already-tracked paths, so it survived every sweep and turned any blanket `__pycache__` clean-up into a spurious deletion in `git status`. Its own commit because a tracked build artifact is worth seeing go, not worth finding folded into an unrelated diff. Nothing imports it; `edge/` is the previous serving stack and is historical reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZbJV3yRcxhRHBzEBannhP
`MIN_FREE_VRAM_MIB` was inferred from two data points -- a segfault at 6561 MiB free and a success at 6721 -- and its own comment admitted the value chosen inside that band was "slightly optimistic". Nobody had watched the card DURING a load, so the requirement had never been observed, only guessed from whether the process survived. `tools/vram_probe.py` samples nvidia-smi at 5 Hz across a load and reports `peak = free_before - min(free)`, in the same source and unit the gate compares against. Six loads peaked at 6056-6239 MiB, spread 183. `LLM_MIN_FREE_VRAM_MIB = 6420` (max peak + spread) now lives in config and is read by both s19 and the demo service, which had drifted 1175 MiB apart -- s19 still asserted `vram_free_gb > 5.5`, a level already known to segfault, so it protected nothing. The sampler runs in the PARENT process. A load that does not fit segfaults, so a sampler started by the child is orphaned by exactly the outcome most worth recording; measured, that leaked three nvidia-smi processes across three failed runs. A crash is now a measurement, with its trace. Also adds PM_LLM_EMBED_DEVICE, defaulting to cpu. `embed_tokens` and `lm_head` are 545 M parameters each over a 152,064-token vocabulary and bitsandbytes quantises neither, so 2080 MiB of an 8 GB card is unquantised 16-bit. Moving the embedding to the host after the load frees 1039 MiB in torch's books and returns only 130 to the driver -- the rest is a hole inside a partially-used segment that empty_cache cannot return, because expandable_segments is a no-op on Windows. Reading there says "not worth it" and that is the wrong place to read: the 909 MiB stays as reusable arena, so generation allocates inside it instead of asking the driver for new segments. Three loads each way, embed=cuda finished with 119/97/117 MiB free in 25.6/17.8/21.6 s; embed=cpu with 770/692/678 MiB free in 14.4/13.2/13.3 s. One sha256 over all six runs, so the placement is output-neutral rather than a quality trade. It does NOT move the load peak. Two traps the placement sets, both handled. Tied embeddings share one storage, so moving the input embedding would drag lm_head to the CPU silently -- every Qwen2.5 below 7B ties, and config's documented fallback is "a 3B model", so it refuses rather than degrades. And `model.device` reports the FIRST parameter's device, which is now the host: anything routing inputs through it sends them to the CPU, cache_position follows, and the rotary embedding dies on `mat2 is on cpu`. The load succeeded and the first forward failed. `Generator.device` carries the compute device, read before the move makes the question ambiguous.⚠️ 6420 is the lowest-that-loads setting, chosen deliberately. It sits below the 6561 that segfaulted once, so it will permit a configuration that has failed; warm the explainer before a demo, when a crash costs a restart rather than an audience. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
torch.cuda.mem_get_info()reports what the CUDA runtime believes a new allocation could obtain. On this WDDM setup that is not what the device has free, and it errs optimistic. Measured 2026-08-19 with the 7B resident:s19_generategates onvram_free_gb > 5.5. Torch would have cleared that gate with a third of a gigabyte actually available, which is the mechanism behind the load segfaulting after a preflight said there was room.vram_status()asks nvidia-smi, falls back to torch when the driver is not reachable, and names which one answered so a report cannot quietly carry the optimistic figure.capability_checkkeeps itsvram_free_gb/vram_total_gbkeys and their units because s19 reads them -- what changed is that they are now true -- and gains MiB values and the source alongside.Totals were never in dispute: 8151 MiB is 7.96 GiB is 8.55 decimal GB. The card is an 8 GB card, and the "8.5 GB" and "8.15 GB" figures in the notes were that same measurement with its units mangled.
Claude-Session: https://claude.ai/code/session_01LZbJV3yRcxhRHBzEBannhP