Add GGUF VRAM Calculator and two quantization guides - #47
Open
McFatcop1 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding one interactive tool and two guides. Disclosure: these are from my own site, so please treat this as a suggestion rather than a request.
Tools
A free, no-signup VRAM calculator. It differs from the existing entry in two ways that I hope earn it a place alongside rather than duplicating it.
Weights come from measured GGUF file sizes read from the Hugging Face API, not from nominal bits-per-weight. llama.cpp keeps token embeddings and the output tensor at higher precision than the model body, and that overhead scales with vocabulary rather than parameter count, so calculated sizes drift. The 2026 Qwen models carry a 248k vocab. Measuring also makes MoE correct for free, since the file already contains every expert.
The KV cache is counted per layer type. Most 2026 models are no longer uniformly full-attention. Qwen3.5 and later interleave linear-attention layers whose state does not grow with context, and Gemma 4 uses sliding-window layers capped at 1024 tokens. Assuming every layer caches the full context overstates Qwen3.8-27B at 128k context by roughly 4x.
23 models built in, 53 GPUs including AMD, Intel Arc and Apple silicon, a multi-GPU count, and you can paste any GGUF repo path to load it live.
As a sanity check on the method, the effective bits-per-weight it derives comes out at Q8_0 = 8.51 and Q4_K_S = 4.50, which match the published llama.cpp figures.
VRAM Requirements
Two guides that I think fill gaps rather than repeat what is already listed. Happy to drop either if you disagree.
On eligibility
CONTRIBUTING.md asks for locally executable resources, and this is a browser tool, so I understand if that rules it out. I read the existing Tools entry as setting a precedent that a hosted calculator counts, but it is entirely your call and no hard feelings if you close this.