Skip to content

Add GGUF VRAM Calculator and two quantization guides - #47

Open
McFatcop1 wants to merge 1 commit into
msb-msb:mainfrom
McFatcop1:add-gguf-vram-calculator
Open

Add GGUF VRAM Calculator and two quantization guides#47
McFatcop1 wants to merge 1 commit into
msb-msb:mainfrom
McFatcop1:add-gguf-vram-calculator

Conversation

@McFatcop1

Copy link
Copy Markdown

Adding one interactive tool and two guides. Disclosure: these are from my own site, so please treat this as a suggestion rather than a request.

Tools

A free, no-signup VRAM calculator. It differs from the existing entry in two ways that I hope earn it a place alongside rather than duplicating it.

Weights come from measured GGUF file sizes read from the Hugging Face API, not from nominal bits-per-weight. llama.cpp keeps token embeddings and the output tensor at higher precision than the model body, and that overhead scales with vocabulary rather than parameter count, so calculated sizes drift. The 2026 Qwen models carry a 248k vocab. Measuring also makes MoE correct for free, since the file already contains every expert.

The KV cache is counted per layer type. Most 2026 models are no longer uniformly full-attention. Qwen3.5 and later interleave linear-attention layers whose state does not grow with context, and Gemma 4 uses sliding-window layers capped at 1024 tokens. Assuming every layer caches the full context overstates Qwen3.8-27B at 128k context by roughly 4x.

23 models built in, 53 GPUs including AMD, Intel Arc and Apple silicon, a multi-GPU count, and you can paste any GGUF repo path to load it live.

As a sanity check on the method, the effective bits-per-weight it derives comes out at Q8_0 = 8.51 and Q4_K_S = 4.50, which match the published llama.cpp figures.

VRAM Requirements

Two guides that I think fill gaps rather than repeat what is already listed. Happy to drop either if you disagree.

On eligibility

CONTRIBUTING.md asks for locally executable resources, and this is a browser tool, so I understand if that rules it out. I read the existing Tools entry as setting a precedent that a hosted calculator counts, but it is entirely your call and no hard feelings if you close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant