Skip to content

docs(gemma4): add the 26B-A4B line to the Gemma 4 page - #25

Open
FeathBow wants to merge 1 commit into
pegainfer-project:mainfrom
FeathBow:feat/gemma4-26b-page
Open

FeathBow wants to merge 1 commit into
pegainfer-project:mainfrom
FeathBow:feat/gemma4-26b-page

Conversation

@FeathBow

Copy link
Copy Markdown
Contributor

Description

Extends /models/gemma4/ to cover the 26B-A4B routed line beside the 12B dense one: launch with the NVFP4 checkpoint, its measured footprint, and a long-context comparison against vLLM at like-for-like KV precision.

  • Keeps one page and one route for the line. 26B-A4B shares the attention stack, the dual paged KV, the scheduling and the four opt-in profiles the page already documents, so it joins them rather than restating them; the sidebar entry and the overview row become Gemma 4.
  • Adds what the routed line adds: 128 routed experts at top-8 beside a dense branch on every MoE layer, served from the published NVFP4 checkpoint with the experts kept packed on the device, the manifest validated before the first upload, and the dense projections left BF16.
  • Adds Performance: the long-context comparison against vLLM 0.23.1rc1 on one 48 GB card, pegainfer e7a41975. The protocol is the 12B one and is now stated once for both lines, with ## Performance splitting into 12B against vLLM and 26B-A4B against vLLM. vLLM's own vllm bench serve client drove both engines. One flag is new: --kv-cache-dtype bfloat16, because this checkpoint's quant config makes vLLM store its KV cache as fp8 by default while pegainfer stores bf16; the page reports both configurations and says which is which.
    • At like-for-like precision, E2EL −2.9% / −13.8% / −24.1% / −39.8% across 10,602 / 40,002 / 81,653 / 163,336 prompt tokens, round-to-round spread at most 3.4%. Peak GPU memory 29,917 MiB against 45,641 MiB, the latter being vLLM's 0.92 utilization reservation.
    • At vLLM's own default (fp8 KV), the second table: +3.4% / +0.4% / −1.6% / −4.3%, with vLLM ahead at the shortest prompt. The text says that the whole difference between the tables is what the fp8 cache buys vLLM's Triton attention at depth.
  • Puts the backend caveats in the shared Caveats section: vLLM's configuration forces its Triton attention path for Gemma 4's mixed head dimensions on both lines, and its Marlin NvFp4 path on a card without native FP4; pegainfer's expert GEMM is a Marlin path too.
  • The figure is an SVG at public/models/gemma4/perf-26b.svg, beside the 12B one and referenced by absolute path per AGENTS.md. It is self-contained (no scripts, no external references, no webfont) and stays sharp at any width.
  • Updates the Limits section: the routed line serves the NVFP4 checkpoint only, both checkpoints are text-only, and 31B stays on the roadmap.
  • Validation: npm run build (13 pages) and the static tests 23/23. Both figures were checked to load and the page checked in both themes at 1280px, with no horizontal overflow at 390 / 700 / 810 / 1024 / 1440 / 1920px.

Signed-off-by: Feathbow <feathbow@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant