Skip to content

docs: measure the ModelOpt checkpoint on one GB10 - #1042

Open
akol1 wants to merge 2 commits into
ashhart:mainfrom
akol1:docs/flash-next-modelopt-gb10
Open

akol1 wants to merge 2 commits into
ashhart:mainfrom
akol1:docs/flash-next-modelopt-gb10

Conversation

@akol1

@akol1 akol1 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Adds the ModelOpt nvidia/Qwen3.8-Flash-Next-FP4-Local checkpoint to docs/recipes/flash-next-cuda.md. The recipe's
measurements covered only the MLX affine checkpoint; this is one new "### The ModelOpt checkpoint on one GB10"
section with the decode and concurrency table, the cold first-token timings, the served exactness checks and the
memory seams.

The commit is unchanged from #952 (b6fd0d92, authored by @balancedscorpion). The 1.1.0 merge 53abe455 took the
earlier head d71a2361, so this docs commit and the experts-rows test fix were left out. The test fix is going in as
its own test-only PR so this stays one change. Documentation only: no source, tests, fixtures or generated files
change.

Review follow-up

The concurrent-chat row is dropped: the #952 write-up records the 1 / 4 / 8-stream aggregate for code only, so there is no receipt for the chat numbers. The section now dates the measurements as @balancedscorpion's, on beec1aad6, the #952 branch build before 1.1.0, not remeasured on 1.1.2. Diff: c44892e5f..1613ff9d7, documentation only.

Receipt

  • Environment: b6fd0d92 on top of 76aafcf27; documentation only. The measurements are the commit author's:
    GB10 (128 GB, driver 580.178, CUDA 13.3), beec1aad6 built unpatched for sm_121, checkpoint
    nvidia/Qwen3.8-Flash-Next-FP4-Local at b9ce18a6, --prompt-cache-gib 0, with the recipe's bench_openai,
    bench_concurrent and bench_cache. They are quoted as measured; nothing was generated here.
  • Exactness, decode speed, prompt speed: n/a for the diff, documentation only. The added section records the
    author's receipts (each concurrent reply equals its solo run, drafted equals "draft": false, cancelled/restarted
    equal fresh, cold ttft_s, peak host and device memory).
  • Tests run: none - documentation only, which this template exempts ("A change to documentation alone needs no
    receipt"). git diff --check is clean. The lean checker does not read docs/.
  • Not run, and why: a GPU re-measurement. No NVIDIA host here, and the section reports the author's run rather than
    a new claim. Host and CUDA CI on the PR covers the tree.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (docs only)
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (docs only)
  • No precision traded for speed. If the bits change, the description says which sums change and why. (docs only)
  • A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (no kernel or backend change)
  • Prompt processing is no slower than the last release. (docs only)
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (the added section names the author's GB10 CUDA host; no platform was run for this diff)
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. (no new tests)
  • Comments and docstrings are one line. No measurements or history in the source. (recipe prose only; no source comments change)
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

The recipe's measurements covered only the MLX affine checkpoint. Add the ModelOpt one, measured with
beec1aa built unpatched for sm_121: decode and concurrency from bench_openai and bench_concurrent, cold
first tokens from bench_cache with exact token arrays, the served exactness checks, the seams at fixed
histories, and what the 128 GB machine admits.
@akol1

akol1 commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

The zig-windows failure here is inherited from 76aafcf27, not this docs change. On current main that job fails 15 build steps in main's own files (zig/src/server/speech.zig:279 and the host-test suite); #1034 ("Windows: main's host tests build again on Windows hosts") fixes exactly that and has zig-windows green on the same base, with speech.zig and the other affected files in its diff.

desktop, zig, fmt and mobile are green here. This branch is one commit and touches only docs/recipes/flash-next-cuda.md, so nothing in it can change the Windows build. If you want the job green before merging, rebase this after #1034 lands; otherwise the failure is the base's, and it will disappear for every PR based on 76aafcf27 once #1034 is in.

@ashhart

ashhart commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Thanks @akol1 for carrying @balancedscorpion's measurements with their authorship. The supplied #952 write-up supports the code rates, cold prompts, memory figures and equality checks; I couldn't match the concurrent-chat rates to a receipt there, so please attach that receipt or omit that row. We'll add: “These are @balancedscorpion's measurements on beec1aad6, the #952 branch build before 1.1.0, and have not been remeasured on 1.1.2.” That also dates the three-stream drafting limit. This is for 1.1.2 after the receipt check and our gates.

Posted by Ash's AI Assistant Synthena

@ashhart ashhart mentioned this pull request Oct 19, 2026
…ncy row

The ashhart#952 write-up records the concurrent aggregate for code at 1 / 4 / 8 streams,
not for chat, so the chat row is dropped rather than carried without a receipt.
The section now says whose measurements these are, which build they ran, and that
1.1.2 has not remeasured them.
@akol1

akol1 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Row dropped and the provenance sentence added at 1613ff9d7 (documentation only, docs/recipes/flash-next-cuda.md).

The #952 write-up's concurrency receipts are bench_concurrent.py --alone --serial --levels 1,4,8 for code — 64.1-64.4 / 44.9-45.2 / 66.2-66.8 tok/s — plus the 78/78 concurrent-equals-solo and drafted-equals-"draft": false checks. It has no 1 / 4 / 8-stream aggregate for chat, so that row had nothing to cite and is gone; the solo chat number (40.7-41.1) stays in the one-stream row.

The section now reads: "These are @balancedscorpion's measurements on beec1aad6, the #952 branch build before 1.1.0, and have not been remeasured on 1.1.2." That also dates the three-stream drafting limit.

Documentation only, so no test receipt; git diff --check is clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants