Repository navigation
Proposal: restore single-Mac GLM-5.3-Flash on the M3 Ultra #743
Description
Problem
Python 0.6 serves GLM-5.3-Flash on one 256 GB Mac, and the 0.6.6 recipe reports M3 Ultra measurements. Native 1.x qualifies GLM on one 256 GB M5 Ultra or two M5 Ultras (#286). The M3 Ultra is the only pre-M5 Mac with the memory for it, so its owners lost their supported path in 1.x.
What works with a raised load limit
Most of it. On an M3 Ultra (80-core GPU, 256 GB, macOS 27.0.1) with GLM_LOAD_LIMIT_GB=200, native 1.0.7 (e7c523e) loads and serves TensorFold/GLM-5.3-Flash-MLX-4bit-MTP (d2b17f5) with its MTP head. Everything below was measured that way. On main (0efbd44) the replies have the same hashes, and decode is within a run's spread.
- Drafted replies match
"draft": falseon thetools/zig/glm_ref.py --long 2048prompts (tf-glm-run, depths 0 and 3, 256 tokens). - Accuracy matches Python. With teacher forcing over 2,048 tokens from a float32-activation reference of the same weights, native bf16 predicts the reference's next token 2,006 times. The Python bf16 family predicts it 1,997 times.
- Decode runs about 55 tok/s plain. With MTP drafts,
tools/bench_openai.py --tokens 256 --temperatures 0gives medians of 76.0 and 67.4 tok/s. - A 16,384-row prompt chunk of code takes 31.1 s, about 525 tok/s.
- Load takes 29.5 s. This Mac has run 2.4 hours of agent sessions (38 tasks) at this limit with no watchdog stall. I haven't measured peak footprint yet.
Gaps
- Default
serverefuses on a 256 GB Mac. 601417f (Native engine: GLM-5.3-Flash on one 256 GB Mac - the fixed 70% load limit misses by 1.4 GB #498) keeps the 70% default and leaves the MTP head on disk when only the head is over.serve's default 131,072-token context leaves no room for that: without the head the weights take 176.4 GB, and their caches at that context pass the 179.2 GB limit. The server then refuses, citing the 180.6 GB plan with the head:180.6 GB of weights and 3.3 GB of caches pass this Mac's 179.2 GB load limit (70% of RAM). On main (0efbd44) with--context 65536,serveleaves the head on disk and loads in 28.7 s with one stream, decoding at 56.2 tok/s. The check depends only on the checkpoint, the context and RAM, so I expect the same on a 256 GB M5 Ultra. When--contextisn't given, I'm proposing the server use the largest default context that fits, and name--contextin the refusal. I'm not proposing to change the 70% default. - Pre-M5 kernel checks. GLM doesn't run the fragment-layout check at load, and the check can't detect a transposed operand. core: make the fragment check catch a transposed operand #744 and GLM: run the fragment check at load #745 fix that. Some GLM kernels also launch past the pipeline thread limit before M5, which GLM: fix sparse attention thread limit on M1 #704 and GLM: fix KDA and router thread limits on M1 #708 fix on M1. The GLM: fix sparse attention thread limit on M1 #704 sparse-attention failure also reproduces on an M2 Max, and passes on the M3 Ultra and an M4.
- Prompt chunks vs. 16-row windows. On the M3 Ultra, a chunked prompt's reply diverges from the windowed reply within 25 to 72 tokens on my test prompts, with stock kernels too. 3fc99e1 made them equal on M5. Question: before M5, should these match bit for bit, or follow a documented tie policy?
- Prompt speed. My branch takes the 16,384-row chunk from 31.1 s to 27.3 s, mostly through a simdgroup-matrix
affine_mm, 32-row expert tiles, and an early stop in sparse prompt attention. The kernel changes reduction order, so it will ship with the FP64 referencesCONTRIBUTING.mdrequires, prompt and decode separately. - Checkpoints. Native only loads the uniform 4-bit checkpoint. Support oQ/oQe mixed-precision quants in the Zig engine #716 covers oQ mixed widths.
Proposed gate
On a pinned checkpoint revision with the qualified settings:
- Drafted equals
"draft": false, resumed equals fresh, and concurrent equals solo (token_sha,tools/bench_concurrent.py --alone --serial). - Teacher-forced agreement with the float32 reference is no worse than the Python family's, with every top-token difference and margin recorded.
- Cold prompt receipts at 2k, 8k, 32k, and 64k, plus decode receipts, against 0.6.6 and the previous native release.
- Peak footprint at the agreed load limit, with nothing else loaded.
Plan
I'll work through the gaps in order, starting with the default context. The one decision I need first is item 3. If leaving single-Mac GLM out of 1.x was intentional, let me know before I go further.
Activity
grantoverton commentedon Oct 13, 2026
Hi @harrisonfriia, I'm working on #716 (oQ mixed-precision support) and it overlaps with this in two places, so I wanted to line up before we collide:
affine_mmto read 2-bit weights, and my follow-up GLM speed work adds 3/5/6-bit reads and faster 2-bit expert kernels. That's the same code as your simdgroup-matrixaffine_mmand 32-row expert tiles.Happy to rebase onto your
affine_mmchanges if they land first, or shape mine so the widths drop into your tiles. Whatever's easiest for you. Your #744/#745 fragment checks would help my kernels too.harrisonfriia commentedon Oct 13, 2026
Thanks for the heads up. Go ahead and land yours first - they're already up, and I need #755's tf-affine-ref for my affine_mm changes anyway. Once they merge I'll rebase my simdgroup kernel and expert tiles on top and make them work with your widths.
PHASE 5 GATES COMPLETE (all green, first full pass): (1) PARITY 8/8 d…
ashhart commentedon Oct 14, 2026
Hello @harrisonfriia,
Thank you for waiting on #755 and #716 before the simdgroup kernel. That order is the right one.
Qualified GLM-5.3-Flash is Metal on two M5 Ultras, 4-bit, group 64 (README.md:72, docs/recipes/glm-5.3-flash.md:9). The load limit defaults to 70% of RAM, 179.2 GB at 256 GB (zig/src/families/glm/load_plan.zig:115). A single M3 Ultra is not restored on this main.
What would move a single M3 Ultra is a checkpoint that fits that Mac's 70% limit, or the oQ widths from #716 once they land. Rebase the simdgroup kernel after #755, as you said.
Thank you.
ashhart commentedon Oct 17, 2026
Thank you, @harrisonfriia, for separating memory fit, fragment checks, prompt arithmetic and full-model qualification.
1.0.12 now runs the fragment check at load (
zig/src/families/glm/engine.zig:175), but the default context is still 131,072 tokens (zig/src/native/metal.zig:420), without automatic fitting. Your default-context proposal remains open under the existing 70% limit.Please coordinate the kernel work with #879 and the mixed-width prompt reports #965/#968. Chunk/window equality still needs checking on pre-M5 Macs; we have not granted a tie exception. Changed sums need their FP64 references, followed by full-model draft, resume and concurrency checks. The eight-layer release checks do not settle that qualification or your whole-model memory case.
Posted by Ash's AI Assistant Synthena
heitke commentedon Oct 17, 2026
@harrisonfriia I just wanted to chime in here as I have a M3 Ultra 256gb and have been running this exact model GLM-5.3-Flash-MLX-4bit-MTP for awhile now just fine. (I wrote the PR to add the GLM_LOAD_LIMIT_GB support.) I've set GLM_LOAD_LIMIT_GB=235 and context to 500k. I haven't had a single crash yet. So, I still feel like we should revisit the 70% cap. Reserving over 70gb of ram still seems excessive to me when it seems to run fine. But I might be missing something!
ashhart commentedon Oct 17, 2026
Thanks @heitke! A 235 GB load at 500k context with no crashes is a useful data point, and thank you again for GLM_LOAD_LIMIT_GB. We're keeping 70% as the default: it sits under macOS's default GPU working-set limit (about 75% of memory), so it's safe on every Mac without changing system settings. Raising it stays your choice. We'll document GLM_LOAD_LIMIT_GB together with the
iogpu.wired_limit_mbsetting it needs above that limit, and fit the default context to memory automatically, so a large Mac gets a longer window without hand tuning.Posted by Ash's AI Assistant Synthena
glm: prepare bounded host context and stream fit selection
glm: prepare bounded fit candidate census
glm: prepare checked free-memory counter conversion
ashhart commentedon Oct 21, 2026
Thank you for the report. We are keeping the existing default load limit while finishing automatic default-context fitting. The current release still uses a fixed default GLM context, and single-M3 qualification needs the pinned memory, reference, draft, resume and concurrency checks.
- Synthema, Ash's AI Assistant.
harrisonfriia commentedon Oct 22, 2026
@ashhart I checked chunk/window equality on an M3 Ultra, on main at
d1018b31(1.1.1's engine source), withtf-glm-run: plain greedy, no drafts, 128 tokens, on 2,048-, 8,192- and 32,768-token prompts cut from the GLM family's source files.Replies with the prompt in 16-row windows (
GLM_PROMPT=0) and in chunks first differ at token 0, 45 and 118 on oQ4-MTP, and at 1, 24 and 97 on iQ-MLX-5.1bpw. At each split both runs' top-two margins are 0.625 or less. 16,384- and 2,048-row chunks give the same tokens and margins.On oQ4-MTP the first difference is layer 0's KDA input projection (8-bit/64). Its input is identical in every row, and 6,441 of its 50,987,008 outputs differ: 6,146 by one bf16 step, 161 by two or three, and 134 by more, all of those below 2e-6 in magnitude. Windows run
glm_rows<8>and chunks runaffine_mm's 8-bit matmul, whose fp32 sums take another order. With layer 0's input and output projections forced through the row kernel, the KDA output still differs from row 1 on, so the chunked KDA step (f_bandg_bthroughaffine_mm, then prep, scan and post) differs from the windows' too. I have not split those apart.On that projection, over 2,048 rows of its real input (GPU time, best of three batches):
affine_mm, with its row sumssimd_qmm_bits(tiles; its twin at 1 and 2 rows)glm_rows<8>(16-row slices at 2,048 rows)affine_mmandsimd_qmm_bitseach gave a row the same bits at 1, 2, 4, 8, 16 and 2,048 rows. A prompt kernel onglm_rows<8>'s own arithmetic got to 60.2 ms at best.So running windows and chunks on
simd_qmmandsimd_qmm_bits, as Qwen3.8-27B does before the M5 (faf1295), would keep this projection's prompt speed and make it chunk-invariant, and its windows of 4 or more rows get faster. It changes decode's bits too, so replies would differ from 1.1.1's, and the KDA step's own difference remains. Is that the direction you want for GLM before M5? If so, I would start with the dense projections and the KDA step, with FP64 references where the contract has none, then run the pinned memory, reference, draft, resume and concurrency checks.germpharm commentedon Oct 23, 2026
I run two M3 Ultra 512GB Mac Studios (bare metal, SSH over Tailscale, macOS 27.0.1). If it helps qualify the M3 Ultra path, including 512GB, I can give a maintainer or contributor a free 3-day window to run the qualification suite, as long as the results land in this issue. maccomputerentals.com