M3 Ultra benchmark re-run + bring-up kit Phase A + Qwen3.8-Flash-Next W2 installer - #24
Merged
Merged
Conversation
…ethods Second full session on the 256 GB M3 Ultra at d5552a0, appended to the existing record: Qwen 3.6 decode +63-85% on the 96-slot auto rung, Inkling streamed prefill ~11x faster (marginal ~0.0105 s/tok), DSV4 decode +5-10% with prefetch default, Gemma 4 long prefill halved. First big-host Qwen 3.8 numbers: 38-39 tok/s plain greedy, MTP no-gain finding on bandwidth-rich hosts, paged KV + SSD needle exact and byte-identical at ~2% cost. Inkling !!!! corruption defect annotated resolved (536266d) and verified fixed on this host. Install table, HF CDN 429 throttling guidance, CLT test.sh flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ossier Workstream 1.1: docs/FAMILY_CONTRACT.md enumerates all 75 ArchConfig axes (48 top-level + 27 nested), generated-checked by FamilyContractDocTests so the doc cannot rot (verified the check bites, then full suite green: 1096 tests / 176 suites). Workstream 3.3/3.4: ./bringup-check.sh runs preflight, family toy suites, install verify, a 16/32/auto ladder smoke with a byte-identity gate, and the protocol scaffold — verdict PASS on qwen36 end-to-end with zero skips. docs/families/TEMPLATE.md is the fill-in per-family page. Workstream 4 rehearsal retargeted to Qwen3.8-Flash-Next (owner decision): Day-0 dossier docs/families/QWEN38_FLASH_NEXT.md pins revision de4b8e4d, closes a 179.5B-param sizing model against the 360.0 GB shard index (±0.3%), maps every tensor pattern to gturbo roles (qwen38flashnext.tensors.json, the first W1.2 mapping-as-data artifact), and prices the port: GDN/MoE/MTP axes covered by existing kernels; three new axes (low-rank hyper- connections, DSA-style attention indexer, 20M-row n-gram PLE table). No faithful MLX conversion exists, so this family forces W2. Rehearsal clock started 2026-08-31T19:42:22Z. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…8flashnext Workstream 2 of the bring-up kit, all four deliverables, targeted at the Qwen3.8-Flash-Next rehearsal family: - StreamingInt4Quantizer: bounded-scratch bf16->int4 g64 transforms built on the extracted Int4AffineEncoder.encodeGroup nucleus, so streaming bit-exactness is structural; W2.1a parity gate across awkward shapes and tile sizes (8 tests). W2.1b (model-level KLD vs the Qwen 3.6 control) is still open and recorded in the quantizer TODO, the dossier, the spec, and every install's manifest.quantizedAtInstall.qualityGate. - SupportedModelSource.Kind.originalRepoQuantize; qwen38flashnext entry pinned to revision de4b8e4d and index sha256 99e81524...c590de (hard gate). - FlashNextPlanner: fused-expert split ([512,1280,2560]/[512,2560,640], VERIFIED against ranged shard-header reads), PLE row-lookup pool (new additive manifest.plePool kind; storage follows the row width — the real table is 128 x [2,500,012 x 160] BF16, 160 % 64 != 0, so rows stay BF16 at ~102.8 GB; 51 rows per 16 KiB block, 0.4% slack), I64 tables carried raw, vision skipped, MTP carried by default (--skip-mtp), MTP's own 512-expert pool as an additive auxiliaryExpertPools entry. - Runtime gate: ManifestReader.familiesWithoutRunner refuses the family at every entry point with a named error listing the missing axes instead of a silent wrong path. No runner code. - Synthetic end-to-end install test (3-shard BF16 snapshot -> temp .gturbo: verify-install green, layout validated, byte-identity against the reference quantizer, sidecars honored both ways). - ArchInfo config keys verified against the real config.json (indexer_n_heads, indexer_kv_heads, split_ngram_parts, ngram_vocab_size_base) — the original guesses would have silently read zeros. - Dry-run against the real repo at the pinned revision: plan builds across 131 shards, 5519 range requests, output 175,106,382,264 bytes (~175.1 GB), 1099 resident entries, 333 multimodal tensors excluded. Full suite green: 1113 tests / 178 suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
359 GB streamed from Qwen/Qwen3.8-Flash-Next @ de4b8e4d and quantized in flight in a single 2h35m attempt with no CDN throttles; verify-install green over 57 files / 175,173,302,167 bytes; dry-run weight accounting matched to 0.04%. Runtime capability gate exercised against the real artifact: named refusal listing the three missing axes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c87209a25
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Stage 0 now checks free disk before any model executes — MIN_FREE_GB (default 5 GiB, run-benchmark.sh convention) on the results volume plus 1 GiB beside the model for the verify receipt. Every non-dry run writes <results>/bringup-report.txt carrying the provenance AGENTS.md requires: commit (and dirty state), hardware and RAM, macOS, Swift, the exact invocation, per-stage exit codes, the model manifest SHA, and deviations (skipped stages); written on FAIL as well as PASS. Re-validated: bash -n, --dry-run listing, and a full real run on qwen36 (PASS, 0 skips, report written). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_exp Implementation contract for the runner, read from the reference implementation (transformers main, modular_qwen4_exp.py + qwen3_5/qwen3_next parents), not inferred from shapes: 4x2560 hyper-connection residual stream (repeat at embed, no final norm — lm_head on the global mixer's output), zero-centered (1+w) RMSNorm baked at load time, exact HC mix/inject math, QSA indexer semantics (raw-key cache, fp32 block pooling, norm-then-rope at block starts, relu-sum scoring, always-selected tail, dense-equivalent within budget), the full PLE n-gram pipeline (splitmix64 multipliers loaded from checkpoint, per-head prime vocabs, EOS-segmented shifts, signed-sqrt gating, dilated depthwise conv), MoE router fp32-softmax->top10->renorm with gate-first fused experts, cache inventory, port plan and gates. Owner-approved scope: full indexer (long context) in v1; MTP deferred; stop-and-report on parity failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PLE row pool Two coordinated tranches toward the qwen38flashnext runner: Reference-parity harness (kit W3.2, first instance): Scripts/parity/ qwen4exp_make_goldens.py builds a deterministic toy Qwen4ExpTextConfig, emits a bf16 sharded toy checkpoint (plus a production-layout variant with the model.language_model prefix and re-fused experts), and captures golden per-module tensors, indexer selection sets, PLE ngram ids, router topk, and greedy rollouts (short=dense-equivalent, long=genuinely sparse) against transformers@4da05482 — 2.5 MiB of committed fixtures under Tests/Mference/Fixtures/qwen4exp/. The harness independently re-derives the splitmix64 multipliers and prime head vocabs and matches the model buffers; byte-identical across regenerations. Resolves the spec's dtype questions: RMSNorm upcasts fp32 with (1+w); the GDN gated norm is ones-initialized (excluded from any bake); q_proj packs query|gate per head (the shipped split_q_gate kernel already matches); a query is not guaranteed to select itself; the ngram mix has exactly 63-bit headroom (exact integer math only). Runtime skeleton: ModelFamily.qwen38flashnext + FlashNextConfig (12 axes, documented in FAMILY_CONTRACT.md), manifest parsing/validation for the new arch axes and plePool/auxiliaryExpertPools/sidecars blocks, resident accessors incl. typed Int64 hash tables, the family-gated load-time (1+w) norm bake (GDN norm excluded; stands down if a future install sets zeroCenteredNormsBakedAtInstall), PleRowPool with a bounded LFU row cache, and 512-expert stream-layout regression coverage. The capability gate is byte-unchanged and re-tested: a baseline does not make the family loadable. Two latent family-neutral trunkPrefix bugs fixed (linear_attn accessors, sharedExpertScalarGate), bit-identical for shipped families. Known blocker for the kernel phase, documented in the dossier: router kernels assert numExperts <= 256 and select k=8; 512/top-10 needs widening. Full suite green: 1155 tests / 182 suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FlashNextReferenceRunner (float32, test tree per repo convention): the full qwen4_exp forward — 4x2560 hyper stream, PLE n-gram embedding with exact int64 hashing from the install's I64 tables, low-rank hyper-connections, GDN via the recurrent reference rule, gated attention with the QSA indexer, fp32-softmax top-10 MoE with sigmoid-gated shared expert, global mixer, no final norm — loaded through the real ManifestReader/Model/PleRowPool path from a .gturbo written with the real index encoder and pool geometry. Parity vs the reference implementation's goldens: PLE ngram ids and row indices EXACT; indexer selected sets EXACT (prefill short+long and all 15 decode steps, both attention layers) including torch.topk's libc++ nth_element tie-breaking at relu-zero boundary ties (a lowest-index-first port measurably fails and cascades to 3.6e-2 on logits — reproduced, fixed, and every tie recorded); router top-k indices EXACT; per-layer tensors and logits within 1e-4 with observed margins 3e-6..3e-5; both 16-token greedy rollouts TOKEN-EXACT; cached decode == re-prefill. Goldens correction: the original fp32 captures were unreachable from the lossy bf16 checkpoint (measured 1.16e-3..3.09e-2 with integer-set flips) — added --weight-dtype bf16 and a second byte-reproducible golden set (Tests/Mference/Fixtures/qwen4exp-bf16/); the install is then a byte-exact BF16 passthrough and the 1e-4 gate is honest. INT4 g64 rung measured, not assumed: 5.1e-2 max-abs on toy logits — the real install's quality gate remains W2.1b (KLD), unchanged and open. Toy production-repack blockers pinned by tests, not worked around: the harness checkpoint's text-only config and layer_types spelling (repaired by a production-shaped config the tests prove ArchInfo accepts), and the structural one — 32-wide toy experts cannot be group-64 quantized (owner decision queued). Router kernels still cap at 256 experts / k=8: the Metal phase's first job. Full suite green: 1170 tests / 185 suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…all gated
Stage 1, router widening (family-neutral): moe.metal selection templated on
K (k8 instantiations keep their exact register footprint; k10 added with a
separate kRouterWideMaxPerLane), MoE.maxRouterExperts 512, prefill router
cap 512. Gates: full suite green with RouterTopKParityTests byte-unchanged;
512/k10 kernels index-exact vs a CPU oracle that itself bit-reproduces the
reference runner's captured router indices/weights on both prompts; par
kernels bit-identical to serial across 10..512-expert widths on
tie-saturated fixtures; live byte-gate on a shipped family:
bringup-check qwen36 verdict PASS, ladder stdouts byte-identical
(28.2/30.7/32.1 tok/s at 16/32/auto slots). Semantics note recorded:
softmax-then-topk-then-renorm equals the shipped topk-then-softmax because
softmax is monotone — no 512-wide prob vector needed.
Stage 2a, grouped RMSNorm + low-rank hyper-connections on Metal:
rmsnorm_bf16w_grouped, FlashNextMatVec (BF16 and INT4 g64 paths — the
parity install is BF16-passthrough, production is INT4), HC mix/inject
kernels. Kernel-only error isolated by pre-rounding inputs: mix 4.5e-4,
inject 1.4e-4, accumulate 2.0e-3 against stated bounds; the HC oracle
bit-reproduces the reference runner's captures on both prompts.
Stage 2b, PLE on Metal + exact hash: FlashNextPleHash (production code,
gated directly against the runner's captured ngram row ids — exact through
prefill and 8 cached decode steps), gather/gate/conv kernels; chunked
prefill vs stepped decode BIT-IDENTICAL at toy and production widths; the
verified update_conv_state contract (retained-length semantics, zero
left-pad, EOS id pre-fill) recorded in the flashnext.metal header.
Also pinned: descendingTopK's k>=n identity-order divergence (unobservable
in the runner, excluded from agreement tests, left unfixed by design — it
is oracle code) and a fixture lesson worth keeping: BF16 scale/bias
quantization collapses INT8 rows whose gains differ by <~0.4% into exact
logit ties.
Open for the next stage, recorded in the dossier: expert COMPUTE is still
8-wide (RoutedBlobs, moe_phase2_down_reduce_k{6,8}) — top-10 needs a k10
reduce or two-pass; per-token dispatch batching for long prefill chunks;
the fp16-stack tolerance question for full-forward token exactness.
Full suite green: 1194 tests / 191 suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…own on toy near-tie Stage A, expert compute at top-10: RoutedBlobs to 10 slots + reduce_k10 (k6/k8 byte-identical; kMaxStreamedExperts stays 8 — it bounds the DSV4 prefill tile, not the array), dense BF16 expert path for the g64-unquantizable toy. INT4 top-10 vs CPU oracle rel 2.8e-4, BF16 5.0e-4, k10-with-2-zero-ranks bit-identical to k8, oracle tie-back bit-exact. bringup-check qwen36 PASS. Stage B, indexer + attention: FP32 end-to-end indexer (selection, not a tensor — relu-zero ties are real, nothing for a tolerance to absorb), GPU scores in the oracle's accumulation order, CPU top-k via the shared descendingTopK, attention gathers the <=2051 selected KV into scratch so the shipped dense kernel runs unchanged. Prefill AND decode selection sets EXACT vs oracle and vs the runner's captures (incl. all 4 boundary ties); gated attention rel 8.4e-4/5.8e-4. Stage C, FlashNextForwardRunner (FP16 activations, FP32 for HC gates / shared scalar gate / indexer; GDN gated norm sigmoid via FC_GDN_GATE_SIGMOID, unset = silu so shipped pipelines are byte-identical; fused GDN decode bit-identical under sigmoid). Integration gates on the BF16 toy: PLE row ids exact everywhere; router/indexer exact through position 16 (short) / 10 (long) then near-tie flips; short rollout 8/8 token-exact, long 7/8 — the one flip at a reference top-2 margin of 1.31e-4 against 6.84e-3 measured drift. ATTRIBUTION (load-bearing): fed the runner's own input+routing, the GPU MoE reproduces the CPU oracle to 7.8e-7 over 354 (position,layer) pairs — the kernels are not the source. The amplifier is the toy: its blocks attenuate ~1000x (block out ~1e-3 vs input ~1.1) with an O(1) Jacobian, so the FP16 input floor surfaces as an output-scale perturbation into a 4e-2 residual stream at ~1%/layer. A trained model does not attenuate its blocks 1000x. GATE NOT LIFTED: familiesWithoutRunner still refuses qwen38flashnext; FlashNextCapabilityGateTests unchanged. Per the port's small-margin-flip rule: report and hold for owner review. Stage D (first light) blocked by the lift, by design. Oracle untouched; no existing family runner changed; every shipped-visible edit additive or defaulted-off; qwen36 ladder byte-gate re-run after the last. Full suite green: 1210 tests / 198 suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FlashNextGoldens.floats and FlashNextParityFixture.floats assembled a
UInt32/UInt16 bit pattern in one fused `UInt32(...) | ... << ...` map
closure. That compiles under this host's CommandLineTools Swift but times
out the type-checker on CI's Xcode toolchain ("unable to type-check this
expression in reasonable time"), failing test (macos-15) and test
(macos-26) since 8df5528. Broken into explicitly-typed intermediate steps;
no behavior change. Toolchain-specific, so CI is the verifier — local CLT
accepted the original.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ys down) Env-gated (MFERENCE_FLASHNEXT_GTURBO) measurement-only entry point that loads the real 175GB install through the internal ungated Model.load door — the same door the parity tie-back tests use — and drives the production FlashNextForwardRunner. Never touches familiesWithoutRunner: the production gate still refuses this family for CLI/server/app, asserted by productionDoorStillRefusesRealInstall. Skips silently without the env var. Carries the standing caveat that greedy token-exactness cannot be checked at 180B scale (no reference rollout), so coherent output is a read of the kernels at scale, not a proof — W2.1b KLD remains the quantitative gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…trieved, 2.4GB RSS Measured 2026-09-01 on the M3 Ultra via the measurement-only door (gate unchanged). Greedy 'capital of France' -> 'Paris. ...Berlin...Rome...Madrid' (correct); 128-token chat -> fluent accurate storm-surge explanation; needle in a 3247-token prompt (> the 2048 indexer budget) retrieved exactly (739215) at 12.3 tok/s decode with no long-context penalty. ~2.39 GB peak RSS for a 180B model (~75x). Confirms the toy near-tie was the toy's ~1000x block attenuation, absent in a trained model. Quality caveat stands: no 180B reference rollout, so W2.1b KLD remains the quantitative gate; the production gate stays down pending it and a maintainer decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related tranches from one testing-and-bring-up session on the 256 GB M3 Ultra:
Benchmark re-run at HEAD — a second full session appended to
docs/BENCHMARKS_M3_ULTRA.md: all six families installed and hash-verified, the frozen protocol on the four protocol-compatible families, family-doc methods for the two think-block families. Headlines: Qwen 3.6 decode +63–85% on the 96-slot auto rung, Inkling streamed prefill ~11× faster (long-case prompt processing ~7 min → ~29 s), DSV4 decode +5–10%, Gemma 4 long prefill halved, and the Inkling!!!!corruption defect annotated as resolved (536266d) and verified fixed on this host. First Qwen 3.8 big-host numbers, including two findings: MTP speculative decode gives no speedup on this bandwidth-rich host, and the community protocol cannot measure think-block families (documented, not worked around).Bring-up kit Phase A, first tranche (spec
docs/superpowers/specs/2026-08-08-family-bringup-kit-design.md):docs/FAMILY_CONTRACT.mdenumerating all 75ArchConfigaxes, generated-checked byFamilyContractDocTestsso it cannot rot;./bringup-check.sh(preflight → toy suites → install verify → 16/32/auto ladder smoke with a byte-identity gate → protocol scaffold; verdict PASS on qwen36 with zero skips);docs/families/TEMPLATE.md.Workstream 2 (repacker independence) for a new
qwen38flashnextfamily — the W4 rehearsal retargeted to Qwen3.8-Flash-Next (released 2026-08-27; no faithful MLX conversion exists, so this family forces W2):StreamingInt4Quantizerbuilt on the extractedInt4AffineEncoder.encodeGroupnucleus (bit-exactness structural, parity-gated);SupportedModelSource.Kind.originalRepoQuantizepinned to revisionde4b8e4dand index SHA-25699e81524…c590de(hard gate);FlashNextPlannerwith the fused-expert split, a new additivemanifest.plePoolrow-lookup pool kind (width-driven storage: the real 320M × 160 BF16 table cannot be group-64 quantized, so rows stay BF16), I64 tables carried raw, vision skipped and MTP carried by--skip-mtpsidecar policy; and a runtime capability gate (ManifestReader.familiesWithoutRunner) that refuses the family at every entry point with a named error listing the three missing axes instead of a silent wrong path. Day-0 dossier:docs/families/QWEN38_FLASH_NEXT.md.Validation
./Scripts/test.sh -Xswiftc -Xfrontend -Xswiftc -disable-cross-import-overlays— full suite green at every commit; final state 1113 tests / 178 suites (the flag is required on CommandLineTools-only hosts, where_Testing_Foundation.framework/Modulesis a dangling symlink; documented in the benchmark doc)../run-benchmark.sh <label> <gturbo> 3per family on a quiet machine (the runner's preflight enforced no concurrent installer/model process); 27/27 measured protocol runs reached a natural end of turn; tables regenerated by./summarize-benchmarks.sh../bringup-check.sh qwen36 scratch/qwen36.gturbo— PASS, 0 stages skipped, ladder outputs byte-identical across 16/32/auto slots.--dry-runagainst the real repo (plan across 131 shards, 5,519 range requests, 175,106,382,264 output bytes); and a real install: 359 GB streamed and quantized in flight in one 2 h 35 m attempt,--verify-installgreen over 57 files / 175,173,302,167 bytes, capability gate exercised against the artifact.Memory and performance
The quantize-in-flight path holds the installer's bounded-memory discipline: transforms run on bounded scratch tiles (unit-tested down to one group per tile), no checkpoint, shard, or large tensor is materialized. PLE rows are packed 51 per 16 KiB page-aligned block (0.4% slack) so one row costs one page fault; per-row page alignment (which would have inflated the ~102 GB table to ~5 TB) was rejected. Performance deltas measured in this PR are documented in
docs/BENCHMARKS_M3_ULTRA.md's re-run section with the protocol and deviations stated.Remaining limitations
W2.1b (model-level KLD vs the mlx-community Qwen 3.6 control) is open — recorded in the
StreamingInt4QuantizerTODO, the dossier, the spec status, and every install'smanifest.quantizedAtInstall.qualityGate. It should run before quantized-at-install weights are trusted for quality claims.The
qwen38flashnextrunner is intentionally not implemented: loading the install fails with the named missing-axes error (hyperConnectionsLowRank,attentionIndexer,pleNgramEmbedding). Gate-vs-up half order within the fused 1280 expert rows is a runner-time question; the byte layout is settled either way.W1.2 mapping-as-data (first data file written; the repacker still hard-codes the same mapping), W1.3 as a general mechanism, W3.1 toy generator, and W3.2 parity harness remain open per the spec status line.
No Flash-Next-specific cancel/resume test (the Maple suite covers transformed-range resume generically); the int4 row-pool branch is currently exercised only by unit tests since the production table is BF16.
Gemma 4's decode medians in the re-run carry wide spread from post-install page-cache churn and are flagged as cache state pending a controlled re-run, not a regression.
The change does not load a complete checkpoint, shard, or large model
tensor into Swift heap memory.
Logs and artifacts contain no credentials, private paths, or model
weights.
🤖 Generated with Claude Code