NVMAI: Qwen 3.6 35B-A3B port with measured decode optimizations - #105
Open
Pummelchen wants to merge 503 commits into
Open
NVMAI: Qwen 3.6 35B-A3B port with measured decode optimizations#105Pummelchen wants to merge 503 commits into
Pummelchen wants to merge 503 commits into
Conversation
Author
|
Estimated tok/s increase over the original repo: ~+55-65% (compounded decode-rate wins). One important note on the baseline: the original repo currently runs Gemma 4 26B-A4B — the Qwen 3.6 port is upstream PR #29, not yet merged there — so a direct head-to-head requires the original to merge the Qwen integration first. This estimate compares against the fork's own pre-optimization Qwen state on the same model and hardware, not against the original's Gemma runtime. |
Two measurements, both against a purged page cache (the user ran sudo purge; it
cannot be driven from here).
Bounded footprint, with both arms matched on slot count and context size -- which
the first attempt was not:
32 slots, default context wall (192 tok) RSS
cached 26.43 / 23.20 / 22.82 1.82 GB
bounded (F_NOCACHE) 29.05 / 29.06 / 29.18 3.74 GB
So ~20%, not the 2.3x an earlier unmatched reading suggested; that gap was mostly
context size rather than cache policy. The RSS inversion is the point rather than an
anomaly: bounded is higher because F_NOCACHE forces every expert into our own slots
where it is counted, while cached leans on the unified buffer cache which never
appears in process RSS. Bounded's 3.74 GB is the true machine cost; cached's 1.82 GB
is 1.82 GB plus whatever the OS holds. 20% for a footprint that is actually bounded
is a good trade, so NVMAI_BOUNDED_IO should be the v4.0 default.
Found while matching those arms, and unrelated to streaming: the default
--max-context costs 1.6x decode throughput.
max-context wall (192 tok) RSS
8192 15.42 / 13.80 s 3.68 GB
32768 15.72 / 14.19 s 2.67 GB
262144 25.92 / 22.44 s 1.84 GB
Same 32 slots, same 25-token prompt, same 192 generated tokens. KV strides are sized
by max-context rather than by the live sequence, so attention walks a buffer two
orders of magnitude larger than the data in it and loses all locality. RSS falling as
context grows fits: more of the reservation is never touched.
That is a v3.x defect rather than a v4.0 design question, and it is worth more than
most of this document -- every user on the default pays 1.6x for context they are not
using. Likely backportable.
…ult context 18 configs: 4-bit and 8-bit x RAM budgets 1/2/4/8/16 GB x bounded/cached x prompts of 25/452/3532 tokens, all at the default --max-context 262144 since that is a product requirement rather than a tunable. The headline reverses an earlier curve in this document. More slot RAM is slower: 4-bit at a 1 GB budget beats 4 GB by ~35-40% on short and medium prompts in both cache modes (13.61 vs 9.85 tok/s cached, 11.10 vs 8.36 bounded). The earlier curve that showed the opposite used --max-context 8192; at the 262144 default the KV reservation is already large enough that adding slot memory pushes the machine into pressure. So the shipped default of 64 slots is wrong twice over -- 16 slots is ~35% faster and uses a quarter of the RAM. One-line change, and the most valuable finding here. Cached beats bounded by 15-30% consistently across every budget and prompt size, firming up the ~20% measured earlier. 8-bit costs 2-2.5x against 4-bit at a comparable budget (5.64 vs 13.61 tok/s at 1 GB, short prompt) but stays usable when streamed at a small budget, which is the point. 8-bit at 16 GB collapses to 0.46-1.22 tok/s: 15.94 GB of slots plus a 262144-token KV reservation does not fit 24 GB. Large budgets are a trap. Prefill is insensitive to the budget -- 46.8-47.9s at 4-bit and 60.6-62.6s at 8-bit for 3532 tokens whatever the slot count -- because a wide chunk touches nearly every expert regardless of cache size, and prefill is GPU-bound at 97.4% occupancy. 4-bit cannot reach a 16 GB budget: 128 slots is the allowed maximum, which is 8.44 GB.
…s at 4-bit, 8 at 8-bit) The shipped default of 64 slots was slower *and* four times larger than the optimum. Benchmarked across 18 configs at the shipped --max-context 262144, 4-bit reached 13.61 tok/s at 16 slots against 9.85 at 64 and 8.78 at 128, and the ordering held on long prompts and in both cache modes. A 262144-token KV reservation is already large, so slot memory past ~1 GiB pushes the machine into pressure and costs more than the extra hit rate returns. Rather than hard-code a number per quantisation, the count is now derived from the model's own expert stride against RuntimeConfiguration.defaultExpertCacheBudgetBytes (1 GiB) and snapped to the supported ladder. That lands on 16 slots at a 1.688 MiB stride and 8 at 3.188 MiB -- the measured optimum for 4-bit and 8-bit respectively -- from one rule, and it will keep working for any future stride. The server reads the manifest before Model.load to get the stride, since the slot count has to be chosen before streaming is configured. Precedence is unchanged: --expert-cache-slots, then NVMAI_EXPERT_CACHE_SLOTS, then the derived value. The ready banner now reports expert_slots so the streaming budget is visible rather than something the user has to infer from a flag they did not pass. Omitted on the lazy-load path, where nothing is known until the first request. Tests pin the rule and its edges: both measured optima, that every result is a supported count across four budgets and three strides, that a larger budget never shrinks the cache, that a degenerate manifest falls back to the smallest supported count rather than an empty cache, and that the chosen count stays within 15% of the requested budget -- otherwise the footprint promise is empty. An earlier sweep in this repo suggested the opposite ordering, 128 beating 8, but it ran at --max-context 8192 where the KV reservation leaves far more headroom. The constant carries that warning so it is not re-tuned against a reduced context. 701 tests; lint clean; 4-bit golden baseline unchanged.
Fills the sub-1 GiB gap for 4-bit bounded:
slots RAM short medium long
8 0.53 GB 4.49 4.33 3.69
16 1.05 GB 11.10 10.78 7.16
32 2.11 GB 9.32 10.09 6.42
64 4.22 GB 8.36 9.06 4.52
128 8.44 GB 9.13 13.49 5.34
8 slots is 2.5x worse than 16 and prefill degrades with it (49.2 s against 46.8 s).
The mechanism is exact: with 8 slots and topK=8 the cache holds precisely one
layer's active set, so every layer evicts the previous one, and at the measured 38%
token-to-token expert reuse the hit rate goes to nearly zero.
That matters for the default committed in b15f01c: 16 slots sits on a genuine peak,
collapsing below and degrading above, rather than being the smallest value that
happened to be tested.
Also records that 8 slots is a hard floor rather than a convention. One slot holds
one expert for one layer -- 67.5 MiB at 4-bit, 127.5 MiB at 8-bit across 40 layers
-- and below topK=8 slots executeExpertCachePlan trips
precondition(plan.experts.count <= slotCount), crashing rather than degrading. So
540 MiB and 1020 MiB are the minimum achievable footprints without restructuring the
MoE plan, and KiB-scale budgets are three orders of magnitude below a single slot.
Adds a scope check: v4.0 is currently 606 insertions and 6 deletions across 10 files
against 43,512 source lines, so 98.6% of v3.8 is unchanged. The C reader, the
bounded-IO path and the derived defaults are real; the clean-sheet engine is still a
plan.
… v4.0 rewrite README: replaces the 3.2-build quant tables and the partial 72-combo harness table with one compact 4-bit/8-bit table at the shipped defaults. 214 lines to 166. The new table reports prefill and decode for 25/452/3532-token prompts at the derived expert-cache budget (16 slots at 4-bit, 8 at 8-bit, ~1 GB either way), and states the finding that governs the defaults: larger caches are slower at the shipped 262144 context, 13.61 tok/s at 1 GB against 9.85 at 4 GB. Also corrects a claim the new defaults invalidated. The 3.8 callout reported 6-bit and 8-bit at 6.7 and 1.6 tok/s; those were measured at 64 slots and no longer hold -- 8-bit reaches 5.64 tok/s at its 8-slot default. The callout now says so rather than leaving a wrong number in place. 6-bit remains listed as a supported quantization because 3.8 ships it; it is simply no longer benchmarked. Design doc: adds a 'must survive the rewrite' section. The document described a decode loop, a streamer and kernels, and mentioned none of the five shipped features -- read as a specification it would have produced a faster engine missing the prompt cache, concise mode, the -fast alias and both unload paths. Each is listed with the interaction that makes it the streamer's problem: an S12 prefix hit skips prefill entirely, so the prefetcher cannot assume a prefill has warmed the cache; concise mode moves requests between prompt-size regimes worth 2x in decode rate; CLIStrip exists to cut prompt length and must stay in front of prefill with its version stamped; the idle timer discards slots so a reload pays a cold cache, which the 8-slot measurement prices at 4.49 against 11.10 tok/s; and the API unload can arrive mid-flight, so slots, descriptors and queued prefetch must tear down without a read landing in freed memory. Verified all five present in v3.8 before writing this: /v1/models returns the -fast alias, POST /v1/models/unload returns 200, and the flags are all in --help. Nothing was dropped by the v4.0 work, which is additive.
… 1.6x default-context penalty)
The shipped --max-context 262144 was costing ~1.6x decode throughput on every
request, including 25-token prompts that use none of it. Not the strides, which are
a constant 1024 B/token: the allocation. Linear layers were sized for maxContext, so
262144 tokens is 512 MiB per layer and 20.00 GiB across 40 against 0.62 GiB at
8192. The buffers are storageModeShared and lazily touched, so this barely shows in
RSS -- but on a 24 GB machine the mappings alone cost the throughput.
Linear layers now start at 8192 tokens and double on demand, clamped to maxContext.
Ring-backed SWA layers are untouched; their capacity is the window and is
deliberate. reserve() is called before prefill writes a chunk and before each decode
step, so growth always precedes the write. Buffers are storageModeShared, so
carrying existing KV forward is a memcpy, and callers fetch buffers through the
accessors at use time rather than caching them, which is what makes swapping safe.
Measured, same 25-token prompt at the shipped default:
before ctx 8192 13.80 s wall ctx 262144 22.44 s wall 1.63x gap
after ctx 8192 12.319 tok/s ctx 262144 12.551 tok/s 1.02x -- gone
RSS is 2.43 vs 2.46 GB across the two contexts, confirming the allocation no longer
tracks the advertised limit. A conversation that does reach 262144 pays five copies
in total rather than one reservation up front.
Tests pin the part that could fail silently: that a large maxContext is not
allocated up front, that a small one is not rounded up, that growth doubles and
clamps at the ceiling, that reserving below current capacity neither reallocates nor
discards live KV, that repeated reserves are idempotent, and above all that written
tokens survive growth byte for byte -- losing them would leave a long conversation
generating from corrupted history rather than failing outright.
They locate a full-attention layer rather than assuming layer 0, since Qwen 3.6
interleaves linear-attention layers that hold a page-sized placeholder at capacity
zero.
707 tests; lint clean; 4-bit golden baseline byte-identical.
…icy actually needs Bounded footprint becomes the default (NVMAI_BOUNDED_IO=0 opts out): expert reads bypass the unified buffer cache, so the slot budget is the machine's true footprint rather than a number the OS quietly supplements. Making it the default exposed a coupling the earlier tuning missed. The 16-slot default derived in b15f01c was measured under the page-cache policy, where the OS holds the routing working set and slot memory is redundant pressure -- 4-bit reached 13.61 tok/s at 16 slots against 8.78 at 128. Under bounded reads there is no second cache, so the slot cache must hold the working set itself, and the ordering inverts: slots RAM bounded tok/s io ms 16 1.05 GB 8.73 49.4 32 2.11 GB 8.94 41.3 64 4.22 GB 9.91 28.3 128 8.44 GB 18.91 7.2 The routing trace predicted this: 131 distinct experts per layer over a 128-token window, and 128 slots is the first budget that holds it. Below that nearly every expert is fetched per token, which is why the curve is a cliff rather than a slope -- a smaller budget costs 2.2x throughput to save RAM the OS would have to hold anyway. So the derived budget goes from 1 GiB to 8 GiB, giving 128 slots at 4-bit (8.44 GiB). 8-bit derives 64 slots (7.97 GiB) rather than 128, because 128 would reserve 15.94 GiB and measured 1.22 tok/s thrashing against 5.21 at 64. Live at the new defaults: 4-bit 18.111 tok/s, 8-bit 5.748. Pairing bounded with the old 16-slot default would have shipped users the worst cell in the matrix, so this is one change rather than two. Tests updated to the new expectations, plus one asserting the 8-bit default stays below the thrashing point. The constant carries the warning that it is only correct while reads bypass the page cache, and must be re-tuned at the shipped max-context rather than a reduced one. 708 tests; lint clean; 4-bit golden baseline byte-identical.
Completes the 'RAM is the input' model. --ram-budget takes a size (8G, 2G, 512M, or a plain byte count) and the slot count is derived from it and the model's own expert stride, rather than the user picking slots and discovering the memory afterwards. Verified end to end: 1G -> 16 slots, 4G -> 64, 8G -> 128, and --expert-cache-slots 16 still overrides an 8G budget. Precedence is unchanged -- explicit slots, then NVMAI_EXPERT_CACHE_SLOTS, then the budget, then the 8G default. The parser accepts G/M/K with optional iB and B suffixes, case-insensitively, plus fractional sizes and bare byte counts. It rejects empty strings, bare units, negatives, zero and trailing garbage, because a silently misparsed size would build a tiny cache and cost 2.2x throughput with no error to explain it -- the help text now says as much, including that smaller budgets are markedly slower since expert reads bypass the page cache and have nothing to fall back on. Tests cover the units users type, fractional sizes, the ten malformed inputs worth rejecting, and that the knob moves the slot count monotonically through the measured 1G/4G/8G points. 712 tests; lint clean; 4-bit golden baseline byte-identical.
…d docs
6-bit is removed rather than deprecated. Measured on the ANE its non-power-of-two
packing reached 46.8 GB/s against 60 for both 4-bit and 8-bit, and only 0.12 ms
faster than 8-bit despite storing 25% fewer bytes -- the signature of a padded
packing. On a 24 GiB machine the 26 GB model also does not fit, measuring 6.7 tok/s
against 4-bit's 18.8. It was costing users twice.
Load boundary: ManifestReader.supportedWeightBits is now [4, 8], and a 6-bit slot
throws with an explanation and a route forward rather than a generic rejection. It
raises unsupportedArchitecture, not indexCorrupt -- the payload is intact and telling
someone their file is corrupt would send them re-downloading 26 GB for nothing:
error: unsupported architecture: 6-bit models are no longer supported
(embedding is 6-bit). Its packing is not a power of two, which measured
46.8 GB/s against 60 for both 4-bit and 8-bit, and it does not fit 24 GB.
Install the 4-bit or 8-bit build instead.
Kernels: ten [4, 6, 8] preconditions tightened to [4, 8], so the code and the
product agree rather than leaving a path the loader rejects.
Installer: the qwen36-6bit source is gone, along with its --model option. Its index
fingerprint is now asserted to resolve to nil, so a source quietly reappearing fails
a test instead of shipping an unsupported quantization.
Launchers: the 6-bit menu entry, its argument forms, its port 8082 mapping and its
model directory are removed from both cli_launcher.sh and server_launcher.sh; the
unknown-quantization message now reads (4|8).
Tests: seven files had parameterisations over 6-bit widths; those now cover [4, 8]
or [8]. The repack planner's sixBitPackedShapesUseScaleGrid test is deleted, since
the behaviour it covered no longer exists.
Docs: README no longer advertises 6-bit, and the port list and launcher example are
updated.
711 tests; lint clean; 4-bit golden baseline byte-identical; a 6-bit model now
refuses to load with the message above.
…y done
Item 6 of the v4.0 plan resolves to nothing to build, and the reasoning is
structural rather than a measurement that came out flat.
Predictive prefetch was to use the previous token's routing for a layer, available
40 layers early, on the strength of 38% measured token-to-token expert reuse. But a
token touches only 8 experts per layer, so it can never evict a cache of 16 or more:
an expert used at layer L in token N-1 is still resident at token N. The predictable
set and the miss set are disjoint by construction, so prediction can only fetch what
is already there.
Replayed against the real 383-token routing trace with a per-layer LRU:
16 slots 52.1% miss rate 0.00% catchable by previous-token prediction
128 slots 10.3% miss rate 0.00%
Zero at both budgets, as the argument requires. The 38% reuse is real but already
fully exploited by the cache; the misses are precisely the part no previous-token
signal describes.
Queue depth is already present within a layer -- executeExpertCachePlan batches every
miss into one reader call, which four threads service in parallel, worth 1.4-1.6x at
4-8 misses over serial pread. Across layers it is impossible, since layer L+1's
experts are unknown until layer L's router runs, and the only work available to
overlap the fetch is the shared MLP, which is already committed before the fetch is
issued.
Also records the miss rates for future reference: they confirm the 128-slot default
from an independent direction (10.3% miss against the ~92% hit rate measured live)
and explain the 16-slot cliff (52.1% miss).
No production code changed.
…h building
Item 7 answered with two prototypes rather than argument.
The expert gather is expressible: mb.gather over all 256 experts stacked as one
tensor converts and runs on CPU_AND_NE. Two toolchain notes recorded because each
cost an hour -- gather needs opset_version iOS17 or the compiler demands a
validate_indices parameter the MIL builder refuses to emit, and
scaled_dot_product_attention needs iOS18.
So expressibility was never the blocker; residency is. A Core ML graph holds its
weights with no streaming, and 256 experts across 40 layers is ~16 GB palettised to
4 bits, all resident. On 24 GB that is the configuration measured at 0.2 GB/s of
thrash, so a full-model Core ML prefill contradicts the streaming architecture the
project exists for.
Attention alone needs no expert residency, and it is where the win is. A real
Qwen 3.6 block in MIL -- RMSNorm, packed q+gate/k/v (2048->9216), GQA 2->16 heads,
SDPA, output projection (4096->2048) -- 4-bit palettised, marginal cost by slope:
width 256 GPU ~28.9 ms/block ANE 1.50 ms 19.2x
width 1024 GPU ~138.8 ms ANE 8.99 ms 15.4x
The GPU column derives from the measured 742 ms/block at 3532 tokens, split equally
between projections and SDPA (~205 GFLOP each at that width) and rescaled linearly
and quadratically. The ANE figure works out to ~8.5 TFLOP/s, matching the
8-17 TFLOP/s measured on isolated shapes, so two independent measurements agree.
Why this and not the decode hybrid: decode alternating ANE and GPU costs 40 handoffs
per token, which is more than the work moved. Prefill alternates 40 times per chunk,
and the chunk is 4096 tokens, so the same overhead spreads across four thousand
tokens. The objection does not transfer.
Worth: attention is 29,687 ms of a 52,350 ms prefill. Discounting 15x to 10x for the
prototype's omissions, a 3532-token prefill goes 52.4 s -> ~25.6 s, a 2.0x
improvement on the largest user-visible cost in NVMAI.
Unsolved and recorded: the cross-framework KV handoff, a second weight artifact with
installer and receipt work, the prototype's missing RoPE/output gate/KV write, and
whether the +89% GPU-busy measured under concurrent ANE load applies when the two
alternate rather than overlap.
Leads with the four shipped changes: on-demand KV growth (default-context penalty 1.63x -> 1.02x), bounded expert reads by default, --ram-budget as the knob with per-quantization derived defaults, and the withdrawal of 6-bit. The benchmark table is left in place but explicitly marked as 3.8 figures, taken under the page-cache policy and a 1 GB expert budget -- both of which 3.9 changes. The development machine is not currently quiet enough to re-measure honestly (over 200% of CPU going to other applications and swap nearly exhausted, with absolute throughput swinging ~2x on machine state), so republishing those numbers as 3.9's would be inventing them. The relative figure quoted in the callout is a same-conditions before/after and does not depend on machine state, which is why it is the one number the notes lead with.
Announce that NVMAI will work on support for Ornith-1.5-35B-A3B as its new default AI model, replacing Qwen 3.6 35B-A3B. Add the LLM Performance Evaluation image (assets/stats.png) with links to the Ornith-1.5 blog and the HuggingFace model card.
Updated image source and corrected roadmap title.
A CPU sample of the server during AgentWorld decode put Swift String indexing among the hottest frames. The callers were today's flags -- NVMAI_KERNEL_SPLIT (checked at every GDN kernel through rotate), NVMAI_ABLATE, NVMAI_HC_FUSED, NVMAI_QSA_GPU_SELECT, NVMAI_ROUTER_TOPK_SIMD, NVMAI_ATTN_SIMD_PARTIAL -- each reading ProcessInfo.processInfo.environment per call, which rebuilds a dictionary from environ: ~800 rebuilds a token. Hidden inside Qwen3.8's 160 ms token, exposed on the 35B family's 65 ms one. All are statics now, as the pre-existing flags already were; the streamer's per-miss-batch NVMAI_PARALLEL_IO read joins them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Story prompt, fixed binary: AgentWorld 4-bit 18.57 tok/s (median of 18.26 / 18.57 / 18.75), 8-bit 9.46 (9.81 / 9.10); the Qwen 3.6 control 22.30 against 6.67 on the 5.0 binary; Qwen3.8 5.05, unchanged within noise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The family that regressed in 5.0 had no golden on this machine. qwen36-4 and qwen36-8 join the target table and the release gate; the 4-bit baseline is captured on the fixed binary (coherent mutex answer, greedy). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The converter takes --model {agentworld, qwen36}, each a pinned repo and
commit, and skips the vision tower and the MTP draft head that ride in
Qwen3.6's index (the text model is 733 tensors in both). install_models.sh
routes qwen36 / qwen36-8bit through it -- both widths from one download,
the same bf16 keeps -- instead of repacking mlx-community's quantization.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
No catalogue entry repacks a third-party quantization any more:
ornith15 / ornith15-8bit prepare_agentworld.py --model ornith15
(ornith-ai/Ornith-1.5-35B-A3B @ 10fbf86)
qwen36 / qwen36-8bit --model qwen36
agentworld / -8bit --model agentworld
qwen38flash / -8bit prepare_qwen38.py, now run rather than printed
qwen36-mtp the converter's new --draft-head mode: the 19
mtp.* tensors from two shards of Qwen's original,
as a qwen3_5_mtp sidecar with prepare_ornith_mtp's
contract (prefix stripped, every weight at the
build width, every norm bf16 with the unit offset
folded, pre_fc_norm_* included)
qwen38flash-mtp prepare_qwen38_mtp.py instead of the RockTalk repack
ornith15-mtp prepare_ornith_mtp.py, with its pinned shard
fetched, instead of a message
The help text says why: third-party group sizes, widths and norm
conventions are theirs; here the router, scalar gate, DeltaNet gating
projections and norms stay bf16 in both widths. NVMAIRepack --model and the
Mac app's descriptors still name the mlx-community repos; the CLI no longer
uses them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both widths built by prepare_agentworld.py --model qwen36 from Qwen's own release; both answer the mutex prompt coherently. These replace the baseline captured on the mlx-community repack earlier today. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Default 8, as every install ships. The head is ~0.5 GB at 8-bit on the 35B family and ~3.5 ms of a ~50 ms token; the option exists so that trade can be measured rather than argued. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…m bf16 Qwen 3.6 4-bit 19.21 tok/s (19.21 / 19.17 / 19.26), 8-bit 9.73 (9.74 / 9.71), story prompt, on the installs the script makes today. The source paragraph now describes every model the same way, and the Ornith rows are footnoted as the 2026-08-30 mlx-community measurement until Ornith is rebuilt. The 5.0.1 notes carry the same and the draft-head pairing result. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Qwen 3.6 4-bit with an 8-bit head 19.2 tok/s, with a 4-bit head 23.0; 8-bit stays the default for quality, --head-bits 4 is available. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A single TLS error on the index fetch (curl 35) ended an Ornith build before its first shard; the index, headers and tokenizer fetches now carry the shard download's retry flags. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pkill sends SIGTERM, which Python does not raise as an exception, so the curl child outlived the converter and kept writing the shard the relaunch resumed -- the same corruption as the orphaned download the exception path already handles. SIGTERM now raises SystemExit into that path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
curl's retries cover ~5 minutes now (20 x 15 s); an outer loop of six attempts a minute apart covers longer, resuming the partial file. A DNS failure (exit 6) ended an Ornith build at shard 6 of 16. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removed detailed benchmark settings and explanations for clarity.
ModelProfile keys the runtime's non-architectural choices -- expert-cache budget, prefetch depth, prefill chunk, sampling defaults, and the kernel switches (simd top-k, simd sparse attention, fused hyper-connection gates, GPU key select) -- by manifest modelID and routed-expert width, resolved family default -> the model's table row -> environment. Eight rows, one per shipped install, each spelled out in full so editing one cannot move another; AgentWorld and Qwen 3.6 share a family and now tune apart. Draft heads and unknown ids fall back to their family. The runner, the CLI and the server resolve through it (budget, prefetch, sampling, chunk); the MoE selector switch is an init parameter instead of a static, and the runner logs the resolved profile under NVMAI_RUNNER_STATS so every benchmark log records what ran. Every experiment flag keeps winning over the table. Values are today's measured ones, so nothing moves: both goldens identical, 543 core tests, lint clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removed the link to full benchmark results from the README.
Both widths built by prepare_agentworld.py --model ornith15 from ornith-ai/Ornith-1.5-35B-A3B (commit 10fbf86) through tools/install_models.sh; both answer the mutex prompt coherently and the server path agrees with the CLI. These replace the baselines captured on the mlx-community repacks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…'s size A partial shard 16 left by an interrupted run was skipped as present and failed the Ornith draft-head conversion; the fetch now compares the file size with Content-Length and resumes anything short. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4-bit 19.24 tok/s (19.24 / 19.27 / 19.05), 8-bit 9.72 (9.72 / 9.72), story prompt, on the installs tools/install_models.sh makes from Ornith's own release. Every row in the table is now a first-party build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removed the period at the end of the alt text for the image.
NVMAIBench: NVMAI_BENCH_MOE_SPECIALIZE=1 builds phase 1 with the runtime's function constants; at the Qwen 3.8 shape the production kernel then reads at 41 GB/s and the best layouts (v2, xsh16) at 44 -- a 7% kernel gain worth under 1% of a token, so phase 1 is closed as a lever. The benchmark harness takes NVMAI_BENCH_RAM_BUDGET for slot-count A/Bs, and 48 joins the slot rungs so the 8-bit builds can sit between 32 and 64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ad router Both are flag-gated diagnostics for the decode campaign; neither changes shipped behaviour, and the Qwen 3.8 and AgentWorld goldens are identical. NVMAI_PREFETCH_IO_THROTTLE=1 runs the prefetch ring's reads on the throttled disk tier (setiopolicy_np IOPOL_THROTTLE, set per read on the C reader's worker and restored after), so a deeper ring can be measured without its speculative reads starving demand reads -- the contention that made depth 2 and 4 lose before. NVMAI_PROBE2_TRACE=1 scores layer L+2's router on layer L's residual as a second probe and writes it into the prefetch trace as next2_layer_prediction, to measure whether a two-layer-ahead prediction is accurate enough to widen the prefetch window. Also: 40 joins the slot rungs (the 8-bit Qwen 3.8 cache measured +13% at 48 over 32 but pays in swap; 40 is the middle), and the streamer test suite is serialized -- its tests share one synthetic layer file and failed 5 of 19 in a parallel run on the unchanged tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ector The second router probe is accurate enough to act on: on Qwen 3.8 4-bit, layer L+2's router scored on layer L's residual has top-1 85.6% and set overlap 58% against the next-layer probe's 90.8% / 65%. With NVMAI_PREFETCH_AHEAD=2 the ring is fed from it, so each speculative read gets a whole extra layer of compute to land in. The ring now reclaims by passed layer rather than by whichever layer is being begun, which is what lets a layer-L+1 read survive a layer-L+2 begin, and it is sized for ahead x depth reads. NVMAI_PREFETCH_IO_THROTTLE becomes NVMAI_PREFETCH_IO_TIER=standard|utility| throttle. Throttle measured: depth 1 4.60 / 4.62 against 5.94 / 6.02 (the read lands too late), depths 2 and 4 a wash (6.0-6.2): it removed the contention penalty deeper rings paid but gave nothing back. The lighter tiers are under test. Golden identical; both stay off by default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-bit 40 slots The prefetch disk-I/O tier is a ModelProfile field now, carried by the prefetch ring into each streamer read, instead of a process-wide static read from NVMAI_PREFETCH_IO_TIER (the env still overrides the row). Measured on Qwen3.8-Flash-Next 4-bit, interleaved story runs: shipped depth 1 5.21 / 5.33 tok/s; two-layer-ahead prefetch 5.34 / 5.17 (M=1) and 5.41 / 5.24 (M=2), a wash; utility tier depth 2 5.46 / 5.40 (+3%), depth 4 5.37 / 5.33. 8-bit: 32 slots 2.05 / 2.06, 40 slots 2.18 / 2.27 with no paging, 48 slots ~1 GB swap growth per run, so the row takes 9.5 GiB. Goldens qwen38-4 and agentworld-4 byte-identical; the 35B rows keep tier 0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removed picture element and kept the project description.
Corrected wording and improved clarity in the benefits section. Removed unnecessary lines and streamlined the supported LLMs section.
Removed the default 8-bit storage mention for compressed KV cache in the README and adjusted the token speed table for Qwen3.8-Flash-Next.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a pointer to the Pummelchen/NVMAI fork; the full code, wiki, and benchmark suite live there.
Improvements over the original repo (measured on M3 24 GB, 4-bit Qwen 3.6 35B-A3B):
Estimated tok/s increase over the original repo: ~+55-65% (compounded decode-rate wins; the original predates the Qwen 3.6 port, so the baseline is the fork's pre-optimization state on the same model and hardware).