From b3dd16462a761c589bb774b6b040622e5966df77 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 29 Aug 2026 15:34:07 +1000 Subject: [PATCH 1/3] docs: table the concrete differences from upstream The banner said what areas the fork touches; it did not say what actually differs. Adds a table with the specific limits, each checked against upstream at b10630 rather than recalled: nemotron-3 fixed 512x512 canvas, 256 tokens/image -> 256-3328, native aspect (upstream has no set_limit_image_tokens for NEMOTRON_V2_VL at all) gemma4 caps at 280 tokens, under-budget images keep their natural rounded grid and are letterbox-padded -> scaled to fill the budget, snapped to the supported ladder, never padded qwen2.5-vl fp16 accumulation in the vision tower overflows on some ordinary images -> fp32 forced for qwen25vl runners MoE + MMQ ids-path padding sized from ne11, no padding under broadcast -> sized from the flattened row count regression none in-tree -> preflight harness, generated triggers, node meter Placed outside the banner's blockquote: a table nested in a blockquote renders inconsistently, and a broken table on the landing page is worse than none. Closes with the point the table should make -- every row is a delta we would rather not have, offered upstream where it is upstream's to take and deleted here once it lands. Co-Authored-By: Claude Fable 5 --- README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.md b/README.md index 5c42e8e25d1..c9e3810cd05 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,26 @@ > Fork-specific documentation, ADRs and measurements live in > [`docs/maxusai/`](docs/maxusai/). + +### What differs from upstream, concretely + +Measured against upstream ollama at llama.cpp `b10630`, the pin this fork +currently builds. + +| | upstream ollama | this fork | +|---|---|---| +| **nemotron-3 vision** | fixed 512×512 canvas — **256 tokens per image**, whatever the aspect ratio | native-aspect dynamic resolution, **256–3328 tokens**, position embeddings interpolated to the patch grid in-graph (`002`) | +| **gemma4 image sizing** | **caps at 280 tokens**; an under-budget image keeps its natural rounded grid and is letterbox-padded | every image scaled to *fill* the budget and snapped to gemma4's supported ladder (70/140/280/560/1120), never padded (`004`). Off-ladder grids measurably break `box_2d` vertical grounding | +| **qwen2.5-vl on CUDA** | f16 vision matmuls accumulate in fp16; on some ordinary images a few elements of millions reach `inf` at `v.blk.31.ffn_down` and the caption collapses into one repeated glyph | fp32 accumulation forced for `qwen25vl` runners. Offered upstream as [ollama#18070](https://github.com/ollama/ollama/pull/18070) | +| **MoE + MMQ on CUDA** | ids-path tail padding sized from `ne11`; under broadcast `ne11 == 1`, so the buffer gets no padding and the kernel overruns by up to a 512-row tile | padding sized from the flattened row count (`903`). Reported as [llama.cpp#27044](https://github.com/ggml-org/llama.cpp/issues/27044) | +| **Vision regression testing** | none in-tree | preflight harness with recorded per-model expectations, generated (public) trigger images, and an env-gated node meter (`801`), run before every deploy | + +Every row is a delta we would rather not have. Each is offered upstream where +it is upstream's to take, and deleted from here once it lands there — the +`qwen25vl` gate and the MMQ padding fix are both filed and pending. + +--- +

ollama From 41fb41ebad1aa2e5b1fb05d3ff29d9b8a9f6efc7 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 29 Aug 2026 15:36:09 +1000 Subject: [PATCH 2/3] docs: say plainly what the MLX runtime costs The banner called MLX 'experimental' without saying what that means to someone deciding whether to use it. Adds the caveats: - it works, with the one CUDA figure we actually have recorded (gemma4:31b-nvfp4, 41.5 tok/s, 22 GiB, idle host) - on CUDA it is roughly half the GGML path's throughput -- marked as an operational observation, NOT a benchmark, because no matched same-host GGML-vs-MLX CUDA pair is recorded anywhere in this repo - on Metal the ordering reverses: a matched campaign measured MLX ~2.4x faster than llama-server (12b: 121 vs 50 tok/s), so the CUDA gap is CUDA-specific and neither number generalises to the other platform - MLX-vs-GGUF output quality is not a controlled comparison at all, since engine and quantization move together (nvfp4 vs q4_K_M) -- the campaign doc already records this caveat and it belongs where users see it Ends with the operational advice: use GGML/llama-server where throughput or comparability matters. Co-Authored-By: Claude Fable 5 --- README.md | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index c9e3810cd05..2c8d4aa44b3 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,8 @@ > it deploys. > - **Fixes carried until upstream takes them**, each tracked against an > upstream issue or PR, and deleted from here when it lands there. -> - An experimental MLX runtime for Apple Silicon and CUDA — increasingly -> converging with upstream's own MLX work, and expected to be superseded -> by it. +> - An experimental MLX runtime for Apple Silicon and CUDA — see the caveats +> below before using it for anything that matters. > > **Current fold:** [`v0.33.2-dynres`](https://github.com/MaxusAI/ollama/releases/tag/v0.33.2-dynres) > — upstream v0.33.2, llama.cpp `b10630`. `main` moves ahead of this between @@ -45,7 +44,32 @@ currently builds. | **MoE + MMQ on CUDA** | ids-path tail padding sized from `ne11`; under broadcast `ne11 == 1`, so the buffer gets no padding and the kernel overruns by up to a 512-row tile | padding sized from the flattened row count (`903`). Reported as [llama.cpp#27044](https://github.com/ggml-org/llama.cpp/issues/27044) | | **Vision regression testing** | none in-tree | preflight harness with recorded per-model expectations, generated (public) trigger images, and an env-gated node meter (`801`), run before every deploy | -Every row is a delta we would rather not have. Each is offered upstream where +### MLX runtime — experimental, and slower on CUDA + +It works. Models load, stay resident and generate correct output on both +Metal and CUDA (`gemma4:31b-nvfp4` decodes at 41.5 tok/s in 22 GiB on CUDA, +measured on an idle host). But it is not the path to reach for by default: + +- **On CUDA it is roughly half the throughput of the GGML path.** That figure + is an operational observation, not a benchmark — we have no matched + same-host, same-model GGML-vs-MLX CUDA pair recorded, and the ratio will + move with model and GPU. +- **On Metal it is the other way round.** A matched campaign measured MLX + ~2.4× faster than llama-server (gemma4 12b: 121 vs 50 tok/s decode). So the + CUDA gap is CUDA-specific, not an MLX property — do not generalise either + number to the other platform. +- **MLX-vs-GGUF output quality is not a controlled comparison.** Engine and + quantization move together — nvfp4 on MLX against q4_K_M on GGUF — so a + quality difference between the two cannot be attributed to the engine. We + have not separated them, and until someone does, treat "which is better" + as open. +- It is **converging with upstream's own MLX work** and is expected to be + superseded by it; the fork has already retired its constrained-sampling + layer in favour of upstream's engine (ADR 0033). + +Use GGML/llama-server for anything where throughput or comparability matters. + +Every row above is a delta we would rather not have. Each is offered upstream where it is upstream's to take, and deleted from here once it lands there — the `qwen25vl` gate and the MMQ padding fix are both filed and pending. From f88d663a287e5a371c380053b806f5c6b68487b9 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 29 Aug 2026 15:52:22 +1000 Subject: [PATCH 3/3] docs: add the bbox contract to the differences table Upstream has no bounding-box coordinate contract: a caller must trust the frame the model declares, and that is precisely what fails -- pinned to real pixels, qwen3.6 converts 1 of 14 geometries. The fork pins norm-1000 and derives the space from the response rather than a declared ref_size, so the model never names its own frame and its internal resize cannot contaminate the coordinates. 111 of 112 cells convert cleanly across 14 geometries x 4 models x 2 think modes (ADR 0027/0030, SPEC C13-C18). Phrased as a protocol plus its conformance measurements, not a runtime feature, because that is what it is -- no Go enforces the pin, and a caller who ignores the contract gets the unpinned behaviour. The row would otherwise read as something the server does for you. Co-Authored-By: Claude Fable 5 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2c8d4aa44b3..498b364d6b9 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ currently builds. | **gemma4 image sizing** | **caps at 280 tokens**; an under-budget image keeps its natural rounded grid and is letterbox-padded | every image scaled to *fill* the budget and snapped to gemma4's supported ladder (70/140/280/560/1120), never padded (`004`). Off-ladder grids measurably break `box_2d` vertical grounding | | **qwen2.5-vl on CUDA** | f16 vision matmuls accumulate in fp16; on some ordinary images a few elements of millions reach `inf` at `v.blk.31.ffn_down` and the caption collapses into one repeated glyph | fp32 accumulation forced for `qwen25vl` runners. Offered upstream as [ollama#18070](https://github.com/ollama/ollama/pull/18070) | | **MoE + MMQ on CUDA** | ids-path tail padding sized from `ne11`; under broadcast `ne11 == 1`, so the buffer gets no padding and the kernel overruns by up to a 512-row tile | padding sized from the flattened row count (`903`). Reported as [llama.cpp#27044](https://github.com/ggml-org/llama.cpp/issues/27044) | +| **Bounding boxes** | no coordinate contract — a caller must trust whatever frame the model declares, which is what fails: pinned to `real` pixels, qwen3.6 converts **1 of 14** geometries | requests pin **norm-1000** and the space is derived from the response, never from a declared `ref_size`, so the model's internal resize cannot contaminate coordinates. **111 of 112** cells convert cleanly across 14 geometries × 4 models × 2 think modes (ADR 0027/0030, SPEC C13–C18). A protocol and its conformance measurements, not a runtime change | | **Vision regression testing** | none in-tree | preflight harness with recorded per-model expectations, generated (public) trigger images, and an env-gated node meter (`801`), run before every deploy | ### MLX runtime — experimental, and slower on CUDA