TASK: merge upstream v0.33.3 into main (D1-A: fork gemma4 vision kept; D2: ADR 0010 on upstream plumbing) - #264
Merged
Merged
Conversation
* fix(docs): correct typos found during code review Non-functional changes only: - Fixed minor spelling mistakes in comments - Corrected typos in user-facing strings - No variables, logic, or functional code was modified. Signed-off-by: Marcel Petrick <mail@marcelpetrick.it> * fix additional typos and shell-unsafe example in docs --------- Co-authored-by: Patrick Devine <patrick@ollama.com>
Put license files in lib/ollama so runner dirs don't duplicate them, and package in the base package(s).
* build: go deps Gather go dep licenses * address comments
* Honor model generation defaults Model-authored sampler defaults from GGUF metadata and HF generation_config.json were ignored, so built-in Ollama defaults could override model intent unless parameters were set in the Modelfile or request. The fix parses those defaults into model config and applies them before Modelfile/request options, preserving the expected precedence order. * review comments * address comments
* MLX, MLX-C: version bump * resolve comments * address comments
* llama.cpp: version bump b10729 Regenerate the compat hooks patch for b10729: upstream removed the whole-tensor load_data_for read (last consumer was llama-quantize, which now reads slabs via load_data_range). Keep the existing hook surface (constructor, skip loops, load_all_data, mtmd/clip) unchanged and add maybe_load_text_tensor_range, which materializes a text load op's output once per tensor and serves the new (offset, size) slab reads from that cache. * address comments
* ci: wire up MLX unit tests for PR runs Download the latest Darwin release payload matching the current MLX and MLX-C revisions so macOS PR tests can exercise MLX without rebuilding it. If no matching release exists after a pin bump, leave MLX tests skipped until the next release. Add whole-tree race coverage and smoke-run committed benchmarks. Verify generated UI types, and stabilize tests exposed by the broader CI coverage. * review comments * mlx: run tests on one pinned worker Keep MLX tests and benchmarks on a shared pinned thread while preserving Fatal, Skip, and Cleanup semantics. Also clean stale CI payloads and ensure updater workers shut down cleanly. * addres comments
* Report cached prompt tokens Add prompt_eval_cached_count to native responses and expose equivalent cached-token fields through the OpenAI- and Anthropic-compatible APIs. Keep prompt_eval_count as the logical input total while excluding cache hits from CLI and benchmark prefill rates. Surface processed and cached prompt counts in benchmark output. Collect cache counts from llama-server and MLX, preserve coherent metrics across two-pass structured generation. Fixes ollama#8008 Related to ollama#15758 * review comments
MLX runs on one goroutine locked to its OS thread, so the thread-local error buffers and closure-based check helpers defended against a calling pattern that is already invalid. Replace them with a single buffer that the handler fills and Go reads after every call. mlxError returns the captured message; mlxCheck panics on it and passes the call's result through, so a checked call is one expression. Only an int status carries a failure signal, which lets a message next to a zero status be reported as an earlier unchecked call. Fix two tests that relied on errors being dropped: the laguna mixed-precision fixture used an unsupported quantization group size, and the compile callback test expected the callback's own panic.
The bindings captured MLX error messages but checked almost no calls, so a failure continued with a null output and surfaced later as zero results, skipped evals, or an unrelated crash. Wrap every call in mlxCheck. Paths that return an error or disable a GPU kernel backend use mlxError instead, and the lookups where a non-zero status means a miss read the buffer first and then treat the status as data. Also free the string handles behind Array.String and the log values after the call that fills them; they were freed before it and leaked.
Audio-capable models need mono PCM at their expected sample rate before model-specific feature extraction. Like the image decoder set in base, the supported audio containers are decided once here so every model accepts the same formats: WAV, covering integer and float PCM, extensible headers, and multi-channel downmix. Anything else is rejected as unrecognized; supporting another container later means one new decoder here, with no model or runner changes. Input at other sample rates is resampled through a band-limiting filter, so mismatched rates degrade gracefully instead of aliasing. Decoded clips are capped at ten minutes: the declared rate comes from an untrusted header, and the cap is what keeps a small file claiming an absurdly low rate from resampling into an enormous allocation. Models whose encoder takes clips only up to a fixed length split longer clips with Split. Chunks are sized evenly, and each cut moves to the quietest point within a few seconds of its even share, so a boundary lands on a pause where the clip has one instead of severing a word between two independently encoded chunks. The even sizing bounds the search so no chunk can end up over the limit.
Safetensors gemma4 imports served by the MLX engine now answer image and audio chats. Images run through both vision architectures: the transformer tower (26B, 31B, e-series) and the 12B's encoder-free unified embedder. Audio arrives through the same intake the ollama API already accepts for gemma4 GGUFs — WAV bytes in the images field, OpenAI input_audio parts, and /v1/audio/transcriptions uploads — with the e2b/e4b checkpoints running clips through their conformer audio encoder and the 12b unified checkpoint embedding the raw waveform directly. Clips longer than 30 seconds are split evenly into chunks of at most 30 seconds, cut at pauses, and encoded independently. Each modality serves only checkpoints that carry it: 26B/31B have no audio config and reject audio input, and checkpoints with an unrecognized vision architecture still load as text-only models and reject image requests. The server previously hid the vision and audio capabilities for gemma4 safetensors because the engine served neither. Both suppressions are removed, and existing imports start advertising the capabilities without re-importing since import already records them.
… metrics) need a decision Plan only; main untouched. 14 commits, both payload pins and MLX-C move, 28 conflicts (5 add/add). Verified against real b10630/b10760 checkouts: upstream's re-cut 001 plus our 002/004/005/801/903 apply in sequence; 903 still required (#27044 open); the #215 cuBLAS knob survives; GGUF generation defaults are inert on the deployed fleet; prompt_eval_count semantics unchanged; four compile breaks hide behind clean auto-merges. Recommends keeping the fork's gemma4 vision for this fold (upstream's has no per-request budget) and deciding adoption on a measured spike. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (ADR 0010 arithmetic on upstream plumbing) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… memory.go, mlxtest, D2 union) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gates 1 and 2 of the v0.33.3 fold: the mechanical merge and the no-GPU test set. 28 conflicts, resolved per docs/maxusai/tasks/upstream-sync-2026-09-04.md and Glenn's decisions D1-A and D2. Payload pins take upstream's values: LLAMA_CPP_VERSION b10760, MLX_VERSION 37c26e5755da..., MLX_C_VERSION c74db5307cc8... (the mlx/compat/ carry patch is deleted with upstream's bindings regeneration). D1-A — the gemma4 MLX package stays the fork's --------------------------------------------- media.go / vision.go / media_test.go / vision_test.go: ours whole-file (add/add). gemma4.go: ours, taking only upstream's package doc line — every other upstream hunk there is vision/audio wiring (parseMultimodalConfig, buildMasks threaded through DecoderLayer/Attention, the media-placeholder embed masking, loadAudioWeights) and is incompatible with the fork's visionChunkMask design. gemma4_test.go / gemma4_moe_test.go: upstream's (pure mlxtest port, no vision content); vision_test.go's nine useMLXTestThread sites ported to mlxtest.Run since upstream deleted that helper with gemma4_moe_test.go. Excluded under D1-A (they reference upstream media.go symbols the fork does not define — multimodalConfig, MultimodalEmbedder, makeClippableLinear, m.MM / m.Vision / m.Audio, visionSoftTokenBudget — and do not compile here): x/models/gemma4/audio.go x/models/gemma4/audio_test.go x/models/gemma4/process_image.go Kept, and dead under D1-A: x/models/gemma4/process_audio.go and process_audio_test.go compile standalone (they depend only on the new x/mlxrunner/model/audio package), so they stay as the audio front-end a D1-B spike would need. x/mlxrunner/model/audio is a clean add and is likewise unused. isGemma4Renderer is restored in server/renderer_resolution.go: upstream deleted it when it started advertising gemma4 audio, and this fork ships gemma4 vision without audio, so server/images.go's suppressAudioCapability keeps its gemma4 branch (the auto-merge had silently dropped it) and server/model_list_cache.go keeps its mirror. Upstream's removal of the gemma4 *vision* suppression is taken in both files; server/images_test.go keeps the fork's "keeps vision, suppresses audio" cases. The per-request budget path is unchanged and asserted at compile time (gemma4/media.go's `_ base.MediaBudgetModel = (*Model)(nil)`): PrepareMediaWithBudget, resolveImageBudget, base.MediaBudgetModel and x/mlxrunner/media.go's bm.PrepareMediaWithBudget dispatch all survive, so image_max_tokens cannot fall through to PrepareMedia for gemma4. D2 — ADR 0010's arithmetic on upstream's plumbing ------------------------------------------------ Upstream's shapes are kept everywhere: the IncludeIntermediateMetrics request field (api/llm/mlxrunner client + pipeline), ChatHandler's `includeIntermediateMetrics := req.Format != nil && currentFormat == nil`, the firstPassMetrics capture with its non-terminal blanking, and the PromptEvalCachedCount pass-through (routes both handlers, llama_server completion + chat, mlxrunner client, pipeline's `len(session.inputs) - len(session.remaining)`). The fold stays the fork's. Upstream's `else if Applying && r.Done` block is dropped because ADR 0004's pass1 summing below it does the same job; running both would count pass one twice. What upstream's plumbing buys is better inputs, not a different fold: at the ChatHandler transition site pass one now has a report of its own (every chunk of a deferring pass carries metrics), so reportedPassMetrics() prefers it over ADR 0010's textual reconstruction — that report is the runner's cache-inclusive prefill, image-embedding tokens included, which is exactly what ADR 0010's subtraction exists to recover. transitionPassMetrics()/transitionPromptDelta remain as the fallback for a runner that reports nothing, and the delta is still computed only from a reconstructed pass (it needs the textual count on both sides). PR #238's `deferring` gate is kept on both ChatHandler transition sites, and llm/llama_server.go keeps result.final, applyCompletionFormat, visionServerArgs, kvCacheFlagValues and ggmlCublasComputeTypeEnv. One assertion in upstream's new restart test is re-expressed for the fork (server/routes_generate_test.go, wantMetrics): upstream reclassifies the continuation's prefill as generation work, so it reports prompt duration as pass one's alone and folds pass two's prefill into eval duration. The fork keeps ADR 0004's summing — each pass's prefill is prefill, each pass's decode is decode. Counts, cached count and eval count now match upstream's expectations exactly; only that duration split differs, and adopting upstream's would move every recorded think+format cell's tok/s, which is a measured surface and a decision of its own. New test TestChatThinkFormatTransitionMetricsReportedPassOne asserts the folded count on a vision-shaped request: it fails if the fold ever falls back to a textual count that cannot see the image surplus. MLX bindings and the test harness --------------------------------- x/mlxrunner/mlx/{mlx,stream}.go: upstream's single-buffer error contract (lastError, mlxError[T], mlxCheck[T]). ClaimOSThread and the `__thread _mlx_thread_owned` flag are deleted, with their callers in x/create/mlxthread.go, x/mlxrunner/server.go and the two vision oracle tests. ADR 0017's guarantee is re-expressed on x/internal/mlxthread, whose Start locks the worker goroutine and deliberately never unlocks; the ADR carries a status amendment saying so, and AGENTS.md / docs/development.md now point at mlxthread and mlxtest.Run. memory.go's MemoryLimit / SetMemoryLimit / SetCacheLimit are ported off mlxCall onto mlxError — mlx_get_memory_limit, mlx_set_memory_limit and mlx_set_cache_limit all survive the MLX-C regeneration with unchanged signatures (x/mlxrunner/mlx/generated.h:5301-5316, include/mlx/c/memory.h:34-38). x/mlxrunner/client.go is mechanical: mlxRunnerEnvDefaults, CacheThrashingCheckEnv, budgetWithOverride and MemoryLimitEnv are untouched. x/mlxrunner/pipeline.go keeps every deferred cleanup on guardClose and keeps the stopper block, with upstream's cachedPromptCount and per-chunk metric enrichment folded in. x/internal/mlxtest: upstream's Run / RunSubtest / SkipIfUnavailable. Of the 18 mlxtest.Setup sites, 17 were in upstream-owned files and upstream ported them itself; the two vision oracles (vision_golden_test.go, vision_e2e_test.go) already drive mlxthread.Start and only lost their ClaimOSThread line — both still compile and skip cleanly without a payload. x/mlxrunner/mlx/memory_test.go's fork-only SetCacheLimit test is ported onto upstream's withMLXThread(t, func(*mlxthreadtest.T)) signature. x/mlxrunner/constrain_bench_test.go and constrain_test.go are DELETED rather than ported: the bench calls mlxtest.Setup(b) and the new API takes only *testing.T, and constrain_test.go's skipIfNoMLX came from an upstream file that no longer defines it. Both drive MLX from a plain test goroutine, which is invalid under the shared-thread model. This brings forward part of ADR 0033's stated follow-up; constrain.go itself and the four non-MLX constrain tests are left for that PR. Build, CI and docs ------------------ cmake/mlx/CMakeLists.txt is the union: the fork's $ORIGIN RPATH block and quadmath in MLX_INCLUDE_REGEXES, upstream's cusolver / cusparse / nv[Jj]it[Ll]ink, OLLAMA_LIB_DIR destinations and license installs. .github/workflows/test.yaml is the union of the fork's path filters (as #232) with upstream's go_mod_changed filter, go_license job and MLX Darwin payload steps. Dockerfile auto-merged correctly — upstream's COPY mlx mlx removal and go-license step, the fork's ccache block. llama/compat: upstream's re-cut 001-llama-cpp-hooks.patch wins verbatim, 002/004/005/801/903 are untouched, and the README merged upstream's b10729 load_data_range paragraph into the fork's band text. docs/api.md takes upstream's prompt_eval_cached_count line and its reworded prompt_eval_duration, and keeps the fork's eval_count note; the fork's image_min_tokens / image_max_tokens and kv-cache-type docs live in docs/maxusai/ and were never in docs/api.md, so nothing was re-added. Gate 2 (golang:1.26.0, -u 1000:1000, caches on the 8 TB array) -------------------------------------------------------------- go build ./... PASS go vet ./... 1 finding, pre-existing go test ./llm/ -run 'TestImageTokensForSize|TestKVCacheType' PASS go test ./server/ ./model/... ./llm/ ./api/ ./convert/ \ ./x/structured/... ./x/mlxrunner/... ./x/internal/... \ ./x/models/... ./x/create/... PASS 4230 pass, 0 fail, 237 skip python3 docs/maxusai/vision-suite/preflight/test_verdicts.py 94 tests OK, 6 skipped (quality arm, pre-existing) The vet finding is tokenizer/bytepairencoding_test.go:542:5 "result of slices.Collect call not used". It is not this merge's: the file is byte-identical to v0.33.3's and unchanged since before the merge base, and a control `go vet ./...` on origin/main in the same image reports the same single finding. No new vet findings. 188 of the 237 skips are "MLX not available: failed to load MLX dynamic library" — the golang image has no native payload — and a skip is not a pass: every MLX kernel path, including TestVisionGoldenParity and TestVisionEndToEnd (which also gate on OLLAMA_VISION_E2E), is unexercised here and is Gate 4's job. gofumpt -l over the 120 Go files this merge touched lists one: server/routes_generate_test.go, whose two hunks are pre-existing fork code in TestChatFormatPassthrough's table (identical hunks on origin/main), outside anything changed here; left alone rather than smuggling unrelated reformatting into a merge commit. SPEC H11 -------- Once the tag is cut, this fold's build identity is 0.33.3-dynres-0-g<this merge's sha>. That is a NEW build identity, not an equivalence to any 0.33.2 stamp: the payload moved (b10630 -> b10760) and the MLX/MLX-C pins moved with it, so no 0.33.2 measurement carries over and nothing recorded here may be folded into a 0.33.2 table (ADR 0032, ADR 0011 rule 5). Not verified here: no native build, no GPU, no preflight. Gates 3-6 stand.
…results Criteria 3 and 4 ticked with the facts. The Conflicts table gains a "resolved as" column recording what was actually done per file rather than what was planned, including the two things that differed from the plan: - 17 of the 18 mlxtest.Setup sites were in upstream-owned files and upstream ported them itself; the 18th (constrain_bench_test.go) cannot be ported because the new API takes only *testing.T, so it and constrain_test.go are deleted, bringing forward part of ADR 0033's follow-up. - server/images.go's suppressAudioCapability had silently lost its gemma4 branch in a clean auto-merge — a fifth hidden break, alongside the four the plan predicted — which would have advertised gemma4 audio this fork cannot serve. Restored. New "Excluded under D1-A" section lists the three upstream gemma4 files the merge removes (audio.go, audio_test.go, process_image.go) with what each contains, so the D1-B spike starts from it, and records the two that were kept but are dead (process_audio.go, process_audio_test.go), plus TestMLXOperationsSurviveRescheduling going with ClaimOSThread. Gate 2's numbers are recorded with their controls: 4230 pass / 0 fail / 237 skip, 188 of the skips being "MLX not available" — a skip is not a pass, and the vision goldens are among them. The one go vet finding and the one gofumpt finding are both shown to be pre-existing by a control run of origin/main in the same image. Also records that this branch is 9 commits behind origin/main (#266, #213, #268 landed after it was cut), that none of the six files they touch is touched by this merge, and that Gate 2 therefore ran without #266's six new test_verdicts.py cases (94 here vs 100 on main). ADR 0021 gets a status note: D1-A upholds it, and its decision 6 / consequence naming mlx.ClaimOSThread and mlxtest.Setup now point at mlxthreadtest.
glennneuber
marked this pull request as draft
September 4, 2026 02:36
glennneuber
changed the title
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…igns Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…332nv1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kept as dead-but-compiling in the merge; golangci-lint's unused check flagged processUnifiedAudio at repo root, which would keep the test workflow red. Under D1-A the fork ships no gemma4 audio, so the D1-B spike restores all five upstream audio/vision files from the tag together. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-measured first: the three arch ladders on the sync-0.33.3 canary are byte-identical to the b10630 rows (nemotron [266, 266, 578, 2306, 3270], gemma4 [1102 x5], qwen35 [1034, 1034, 1034, 2306, 4082]) -- the third consecutive pin bump with unchanged numbers. Only the identity moves; no expected value is edited, as in #239. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…der shared-GPU contention, capped re-run pending Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e fold Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
glennneuber
marked this pull request as ready for review
September 4, 2026 11:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR started as the fold plan (still in
docs/maxusai/tasks/upstream-sync-2026-09-04.md, decisions D1/D2 recorded) and now carries the merge itself, as #232 did for v0.33.2. Draft until Gates 3–5 pass (image, preflight, campaign spot-check).Merge:
b81656ae=Merge tag 'v0.33.3'(14 upstream commits). Pins: llama.cppb10760(0f3a71be1), MLX37c26e5755da…, MLX-Cc74db5307cc8…;mlx/compat/carry patch gone; upstream's re-cut001hooks patch taken, our002/004/005/801/903unchanged (verified applying in sequence on a real b10760 checkout).D1-A (Glenn: keep
image_max_tokens).x/models/gemma4/{media,vision,*_test}.goare ours;gemma4.gokeeps our vision/media wiring (upstream'sbuildMasks/parseMultimodalConfig/loadAudioWeightshunks excluded). Excluded: upstreamaudio.go,audio_test.go,process_image.go(the direct competitor to ADR 0008's ladder — the D1-B spike's file).process_audio*.gokept (compile standalone, dead).isGemma4Rendererrestored; gemma4 audio stays suppressed in/api/tags/show(we do not ship it); the vision-suppression removal is upstream's.PrepareMediaWithBudget→ gemma4 still wired, no silent fallthrough.D2 (Glenn: ADR 0010's arithmetic on upstream's plumbing). Upstream's
IncludeIntermediateMetrics,includeIntermediateMetrics := req.Format != nil && currentFormat == nil,firstPassMetrics,PromptEvalCachedCounteverywhere, pipelinecachedPromptCount— all kept. NewreportedPassMetrics()prefers the runner's own pass-one report (cache-inclusive, image tokens included) and falls back to ADR 0010'stransitionPassMetrics(); done-time summing stays ADR 0004's (upstream's reclassification of the continuation prefill as generation time was not adopted — it would move every recorded think+format tok/s); #238'sdeferringgate kept on both transition sites. Upstream's new restart test re-expressed for its two duration assertions (comment names D2); newTestChatThinkFormatTransitionMetricsReportedPassOnepins the vision-shaped fold.MLX bindings / harness. Upstream's single-buffer error contract taken;
ClaimOSThreaddeleted (ADR 0017's guarantee is now carried bymlxthread.Start; ADR 0017/0021 carry dated status notes; AGENTS.md anddocs/development.mdupdated);MemoryLimit/SetMemoryLimit/SetCacheLimitported ontomlxError(MLX-C symbols confirmed present); 18mlxtest.Setupsites ported tomlxtest.Run/RunSubtest;guardClose×5,stopper,mlxRunnerEnvDefaults,applyArchServerEnvs,visionServerArgs,kvCacheFlagValuesintact;cmake/mlxunion keeps$ORIGINRPATH +quadmathalongside upstream'scusolver cusparse nvJitLinkandOLLAMA_LIB_DIR.Fifth hidden break beyond the four the plan predicted:
suppressAudioCapabilitylost its gemma4 branch in a clean auto-merge — restored.Gate 2 (golang:1.26 container,
-u 1000:1000):go build ./..., fullgo test ./...= 67 packages ok, 0 fail (237 skips, 188 of them "MLX not available" — payload paths are Gate 4's);test_verdicts.pygreen. Pre-existing, not from this merge: onego vetfinding intokenizer/bytepairencoding_test.go(identical upstream); gofumpt onserver/routes_generate_test.go(fixed by #269).Before undraft: merge
main(#266/#213/#268/#267/#269) in; Gate 3 image onbigdisk; Gate 4 CUDA preflight re-measure-then-repin +mlx-metal-0-33-3(Metal host is Glenn's); Gate 5 think-off nvfp4 spot-check; then tagv0.33.3-dynreson the merge commit, release + generated matrix, README pointer, deploy from the tag.🤖 Generated with Claude Code