release: sync fork and add signed notarized macOS DMG pipeline - #2
Merged
Merged
Conversation
mlx-lm's sharded_load maps every constructed parameter name through model.safetensors.index.json before downloading, but an architecture whose sanitize() fuses weights at load time (glm_moe_dsa's DSA indexer fusing wk and weights_proj, deepseek_v4 stacking expert biases) constructs names the index cannot contain, so distributed loading of a local checkpoint raises 'Pipeline loading is only supported for MLX converted models' even though the normal loader handles it fine. The gate only exists to pick which files to download; when it rejects a local directory the wrapped sharded_load now repeats the load-and-shard tail on the local path, where load_model performs the sanitize fusion exactly as it does for a single process. Installed by the glm_moe_dsa and deepseek_v4 patches, idempotent, and rebinds the server's import-time alias too. Verified that both ranks of a 2-rank GLM-5.2 launch get through the gate and into weight loading; the full serve on one 512 GB Mac then dies to jetsam because two 200 GB shards plus double page cache exceed the machine, which is a placement the production admission planner rejects, so end-to-end engaged-regime serving stays covered by the planned two-Mac validation.
…e copies A boundary snapshot used to hold the whole prompt cache, so a long conversation settled into dozens of near-duplicate full copies (the last 64 boundaries of a 128k chat differ by 2048 tokens each) and one 128k prefill wrote a triangular sum of bytes. The store now mirrors the local paged SSD policy: a boundary file holds what that boundary added. Plain KVCache members in step with the token stream store only their newest step-sized slab through a KVCacheSegment stand-in, positionally immutable and byte-exact under concatenation, with a layout entry carrying MLA-style zero-width halves that safetensors cannot hold. Non-sliceable members keep their proven full-state serialisation, which for rotating windows and recurrent slots is constant-sized and for the pooling cache remains its only representation. Restoring boundary B assembles the chain of files at step..B; a hole simply hides the deeper boundaries from that rank's vote, and prompts sharing a prefix share their early files like local blocks do. Eviction stays a deterministic LRU, now with an optional byte budget, and orphaned deeper files age out on their own. The GLM-5.2 engaged-regime chain drops from about 6.5 GB cumulative to 1.17 GB linear at 12k tokens and the gap grows quadratically with context. Re-verified everything on the new format: greedy parity on gemma-4-26B, Qwen3.6-27B, DeepSeek-V4-Flash and GLM-5.2 (short and engaged regime), the real-server end-to-end run, and the three 2-rank loopback profiles.
…snapshot-cache feat(cluster): add SSD boundary snapshot prompt cache for distributed ranks
* feat(scheduler): decode fairness for concurrent prefill (jundot#2031, jundot#2622) * fix(scheduler): share the prefill hold deadline across engines --------- Co-authored-by: jundot <jundot@users.noreply.github.com>
Claude Code sends periodic role=system reminders after tool results, which the anthropic adapter surfaces as tool -> system -> assistant runs. The V4 relocator only accepted user-adjacent runs, so these conversations fell back to front-consolidation and every new reminder rewrote the rendered prompt near the system section, invalidating the whole prefix and forcing 100k+ token re-prefills. Relocate such runs as a latest_reminder before the contiguous tool run so the rendered prefix stays byte-stable across requests.
…ocks The supersede-on-extend lineage assumed the block before the first new block was the previous store's tip, but on a store that diverged from an existing chain it is a shared-prefix interior block; stripping it two stores later permanently broke partial-match walk-back restores. Track real store tips and record lineage only when the predecessor was one. Also, dedup'd blocks were never rewritten, so blocks first stored without snapshot coverage kept placeholder payloads forever; backfill them from the current store's boundary snapshots so walk-back can restore at those boundaries again.
Quantizing a model directly from the Hugging Face cache used the route-safe cache ID as the output name, which produced invalid default repository IDs during upload. Derive the output base from the bare canonical repository name while preserving the full source repository ID for display.
…jundot#2644) * feat(cache): add GDN sidecar state codecs Persisted GDN sidecars are dominated by the fp32 recurrent state — the largest single payload in a full checkpoint. Encode it only at the serialization boundary (bf16, row-wise int8, and a randomized-Hadamard int8 variant) and restore to fp32, so live inference never sees reduced precision. Unknown codecs, malformed metadata and non-finite values fail closed: a corrupt sidecar becomes a cache miss, not silent garbage. * feat(cache): add RHT-INT16 GDN sidecar codec Row-wise int8 keeps the codec cheap but its reconstruction error is still visible at the per-row scale. Spreading the same quantization across an int16 payload after a randomized Hadamard rotation drops relative L2 to ~0.0025% — near-lossless while still ~2x smaller than fp32 — which makes it a safe default. The fixed sign diagonal is deterministic and never touches the generation RNG. * feat(cache): add GDN snapshot storage policy Replace the split boolean with auto|ssd_sidecar|embedded. Reduced sidecars are namespaced apart from fp32 but still fall back to a legacy fp32 sidecar at the same endpoint, so an experiment's warm cache never silently reuses another arm's checkpoint. Admin changes update the scheduler template and unload models so the selection applies on the next load.
…jundot#2643) Sync the vendored Muse Glimmer implementation with upstream mlx-vlm numerics and reasoning configuration while preserving oMLX-specific compatibility hooks.
Add an Enhanced Readability preference for web admin pages. Co-authored-by: LXD-8 <308763019+LXD-8@users.noreply.github.com>
…calls (jundot#2617) Snapshot/restore the budget processor around speculative token calls in the DSpark verify chain and draft generation, so only emitted tokens count toward the thinking budget.
…ot#2623) DeepSeek V4-style chat templates can end the rendered prompt with <think>, so initialize Responses streaming from the rendered prompt state. Reuse the existing detection to separate reasoning deltas and restore reasoning token accounting. Fixes jundot#2584
…#2593) Fixes jundot#2545. Co-Authored-By: discoStew <discostew6082@pm.me>
Hoist the memory helpers' imports to the top of the module, fold the headroom check into the snapshot helper, and return directly on a memory skip so the ladder no longer logs a compile failure that never ran.
…ndot#3105) The single-instance (AneHybridQ4Primitive) and dual (DualAneHybridPrimitive) ANE dispatch paths retained the producer command buffer and called model_->begin() outside any try. An exception thrown after begin() but before the detached evaluation thread was spawned — most realistically from the qmm device.get_kernel() calls that sit in that window — orphaned the ticket: submitted_ stayed ahead of completed_ forever, so the next begin() threw "overlapping evaluations" and the program stayed wedged until the process restarted (finding C1). The retained command buffer also leaked. Introduce a small AneDispatchGuard RAII helper that, on unwind between begin() and the spawn, releases the producer buffer (unless the pack step already retired it via producer_released()) and cancels the outstanding ticket(s) to rebalance the counters. begin() itself is wrapped in a narrow try that only releases the buffer — begin() throws before it increments the counter, so there is no ticket to cancel there — mirroring the existing fused-path pattern; the guard then covers the get_kernel window and is disarmed once the evaluation thread(s) own the ticket. The dual path disarms before the first spawn so it never double-counts a ticket a thread already owns. The fused path (AneHybridQ4SwiGLUDownPrimitive) already guards retain+begin but shares the same residual get_kernel-after-begin gap; it is left unchanged here and called out for a focused follow-up. Claude-Session: https://claude.ai/code/session_01GENz3t3tZVc8En54DxbZ9w Co-authored-by: Alyta Phoenix <alyta@phoenixes.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Extend the C1 dispatch guard from jundot#3105 to the fused SwiGLU/down path: construct it right after begin, fold the manual pack-failure rollback into it, and disarm before the evaluation threads take ticket ownership. Closes the same orphan-ticket wedge the single and dual paths already guard.
_prepare_fused_down_for_bank dequantized down_proj's ENTIRE packed weight to a dense fp32 array (dense_down), but only columns [0:gpu_start] were ever consumed from it -- down0/down1 and, when cpu_hidden>0, cpu_down_weight. The GPU portion (columns [gpu_start:hidden]) reads down.weight directly, still quantized, a few lines later for state.down_weight/down_scales/down_biases. The [gpu_start:hidden] suffix of dense_down was therefore computed and immediately discarded on every fused-down MLP compile -- a pure-waste transient of roughly (hidden - gpu_start) * out_features * 4 bytes (~0.5GB/layer at typical Qwen3.5/3.6 MLP dimensions). Fix: slice down.weight/scales/biases to [:gpu_start] (in packed-axis terms: gpu_start // 8 for the 4-bit-packed weight, gpu_start // 128 for the group_size=128 scales/biases) before calling mx.dequantize, instead of after. gpu_start is always a multiple of 128 (per_ane and cpu_hidden are both rounded down to 128 via the existing `// 128 * 128` pattern), so the packed-axis slice boundaries land exactly on quantization group boundaries -- the sliced-then-dequantized result is bit-identical to the old dequantize-then-sliced one, verified directly against a full-dequant reference in the new correctness test. Co-authored-by: Alyta Phoenix <alyta@phoenixes.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…pressure (jundot#3103) The compiled ANE procedure banks keep their packed weight blobs mapped for the model's whole residency (~13 GB for a 27B at mlp 0.35 / gdn 0.45). A config tuned at a short calibration length therefore competes with the KV cache at long context: once usage crosses the prefill guard's sizing target, chunks collapse 2048 -> 512 and a 57K prefill runs ~40-45% slower than with ANE disabled outright. The existing headroom ladder cannot recover this memory: idle-model eviction skips the requesting model, and the pooled-buffer reclaim keeps 'succeeding' marginally on every pass (prefill continuously refills MLX's buffer cache) while never touching the banks. Three changes: - release_qwen35_ane_prefill(model): latch every sliced module through the existing per-module failure flags (the dispatch sites fall back to stock GPU compute and never lazily recompile), drop the state references, zero the status counters. The native programs free their mapped blobs when the last reference dies; the next load rebuilds them from settings. - EnginePool ladder: a new last rung sheds the requesting model's banks on its own MLX thread (step loop parked, nothing mid-dispatch), and a request whose earlier pass already got a reclaim escalates straight to the release instead of reclaiming again first. - Scheduler: allow a second eviction pause per request, so recurring pressure can reach the deeper rung after a marginal first-pass reclaim. Verified on an M5 Pro 64 GB (57K prompt, banks resident 13.95 GB by footprint delta): under an enforced ceiling the second pause logs the release, the prefill finishes at full 2048-token chunks, /api/status reports the shed state, and subsequent requests serve GPU-only. With no memory pressure the rungs never run and behavior is unchanged.
… terms (jundot#3082) Chunked-prefill rejections logged one aggregate cap comparison with no way to tell which contributor actually bound. Add a second warning line on every rejection that breaks the admission bound into its separate terms: resident usage, the predicted per-chunk transient, the session observed-max ratchet, and the ANE prefill transient reservation.
After jundot#3103 sheds the banks, drop the load-time ANE I/O surface reservation from the memory monitor so admission stops pausing for memory that no longer exists, promote the shed log to a warning, and report an explicit shed flag in /api/status so it does not read like a load-time compile failure. A fresh enable clears the flag.
…st prefill-memory rejection (jundot#2992) _with_json_keepalive wraps non-streaming completions in a StreamingResponse so long prefills don't hit client read-timeouts, but it yielded its first keepalive byte -- and therefore committed the ASGI response to whatever status StreamingResponse was built with (always 200, none of the 6 call sites passed an explicit status_code) -- before knowing whether the wrapped request would succeed or fail. A memory-guard rejection that happens quickly (the common case for most prompt sizes) still surfaced a well-formed error body, but with a 200 status silently telling any client that only checks the status code that the request succeeded. Adds _json_response_or_keepalive, which races the wrapped coroutine against a short grace period. A fast outcome (success or PrefillMemoryExceededError) returns a plain Response/JSONResponse with the real status code, bypassing the streaming wrapper entirely -- reusing the same body-construction helper the synchronous preflight-rejection path already uses. Only requests still running past the grace period fall back to the existing keepalive-streaming behavior, which still commits to 200 on any later failure once streaming has begun -- an HTTP/1.1 constraint once headers have shipped, not something this fix can close, but that only affects the far less common case of a failure minutes into a long-running request rather than the ordinary fast-reject case this fixes. Applies to all 6 previously-affected endpoints: markitdown completions, embeddings, completions, chat completions, Anthropic messages, and the Responses API -- preserving engine-lease release semantics for the four that hold one. Claude-Session: https://claude.ai/code/session_01GENz3t3tZVc8En54DxbZ9w Co-authored-by: Alyta Phoenix <alyta@phoenixes.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…reaming paths (jundot#3060) The streaming generators' blanket except is the innermost handler, so a PrefillMemoryExceededError raised during prefill was flattened to a generic server_error before the correct handler in _with_sse_keepalive could run (jundot#3036) - the structured body (root type, omlx_code, estimated_bytes, limit_bytes) was built by unreachable code. Route the classification through the generators themselves: the OpenAI chat and completions generators share a _streaming_error_payload() helper that keeps the structured body for guard rejections and the exact previous behavior for everything else. The same shadowing existed on the two paths the issue left unchecked: /v1/messages now emits an Anthropic-native invalid_request_error event with the guard's detailed message, and /v1/responses now carries {code, message} in the response.failed error object instead of failing with no error at all. Regression tests pin the exceeded/aborted/generic payloads.
…iler is unavailable (jundot#3067) * fix(ane-tuner): return a completed GPU-only verdict when the ANE compiler is unavailable On a machine without the private ANE runtime (or a build without the procedure-bank compiler), the tuner previously discovered the gap only deep inside the bank-split ladder: every width failed with the same 'compiler is unavailable' error, the run ended in status error, and the model was left unloaded (jundot#3044, M2 Ultra). The serve path has always handled the same condition gracefully by skipping ANE. Probe availability at the top of run_tuning, before anything is pinned or unloaded, and return a completed run recommending GPU-only: on such a machine that IS the tuning answer, not an error. The probe is shared with qwen35_ane_compile_linear_bank's own gate so the two cannot drift. * test(ane-tuner): stub the compiler preflight in mocked pipeline tests The new run_tuning preflight consults the real probes, which return False on runners without the extension and short-circuit every mocked pipeline test to a GPU-only verdict. Pretend the compiler exists in the shared fixture; the unavailable-path tests re-stub the probes and the parity test pins the real implementation explicitly. --------- Co-authored-by: jundot <jundot@users.noreply.github.com>
The jundot#3067 preflight verdict ships processing_tps/speedup_percent as null for the first time; the mac app DTO declared them non-optional, so the status decode threw on exactly the machines the preflight serves. Make both fields optional and have the mac and web recommendation cards drop the tok/s segment when nothing was measured.
…ble-specprefill fix(ane): disable SpecPrefill during ANE split tuning
* perf(ane): reuse compiled Qwen programs across loads Opt-in with OMLX_QWEN35_ANE_COMPILE_CACHE=1. Uses stable per-OS model URLs and compiledModelExists to reuse Apple's AOT programs, with cross-process locking, fail-open temp compilation, and one corrupt-hit recompile fallback. Covers individual linear, fused SwiGLU/down, and procedure-bank programs.\n\nMeasured fresh-process load reductions: M1 Ultra 65.17s -> 22.42s, M2 Max 55.74s -> 20.08s, M1 Max 54.37s -> 25.55s. Output parity was byte-identical; oMLX persistent data footprint is zero bytes (lock files only). * perf(ane): bound compile-cache lock acquisition and harden staging delete flock(LOCK_EX) blocked forever when a suspended process held the entry lock, hanging every later load of that identifier. Take the lock non-blocking with a 30s deadline and fail open to the temp path on timeout. Resolve symlinks before the cache-root prefix check so a symlink under the cache root cannot redirect the staging delete outside it. * test(ane): assert bounded lock acquisition and symlink-resolved staging delete --------- Co-authored-by: jundot <jundot@users.noreply.github.com>
The reuse landed in jundot#2975 behind OMLX_QWEN35_ANE_COMPILE_CACHE, which nobody discovers. Add a toggle in the advanced cache group that exports the env var before the first ANE compile, so it applies on the next restart, and keep it off by default. An explicit env var still wins on the serve path.
A staging path under the cache root that resolves outside it fell through to the unlocked delete, which followed the symlink out of the cache. Skip those instead, and drop the entry's 0-byte lock file along with the entry it guards so one file per compiled procedure does not accumulate.
…hing A gdn_fraction under the bank floor makes _prepare_gdn_for_bank reject every layer, and the only symptom was gdn_layers=0, which reads the same as a compile failure. Warn at enable time with the model's actual floor, and move the floor rule next to the bank code so the tuner's grid clamp and this warning share one implementation. Based on jundot#2905 by finaltv971.
Bring README.ko.md up to date with README.md. Sections and assets now match 1:1 (32 headings, 11 images, 20 code blocks, 22 table rows) and every shell command is identical to the English version. Added (missing entirely): - "실험적 멀티 Mac 추론" section - Native custom kernel note blockquote (from jundot#2209) - --hf-endpoint example under CLI 설정 - MTPLX, mlx-serve, SiliconScope acknowledgments Corrected (stale vs English): - CLI memory flags: --max-model-memory / --max-process-memory are no longer registered in omlx/cli.py; replaced with --memory-guard and --memory-guard-gb - Requirements: Python 3.10+ -> 3.11-3.13, added macOS 15.0+ (Sequoia) - brew install omlx -> brew install jundot/omlx/omlx (both occurrences) - Tool calling streaming described the opposite behavior (buffer-then- send); now incremental emission with tool-call markup suppressed - Sparkle-based updater -> built-in auto-update - Custom kernels marked optional -> strongly recommended, added Qwen3.5 - Admin dashboard language list extended to match English - Fixed a stray space in the built-in chat paragraph Note: the python badge line duplicates a change in the stalled draft jundot#2574; this version also carries macOS 15.0+ and keeps M5 from jundot#2925. Other translations are intentionally untouched and left for contributor review.
# Conflicts: # omlx/_version.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
bash -n scripts/release_macos_dmg.shplutil -linton Info.plist and release entitlementsReleasesCheckerTests,UpdateInstallerTests)Required operator setup after merge
Create the protected
macos-releaseenvironment and add the five Apple secrets plusAPPLE_TEAM_IDdocumented indocs/release-public-checklist.md. The workflow will remain safely blocked until those values exist.