TASK: merge upstream v0.33.0 (ebf200f9) — mlxrunner prefix-cache robustness, Claude Desktop subsystem; pins unchanged - #217
Conversation
A prefill settles the drafter with the seed token after its last chunk, leveling the draft caches with the targets; a cancelled prefill returned before that, leaving the targets one token past the draft caches and the recorded keys. The next request then had to move every cache, and models with recurrent layers, which cannot rewind, fell back to the last snapshot: a retry after a client timeout lost up to a full snapshot interval of the prompt it had just evaluated. Settle with the next prompt token on the cancelled path too. The caches then rest level with the recorded keys, and a retry resumes exactly where the prefill stopped.
Page-in restores a path node by node and trusts each stored snapshot to cover its node's whole edge. A capture taken during prefill spans from the previous capture or the prefill base, which need not line up with the node it lands on: when a prefill resumes partway into cached history, a capture can reach back before its node's start, and a capture landing on a node that already has snapshots replaced them with a shorter span that page-in then could not serve. Clip each capture to its node's edge on attach, and keep the snapshots the node already has instead of replacing them.
When a session closes, every cache rests exactly at the end of the segment the trie is about to record. That is the one moment the segment's state can be captured for every layer, so close now pages the new segment out itself instead of recording it without snapshots and leaving the capture to a later path switch. Path switching then has nothing left to capture and only rewinds and pages in. The whole-state entry taken at close is released when the next request grows past the segment; sliding-window layers pay the same window copy a scheduled capture already costs.
…s survive resumed prefills A prefill that resumes partway into cached history — routine once client timeouts interrupt long prompts — used to attach its captures onto a node extended in place, so the stored snapshot spanned only the tokens the prefill evaluated while the node's edge reached further back. Restores walk node by node and trust each snapshot to cover its node's edge; the short snapshot stranded the caches at mismatched offsets and, on models with recurrent layers, ended up freeing all cache state — a request matching 46k of a 47k-token prompt reprocessed from zero. Growth now never extends a node underneath its snapshots. New tokens become a child node that carries exactly its own captures, and the path stays compressed because non-user segments merge back into their parent through the caches' snapshot Merge. Close already pages out what it records, so every merge combines adjacent covered snapshots and every stored snapshot spans exactly its node's edge.
A long prompt records restore points during prefill, but they only reached the prefix trie when the prefill completed; a cancelled request closed and released everything it had captured. Agent clients routinely cancel long prefills — their timeouts are shorter than the minutes a 40k-token prompt takes — so every retry started the whole prompt over and never got further than the timeout allowed, which presents as the model hanging forever. Closing a session now attaches every snapshot the prefill crossed, so a retry resumes from the last one and makes progress across timeouts. Scenario tests cover retries resuming exactly where a cancelled attempt stopped and cancellations on divergent conversation variants. Fixes ollama#17839
…ama#17918) Claude Code adds a "tokens left" system message after every tool result. Since ollama moves system messages to the front of the prompt, this breaks the KV cache on every request.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
* app: fix desktop interaction regressions * app: serialize settings reset updates
* app: improve desktop integration responsiveness * app: reconcile delayed Claude connection results * app: preserve delayed Claude action errors
Assessment for merging upstream tag v0.33.0 (ebf200f): 20 commits from base v0.32.15-6-g8f912415, dry-run merge-tree clean (zero conflicts), llama.cpp and MLX pins UNCHANGED so compat patches and preflight expectations carry without re-measurement. Substance is five mlxrunner prefix-cache robustness commits; semantic gate is the fork's mlx suite + vision goldens. Build/preflight deferred per operator instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…xrunner prefix-cache robustness; pins unchanged 20 commits from v0.32.15-6-g8f912415; zero conflicts, textual or otherwise, including against the qwen25vl cuBLAS gate and the preflight poison_probe already on main. LLAMA_CPP_VERSION stays b10488 and the MLX pin stays 27fec909 — every llama/compat patch and every preflight expectation (payload_pin 9d77fa172) carries unchanged. The five mlxrunner commits (prefill snapshots on cancel, trie-edge capture clipping, generated-token page-out, whole-child-node trie growth, draft-cache settling) overlap fork work only in pipeline.go and prefix_cache.go; the mlx suite + 12b/26b/31b vision goldens are the semantic gate before deploy. Stock 0.33.0 still carries the qwen2.5vl fp16-accumulate class (#216), so the gate stays load-bearing. SPEC H11 server_version remains the comparability boundary for benchmark cells measured on builds from this merge; vsuite's interim global GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 workaround retires when a build from this lineage deploys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewing as consolidator. I ran the check I asked for on #208 rather than asking again — it is clean. Independently, from a That matters because the acceptance criteria scope tests to None of the four listed packages would have caught it. This sync does not carry that class, and now there is a record saying so rather than an absence of evidence. Zero conflicts on +14072/-893 is worth a second look for the same reason — a clean auto-merge is exactly the condition under which a duplicated symbol slips through unexamined, because there is no conflict to draw the eye. The sweep above is the substitute for that attention. Two process notesCI is still in flight. Criterion 4 is the one not to let slide. The image build plus full CUDA preflight and On the MLX half
|
…pass) The targeted semantic gate for the two overlapping files ran on the merged tree without Metal: 244 x/mlxrunner tests pass incl. upstream's prefix_cache_scenario_test.go. Remaining Apple-host residue narrowed to the native-runtime SKIPs (TestDFlash* — the draft-cache-settling paths) and the vision goldens, bundled with the mlx-metal preflight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-off Picks up the upstream mlxrunner prefix-cache rework alongside this branch's pipeline.go changes; dry-run and actual merge both clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Criterion 4, CUDA half:
|
6 participants
The merge is done on this branch — zero conflicts — with the local test gates green; what remains before deploy is the Metal-host gates and the (deliberately deferred) image build + preflight. Full assessment:
docs/maxusai/tasks/upstream-sync-2026-08-26.md.Scope
Merge upstream release tag
v0.33.0(ebf200f9) intomain. Merge-base8f912415(v0.32.15+6); exactly 20 upstream commits (2026-08-21 → 08-25). Dry-run and actual merge both clean — zero conflicts, including against the freshly merged qwen25vl cuBLAS gate (#215) and the preflightpoison_probe.Why this fold is nearly free
Both payload pins are unchanged:
LLAMA_CPP_VERSIONstaysb10488/9d77fa172, MLX stays27fec909…. Everyllama/compat/patch carries unmodified (clean-room validated against this exact pin on 2026-08-26; 903 still required — llama.cpp#27044 remains open), every preflight expectation stays valid with zero re-measurement, and the fork-tag version lineage keeps matching thecuda-dynres-903profile.What's in it
x/mlxrunner/pipeline.go+prefix_cache.go(auto-merged).internal/proxy/claude_desktop*+app//cmd/launch/): new self-contained files, inert to the serving image.Stock 0.33.0 still carries the qwen2.5vl fp16-accumulate class (#216) — the gate stays load-bearing and
poison_probeasserts it on every preflight.Acceptance criteria
go test ./server/ ./model/renderers/ ./model/parsers/ ./llm/— all green (golang:1.26 container).mlx-metalpreflight — deferred until the rebuild is green-lit; vsuite's interim global-f32 workaround retires when this deploys.server_versioncomparability note.Follow-up owed after landing: re-merge
fix/mlx-thrash-check-default-off(touchespipeline.go, upstream's delta there is +4 lines).🤖 Generated with Claude Code