Repository navigation
2.12.0 Flash-Next: SSD block-prefix restore installs recurrent state from a different conversation (row token_ids match ≥65536 tokens, answer cites facts absent from the prompt) #635
Description
On 2.12.0 (Qwen3.8 Flash-Next, Youssofal--Qwen3.8-Flash-Next-MTPLX-Optimized-Speed), an SSD-tier block-prefix restore produced a recurrent state that belonged to a different conversation. The restored row's token_ids matched the prompt for at least 65,536 tokens. The model then answered using facts that appear nowhere in the request, and lost the user's actual request.
This looks like the same failure family as #130 (fixed for SSD in 2.0.0) and #247 (in-memory bank). Here it happens on the SSD block_prefix path, with boundary_restore: true.
Setup
Mac Studio M3 Ultra, 256 GB. MTPLX 2.12.0 from PyPI, served by mtplx.server.openai.
Flags: --scheduler-mode serial --batching-preset latency --profile turbo --reasoning-mode auto --reasoning-effort medium --preserve-thinking auto --tool-prompt-mode hybrid --ssd-session-cache on --ssd-session-cache-max-size 80GB --ssd-session-cache-min-prefix-tokens 512.
The SSD bank had grown to about 82 GB, with 256 rows, over about 17 days of production traffic. One server restart happened the day before the failure.
Workload: an agent with a long shared prefix of about 61k tokens (system prompt and tool schemas). Each project then adds its own data block and a short user message. Prompts are about 65.5k tokens. Many different conversations share the first ~61k tokens and then diverge.
What happened
The request was a fixed test prompt of about 65,575 tokens, ending in a user message that asks the agent to move a task. /metrics reported:
session_restore_mode: ssd_block_prefix_clone
session_restore_served: {entry_prefix_len: 68126, requested_matched: 65536,
actual_restore_point: 65536, boundary_restore: true, storage_restore_mode: clone,
candidate_index: 1}
session_prompt_prefix_bank_commit: {cached_tokens: 65536, suffix_tokens: 37}
The answer was "No request in that message yet". The reasoning quoted a schedule span and a contract amount that occur 0 times in the request body; they belong to another project's data. Only 37 tokens were re-prefilled. Everything before token 65,536 came from the SSD row (entry_token_hash dffbf187…, prefix_len 68126).
Because the cold-tier lookup matches on real tokens (common_prefix_len against token_ids_json, floored to the block), the row's stored token_ids must have equalled our prompt up to at least 65,536. The state that was installed did not correspond to those tokens. The likely suspect is the GDN boundary record used at 65,536, or the state snapshot behind it.
A/B arms on the same server (same request body unless noted)
arm restore served result
SSD ssd_block_prefix_clone from row 68126 at 65536 ssd, boundary_restore wrong: "no request", foreign facts
RAM exact restores that descend from that bad SSD restore (clone, exact prefix) ram mostly wrong; the bad state propagates into the RAM bank via the next commit. Across this restore and its descendants: 1 of 7 correct
Same SSD row, prompt variant diverging earlier: ssd_block_prefix_boundary_clone at 63488 ssd correct
Cold prefill, clean RAM cache warmed fresh, block-prefix restores at 57344 / 63488 none / ram correct 14 of 14
Same body on a different engine (vLLM, same weights) n/a correct
After --ssd-session-cache off and a restart: 19 replays and RAM block-prefix/boundary restores ram correct, all of them
Thinking on/off, reasoning effort, the tool contract and our client code were each ruled out as causes. "Thinking off fixes it" only held because it took a cold path.
What I cannot give you, sorry
I deleted the 82 GB session-bank directory after switching the SSD tier off, so the offending row is gone. What I do have is the /metrics records of every arm above, and the exact request bodies, which contain private project data and can't be shared publicly. I can run instrumented builds or env flags on this machine if you tell me what to capture next time.
Side note on diagnostics
divergence_at_token in the /metrics prefix diagnostic is computed in engine_session.py::_prefix_diagnostic against the nearest live engine session (best_session_id). It is not computed against the SessionBank entry that was actually restored, while actual_restore_point comes from the bank. I built an alarm on actual_restore_point > divergence_at_token and it produced false positives. One example is three identical 65,968-token prompts restored at 65,792 (the block floor). It would help to also expose the true common prefix between the prompt and the served entry (common_prefix_tokens of the served candidate) in session_restore_served. Then a client could check "restore point ≤ real match" directly.
Current mitigation
--ssd-session-cache off. The RAM bank alone has shown no contamination since (all arms correct). We only lose the warm-after-restart benefit; the shared prefix re-enters RAM after one cold request of about 92 s.
Questions:
Is there a known path where an SSD row's GDN boundary records, or its snapshot, can be inherited from or written for a different token sequence? Candidates: prefix_donor inheritance, supersede/collapse of contained prefixes, deferred cold encode re-dispatch after a foreground preemption, or boundary thinning.
Do 2.12.1 or 2.12.2 change anything here?
Activity
bjornnelson76 commentedon Oct 15, 2026
Related evidence from a production deployment of the same disk tier, from an adjacent failure class. Two things our logs add to the #635 picture: the same corruption signature at load on a different workload shape, and what looks like a second failure mode — cross-session entry sharing — which a fix scoped to the restore path alone may not close.
Our setup
turboquant_disk_kv/m32_disk_kv/m32_gpu_kv/session_prefix_kv/turbo_kvall present inmtplx.log), serving continuously since 2026-09-301. Same corruption signature at load, on a different workload
Our wedge-repro probe is the single ~429-message conversation above. Against the disk-enabled stack it wedges at 0 tokens, 180 s timeout — every time (3 reproduced attempts over two weeks: 09-30, 10-06, 10-14). The log shows restore events oscillating between "everything" and "nothing" for the same content:
We also have 46
session_prefix (dirty)source entries with zero correspondingstored/insertlines ever logged — a cache tier that was polluted, never cleanly populated.A/B: flipping the disk tier
write → read_only(plus server restart) made the identical probe complete cleanly — 656 tokens, exit 0, ~120 s — the first clean pass in two weeks. Honest caveat so nobody re-tests on a stale bank: the observed fix was disk off + restart, i.e. it removes the shared disk pool; it is not evidence that the restore path itself was repaired.2. Cross-session entry sharing (beyond what #635 shows)
In one 9-second window — 2026-10-13 03:20:33–03:20:42 UTC — four concurrent session lanes were served by the same entry IDs (
68820/68819/68867) mid-flight:Entry
68820was created on 10-12 17:10 inside a different session's lane (a8181832), then reused by two other sessions the next day.Correlation with output corruption: two of our three output-corruption incidents — fabricated claims that looked like normal prose — were generated inside clusters where entry IDs were shared across sessions (this 9-second cluster, and a 09-30 02:25–02:35 UTC disk/prefix-restore cluster). Every window where a session held its entries exclusively was clean. This suggests the entry key ignores session identity under concurrency — state corruption via cross-contamination, not just truncated restore — so a fix scoped to the restore path alone might not close it.
Offer
Happy to run any instrumentation build against the probe workload: it wedges deterministically on the disk path and passes deterministically with the disk tier disabled. Full
mtplx.log(2026-09-28 → 10-14) and the quarantined session bank are preserved and available as evidence.goodbadwolf commentedon Oct 20, 2026
I found a cause that fits this report, and I opened #650 with a one-line fix. I reproduced it on 2.12.2 with the real class and a real stored entry, and end to end on one machine. I did not test 2.12.0.
On Flash-Next the 12 full-attention layers use
QSACache: the keys and values, a raw indexer key stream, and a pooled table of block keys. A sub-prefix restore from the SSD tier goes throughdecode_payload_prefix, which cuts the keys and values to the restore point. The raw and pooled tensors cannot be cut there, because the encoder only splits a tensor into blocks whenshape[2] >= 2 * block_size, and these are(1, T, 128)and(1, T/4, 128). They arrive at the donor entry's full length. TheQSACache.statesetter then setspooled_len = pooled.shape[1], sopooled_lenis the donor's block count, notoffset // ratio.On the next forward,
QSAIndexer._extend_pooledcomputesnb_old = min(cache.pooled_len, nb_total). With the donor's count,nb_old == nb_totaland no pooled block is computed for the new tokens, so selection scores against the donor conversation's block keys. On a stored 58,073-token entry restored at 2,048, the old setter gavepooled_len=14518and the next 4,096-token chunk recomputed 0 blocks where 1,024 were needed. The RAM path loads the full state and callstrim, which clampspooled_len, so it never shows this.That matches the arms in this report: the SSD
block_prefixrestore is wrong and a cold prefill or a RAM restore is right. It also explains the propagation. The state getter emitspooled[:, :min(pooled_len, offset // ratio)], so a restored cache stores the donor's table again on the next commit, and descendants keep it. When the donor's text is nearly the same as the new prompt, the stale blocks are nearly right and nothing visibly breaks, which is why a restore can look fine for weeks and then fail on a different conversation.The fix clamps
pooled_lento the restored keys in the setter, which is a no-op for a consistent state.Evidence, 2026-10-20, M5 Max 128 GB,
Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speedat34ae535f: with the RAM bank cleared before each run and thinking on at temperature 0, an SSD boundary restore at 8,192 tokens from an entry holding a different conversation found the task 5 of 5 times on the fixed build. On v2.12.2 it answered that there was no task 2 of 2 times, and the same request withx-mtplx-cache-mode: bypassfound the task.Not examined: the cross-session entry sharing in the second comment, and cache families other than QSA. Entries written before the fix carry pooled tables that were never recomputed, so they should be dropped when upgrading.