QVAC-23752 feat[bc]: remove sliding-context support from the llm-addon

[yingying0906]

🎯 What problem does this PR solve?

• The addon could evict tokens from the middle of the KV cache once the context filled, and tracked firstMsgTokens so the system prompt was never what got dropped. It was opt-in through n_discarded and defaulted to off, so the shipped behaviour was already the non-sliding path while the slide machinery added state every other path had to keep correct.
• Reasoning-block compaction kept shifting the cache even with sliding gone: seq_rm the <think> span, then seq_add the tail down over it. That is context shifting under another name, so it goes too.

📝 How does it solve it?

Sliding removed. ContextSlider and ContextShifter are gone, along with firstMsgTokens / protectedPrefix, slideCapable admission, applySlide, supportsSliding and SequenceStepResult::discarded.

Overflow is now the single path n_discarded=0 already took. A prefill that does not fit throws ContextOverflow. A generation that fills the window stops with stopReason=contextOverflow and still returns what it produced, so a caller can tell a full context from a prediction-limit cutoff. Both prefill guards name the quantity they report.

Compaction replays instead of shifting. Every model rewinds to the end-of-prefill boundary and re-decodes what it keeps, which is what the recurrent path already did. Pure attention does not need that path's full-state snapshot: its cells are positionally indexed, so the boundary is just a position and rewinding is a tail trim. KvCacheOps.hpp/.cpp and compactKvRange are deleted, so no seq_add remains in the addon and the deferred K-shift never runs. Outcome::Kind collapses to one Compacted, and FailedKvIntact goes with both drivers' roll-back-to-pre-request-cursor recovery: compaction rewinds before it replays, so by the time anything can fail only a wipe leaves the cache coherent.

Compaction was pointer math over cells and is now a re-decode of the kept tokens, once per reasoning turn. Measured on Qwen3-0.6B on Metal that is 0.85 ms to replay 30 tokens and 2.2 ms to replay 71, so it scales like prefilling those tokens rather than like generating them.

A multi-token reasoning close marker works now. Replay seeds the close marker to keep the restored span balanced and used to seed a single token, so the policy refused markers that tokenise to several pieces. Survivable while pure attention used seq_rm + seq_add; not survivable once every model replays. The whole cached_close_tag_tokens sequence is seeded instead, and the refusal path is deleted with it.

An unfinished reasoning span no longer replays the tokens that opened it. The seeded prefix runs up to and including the open marker, and those pieces sit inside the range compaction drops. Nothing balances them when generation never reached the close, so replaying them rebuilt a <think> the next turn resumed from with nothing to close it. clipSeededPrefix caps the prefix at what sat before the span. Only pure attention could reach this: the interrupted-reasoning rollback covers recurrent and hybrid, and pure attention used to drop the whole span with seq_rm + seq_add, so it appeared when every model started replaying. A closed span is unaffected, its opener is balanced by the close marker the replay also seeds.

Compaction replay runs outside the scheduler mutex. finalizeTerminalDriver reaches compactThinkSpan, so on a reasoning turn the drain now runs a real llama_decode. stepLocked already drops the lock for the main decode and for media eval; drainFinishedLocked did not, so a replay stalled every co-tenant slot and blocked a cross-thread cancel() for its whole length. This window holds a reference into slots_ across the unlock, so the usual reconcile on every reacquisition is wrong here: a cancel recorded during it still passes slotOwnedByLocked and would run onCancel on a driver mid-finalize. TeardownDeferGuard suspends application for the window without dropping anything, and the worker applies the records at its next loop top where the ownership re-check discards the stale ones.

generatedTokens no longer comes from llama's perf counters. llama_perf_context keys on batch size, not meaning: n_queued_tokens == 1 bumps n_eval, larger bumps n_p_eval. Generation decodes singly, so the two agreed by coincidence until replay started decoding in batches. The stat is counted in the generation loops now, where a token is committed. On the single-prompt path that restores the old numbers rather than redefining them. The batch path used to count whatever the sampler returned, which included the sample that ended the sequence, so it now counts at the commit site too and reports one less. See Breaking Changes.

Batch path. advance marked a slot finished the moment currentPos hit maxTokensPerSequence, so it was filtered out before the driver's overflow check and ordinary generation reported sequenceLimit for a full window. That limit IS the slot's share of the context, and submit rejects any request that would not fit, so a full window is the one thing it can mean. It now says so.

Shared helpers. contextWindowFull(pos, ceiling) replaces five hand-rolled context-full checks that had drifted between >=, +1 >, raw llama_n_ctx and ctxCeiling(). SessionMetadata moves into LlmContext.hpp so the {nPast, nPast, cacheTokens, cacheTokens} layout has one home instead of three.

Session metadata keeps its four-slot width so files written by either build still load. Slots 1 and 3 are retired and this build's readers ignore them, but they are not written as 0: an older build reads slot 1 as its protected prefix and evicts from there, so a 0 would point that at position 0 and silently drop the system prompt and tool definitions. They mirror the live cursors instead, which drives that build's slide guard negative so it refuses the slide and reports a context overflow with the cache intact.

The SDK half is in #3999, and the two can land in either order: sdk and inference pin @qvac/llm-llamacpp@^0.45.0 from npm, and #3999 still parses the published wordings. A fabric bug found on the way is fixed in tetherto/qvac-fabric-llm.cpp#213; this PR does not depend on it, since it removes the addon's only route into that code.

🧪 How was it tested?

• C++ unit: 839 pass, 2 skipped for missing model fixtures. New coverage pins the compaction contract: a failed restore must not go on to replay, a failed replay wipes, and a span with no boundary is a clean no-op rather than a wipe. Also pinned: an unfinished span does not replay its opener, a second close marker is ignored, a sample that ends the sequence is not counted, and deferred cancels survive the finalize unlock window. Each of those was confirmed to fail with its fix reverted, since two tests this PR inherited passed only because they never set up the state production has.
• Desktop integration on Metal: reasoning 15/15 with 10 real compactions through the replay path, api-behavior 14/14, cache-state-machine 21/21, generation-params 2/2.
• api-behavior.test.js covers both prefill guards, including a cached follow-up that only the second guard can reject, and pins each guard's wording since the SDK parser matches them separately. test/integration/_context-overflow.js owns the tokenizer-sensitive sizing both files share.
• sliding-context.test.js and mrope-sliding-context.test.js deleted, with all three mobile registries updated so a run does not abort on an unregistered test.
• clang-format and clang-tidy clean on every changed addon/src translation unit.

💥 Breaking Changes

n_discarded is no longer consumed, so it reaches llama's own argument parser and fails model load as an unknown option.

BEFORE:

await model.load({ ctx_size: 2048, n_discarded: 256 })

AFTER:

await model.load({ ctx_size: 2048 })

A batched sequence that fills its window reports the full context instead of the per-sequence cap.

BEFORE:

// parallel: 4, predict: -1, slot window fills
stats.stopReason // 'sequenceLimit', same value a predict cutoff gives

AFTER:

stats.stopReason // 'contextOverflow'

A batched request reports one less generated token. The sample that ends a sequence is never decoded, so it never reached the cache and is no longer counted. This matches what the single-prompt path has always reported. TPS shifts with it.

BEFORE:

// parallel: 4, sequence stops on EOG after 64 committed tokens
stats.generatedTokens // 65, counting the EOG that was never decoded

AFTER:

stats.generatedTokens // 64

🔌 API Changes

contextSlides is gone from the runtime stats snapshot and from RuntimeStats in the type declarations.

const stats = inference.runtimeStats()
// stats.contextSlides no longer exists

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./.github/actions/release-merge-guard
• ./docs/website
• ./packages/ggml-coload-smoke
• ./packages/fabric/test/integration
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk/e2e
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/asr-ggml/benchmarks/server

[aegioscy]

The end − start arithmetic is the textbook llama hole-punch, but the range/branching around it can leave a hole without a shift, abort on M-RoPE after seq_rm already mutated cache, or OOM on the next decode's K-shift graph. Inline comments on the spots that need to change.

[aegioscy]

Please don't treat every out-of-range input as a silent NoOp.

endPos > nPast is a stale span, not an empty one. llama's own p1 = -1 means "to the end of the sequence"; if a caller ever passes that as nPast, this guard is true for any endPos >= 0 and compaction never runs. SWA / a cursor that lags seq_pos_max has the same shape: we refuse instead of shifting what is actually resident.

Requested changes:

• Cross-check nPast against llama_memory_seq_pos_max(mem, seqId) + 1 (or seq_add with p1 = -1 and then read newNPast back from seq_pos_max). Do not trust the software cursor alone.
• If endPos > nPast but startPos < nPast, clamp the hole to [startPos, nPast) rather than NoOp. Leave NoOp for empty/inverted/start < 0 only.
• Reject nPast < 0 explicitly so the llama -1 sentinel cannot fall through.

[yingying0906]

Update: removed in 68087763b, so this thread is moot.

Context shifting has been removed from reasoning-block compaction entirely. Every model now rewinds to the end-of-prefill boundary and replays the answer, which is what the recurrent path already did. So compactKvRange and KvCacheOps.hpp/.cpp are gone, along with the nPast < 0 reject, the endPos > nPast question and the seq_pos_max readback we were discussing here. Compaction now rewinds to the end-of-prefill boundary and re-decodes the kept tokens, which on pure attention is a tail trim and needs no cursor arithmetic at all.

Leaving the original reply below for the record.

Addressed in d0bdab79, bb2d1e15 and 6ba83952.

nPast < 0 is rejected now. llama uses -1 as the "to the end of the sequence" sentinel, so it would have made endPos > nPast true for every range and stopped compaction silently.

On not trusting the software cursor, you were right and I went with the readback. compactKvRange takes the new cursor from seq_pos_max after the shift instead of from nPast - discarded, and a disagreement means the shift did not land where the arithmetic said. So a cursor that has drifted from live memory is caught rather than propagated as a cursor no live cell backs.

I didn't add the pre-call seq_pos_max + 1 cross-check. The readback catches the same drift without needing a rule about what a healthy sequence looks like going in, and I didn't want a hard pre-check failing a cache that loaded with gaps. Tell me if you think the pre-check catches something the readback doesn't and I'll add it.

endPos > nPast is still NoOp rather than a clamp. ReasoningBlockCompactor::compact already does end = std::min(recordedEnd, pos) before it calls in, so a range past the cursor means the caller is inconsistent with itself, and a silent clamp in a primitive that owns no policy would hide that.

Worth saying, this function is byte identical to origin/main:ContextSlider.cpp:106. The PR only moved it, so none of it was new behaviour.

[aegioscy]

This is the part that looked funny, and it is not infallible.

llama_memory_seq_add is void, but:

• GGML_ASSERT(n_pos_per_embd == 1) — M-RoPE (Qwen2-VL / Qwen3-VL) aborts after seq_rm already punched the hole. Mtmd's attention path calls this.
• shared/other streams no-op seq_add while seq_rm succeeded → hole, no shift, newNPast still decreases.
• seq_add only updates cell pos / has_shift. K RoPE is rewritten later in llama_decode → init_update → build_graph_shift, which allocates an I32 tensor of size n_ctx * n_stream plus a RoPE pass over every K layer. Compact runs when the window is full of thinking; that graph is the OpenCL/GPU OOM. On alloc failure llama logs and does not reset_shift, so every later decode retries it.
• !get_can_shift() (M-RoPE, Step35) GGML_ABORTs on that next decode.

p1 = nPast also disagrees with llama's own slide examples, which use -1 so stray cells with pos >= nPast still move.

Requested changes, before seq_rm:

1. Refuse unless the memory module can shift (n_pos_per_embd == 1, get_can_shift()). Never punch a hole we cannot shift.
2. seq_add(..., endPos, /*p1=*/-1, -discarded) or assert nPast == seq_pos_max+1.
3. Apply init_update / K-shift here. If the shift graph fails to allocate, report MemoryOperationFailed — do not return Compacted with sticky has_shift and a lying newNPast.
4. Drop the "infallible" comment.

[yingying0906]

Update: removed in 68087763b, so this thread is moot.

Context shifting has been removed from reasoning-block compaction entirely. Every model now rewinds to the end-of-prefill boundary and replays the answer, which is what the recurrent path already did. seq_add no longer exists anywhere in the addon, so the can-shift guard, the seq_pos_max readback and the MemoryInconsistent kind all go with KvCacheOps.hpp/.cpp.

That also settles the K-shift graph point. Nothing in the addon schedules a shift any more, so has_shift is never set and the fabric bug can't be reached from here. It is a real bug on the fabric we pin, and it is fixed in tetherto/qvac-fabric-llm.cpp#213. Worth knowing if you review that one: making apply() return the failure was not enough on its own. llama_context::memory_update logged it and returned true anyway, and decode() discarded that, so the failure still never reached llama_decode. Both layers had the same shape, a bool where false meant both "nothing to do" and "it failed", so it now returns a three-state result instead.

Leaving the original reply below for the record.

Addressed in d0bdab79, bb2d1e15 and 6ba83952.

compactKvRange checks llama_memory_can_shift before seq_rm now, so no hole is opened that the shift can't close, and it reads seq_pos_max back after seq_add instead of trusting nPast - discarded. Shift passes p1 = -1 and the infallible comment is gone.

6ba83952 fixes a mistake in the first version of that readback. It reported MemoryOperationFailed, which maps to FailedKvIntact, but seq_rm has already run by then, so the hole is mid-cache and a tail trim can't reach it. There's a MemoryInconsistent kind now that maps to FailedKvWiped, and compact() wipes.

On the K-shift graph, this is a fabric bug and the addon can't reach it. llama_kv_cache_context::apply() at src/llama-kv-cache.cpp:2742 discards the bool from kv->update() and returns true, so llama_decode returns 0 even when the shift graph failed to allocate, and the decode runs with K unshifted. Silently wrong rather than an error. The failure path at :986 also skips reset_shift(), which only runs at :1005 on success, so has_shift sticks and every later decode retries. And there's no per-sequence reset: llama_kv_cells::rm at src/llama-kv-cells.h:251 clears only the per-cell shift, and whole-cache clear() is the one thing that clears the flag. So the addon can neither detect it nor clear it for one sequence. Your concern stands and this PR doesn't fix it. I'd rather file it against fabric, where apply() propagating the false is the first thing to fix, than work around it here.

Two premises don't match this fabric though. seq_add asserts on get_can_shift(), not n_pos_per_embd == 1, and get_can_shift at :1289 allows n_pos_per_embd == 4 under llama_kv_cache_uses_mrope_shift, so Qwen2-VL and Qwen3-VL don't abort, only LLM_ARCH_STEP35. And shared cells no-op seq_rm and seq_add together at :479 and :667, so no hole is left unshifted.

Four new tests cover the refusal, the readback and the wipe mapping. The fake models a resident sequence and fails the test if one isn't modelled, so a readback assertion can't become a rubber stamp. reasoning.test.js does 10 real compactions on Metal and all of them pass the readback.

[aegioscy]

compactKvRange has three outcomes. The fall-through below collapses NoOp and MemoryOperationFailed into FailedKvIntact, and the driver then removeLastNTokens of the entire request — answer included.

NoOp means the cache was not touched (empty/inverted/out-of-range). That is not a seq_rm rejection. A stale end > nPast in the primitive becomes a hard-fail wipe instead of a clamp.

Requested change: map Kind::NoOp → Outcome::Kind::NoOp, and only Kind::MemoryOperationFailed → FailedKvIntact. Please add a unit test that a primitive NoOp does not take the wipe path.

[yingying0906]

Update: removed in 68087763b.

Context shifting has been removed from reasoning-block compaction entirely. Every model now rewinds to the end-of-prefill boundary and replays the answer, which is what the recurrent path already did. With the shift gone there is no compactKvRange and no attentionOutcomeFor, so the NoOp mapping this thread was about no longer exists. Compaction now has a single path for every model.

Leaving the original reply below for the record.

Addressed in d0bdab79, bb2d1e15 and 6ba83952.

Kind::NoOp maps to Outcome::Kind::NoOp now. MemoryOperationFailed goes to FailedKvIntact, and MemoryInconsistent, the kind 6ba83952 adds for a shift whose removal ran but whose move didn't land, goes to FailedKvWiped since the cache has been written to by then.

For the regression test, I pulled the mapping out into ReasoningBlockCompactor::attentionOutcomeFor, a pure function, and compact() calls it. Three tests under ReasoningBlockCompactorOutcomeMapping pin all four kinds, including NoOpDoesNotBecomeTheWipePath for the one you asked about.

Reaching that branch through compact() isn't possible, setOpenSpan refuses start < 0, the degenerate check catches end <= start, and end = std::min(recordedEnd, pos) caps the rest. But that's a reason to make the mapping testable on its own, not a reason to skip the test.

[gianni-cor]

Inline notes from an automated-assisted review — five consolidation cleanups the refactor left behind. None are blocking bugs; each is a follow-up-sized change that removes a footgun or dead state.

[gianni-cor]

Context-full detection is now hand-rolled at five divergent sites. This check compares against raw llama_n_ctx, as do the seven guards in evalMessageWithTools (~456–471), while onLogitsReady (~1639) and both TextLlmContext sites compare against ctxCeiling(); the sites also mix >= with +1 > and their warning strings have already drifted ("context is full" vs "per-slot context is full"). The next boundary change (e.g. reserving EOT headroom, or a per-seq ceiling path reaching evalMessageWithTools, where perSeqCtxCeiling_ would currently be ignored) has to be applied at all five sites, and missing one yields an off-by-one overflow that manifests on only one model family or only in batch mode. A single shared contextWindowFull(pos, ceiling) helper on the LlmContext base would serve all five.

[yingying0906]

Done. Added contextWindowFull(pos, ceiling) next to exceedsContextWindow in SequenceDriver.hpp, and all five sites go through it.

No behaviour change today. evalMessageWithTools only runs on the single request context, built by the ctor that never sets perSeqCtxCeiling_, so ctxCeiling() there is already llama_n_ctx. It becomes correct if a per-slot ceiling ever reaches it, which was your point.

I left the two warning strings as they are. "context is full" and "per-slot context is full" are different paths and the wording is what tells them apart in a log.

[gianni-cor]

The on-disk 4-slot session-metadata layout {NPast, 0, CacheTokens, 0} is hand-encoded at three sites that this PR had to edit in lockstep. This anonymous-namespace SessionMetadata struct, TextLlmContext::saveCache, and MtmdLlmContext::saveCache (a verbatim copy) each independently encode the "slots 1 and 3 retired, write 0" contract, and TextLlmContext::loadCache hand-reads slots 0/2. Since the PR itself documents that a wrong slot-1 value makes an older build evict from position 0, a future writer that forgets one of these sites is a silent cache-corruption bug. Suggest hoisting a shared capture/apply next to SessionMetadataField in LlmContext.hpp so the layout is written and read in exactly one place.

[yingying0906]

Done. SessionMetadata moved to LlmContext.hpp next to SessionMetadataField, with capture and applyTo. CacheManager, both saveCache and TextLlmContext::loadCache all use it now, so the {nPast, 0, cacheTokens, 0} layout is written and read in one place.

I left mtmdSessionMetadataIsComplete alone. It's == 4 on purpose while the text path takes >= 4, so folding them would change behaviour.

[gianni-cor]

StopReason::LimitReached is dead — no producer remains. The comment itself says sequenceLimit is unreachable while nothing sets this, and grep confirms no production assignment is left. Yet the LimitReached → GenerationStopReason::SequenceLimit mapping in ContinuousBatchScheduler.cpp (~111–112) and the unit test LimitReachedPropagatesSequenceLimit in test_continuous_batch_finalize.cpp (~112–115) both survive, asserting behavior the program can never reach. The enum is in-memory only, so numbering stability buys nothing here (unlike the reserved error code 26, which is externally visible). Suggest deleting the enum value, the switch case, and the dead-path test — and dropping the now-unreachable "sequenceLimit" from the JS-facing stopReason union in index.d.ts alongside the SDK PR.

[yingying0906]

Done for the C++ side. Dropped the enum value, the SequenceLimit mapping in ContinuousBatchScheduler.cpp and LimitReachedPropagatesSequenceLimit.

I kept sequenceLimit in the JS union though. GenerationStopReason::SequenceLimit is ordinal 4, pinned by the static_assert in SequenceDriver.hpp and by the positional STOP_REASONS array in addon.js, so dropping it renumbers contextOverflow from 5 to 4. That's a real break for a string that only can't be produced any more.

[gianni-cor]

Outcome::keptPrefixEnd (declared at line 251) is write-only after this PR. Its production consumers were the firstMsgTokens_ / protectedPrefix_ adjustments deleted from both compactThinkSpan implementations. What remains is exactly: the declaration, the two assignments in ReasoningBlockCompactor.cpp (356, 496), and two EXPECT_EQs in test_reasoning_block_compactor.cpp (916, 1100) — no production reader. This doc comment was reworded to keep mentioning "the kept-prefix end" without the (now deleted) purpose. Dead state the feature removal should have taken with it: suggest dropping the field, both assignments, the two test assertions, and this mention.

[yingying0906]

Done. Dropped the field, both assignments, the two EXPECT_EQs and the doc mention.

[gianni-cor]

This overflow scenario is copy-pasted, and its sizing is tokenizer-sensitive. reasoning.test.js (~1024–1084) hard-codes the same recipe — 'word '.repeat(430) against a 512 window, the same three assertions (stopReason === 'contextOverflow', output survives, oversized prompt refused at prefill matching /context overflow/i) and the same recovery turn. The 430 must tokenize to just under 512 so the prompt is accepted at prefill but generation fills the window; a chat-template or model-pin bump can silently flip both copies into asserting the prefill-rejection path instead, and both then need the same recalibration (the darwin-x64 CPU fallback is already implemented differently in each file — file-wide useCpu here vs a per-test device override there). Suggest a shared helper in the test utils that owns the sizing and assertions.

Related duplication on the C++ side: the RejectingKvCacheOps fake is defined near-verbatim in both test/unit/test_cancel_rollback.cpp (1274) and test/unit/test_reasoning_block_compactor.cpp (861), and test_kv_cache_ops.cpp holds a third IKvCacheOps fake that subsumes both — one configurable fake in a shared test header would cover all three.

[yingying0906]

Done. The sizing and the assertions live in test/integration/_context-overflow.js now and both files call it, so a chat-template or model pin bump gets recalibrated once.

The three IKvCacheOps fakes are one configurable FakeKvCacheOps in test/unit/test_kv_cache_ops_fake.hpp, used by test_kv_cache_ops, test_cancel_rollback and test_reasoning_block_compactor.

Update after 68087763b. The _context-overflow.js half is unchanged. The fake consolidation survived the shift removal but changed shape: with compactKvRange gone there is no IKvCacheOps left to fake, so it is now one FakeReasoningRewindOps in test/unit/test_reasoning_rewind_fake.hpp, shared by test_reasoning_block_compactor and test_cancel_rollback. test_kv_cache_ops.cpp was deleted with the primitive.

[iancris]

The second prefill-overflow guard, whose message this PR rewrote, has no desktop integration coverage after the deletions.

TextLlmContext.cpp carries two prefill guards: :626 reports prompt tokens alone, and this one reports cached tokens %d plus prompt tokens %zu exceed the max context tokens %d. Nothing exercises the second — grepping test/ at this head, no file contains the string cached tokens.

The deleted sliding-context.test.js held two scenarios configured n_discarded: '0' — testing behaviour this PR keeps, not the feature it removes:

• :156 'Generation stops at the context boundary when sliding is disabled'
• :309 'Cached follow-up overflows when sliding is disabled and context is full'

The replacement at api-behavior.test.js:306-320 sends one oversized prompt, which trips the first guard — this PR's CI shows prompt tokens 4013, max context tokens 512, the prompt-alone wording. It also asserts only /context overflow/i (:318), which matches both guards, so it would not pin either wording. Also lost in the deletion: the stats.generatedTokens < n_predict assertion, which proved the prediction cap was not the stopper.

Impact: a change to this guard's wording or boundary would not fail desktop CI — while #3999's overflow parser is being taught to match this exact wording.

Suggested fix: prime a cacheKey with saveCacheToDisk, then send a follow-up that no longer fits. That reaches this guard rather than the prompt-alone one, because nPast_ is non-zero on the second turn:

t.ok(/cached tokens .* plus prompt tokens/.test(err.message))

Restoring the generatedTokens < predict assertion to the generation case would close the other half.

Distinct from the comment on api-behavior.test.js:287: that one is about the two overflow recipes duplicating each other; this is about the guard neither of them reaches.

[yingying0906]

You're right, nothing reached that guard. Added run | context full: cached follow-up is refused at prefill in api-behavior.test.js. It primes a cacheKey with saveCacheToDisk and then sends a small follow-up, which is your suggested shape. Local run lands on it:

[TextLlm] context overflow at batch prefill step: cached tokens 512 plus prompt tokens 17 exceed the max context tokens 512

Both overflow tests pin the wording of the guard they actually hit now instead of /context overflow/i, which matched either one. Also restored the generatedTokens < predict assertion.

[gianni-cor]

Inline reasoning-compaction follow-up.

[gianni-cor]

The fix should preserve the no-context-sliding direction, but compaction still needs a boundary before the reasoning span. Hybrid and pure-attention models should produce the same final cache: no reasoning body and no leftover <think> / </think> scaffold. Right now the shared boundary/replay path can restore from the end-of-prefill boundary, which is after a forced-open <think> suffix, so those opener tokens can survive the compaction. Suggested fix: make the compaction boundary semantically be spanStart for all model kinds, replay only the visible post-reasoning tail, and avoid seeding/replaying structural reasoning markers into the compacted cache. Hybrid/recurrent models can use a different implementation mechanism, such as a full-state snapshot before the span or a sanitized prompt replay, but the final cache should match pure attention. Please add coverage for forced-open pure-attention and hybrid/recurrent paths to verify the next cached turn does not resume inside reasoning.

[gianni-cor]

Inline pure-attention closed-span follow-up.

[gianni-cor]

Closed pure-attention spans have a related issue here: by removing the recurrent-only gate, pure-attention now seeds structural reasoning markers into the replay buffer too. Later clipPostReasoningTokens() preserves the seeded prefix, so a closed span can compact to a cache containing <think></think> plus the answer tail instead of removing the full reasoning scaffold. Recurrent/hybrid models may need structural replay to keep hidden state balanced, but pure-attention models should produce the same final logical cache without leftover reasoning markers. Suggested fix: split the replay policy by memory kind or by compacted-span semantics. For pure attention, trim from spanStart and replay only tokens outside the span, especially the visible answer tail; do not replay <think> / </think> marker seeds. Please add a closed-span pure-attention test that verifies the compacted cache contains the answer tail but no leftover reasoning scaffold.