fix(run): make automatic compaction recover an over-limit conversation - #426
Open
dgruhin-hrizn wants to merge 1 commit into
Open
fix(run): make automatic compaction recover an over-limit conversation#426dgruhin-hrizn wants to merge 1 commit into
dgruhin-hrizn wants to merge 1 commit into
Conversation
Automatic compaction cannot do its job once a conversation crosses the
context window, so the conversation stays there permanently. Observed
against a 1M-token Anthropic window:
1. The summarize call replays the full history. It runs precisely
because that history is too large, so the request is itself over the
limit ("prompt is too long"), or it ends with an assistant/tool
message that Anthropic refuses as a prefill. Either way the run falls
back to the 12K truncated JSON summary, which discards the context.
In one trace the summarizer received 771 messages (2.78 MB) and
returned a single token.
2. The compaction check uses a 10K fixed reserve. The estimate trails
the provider's own count by the request context and provider-side
overhead that the message-tail estimate does not model; a 948K
estimate passed the check and Anthropic counted 1,017,628.
3. When the provider does refuse the prompt, the run retries the same
prompt eight times at 5s intervals and then fails. Nothing compacts.
Fixes, all in server/src/run:
- compaction_history trims the summarizer input to the context budget
at user-turn boundaries (never splitting a tool call from its
results) and guarantees it ends with a user message.
- context_budget keeps 10% of the window free instead of a fixed 10K,
so the reserve scales with the model and absorbs the drift.
- A provider refusal matching is_context_overflow compacts once and
retries the turn instead of failing it.
- 4xx responses other than 408/425/429 are terminal. A rejected request
fails identically every time, so retrying only delays the error.
Separately, Cursor can resume a finished turn whose checkpoint already
ends with the assistant, which Anthropic also rejects as a prefill.
run/history.rs appends a transient user tail to every provider request
that would otherwise end with the assistant. The tail is never
persisted, so committed checkpoints stay an exact prefix of the next
turn and the usage anchor still counts persisted messages only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Once a conversation crosses the model's context window, automatic compaction cannot bring it back, so the conversation is wedged permanently. Observed against a 1M-token Anthropic window over two days of logs:
prompt is too long), or it ends with an assistant/tool message that Anthropic refuses (This model does not support assistant message prefill). Either way the run falls back to the 12K truncated JSON summary, which discards the context. In one trace the summarizer received 771 messages (2.78 MB), returned a single token, and the fallback replaced the whole conversation.400 prompt is too longwas retried at 5s intervals (89 identical 400s in one day across two conversations), then the run failed. Nothing compacted.Fix
All in
server/src/run/:compaction.rscompaction_historytrims the summarizer input to the context budget at user-turn boundaries (never splitting a tool call from its results) and guarantees it ends with a user message.context_budgetkeeps 10% of the window free instead of a fixed 10K so the reserve scales with the model.is_context_overflowrecognizes the provider's over-limit refusal.engine.rsis_context_overflowcompacts once and retries the turn instead of failing it.auto_compactwraps the summarizer history incompaction_history.model_cycle.rshistory.rs(new)user_terminatedappends a transient user tail to any provider request that would otherwise end with the assistant.The last item also covers a separate failure: Cursor can resume a finished turn whose checkpoint already ends with the assistant, which Anthropic rejects as a prefill. The tail is provider-visible only and never persisted, so committed checkpoints stay an exact prefix of the next turn and the usage anchor still counts persisted messages only.
Tests
cargo test -p cursor-server --lib run::— 21 passed (new: proportional reserve boundaries, overflow-message recognition, trim/termination of the summarizer history, 4xx terminal classification)cargo test -p cursor-server --test compaction— 6 passed (new:provider_overflow_refusal_compacts_and_retries_once,assistant_terminated_history_is_sent_with_a_user_tail)--test prefix_stability --test checkpoint_recovery --test error_lifecycle --test interrupt— all greencargo fmt --all -- --checkclean