Skip to content

fix(run): make automatic compaction recover an over-limit conversation - #426

Open
dgruhin-hrizn wants to merge 1 commit into
leookun:mainfrom
dgruhin-hrizn:fix/compaction-recovery
Open

fix(run): make automatic compaction recover an over-limit conversation#426
dgruhin-hrizn wants to merge 1 commit into
leookun:mainfrom
dgruhin-hrizn:fix/compaction-recovery

Conversation

@dgruhin-hrizn

Copy link
Copy Markdown

Problem

Once a conversation crosses the model's context window, automatic compaction cannot bring it back, so the conversation is wedged permanently. Observed against a 1M-token Anthropic window over two days of logs:

  1. The summarize call replays the full history. It runs precisely because that history is too large, so the request is itself over the limit (prompt is too long), or it ends with an assistant/tool message that Anthropic refuses (This model does not support assistant message prefill). Either way the run falls back to the 12K truncated JSON summary, which discards the context. In one trace the summarizer received 771 messages (2.78 MB), returned a single token, and the fallback replaced the whole conversation.
  2. The compaction check uses a 10K fixed reserve. The estimate trails the provider's count by the request context and provider-side overhead that the message-tail estimate does not model. A 948K estimate passed the check; Anthropic counted 1,017,628.
  3. A provider refusal is retried eight times. Every 400 prompt is too long was retried at 5s intervals (89 identical 400s in one day across two conversations), then the run failed. Nothing compacted.

Fix

All in server/src/run/:

File Change
compaction.rs compaction_history trims the summarizer input to the context budget at user-turn boundaries (never splitting a tool call from its results) and guarantees it ends with a user message. context_budget keeps 10% of the window free instead of a fixed 10K so the reserve scales with the model. is_context_overflow recognizes the provider's over-limit refusal.
engine.rs A provider refusal matching is_context_overflow compacts once and retries the turn instead of failing it. auto_compact wraps the summarizer history in compaction_history.
model_cycle.rs 4xx other than 408/425/429 is terminal. A rejected request fails identically every time, so retrying only delays the error.
history.rs (new) user_terminated appends a transient user tail to any provider request that would otherwise end with the assistant.

The last item also covers a separate failure: Cursor can resume a finished turn whose checkpoint already ends with the assistant, which Anthropic rejects as a prefill. The tail is provider-visible only and never persisted, so committed checkpoints stay an exact prefix of the next turn and the usage anchor still counts persisted messages only.

Tests

  • cargo test -p cursor-server --lib run:: — 21 passed (new: proportional reserve boundaries, overflow-message recognition, trim/termination of the summarizer history, 4xx terminal classification)
  • cargo test -p cursor-server --test compaction — 6 passed (new: provider_overflow_refusal_compacts_and_retries_once, assistant_terminated_history_is_sent_with_a_user_tail)
  • --test prefix_stability --test checkpoint_recovery --test error_lifecycle --test interrupt — all green
  • cargo fmt --all -- --check clean

Automatic compaction cannot do its job once a conversation crosses the
context window, so the conversation stays there permanently. Observed
against a 1M-token Anthropic window:

1. The summarize call replays the full history. It runs precisely
   because that history is too large, so the request is itself over the
   limit ("prompt is too long"), or it ends with an assistant/tool
   message that Anthropic refuses as a prefill. Either way the run falls
   back to the 12K truncated JSON summary, which discards the context.
   In one trace the summarizer received 771 messages (2.78 MB) and
   returned a single token.

2. The compaction check uses a 10K fixed reserve. The estimate trails
   the provider's own count by the request context and provider-side
   overhead that the message-tail estimate does not model; a 948K
   estimate passed the check and Anthropic counted 1,017,628.

3. When the provider does refuse the prompt, the run retries the same
   prompt eight times at 5s intervals and then fails. Nothing compacts.

Fixes, all in server/src/run:

- compaction_history trims the summarizer input to the context budget
  at user-turn boundaries (never splitting a tool call from its
  results) and guarantees it ends with a user message.
- context_budget keeps 10% of the window free instead of a fixed 10K,
  so the reserve scales with the model and absorbs the drift.
- A provider refusal matching is_context_overflow compacts once and
  retries the turn instead of failing it.
- 4xx responses other than 408/425/429 are terminal. A rejected request
  fails identically every time, so retrying only delays the error.

Separately, Cursor can resume a finished turn whose checkpoint already
ends with the assistant, which Anthropic also rejects as a prefill.
run/history.rs appends a transient user tail to every provider request
that would otherwise end with the assistant. The tail is never
persisted, so committed checkpoints stay an exact prefix of the next
turn and the usage anchor still counts persisted messages only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant