feat(backend): chat context compaction

[MesoX]

Chat context compaction: keep chats alive past the model's context window

Design doc: docs/adr/0012-context-compaction.md (ADR-0012)

Problem

Chats hard-fail the moment their message history exceeds a model's context window. The AI SDK (ai@6) reports real token usage after each call, but it exposes no context-window metadata on the model interface and no pre-call tokenizer. Providers also disagree on whether the window is discoverable at all: Google, OpenRouter, and vLLM expose it via their APIs, while OpenAI, Anthropic, and Bedrock do not. On top of that, the existing error handling only covered auth, rate-limit, and 5xx failures, so a context-overflow rejection killed the turn outright.

What this PR does

This change adds a two-tier, view-not-delete compaction model. It is fed by a single token estimator, all of its durable state is mutated through a single versioned compare-and-swap (CAS) writer, an always-on recovery net catches the overflow errors that the proactive path misses, and a deterministic context-editing pass prunes stale, bulky tool results without ever calling a model. Because top-level chats and sub-agents both run through the shared agent-runner/ToolLoopAgent, one implementation covers both.

The full rationale — including every alternative we rejected and why — lives in ADR-0012. The key decisions, summarized from it, are below.

Load-bearing principles

• View, not delete. The watermark and summary change what is sent to the model, never what is stored. Raw messages persist in the database untouched, so compaction is non-destructive in the data sense: a user can still read the full history, and a future "expand summary" UI comes for free. This reduces the "irreversible data loss" objection to a matter of UX courtesy rather than correctness. A summarized message is never hard-deleted.
• One estimator. Token counting lives in exactly one function operating over one neutral structure (CountUnit[]). Both Tier 1 (UIMessages) and Tier 2 (ModelMessages) normalize into it, and both count only model-bound parts (text, tool-call, tool-result, file, image). Divergence between the two tiers is therefore impossible by construction, because a tier cannot fire on a number the other tier never sees.
• One durable writer. Every mutation of compaction state (summaryWatermark, contextSummary, compactionDirty) goes through a single compare-and-swap keyed on a version column. Concurrency — for example a trigger run racing a user run — and the interaction between compaction and history-edit invalidation are both resolved by version rather than by comparing watermark values. That matters because a watermark can move backward on a history edit, and a value-based comparison would misread that as "not yet advanced" and write a stale summary over mutated history. On a CAS conflict the loser re-reads the row: if the winner already covered its prefix it skips, otherwise it retries once and then skips with a contended warning. There is no recompute loop and no livelock.
• Recovery is the net. When a 400/413 overflow error is caught, the messages are aggressively trimmed in memory (through the same Tier 2 adapter, so there is no bespoke trimmer) and the call is retried exactly once. Recovery flags compactionDirty and leaves the durable compaction to the next turn. Crucially, recovery stays on even when proactive compaction is globally disabled — it is the last line of defense, not a risk surface.

Mechanisms

• Window resolution. resolveContextWindow(provider, modelId) resolves a window per model, in order: a manual override (provider.modelMeta), then provider API auto-detection (Google / OpenRouter / vLLM), then the community-maintained litellm registry JSON (which covers OpenAI / Anthropic / Bedrock), and finally a conservative 8192 default. We deliberately do not maintain our own context-window table. Key normalization is boundary-safe, results are cached per provider+model with a TTL, the cache is evicted immediately on a modelMeta edit, default results use a short TTL, and API fetches use a 5 s timeout behind a single-flight guard.
• Token estimation. The estimator uses char/4 over text parts only (it never runs char/4 over a base64 image) and a modality table to size non-text parts. It is used only on the first turn, before any provider usage exists; every later turn uses the real, provider-reported usage.inputTokens. We accept the first-turn imprecision — guarded by a 1.15 cold-start margin and the recovery net — rather than ship a per-provider tokenizer.
• Tier 1 — cross-turn (durable). This runs in prepareChatTurn over the durable history. Its budget math works off the input budget (window − maxOutputReserve − safetyReserve), with a trigger ratio of 0.8 and a target of 0.5. The two ratios are deliberately distinct to give hysteresis, so compaction does not re-fire on the very next turn. The work is staged cheap-first: Stage 0 context-edits (pruning bulky old tool results with no model call), Stage 1 prunes the older prefix, and Stage 2 summarizes the prefix into a single synthetic message only if the conversation is still over target. Tool-call/result pairs stay atomic across the keep boundary, summarization is incremental (only the messages after the watermark), and a visible context-compacted event keeps the behavior fail-loud.
• Tier 2 — intra-turn (in-memory). This handles a single heavy response whose own tool loop bloats the window mid-loop. It runs in the SDK prepareStep hook on both streamText and generateText, summarizing old completed tool results while keeping recent steps verbatim and preserving call/result pairing. It fires only when genuinely near the limit, and it is not persisted — the next turn's Tier 1 folds the result into the durable summary.
• Sub-agents. Sub-agents start fresh on each invocation and carry no cross-turn history, so they use Tier 2 only. Each one resolves its own model's window and output limits, and recovery covers them as well through the shared runner.
• Config and kill switch. Compaction behavior is global (DEFAULT_COMPACTION_CONFIG); only the window/output size is per model. Setting COMPACTION_ENABLED=false disables all proactive compaction in production without a deploy, and recovery ignores that flag. Per-agent tuning was shipped and then removed, because no surveyed tool exposes it and the ratios already self-normalize to the model window.

Frontend

• Context-usage ring. A small ring next to the model selector fills as usedTokens / contextWindow, ramping from green to amber (≥ 0.7) to red (≥ 0.9), and rendering a neutral grey when the window is unknown or defaulted rather than guessing a ramp. The window comes from the currently selected model, and the numerator is the last response's peak contextTokens.
• Per-message stats. An (i) popover under each assistant response shows input/output tokens, time-to-first-token, and total generation time, reusing the existing tool-call timing mechanism.
• Force-compact on demand. The ring is clickable: POST /chats/:id/compact runs Tier 1 once regardless of the threshold and returns the post-compact usage so the ring refreshes immediately. A click is deferred while a response is streaming, and a confirmation dialog appears only when the drop would be significant.
• Compaction trace. Compaction is surfaced as a synthetic compact_context tool-call/result pair that reuses the tool-call UI. The trace part is stripped before convertToModelMessages, so it never replays to the provider as a phantom tool call.

Notable rejected alternatives (full list in ADR §Considered Options)

• Single-tier compaction (cross-turn only). Rejected because it cannot rescue a single response whose own tool loop overflows the window.
• Hard-deleting or truncating old messages. Rejected because it is irreversible and reproduces the "silently drops the middle" failure mode seen in gateway truncation. View-not-delete keeps the data and makes the action auditable.
• A homegrown context-window lookup table. Rejected because it is unmaintainable across providers; the litellm registry is the industry's "don't maintain your own table" answer.
• A real pre-call tokenizer. Rejected for v1 because it is a heavy per-provider dependency for a number the provider returns accurately after the very first call.
• Optimistic concurrency by comparing watermark values. Rejected because it breaks when invalidation moves the watermark backward; versioned CAS removes that monotonicity assumption.
• Compacting only to the trigger threshold. Rejected because it re-fires every turn (the thrash failure mode); distinct trigger and target ratios are what provide hysteresis.
• A token floor on the trigger (max(window × pct, 64000)). Rejected because it overflows sub-64k models, where the trigger would never fire. A floor belongs only on the window fallback, never on the trigger.

Schema changes (additive, nullable/defaulted)

• provider.modelMeta (JSONB) — per-model window/output overrides.
• chat/run gain contextSummary, summaryWatermark, compactionDirty, and version.
• Migration: apps/backend/drizzle/0046_context_compaction.sql.

Rollout is lazy and there is no backfill: existing chats compact only on their next turn. An eager backfill would create a thundering herd of summarize calls.

Cross-tenant safety

The submit route verifies that the body id belongs to the caller's workspace before a run starts. The compaction store is keyed by chat id alone, so without that check one workspace could mutate another's summary and watermark.

Observability

The feature emits structured metric:-tagged log lines: compaction.fired, summarize.latency_ms, recovery.*, context_window.fell_to_default, litellm.key_miss, cas.conflict, and context_edited.

Config

New apps/backend/.env.example keys, all optional with built-in defaults: COMPACTION_ENABLED, COMPACTION_TRIGGER_RATIO, COMPACTION_TARGET_RATIO, COMPACTION_RESERVE_RATIO, COMPACTION_KEEP_RECENT, COMPACTION_MIN_PRUNABLE_CHARS, and COMPACTION_MIN_RECENT_PRUNABLE_CHARS.

Tests

New suites cover compaction.test.ts, context-window.test.ts, token-estimate.test.ts, and recovery.test.ts; existing suites for agent-runner.test.ts, chat.test.ts, chat-execution.test.ts, sub-agent.test.ts, and packages/schemas/index.test.ts were extended. Recovery's per-provider overflow regex is fixture-tested across OpenAI/vLLM, Anthropic, Google, and Bedrock.

Known limitations (ADR §Open/deferred)

• A single oversized newest result. A tool result too large to fit even as the newest message still hard-errors; the fix is an ingestion cap at storage time, which is out of scope here.
• A few items are consciously deferred and are not needed for the "no more hard fails" goal: a live Pending → Running compaction trace (it currently renders post-hoc only), the estimate-vs-real divergence metric (log-only for now), and content-type tool-result media accounting.

🤖 Generated with Claude Code

[MesoX]

Hi @willdady, after few weeks of usage I have noticed few small things which are fixed in latest commit in this branch. Overal implementation is heavily inspired by industry leaders - Anthropic, Hermes Agent and others,

I have also played around with Fable and let it do a thorough review of the process itself and most findings were rather cosmetic ones, so I am pretty confident we are near production quality.

There sure are things which can be expanded later (like handling precalculations for attached files), but I mainly wanted to prepare solid base with two tiered compaction in place which greatly helps for smaller LLMs with lower context.

Thanks.

[willdady]

Thanks for this, @MesoX. It's a serious piece of work and the ADR is genuinely good to read. The two-tier view-not-delete model is the right call, the versioned CAS writer holds up, tool-call/result pairing stays intact across the keep boundary, and the tests actually exercise the logic instead of stubbing it out. The /compact endpoint scopes by workspace correctly, and the data.id check on submit closes a real gap that exists on main today. So no complaints about the craft.

I'm hesitant to merge yet, though, mostly over one schema decision.

modelMeta vs the per-model config we already have. Since you opened this, main swapped modelIds: string[] for modelConfigSchema ({ id, passthroughFileTypes }), and that object's comment already calls itself the home for this kind of metadata: "the intended home for future per-model metadata (e.g. max input/output tokens)". Your PR adds modelMeta as a separate JSONB column keyed by model id instead, so we'd have two per-model config structures shaped differently (an array of objects, and a record) with nothing keeping them in sync. A model can have a modelMeta entry but not be in modelIds, or drop out of modelIds and leave its modelMeta entry orphaned. dedupeModelConfigs only touches modelIds.

Can we put contextWindow? and maxOutputTokens? on modelConfigSchema, right next to passthroughFileTypes where operators already set per-model behaviour, and drop the modelMeta column? That kills the drift problem and follows the direction the code already documents. It's a migration, so it's a lot cheaper to settle before the rest lands on top of it.

Tier 2 prepareStep. I want to flag this rather than assert it, since I was reading against an older SDK than the ai@7 we're on now. My concern: the messages you return from prepareStep build that one step's prompt, but if the SDK doesn't fold them back into the running history, every later step rebuilds from the untrimmed messages, and summarize() (a real model call) fires again on each step with the result thrown away. Not an infinite loop, but on a long tool loop it'd be a quiet cost multiplier. Could you confirm how this behaves on v7? A test over a multi-step loop asserting summarize runs once would settle it.

A few smaller things:

• defaultHttpGetJson in the window auto-detect has a 5s timeout but no size cap, and it hits a provider-configured baseUrl. A misbehaving gateway could stream a huge body inside the timeout and blow up memory. Worth bounding.
• The API-reported window is accepted on if (contextWindow), so a -1 or a float would flow into the budget math. > 0 && Number.isFinite would be safer.
• The PR text says window resolution falls back to a remotely-fetched litellm registry, but the code vendors it as static data with no fetch. That's the safer choice, just worth fixing the description so nobody goes looking for a network dependency that isn't there.
• The overflow regex in recovery.ts has some broad alternates (too many tokens, request_too_large) that could catch an unrelated 400 and trigger a needless trim and retry. Also only 400/413 are handled; a few gateways use 422.

Last thing, and it's about process, not the code: for something this load-bearing I'd rather have hashed out the design in an issue first, mainly so the modelMeta question above got answered before 14k lines were built on it. Can we start there? One way to unblock the rest: split the per-model window config and resolver into their own smaller PR, get the schema shape agreed, then stack the compaction work on top. Happy to pair on the modelMeta change if it helps.

[willdady]

@MesoX please let me know when this is ready for another review 👍

[MesoX]

Hi @willdady,

Thanks for the review.

I'll update it accordingly - just give me a little time to test all the changes.

On the last part: I completely agree, and I've since started raising things as issues up front, because it does feel like the better direction. In this case it initially looked like there wasn't that much code involved - at least from the early findings. Splitting the two tiers also would have felt like shipping an incomplete feature: I try to deliver something MVP-like, and one tier without the other is more of a mess than something useful. Then a lot of small issues cropped up here and there once our users started testing it.

On the split specifically: I'll ship a smaller PR first for the per-model window configuration (the schema shape + resolver + UI), get that shape agreed and merged, then rebase this PR on top of it. That should let us settle the modelMeta → modelConfig question in isolation before the rest lands on it.

On the litellm registry: you're right on both counts - the PR text is wrong (it claims the window falls back to a remotely-fetched registry, but the code vendors it as static data - it originated from LiteLLM registry and I worded it incorrectly), and vendored-static is the safer runtime choice. I'll fix the description regardless.

The one thing I'd flag before we lock it in: the static file is hand-written and only covers ~120 models, while litellm's own list is 300+ and moving. Left as-is it quietly drifts out of date, and a model that's missing silently falls to the conservative 8192 default. So I'd like to keep it static at runtime but stop hand-maintaining it. A few ways to do that:

Build-time generated vendor (my preference). A small script (pnpm update:litellm-registry) fetches litellm's official model_prices_and_context_window.json (MIT-licensed, ~1.4 MB), prunes each entry to {max_input_tokens, max_output_tokens, max_tokens}, and writes a checked-in JSON. Runtime stays exactly as it is now — static import, no network — but refreshing the data is one command (or a periodic CI job that opens a PR) instead of hand-editing.
Runtime fetch + cache, with the vendored file as fallback. Always-latest, but adds a startup egress dependency to GitHub — I don't think it's worth it given your (correct) preference for no runtime fetch.
Drop the registry entirely (close second for me). Now that the per-model window is a real UI field, we could require operators to set it explicitly for providers that can't auto-detect (OpenAI/Anthropic/Bedrock) and fall straight to the default otherwise.
My honest reasoning on (3): if we commit to the principle that an admin should know which model and provider they're connecting, then we don't strictly need a registry at all. But since a well-maintained, community-driven registry already exists and is trivially reachable, it feels daft not to lean on it — which is the only thing that edges (1) ahead of (3) for me. Happy to go either way. What's your call?