Skip to content

runtime: record image omission as a durable projection transition #4569

Description

@Astro-Han

Last piece of #4283. Everything else that issue asked for is on main; this is the one lossy edit to model-visible history that still lives outside the ledger.

Problem

When a provider rejects a mid-turn request as too long, the runtime drops historical image Tool Results from the request and retries once. The set it dropped is kept only in memory:

  • packages/runtime/src/ai-sdk-compaction.ts: the ContextLength branch builds state.omittedImageToolResults on the mid-turn state, and projectHistoricalImageOmissions re-applies that map to every later request in the same turn.
  • packages/runtime/src/provider-image-overflow-recovery.ts: collectHistoricalImageToolResults and omitHistoricalImageToolResults do the selection and the rewrite.

Nothing about the omission is written to the operational ledger. On the next Turn, after a restart, or on a branch, the effective-history reducer rebuilds from the durable projections, the images come back, and the request either overflows again (one more rejected round trip and one more fold) or goes through with content the model had already lost. Either way the prompt-cache prefix the previous request established is gone.

The invariant #4283 was opened for:

A successful model-visible history is append-only. Any lossy change to already-visible history must first become a durable successor in the append-only operational ledger, and no later replay, compaction, branch, or restart may restore the replaced form.

Tool Result archiving already honours it: packages/runtime/src/tool-result-archive-transition.ts builds a ModelProjectionTransition (@maka/core/model-projection-transition) whose replacement is the archived placeholder projection, records it through recordTransition, and lets the reducer apply it everywhere. Image omission is the same shape of change and should use the same record.

Change

  • Each omitted image Tool Result becomes one ModelProjectionTransition against its function_response event, with a replacement projection that carries the omission text omissionText already produces, recorded through the same recordTransition the archive path uses.
  • The reducer applies those transitions like any other, so live continuation, the next Turn, restart and branch all see the omitted form. state.omittedImageToolResults and projectHistoricalImageOmissions go away; the mid-turn re-application falls out of the reducer.
  • Selection stays all-at-once. fix(runtime): let the provider decide fit, anchor estimates on real usage, retire the local verdict #4486 bills every materialized image at one constant, so there is no per-image cost to order by. Do not add one.

Acceptance

  • After a ContextLength rejection that omits images, the ledger holds one transition per omitted Tool Result, and the retried request matches what the reducer now produces.
  • The next Turn, a restart, and a branch from that Session send the omitted form; the image does not come back.
  • No new in-memory map of omissions on the turn state.
  • Existing archive transitions and their reducer tests keep passing; packages/runtime test:dist covers it.

Sequencing

Land this after #4559. That change rewrites the tail and overflow path in ai-sdk-compaction.ts, and a transition-based omission written against today's code would conflict with it. Rebase on whatever #4559 leaves.

Out of scope

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions