Skip to content

Last-resort tool-response truncation is O(candidates x rows) on the journal-backed send path #3857

Description

@acoliver

Problem

On the journal-fed source send route from #854 (PR #3852), the last stage of the context enforcement ladder is truncateSourceToolResponses → truncateOversizedToolResponsesUnified (toolResultTruncator.ts). After every single tool-response replacement it re-projects the whole candidate from disk and writes the journal. With n rows and about n/3 tool-response candidates, a send that reaches this stage does O(n²) work.

It was measured in a test fixture during #854 CI work (tmp bench, not committed). With a 1M-token limit and mostly uncompressible 2 KiB text rows, the send stalled:

  • 1,024 rows took 520 ms.
  • 2,048 rows were still running at 180 s.
  • 8,192 rows never reached provider preparation.

Under the limit, the same send path is linear: about 1.3 s at 8,192 rows.

This stage only runs after compression and the fallback stages have failed to bring the request under the limit, so most sessions never reach it. When a session does reach it, the turn appears to hang instead of failing fast with a context-overflow error.

Proposal

  • Decide all replacements in one pass, then publish a single atomic change. Use a disk-backed running estimate per row, so the loop does not re-project after each replacement.
  • Alternatively, cap the stage's work. When truncating tool responses cannot reach the limit, because the remaining rows are not tool responses, fail fast with ContextOverflowError before doing the per-candidate loop.

Acceptance

  • A counting-seam test shows that projections and journal writes per send in this stage do not grow quadratically with the row count.
  • An over-limit fixture at 8,192 rows either finishes or fails with ContextOverflowError within seconds.
  • Request bytes are unchanged for cases where truncation succeeds.

Follow-up from #854 / PR #3852.

Activity

  1. github-actions commented on Oct 11, 2026

    @github-actions
    Contributor

    LLxprt Issue Planner — infrastructure failure

    The autonomous planner failed with exit code 1. Please inspect the workflow logs (Run planner agent section) and re-run once resolved.

  2. added theissue type on Oct 13, 2026
  3. added this to the 0.12.0 milestone on Oct 13, 2026
  4. added a commit that references this issue on Oct 14, 2026
    ba2947e
  5. acoliver commented on Oct 14, 2026

    @acoliver
    CollaboratorAuthor

    Fixed in PR #3852 (ba2947e). The last-resort stage now ranks and decides every replacement in one pass over a disk-backed running estimate and publishes them as a single change, instead of re-projecting the candidate and writing the journal after each replacement. If shedding every tool response cannot bring the request under the limit, it fails fast with ContextOverflowError before doing any work.

    Measured on the same fixture (mostly 2 KiB text rows, tool-response truncation reached):

    Rows Before After
    1,024 722 s 84 ms
    2,048 did not finish 227 ms
    8,192 did not finish 537 ms

    Checks: a counting-seam test shows row reads and writes grow about 4x for 4x the rows (it failed on the old code, 140 s). An 8,192-row over-limit send fails with ContextOverflowError in about 8.5 s without rewriting the journal. A differential corpus shows the history, pending rows and final token count equal the old per-replacement stage whenever truncation succeeds.

  6. moved this from Todo to In Progress in LLxprt Roadmapon Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Context ManagementIssues related to managing the context sent to the model.

    Type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions