Repository navigation
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 10 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.21.1 · 1m 34s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 21 model calls · 88.5k tokens (84% cached) · $0.0056 All 33 tests in 5 files
e2e 0.21.1 · 2m 58s · web · run artifacts |
Contributor
🟢 e2e ios: 53 passed, 1 skippedAll 54 tests in 7 files
e2e 0.21.1 · 5m 52s · ios-simulator · run artifacts |
Contributor
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 33 model calls · 162.8k tokens (86% cached) · $0.0094 All 20 tests in 1 file
e2e 0.21.1 · 7m 19s · android-emulator · run artifacts |
Contributor
🟢 e2e android: 55 passed, 1 skippedAll 56 tests in 7 files
e2e 0.21.1 · 8m 13s · android-emulator · run artifacts |
Contributor
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 10 replayed from cache · 69 model calls · 430.1k tokens (82% cached) · $0.0299 All 22 tests in 1 file
e2e 0.21.1 · 14m 41s · ios-simulator · run artifacts |
…und to
Claude Opus 5.5, Sonnet 5.5, and Fable 5.1 sign every thinking block over
the system prompt and the messages before it, and reject a replay after
either changed ("Invalid signature in thinking block. The block is bound to
a different conversation"). The act loop edited both mid-step: it appended
the tool-calls-only rule to the system prompt on the switch to auto after
the provider refused or downgraded a forced tool choice, and it elided
superseded screens and screenshots from earlier messages.
The rule is now part of every step's instructions from the first turn, so
the switch changes only tool_choice. When an earlier message is rewritten,
found by object identity against the last request, the Anthropic hints
drop the reasoning parts from that message on and the step's turn notes
it; other providers get the history untouched. The overflow shrink returns
unchanged messages as the same objects. The Anthropic detection also
recognizes Claude on Bedrock and Vertex AI, where the same check runs.
Verified live through the AI Gateway with the binding check opted in as
error: main fails the pixel flows on all three models, the branch passes
all 29 testbed agent tests on each.
Closes ENG-984.
The rewrite check compared message objects against the last request's, which an in-place edit through the object handed to prepareMessages leaves identical. It now keeps the references each message held (string content, text parts' strings, every other part) and compares those.
TheTuso
force-pushed
the
piotrek/eng-984-agent-anthropic-claude-5x-rejects-replayed-thinking-block
branch
from
October 15, 2026 12:31
fa72312 to
428adbb
Compare
Member
|
Deep review: root cause and approach are right (provider-based detection, drops at/after the already-elided message so no extra cache misses, identity-preserving compaction means default executor isn't flagged). Unit suite green, new Anthropic tests fail with main's
This was an automated review. |
The rewrite check kept text strings by value and every other part by reference, so an executor editing a tool result below the part slipped through, and one cloning the history each turn was flagged every turn. Message shapes are now serialized without provider options, with binary payloads compared by bytes. Notes the Bedrock application inference profile limit and shortens the changeset.
Collaborator
Author
|
[claude-fable-5-1] responding on behalf of TheTuso All five taken, in c304977:
Verified on the built tree: Opus 5.5 through the gateway with the binding check enforced, all six testbed agent files, 29 of 29, same call count and cache rate as before the change, five rewrite notes at five distinct elision points. |
Member
|
Automated review (Claude Code on behalf of @okwasniewski) Re-checked c304977. All five asks are done:
Typecheck is clean and all unit tests pass. A six-request run with an early elision showed one note and kept later thinking. Live gateway calls on Sonnet 4.5 (thinking on) and Opus 5.5 accept requests with the thinking dropped. CI is green. Ready for human review. Leftovers:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes ENG-984.
Claude Opus 5.5, Sonnet 5.5, and Fable 5.1 sign every thinking block over the request prefix that produced it, the system prompt and the messages before the block, and answer a replay after any edit to that prefix with:
The act loop edited that prefix in two places:
autoafter the provider refused or downgraded a forced tool choice, it appended the tool-calls-only rule to the system prompt. Recent@ai-sdk/anthropicversions and the AI Gateway downgrade with a warning instead of a 400, so turn 1 succeeds with a thinking block and turn 2 replays it under a changed prompt. This fires on the first step of every worker and is what the reporter hit; the issue text names only the second cause.transcript-compaction.ts) rewrites earlier messages once superseded screens pass their budget. Every block signed before the edit is invalid from then on.flowchart LR A[turn n: history prepared] --> B{a message differs from<br/>the last request's object?} B -- no --> C[send as is] B -- yes --> D[afterEdit: drop reasoning parts<br/>from that message on<br/>Anthropic only] D --> E[note it on the turn] --> Cpackages/e2e/src/agent/tool-loop.ts: the tool-calls-only rule is part of every step's instructions from turn 1, so the switch toautochanges onlytool_choice, which the check ignores.prepareTurncompares what each message holds with what the last request sent, by value with provider options left out and binary payloads compared by bytes, so a replaced message and one a customprepareMessagesedits in place both show up while a moved cache breakpoint or a cloned history does not, and hands the first rewritten index to the provider hints. The overflow shrink now returns unchanged messages as the same objects for the same reason.packages/e2e/src/agent/model/provider-hints.ts:afterEditremoves thereasoningparts of assistant messages from the rewritten one on, and drops a reply that was only reasoning. Anthropic allows removing thinking from the end of a history; a block produced after the elision stays valid because the elided message does not change again. Other providers get the history untouched, since OpenAI Responses needs its reasoning items beside their function calls. The Anthropic detection now also recognizes Claude on Bedrock (eu.anthropic.claude-…) and Vertex AI (vertex.anthropic.messages), where the same check runs; before, only the direct provider and the gateway were matched.tests/unit/tool-loop-prefix.test.ts: the real@ai-sdk/anthropicprovider over a fakefetch, asserting on the request bodies that the system prompt is byte-identical through the switch, that no thinking is replayed after an elided message while later thinking is, and that a non-Anthropic model keeps every reasoning part.prefix_mismatch_behavior: drop_block. It needs a beta header, a provider version and a gateway that forward it, is rejected under Sonnet 5.5'sbetween_tools, and hides the edit instead of removing it. TheactiveToolsnarrowing on forced-conclusion turns stays: Anthropic's docs list the tool set in the bound prefix, but a narrowed and a reordered list both passed the check when probed live, and main's forced turn on Fable 5.1 passed it under enforcement.Verified
Ran it locally: yes
error(our account is not enforced by default) andeffort: 'high'(atmediumOpus 5.5 produced no thinking on these pages), all sixtests-agentfiles,--no-cache:e2e explore --max-steps 6on Opus 5.5 completed 6 of 6 steps on the branch. A step driven into its forced-conclusion turns with a three-call budget passed the check on both builds.openai/gpt-6-luna-fastwith the unmodified agent config, 28 of 29 passed and the keypad flow passed on two reruns.enabledthinking accept a last assistant turn with its thinking stripped.pnpm bench:abon the web benchmark's control inventory, 2 pairs: identical behavior and counters on every run of both builds, timings same.Fresh-context review: one finding rejected, that narrowing
toolson forced turns breaks the check, since the live probe and main's forced turn under enforcement show it does not; one finding fixed, the Bedrock and Vertex detection above.