Add e2e suite 27: retried LLM turn must keep conversation history - #462
Open
ling-senpeng13 wants to merge 2 commits into
Open
Add e2e suite 27: retried LLM turn must keep conversation history#462ling-senpeng13 wants to merge 2 commits into
ling-senpeng13 wants to merge 2 commits into
Conversation
Reproduces orkes-io/orkes-conductor#3876. When a workflow is retried, the retry path re-resolves the task definition's inputParameters instead of running the history assembly the LLM_CHAT_COMPLETE task mapper performs at scheduling time, so the retried LLM turn is dispatched with a bare [system, user] template. The model re-issues a tool call it already made — a duplicate execution of a potentially side-effecting tool. Two tests: - test_retried_llm_turn_keeps_conversation_history — asserts the retried task's inputData.messages still contain the get_customer_info call and its result. Currently FAILS; this is the bug. - test_retried_tool_task_is_unaffected — control from the issue. Retrying an interrupted plain tool task re-runs it with the same arguments and must keep working after a fix. Assertions are algorithmic, on the workflow API's task inputData, with no LLM output parsing. The retried task is identified by taskId rather than by its input, so the suite cannot pass against a broken server by matching the interrupted original, which still holds full history. Skipped by default via SUITE27_RUN so CI stays green while the server bug is open: SUITE27_RUN=1 pytest e2e/test_suite27_retry_context.py -v Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip-by-default gate could not report the fix. Downstream (orkes-conductor PR #3888) checks this repo out at the pinned tag and runs e2e/ in place, and its known-failure mechanism marks entries xfail(strict=False, run=True) so a fixed bug turns the test XPASS. A skipped test never runs, so it can never XPASS — the gate would have suppressed the very signal the suite exists to produce, in this repo's CI and downstream's alike. Mark only test_retried_llm_turn_keeps_conversation_history xfail; the control test passes today and now runs unmarked. Non-strict, so the expected failure is green now and an XPASS is green too, just visible in the summary — strict would turn the suite red the moment the server fix lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ling-senpeng13
added a commit
to conductor-oss/conductor
that referenced
this pull request
Aug 6, 2026
…s conversation Ports python-sdk e2e suite 27 (conductor-oss/python-sdk#462) into the test harness. A chat task scheduled after a tool call has completed must come back from a workflow retry with that exchange intact, not the definition's bare [system, user] template. No LLM is involved: the prior exchange comes from a completed SIMPLE task named in participants, which the mapper folds into a tool ChatMessage carrying the tool name and its output. That is the same assembled-in-Java input the retry path clobbers, so it reproduces the same regression. Assertions are on the task's inputData read back from the store, never on model output. Verified to discriminate: short-circuiting detachAssembledInputFromDefinition fails the spec on the tool-exchange assertion with roles=[system, user], and the retried messages come back without the mapper's media defaults - a freshly resolved template, not a truncated conversation. Needs conductor.integrations.ai.enabled=true, set via @TestPropertySource.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an e2e regression suite for the bug — "LLM_CHAT_COMPLETE retry loses conversation history, causing duplicate tool calls".
The tests follows the steps described below to reproduce the issue. There are 2 test cases:
test_retried_llm_turn_keeps_conversation_historyis expected to fail andtest_retried_tool_task_is_unaffectedis expected to pass.Steps To Reproduce
Minimal repro using a native agent, two sequential tools:
Start the agent. Let it complete its first tool call (get_customer_info) so it's genuinely in-progress on a second LLM turn.
In the Conductor UI, terminate the execution while the second (or any subsequent) LLM_CHAT_COMPLETE task is IN_PROGRESS, not while a tool task is running.
Use the UI's Retry action to resume from the failed/canceled task.
Inspect the retried LLM_CHAT_COMPLETE task's inputData.messages, it contains only [system, user], with none of the already-completed tool_call/tool message pairs from prior iterations, even though those tool tasks show COMPLETED in the same workflow's task list.
Observe the model, having no memory of the already-successful get_customer_info call, calls it a second time. This is a duplicate execution, not just a display quirk: a second get_customer_info task instance actually runs.
The following LLM turn (the next fresh, non-retried dispatch) does correctly receive the full history, including both get_customer_info calls, and produces a coherent final answer, the model itself narrates the duplication (e.g., prefacing its answer with "Continuing from before...").
For contrast, terminating instead while a tool task (e.g., get_weather) is IN_PROGRESS and retrying it behaves correctly: the retried tool task just re-runs with the same arguments, and the next LLM turn correctly sees both the failed and successful attempts in its history (e.g., "Tool execution failed (status: CANCELED)" followed by the successful retry's result), producing a clean, non-duplicated final answer.
Expected behavior
Retrying a canceled/failed LLM_CHAT_COMPLETE task should receive the same fully-reconstructed conversation history (all prior completed tool calls and results) that a freshly-scheduled LLM_CHAT_COMPLETE task for the next iteration would receive, not the bare static template. The retry mechanism shouldn't silently drop context that's demonstrably still present in the workflow's own task execution history.