Skip to content

Add e2e suite 27: retried LLM turn must keep conversation history - #462

Open
ling-senpeng13 wants to merge 2 commits into
mainfrom
test/suite27-retry-context
Open

Add e2e suite 27: retried LLM turn must keep conversation history#462
ling-senpeng13 wants to merge 2 commits into
mainfrom
test/suite27-retry-context

Conversation

@ling-senpeng13

@ling-senpeng13 ling-senpeng13 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Adds an e2e regression suite for the bug — "LLM_CHAT_COMPLETE retry loses conversation history, causing duplicate tool calls".

The tests follows the steps described below to reproduce the issue. There are 2 test cases: test_retried_llm_turn_keeps_conversation_history is expected to fail and test_retried_tool_task_is_unaffected  is expected to pass.

Steps To Reproduce

Minimal repro using a native agent, two sequential tools:

import time
from conductor.ai.agents import Agent, AgentRuntime, tool

@tool
def get_customer_info(customer_id: str) -> dict:
    """Look up a customer's account information, including their home city, by customer ID."""
    return {"customer_id": customer_id, "name": "Jane Doe", "city": "Seattle"}

@tool
def get_weather(city: str) -> str:
    """Get the current temperature, wind speed, and humidity for a city."""
    return f"{city}: 17.7C, wind 6.4 km/h, humidity 73%"

agent = Agent(
    name="retry_context_repro",
    model="OpenAI/gpt-5-mini",
    tools=[get_customer_info, get_weather],
    instructions=(
        "First call get_customer_info to find the customer's city, then "
        "call get_weather for that city, then summarize both."
    ),
)

with AgentRuntime() as runtime:
    handle = runtime.start(agent, "What's the weather like where customer 4471 lives?")
    print(handle.execution_id)

Start the agent. Let it complete its first tool call (get_customer_info) so it's genuinely in-progress on a second LLM turn.

In the Conductor UI, terminate the execution while the second (or any subsequent) LLM_CHAT_COMPLETE task is IN_PROGRESS, not while a tool task is running.

Use the UI's Retry action to resume from the failed/canceled task.

Inspect the retried LLM_CHAT_COMPLETE task's inputData.messages, it contains only [system, user], with none of the already-completed tool_call/tool message pairs from prior iterations, even though those tool tasks show COMPLETED in the same workflow's task list.

Observe the model, having no memory of the already-successful get_customer_info call, calls it a second time. This is a duplicate execution, not just a display quirk: a second get_customer_info task instance actually runs.

The following LLM turn (the next fresh, non-retried dispatch) does correctly receive the full history, including both get_customer_info calls, and produces a coherent final answer, the model itself narrates the duplication (e.g., prefacing its answer with "Continuing from before...").

For contrast, terminating instead while a tool task (e.g., get_weather) is IN_PROGRESS and retrying it behaves correctly: the retried tool task just re-runs with the same arguments, and the next LLM turn correctly sees both the failed and successful attempts in its history (e.g., "Tool execution failed (status: CANCELED)" followed by the successful retry's result), producing a clean, non-duplicated final answer.

Expected behavior

Retrying a canceled/failed LLM_CHAT_COMPLETE task should receive the same fully-reconstructed conversation history (all prior completed tool calls and results) that a freshly-scheduled LLM_CHAT_COMPLETE task for the next iteration would receive, not the bare static template. The retry mechanism shouldn't silently drop context that's demonstrably still present in the workflow's own task execution history.

Reproduces orkes-io/orkes-conductor#3876. When a workflow is retried, the
retry path re-resolves the task definition's inputParameters instead of
running the history assembly the LLM_CHAT_COMPLETE task mapper performs at
scheduling time, so the retried LLM turn is dispatched with a bare
[system, user] template. The model re-issues a tool call it already made —
a duplicate execution of a potentially side-effecting tool.

Two tests:
  - test_retried_llm_turn_keeps_conversation_history — asserts the retried
    task's inputData.messages still contain the get_customer_info call and
    its result. Currently FAILS; this is the bug.
  - test_retried_tool_task_is_unaffected — control from the issue. Retrying
    an interrupted plain tool task re-runs it with the same arguments and
    must keep working after a fix.

Assertions are algorithmic, on the workflow API's task inputData, with no
LLM output parsing. The retried task is identified by taskId rather than by
its input, so the suite cannot pass against a broken server by matching the
interrupted original, which still holds full history.

Skipped by default via SUITE27_RUN so CI stays green while the server bug
is open:

    SUITE27_RUN=1 pytest e2e/test_suite27_retry_context.py -v

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ling-senpeng13 ling-senpeng13 self-assigned this Aug 5, 2026
The skip-by-default gate could not report the fix. Downstream (orkes-conductor
PR #3888) checks this repo out at the pinned tag and runs e2e/ in place, and
its known-failure mechanism marks entries xfail(strict=False, run=True) so a
fixed bug turns the test XPASS. A skipped test never runs, so it can never
XPASS — the gate would have suppressed the very signal the suite exists to
produce, in this repo's CI and downstream's alike.

Mark only test_retried_llm_turn_keeps_conversation_history xfail; the control
test passes today and now runs unmarked. Non-strict, so the expected failure
is green now and an XPASS is green too, just visible in the summary — strict
would turn the suite red the moment the server fix lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@conductor-oss conductor-oss deleted a comment from codecov Bot Aug 5, 2026
ling-senpeng13 added a commit to conductor-oss/conductor that referenced this pull request Aug 6, 2026
…s conversation

Ports python-sdk e2e suite 27 (conductor-oss/python-sdk#462) into the test
harness. A chat task scheduled after a tool call has completed must come back
from a workflow retry with that exchange intact, not the definition's bare
[system, user] template.

No LLM is involved: the prior exchange comes from a completed SIMPLE task named
in participants, which the mapper folds into a tool ChatMessage carrying the
tool name and its output. That is the same assembled-in-Java input the retry
path clobbers, so it reproduces the same regression. Assertions are on the
task's inputData read back from the store, never on model output.

Verified to discriminate: short-circuiting detachAssembledInputFromDefinition
fails the spec on the tool-exchange assertion with roles=[system, user], and
the retried messages come back without the mapper's media defaults - a
freshly resolved template, not a truncated conversation.

Needs conductor.integrations.ai.enabled=true, set via @TestPropertySource.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant