Skip to content

Reconnect interrupted orchestrator streams so inline agents do not stay stuck #82

Description

@MuJiongan

Problem

An inline run_agent execution can complete and persist on the backend while its agent card and nested LLM-call view remain stuck in RUNNING. Refreshing the page reveals the completed state because persisted chat history is correct.

The inner run_agent_event stream and outer tool_call_end share the orchestrator SSE connection. If that connection ends unexpectedly, the durable backend turn keeps running, but the frontend does not reconnect or reconcile the terminal result.

Reproduction

  1. Start a run_agent call that produces visible LLM/tool-call progress.
  2. Interrupt or lose the orchestrator SSE connection while the durable turn continues.
  3. Wait for the agent to finish on the backend.
  4. Observe that the card and/or nested LLM call remains running.
  5. Refresh the page; the completed state appears.

Scope

1. Cursor-based stream reconnection

  • Give each orchestrator turn a stable ID and its events a monotonically increasing cursor.
  • Return the turn ID/cursor to the frontend and reconnect after unexpected EOF or transport failure.
  • Resume strictly after the last applied cursor.
  • Make event application idempotent so reconnects cannot duplicate token chunks, tool calls, agent events, or terminal events.
  • Do not reconnect after an intentional stop, abort, superseding turn, or unmount.

2. Persisted-session reconciliation fallback

  • If reconnect is impossible because the backend restarted, the turn expired, the ID is missing, or retries are exhausted, reload the persisted chat/session.
  • Merge the authoritative persisted tool and agent state into the current conversation without duplicating messages.
  • If persistence still reports an active turn, retry on a bounded cadence until it reaches a terminal state or the user stops it.

3. Surface inline-agent retry progress

  • Forward and persist rate_limit_retry events produced by run_agent.
  • Render retry state in the agent inspector's LLM-call view with the same useful detail available for a normal node, including the retry attempt and wait when available.
  • Keep retry progress live across stream reconnection and correct after persisted-session reconciliation.
  • Preserve the eventual recovered or failed terminal state without leaving the LLM call stuck.

Acceptance criteria

  • A transient SSE disconnect does not require a page refresh.
  • Nested LLM-call progress resumes after reconnect.
  • Agent cards reliably converge to DONE, FAILED, or STOPPED.
  • Replayed events do not duplicate content or tool-call rows.
  • A backend restart or expired resume cursor converges through persisted-session reconciliation.
  • Intentional stop and supersede behavior does not trigger reconnection.
  • A retryable 429 inside run_agent visibly updates the agent inspector's LLM-call view and continues updating through recovery or terminal failure.
  • Add coverage for disconnect/reconnect, replay deduplication, fallback reconciliation, intentional cancellation, and inline-agent retry progress.

Likely areas

  • Orchestrator SSE endpoint/event envelope and durable-turn registry
  • Frontend SSE reader and orchestration stream lifecycle
  • Chat-history hydration/merge logic
  • Inline-agent event state updates
  • Agent inspector LLM-call rendering

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions