Problem
An inline run_agent execution can complete and persist on the backend while its agent card and nested LLM-call view remain stuck in RUNNING. Refreshing the page reveals the completed state because persisted chat history is correct.
The inner run_agent_event stream and outer tool_call_end share the orchestrator SSE connection. If that connection ends unexpectedly, the durable backend turn keeps running, but the frontend does not reconnect or reconcile the terminal result.
Reproduction
- Start a
run_agent call that produces visible LLM/tool-call progress.
- Interrupt or lose the orchestrator SSE connection while the durable turn continues.
- Wait for the agent to finish on the backend.
- Observe that the card and/or nested LLM call remains running.
- Refresh the page; the completed state appears.
Scope
1. Cursor-based stream reconnection
- Give each orchestrator turn a stable ID and its events a monotonically increasing cursor.
- Return the turn ID/cursor to the frontend and reconnect after unexpected EOF or transport failure.
- Resume strictly after the last applied cursor.
- Make event application idempotent so reconnects cannot duplicate token chunks, tool calls, agent events, or terminal events.
- Do not reconnect after an intentional stop, abort, superseding turn, or unmount.
2. Persisted-session reconciliation fallback
- If reconnect is impossible because the backend restarted, the turn expired, the ID is missing, or retries are exhausted, reload the persisted chat/session.
- Merge the authoritative persisted tool and agent state into the current conversation without duplicating messages.
- If persistence still reports an active turn, retry on a bounded cadence until it reaches a terminal state or the user stops it.
3. Surface inline-agent retry progress
- Forward and persist
rate_limit_retry events produced by run_agent.
- Render retry state in the agent inspector's LLM-call view with the same useful detail available for a normal node, including the retry attempt and wait when available.
- Keep retry progress live across stream reconnection and correct after persisted-session reconciliation.
- Preserve the eventual recovered or failed terminal state without leaving the LLM call stuck.
Acceptance criteria
- A transient SSE disconnect does not require a page refresh.
- Nested LLM-call progress resumes after reconnect.
- Agent cards reliably converge to
DONE, FAILED, or STOPPED.
- Replayed events do not duplicate content or tool-call rows.
- A backend restart or expired resume cursor converges through persisted-session reconciliation.
- Intentional stop and supersede behavior does not trigger reconnection.
- A retryable 429 inside
run_agent visibly updates the agent inspector's LLM-call view and continues updating through recovery or terminal failure.
- Add coverage for disconnect/reconnect, replay deduplication, fallback reconciliation, intentional cancellation, and inline-agent retry progress.
Likely areas
- Orchestrator SSE endpoint/event envelope and durable-turn registry
- Frontend SSE reader and orchestration stream lifecycle
- Chat-history hydration/merge logic
- Inline-agent event state updates
- Agent inspector LLM-call rendering
Problem
An inline
run_agentexecution can complete and persist on the backend while its agent card and nested LLM-call view remain stuck inRUNNING. Refreshing the page reveals the completed state because persisted chat history is correct.The inner
run_agent_eventstream and outertool_call_endshare the orchestrator SSE connection. If that connection ends unexpectedly, the durable backend turn keeps running, but the frontend does not reconnect or reconcile the terminal result.Reproduction
run_agentcall that produces visible LLM/tool-call progress.Scope
1. Cursor-based stream reconnection
2. Persisted-session reconciliation fallback
3. Surface inline-agent retry progress
rate_limit_retryevents produced byrun_agent.Acceptance criteria
DONE,FAILED, orSTOPPED.run_agentvisibly updates the agent inspector's LLM-call view and continues updating through recovery or terminal failure.Likely areas