Improve Phase hang diagnostics and Agent process supervision

[maxBRT]

Problem

Ship can appear to hang during an Iteration or Phase with no useful clue about what the external Agent is doing.

The current Run UI keeps the throbber spinning, but the operator often cannot answer basic questions while waiting:

• Which Phase is currently running?
• Which external Agent process is active?
• What was the last tool call or Agent event observed?
• Has the configured Phase timeout actually fired?
• Did the Agent spawn a child process that is still alive?
• Where should I look for live forensic data?

This makes AFK Runs hard to trust. When a Run stalls, the only visible signal may be an indefinitely spinning status line.

Source-level findings

The main Run loop is bounded and straightforward. The likely problem is at the process supervision and observability boundary around external Agents.

Relevant files:

• internal/run/orchestrator.go
• internal/agent/cursor.go
• internal/agent/codex.go
• internal/agent/claude.go
• internal/agent/pi.go
• internal/observe/observer.go
• internal/throbber/line.go
• internal/ticket/github.go
• internal/gitops/branch.go
• internal/gitops/restore.go
• internal/run/shipfile.go

1. Agent output is buffered until the Agent exits

Each Agent adapter currently captures stdout and stderr into buffers, runs the process to completion, then parses the JSON stream only after cmd.Run() returns.

Example shape:

var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr

err := cmd.Run()
...
consumeStreamJSON(stdout.Bytes(), req.Events)

This means Ship does not emit curated events live while a Phase is running. If an Agent emits useful JSON events before hanging, Ship still will not write them to .ship/runs/.../*.jsonl or show them in the observer until the process exits.

Impact:

• .ship/runs/<run-id>/<phase>.jsonl can stay empty while the Agent is actively doing work.
• The Abort banner's last-tool section is often empty for hangs because no live events were emitted.
• The operator sees only the throbber, not the last observed Agent action.

2. Timeout only supervises the direct Agent process

Adapters use exec.CommandContext, which is useful but only targets the direct process. If the Agent starts subprocesses, a timeout may leave child processes alive or keep pipes open.

Possible cases:

• Agent starts a shell command that hangs.
• Agent process exits, but a child keeps stdout or stderr open.
• A child process ignores the parent cancellation path.
• Go waits for pipe closure, making timeout handling look stuck.

Existing timeout tests cover a simple sleeping fake binary. They do not cover an Agent that spawns a long-lived child process.

3. Timeout can be disabled by config

timeout: 0s or a negative duration is accepted by config parsing today. The adapters only install a timeout when req.Timeout > 0, so 0s disables timeout entirely.

If this is intentional, it should be explicit, such as timeout: none. If it is not intentional, config validation should reject non-positive timeouts.

4. Some GitHub and git operations have no local timeout

Several non-Agent operations can also block or wait for prompts:

• gh repo view
• gh api graphql
• gh issue edit
• gh issue close
• gh pr list
• git checkout
• git reset --hard
• git rev-parse

Main passes context.Background() to the orchestrator, and git helpers use exec.Command without context. A stuck credential prompt, network wait, git hook, lock, or keychain prompt can look like a Ship hang outside the Agent Phase.

Desired behavior

When Ship is running a Phase, the operator should be able to tell what is happening without waiting for process exit.

At minimum, Ship should persist and report:

• Run directory path
• Phase name
• Phase start time
• Agent kind and argv summary
• Agent process pid
• configured timeout and timeout deadline
• last observed Agent event time
• last observed tool start and tool completion
• recent stderr lines, redacted if needed
• whether the Phase ended by success, non-zero exit, timeout, cancellation, missing terminal event, or supervisor failure

Proposed implementation

A. Stream Agent output live

Replace stdout and stderr buffers with pipes:

• cmd.StdoutPipe() for Agent JSON events
• cmd.StderrPipe() for recent stderr capture
• Start the process with cmd.Start()
• Parse stdout line by line while the process is running
• Emit curated events immediately to req.Events
• Wait for process completion after readers are running

This preserves the current quiet terminal UX while making .ship/runs live and useful.

The terminal does not need to print every tool live. Durable reports should update live, and the throbber can continue to own the active screen.

B. Add explicit lifecycle events

Extend the observer events with lifecycle records, for example:

• phase_start
• process_started
• tool_started
• tool_completed
• stderr_line
• heartbeat
• timeout
• process_exited
• phase_end

The existing KindPhaseStart constant exists but does not appear to be emitted today. Use it or replace it with a more complete lifecycle model.

C. Kill the full process group on timeout

On Unix, start each Agent in a new process group and kill the group on timeout. This avoids leaving Agent-spawned children alive.

Suggested approach:

• Set process group before start.
• On context cancellation, send termination signal to the group.
• Escalate to kill after a short grace period.
• Include process group cleanup result in the Phase report.

Keep Windows support in mind with a platform-specific implementation.

D. Validate timeout config

Reject timeout <= 0 unless there is a documented explicit escape hatch.

Possible user-facing behavior:

• timeout: 0s returns config error: timeout must be positive; use timeout: none to disable timeouts.
• Or do not support disabling timeouts at all.

E. Add command timeouts for gh and git operations

Introduce helper execution functions with operation names and bounded contexts.

Examples of operation labels:

• gh repo view
• gh api graphql list ship Tickets
• gh issue edit remove ship
• gh issue close
• gh pr list
• git checkout branch
• git reset --hard restore point

Failures should include the operation label and elapsed time.

F. Add a status or doctor surface

Consider one of:

• ship status: print the latest Run report, current Phase, last event, elapsed time, and timeout deadline.
• ship doctor: validate Agent binaries, gh auth, git repo state, timeout config, and whether stale Agent child processes are present.
• A clear stderr line at Phase start: report .ship/runs/<run-id> so the operator knows where to look before a failure.

Acceptance criteria

• During a long-running Agent Phase, .ship/runs/<run-id>/<phase>.jsonl is updated before the Agent exits.
• If an Agent starts a tool call and then hangs, the Phase log records the last observed tool or event before timeout.
• On timeout, Ship reports the Phase name, timeout duration, elapsed time, and last observed event.
• On timeout, Ship attempts to terminate the full Agent process tree or process group, not only the direct child.
• A regression test covers an Agent process that spawns a child which outlives the parent or keeps stdout/stderr open.
• timeout: 0s is either rejected or explicitly documented as disabling timeouts.
• GitHub and git subprocesses have bounded waits or clear operation-level diagnostics.
• Existing terminal behavior remains quiet enough: no raw vendor stream spam during a normal Phase.

Suggested tests

1. Agent emits one JSON tool-start event, then hangs. Assert the event appears in the Phase log before timeout.
2. Agent writes stderr periodically, then hangs. Assert recent stderr is captured in the timeout report.
3. Agent starts a child process that sleeps longer than the timeout. Assert Ship returns near the timeout and cleans up the process group.
4. Config with timeout: 0s either fails validation or is documented and tested as no-timeout behavior.
5. Fake gh or git helper sleeps. Assert operation-level timeout error names the blocked command.

Notes

This issue is not about changing the Run model. The Run loop and Iteration semantics look sound. The problem is that Ship currently observes Agent streams after process exit, which is too late for diagnosing a hung Phase.

[maxBRT]

Roadmap note: this also covers the blog post item "Adding more observability to each phase" from https://maxbrt.dev/blog/meet-ship/.