Skip to content

inflight watchdog false-positive: long silent foreground tool call + queued turns → force_restart kills the tool child #731

Description

@olegbrok

What happened

2026-06-10 ~20:25 PDT, barsik. A foreground gh run watch <id> --exit-status >/dev/null (a deliberate up-to-10-min blocking CI watch) produced no transcript writes for ~614s while two inbound turns queued behind it. The inflight watchdog's wedge heuristic fired and force-restarted the REPL, SIGKILLing the Bash child (exit 137):

tmux[barsik]: verdict_wedged_inputs live_status='working' live_last_updated=1781147818 idle_floor=1781147720 head_dispatched_at=1781147720 transcript_mtime=1781147719 age_s=614.1 depth=3
tmux[barsik]: inflight head aged 614.1s > 600.0s, transcript quiet + REPL not idle — REPL stuck; scheduling force_restart (deque depth=3)
tmux[barsik]: requeued 2 turn(s) for replay after force_restart (tail=2, in_hand=no)

The turn was healthy — the model was awaiting a single long tool call. Recovery worked as designed (turns requeued + replayed, mode=continue relaunch), so impact was only the killed child process + a cold re-read, but the verdict itself was a false positive.

Why the current heuristic misses this

transcript_mtime only advances when a tool call COMPLETES — a single silent long-running foreground call looks identical to a wedge. Harness-tracked background tasks already extend the window (transcript or background task still active — not wedged); foreground calls get no such credit even though the tool child is demonstrably alive.

Possible fixes (in rough preference order)

  1. Liveness via process tree: before declaring wedged, check whether the REPL's active tool child (Bash descendant) is alive/consuming CPU — alive child + working REPL = extend window, same as background tasks.
  2. Tmux pane activity as a secondary signal (pane content hash change vs transcript mtime).
  3. Cheaper stopgap: raise the silent-foreground threshold (600s → 1200s) when live_status='working' and deque depth is from NEW arrivals (queued-behind ≠ stuck-on).

Agent-side mitigation (already adopted)

Long waits (CI watches etc.) moved to run_in_background: true — those correctly extend the window today. This issue is about the heuristic, not urgency.

Related: #713 (watchdog tuning in 26.06.124) — this is a different verdict path (verdict_wedged_inputs), observed on 26.06.124.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions