Skip to content

fix(android): hold a long idle lease while a tool call is in flight - #662

Closed
trevornk wants to merge 1 commit into
Codename-11:devfrom
trevornk:fix/tool-in-flight-watchdog
Closed

trevornk wants to merge 1 commit into
Codename-11:devfrom
trevornk:fix/tool-in-flight-watchdog

Conversation

@trevornk

@trevornk trevornk commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Summary

The Gateway emits no events between tool.start and tool.complete. GatewayChatClient's idle watchdog (TURN_TIMEOUT_MS, 180s) is re-armed only on events, so any single tool call that runs silently for longer than 180s (for example a long terminal command) is treated as a dead turn: the client sends session.interrupt, the server logs Interrupt requested (hard): explicit stop requested, and the healthy tool is killed (exit 130). The user sees "Gateway turn timed out" / "Operation interrupted" mid-task without having pressed stop.

Asks and compaction already get longer leases for exactly this reason (see the compacting lease); tool execution had no equivalent. This PR holds a longer lease while any tool call is in flight.

Changes

  • GatewayEventMapper tracks open tool calls (id → name): tool.start adds, tool.complete removes (same id/name matching as the existing complete handling; an id-less complete retires the oldest open call of that name). Exposes hasOpenTools / clearOpenTools().
  • GatewayChatClient adds TOOL_IN_FLIGHT_TIMEOUT_MS (1800s, matching PROMPT_SUBMIT_REQUEST_TIMEOUT_MS and the backend's agent turn ceiling) with a toolInFlightTimeoutMs test seam. watchdogTimeoutFor / default armWatchdog() use it while any tool is open; ask and compaction leases keep priority.
  • Open-tool state is cleared on mid-turn socket rejoin (a tool.complete missed during the gap cannot be replayed), so the normal idle timeout applies after a rejoin.
  • Dead-turn detection is unchanged when no tool is in flight.

Verification

  • ./gradlew :app:testSideloadDebugUnitTest --tests "com.hermesandroid.relay.network.upstream.GatewayChatClientTest" --tests "com.hermesandroid.relay.network.upstream.GatewayEventMapperTest" on the PR head (rebased on current dev): BUILD SUCCESSFUL — GatewayChatClientTest 199 tests, GatewayEventMapperTest 77 tests, 0 failures, 0 errors.
  • Four new tests: silent in-flight tool is not interrupted past the idle window; idle timeout resumes after tool.complete; with parallel tools the lease holds until every call completes; a dead turn with no tool in flight still times out and interrupts even with a long tool lease configured.
  • Negative control: with the in-flight check disabled, the silent-tool and parallel-tool tests fail.
  • Not run: lint, full unit suite, Gateway contract fixture/instrumentation lanes, on-device check.

Screenshots

No visual change.

Compatibility / risk

  • Client-only; works against unmodified upstream Gateway (uses existing tool.start / tool.complete events). No relay-plugin or fork-only behavior.
  • A tool that hangs without ever emitting tool.complete is now cut off client-side after 1800s instead of 180s; server-side tool timeouts still apply first.
  • If a tool.complete is lost without a socket rejoin, the long lease persists until the next event or turn end.

Lineage / contributor credit

  • Source PR(s): N/A (same pattern as the compacting lease)
  • Attribution preserved by: N/A

Checklist

  • Target branch is dev
  • Scope is focused and related issues/PRs are linked
  • Android changes: focused tests ran; lint not run (listed above)
  • Translation changes: N/A
  • Server/plugin changes: N/A
  • Desktop changes: N/A
  • Docs/site changes: N/A
  • UI changes: N/A (no visual change)
  • Commit messages follow Conventional Commits
  • CHANGELOG.md is updated
  • Public writing hygiene checked
  • Salvaged/replacement work: N/A

The Gateway emits no events between tool.start and tool.complete, so the
180s idle watchdog treated any single tool call that ran longer than that
as a dead turn and sent session.interrupt, killing a healthy tool
(exit 130).

Track open tool calls in GatewayEventMapper (id -> name, matched the same
way as tool.complete handling) and, while any is open, arm
TOOL_IN_FLIGHT_TIMEOUT_MS (1800s, matching the prompt.submit ceiling and
the backend gateway timeout) instead of the idle window. Ask and
compaction leases keep precedence. Open-tool state is dropped on a
mid-turn rejoin since a missed tool.complete cannot be replayed. The
ordinary idle timeout applies again once no tool is open.

Adds a toolInFlightTimeoutMs constructor seam and tests for: silent
in-flight tool, normal timeout after tool.complete, parallel tools, and a
dead turn with no tool in flight.
@Codename-11

Copy link
Copy Markdown
Owner

Sorry this was cutting your tools off, @trevornk, and apologies for the delay. #694 removes the cutoff entirely and includes regression coverage for the long-tool case you found. Closing this one in favor of that fix. Thanks for tracking it down!

@trevornk
trevornk deleted the fix/tool-in-flight-watchdog branch October 16, 2026 14:53
sacgsxr pushed a commit to sacgsxr/hermes-relay that referenced this pull request Oct 16, 2026
Extend the preserved PR Codename-11#675 commits with read-only quiet probes, exact unsolicited settlement, passive reconnect and shutdown, native ask replay, and missing-final history reconciliation. Cover owner and observer paths through real Gateway transport and rendered fixture tests. Update current-upstream conformance for extracted helpers and additive requests.

Adapt cancellation-only diagnostics from PR Codename-11#668. Supersede PR Codename-11#662's eventual interrupt lease; quiet tools use server-owned liveness instead.

Co-authored-by: ophirhan <ophirhan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants