Skip to content

An in-process tool call cannot be cancelled #26

Open

Description

@danczw

#515 closed the bash half of this issue: a tui stop now raises a per-turn Cancel that reaches kill_group for the running command, waits for the call to settle, and starts none behind it. What is left is the six in-process tools.

An in-process tool call cannot be cancelled

read, write, edit, ls, grep and find run through BuiltinTool::execute on tokio::task::spawn_blocking (crates/sandbx-turn/src/tools.rs), which cannot be cancelled: dropping the JoinHandle leaves the task running. So a stop that lands in one of them waits for it, and its effect stays — a write under way lands, and a read on a stalled filesystem holds the stop for as long as the filesystem takes.

There is no kill to reach for as there is with bash: these tools run inside sandbx's own process, so there is no process group to signal. Cancelling one means the tool itself observing the token between units of work — between files for grep and find, and not at all inside a single write or read.

Constraints any design must keep

  • The gate stays ahead of the spawn. approve is asked between resolving the name and spawn_blocking, and context/decision-approval-gate.md ties that position to blocking tasks being uncancellable. A tool that can be cancelled mid-write also has to say what a half-applied write or edit leaves behind; stopping partway is a different hazard from finishing.
  • tokio::process::Command was rejected and the reasoning stands (it would rewrite the verified spawn path for a disclosed non-weakness), and it does not apply here anyway: these tools do not spawn.
  • The bounds No run_turn: every consumer will reassemble the event stream itself, and the blocking boundary has no owner #58 shipped stay: max_rounds, stream_timeout, and a sequential answer_calls.

Where this is disclosed

SECURITY.md — "Only a spawned command can be interrupted.", and the wall-clock bullet's "on a thread that cannot be cancelled". Both stay accurate until a tool can observe the token, and want a look when one does.

Cancel and ExecutionContext::cancelled are already in place for a tool to poll, so the open question is per tool, not mechanism: which of the six has a safe point to stop at, and is a stop worth the partial state it can leave.

Not labelled security

Per #22: a limitation of the harness, not a weakness in the sandbox boundary.

Activity

  1. self-assigned this
    on Sep 25, 2026
  2. danczw commented on Oct 1, 2026

    @danczw
    OwnerAuthor

    2a is decided: spawn_blocking. #58 was the Phase 5 caller this issue said to wait
    for, and it landed in #86 with the boundary owned by run_turn and nowhere else.

    The reasoning, so it does not have to be re-derived:

    tokio::process::Command would buy cancellation and parallelism, but it would do so by
    rewriting the spawn path — the machinery behind the process-group SIGKILL, the PID
    namespace reaping, and PR_SET_PDEATHSIG. Both this issue and #22 are explicit that
    what that buys is not a sandbox-boundary weakness: it is a denial-of-service surface
    against the harness, which SECURITY.md already discloses honestly in its ninth
    does not claim bullet. Rewriting verified enforcement to fix a disclosed
    non-weakness is a net loss, so the restrictive-by-omission option won.

    What #58 shipped instead, which narrows this issue rather than closing it:

    • a round cap, so a looping or injected-into model cannot drive tool execution without
      bound
    • a per-round streaming bound, discharging the obligation anthropic.rs hands upward
    • a sequential answer_calls, with a comment naming this issue as the reason it is
      sequential

    So a turn's cost is now derivable — max_rounds * (stream_timeout + calls * tool timeout) — without anything becoming cancellable.

    Still open here, unchanged: a running tool call cannot be killed from outside, and
    tool calls do not run in parallel. The ordering semantics of two tools sharing one
    ExecutionContext are also still undefined; run_turn sidesteps the question by
    running calls in order rather than answering it.

  3. danczw commented on Oct 2, 2026

    @danczw
    OwnerAuthor

    Concrete consequence found while reviewing feat/run-turn, recorded here rather than fixed, since it needs the cancellation this issue tracks.

    answer_calls runs each tool via tokio::task::spawn_blocking(..).await (crates/sandbx-agent/src/turn.rs:226). That is not cancel-safe: dropping the run_turn future drops the JoinHandle while the blocking task runs to completion. So a turn abandoned mid-tool — a TUI cancel, a losing select! branch, an outer deadline — still:

    • applies a write or edit after the caller stopped waiting, or
    • lets a bash command run out its full timeout (90s by default).

    The transcript that would have named the call is dropped with the future, so the model-facing record of it is gone. The side effect itself is still in the audit trail (AuditEvent::allowed from FsGuard::permit, AuditEvent::spawned from SandboxedCommand), which is what keeps this an observability gap rather than a silent one.

    No fix attempted: an outer tokio::time::timeout around the spawn_blocking would bound when the caller gives up, not when the tool stops, and would leave runaway work parked on the blocking pool — so a time bound here is misleading until tool calls can actually be cancelled.

    Documented in the meantime, so nothing claims otherwise:

    • run_turn’s “blocking boundary” section now states it (c7762ae).
    • SECURITY.md’s wall-clock bullet now says tools run on a blocking thread that cannot be cancelled.
  4. added this to the milestone on Oct 2, 2026
  5. danczw commented on Oct 5, 2026

    @danczw
    OwnerAuthor

    Triage against 44fc1ea (2026-10-05) — still open and unchanged in substance; one line reference in the comment above has drifted.

    The remaining scope is intact and nothing has fixed it:

    • run_with_deadline still exists (crates/sandbx-core/src/command.rs:194), still polls try_wait() (:234) and still SIGKILLs the process group (killpg(Pid::from_raw(pid), Signal::SIGKILL), :341).
    • answer_calls still runs tools one at a time through tokio::task::spawn_blocking(move || tool.execute(input, &context)).await, and its doc still names this issue as the reason: "Sequential: concurrency would need the ordering semantics of two tools sharing one ExecutionContext settled first (An in-process tool call cannot be cancelled #26)."
    • run_turn's doc still states the cancel-safety gap (turn.rs:203-208): "spawn_blocking cannot be cancelled: dropping this future drops the JoinHandle while the blocking task runs to completion, so a turn abandoned mid-tool still applies the write, recorded only in the audit trail (An in-process tool call cannot be cancelled #26)."
    • SECURITY.md still discloses it: "…no deadline anywhere cuts it short, because tools run on a blocking thread that cannot be cancelled."
    • The bounds No run_turn: every consumer will reassemble the event stream itself, and the blocking boundary has no owner #58 shipped are all present: TurnLimits::max_rounds (turn.rs:130, default 8), enforced at :245, with TurnError::RoundLimit at :333; stream_timeout (:144, default 300s) applied at :274/:279.

    Correction to the comment above: the spawn_blocking call site is crates/sandbx-agent/src/turn.rs:364, not :226. answer_calls itself begins at :341. Line 226 now falls inside run_turn's round loop, near the compaction code. The drift is from 94e5001/55b2211/dd5717a (doc compression and test renames), none of which changed behaviour.

    Also note context/TOOLS.md is now context/guide-tools.md. It still commits to spawn_blocking (:126) and carries the honest row | cancellable from outside | **partly** — the turn can be abandoned; the running tool still completes (#26) | (:134).

    On the relationship to #128: that issue originally listed this one as blocked on a run_turn caller. That claim has been withdrawn there. Of the three parts left here, parallelism and the ExecutionContext ordering semantics need no caller at all — they are design questions answerable today — and cancellation needs an interactive caller, which #128's single-shot non-interactive subcommand is explicitly not. So nothing here is waiting on #128.

  6. danczw commented on Oct 6, 2026

    @danczw
    OwnerAuthor

    The precondition this issue deferred on is now met, and the gate added a constraint the redesign has to keep.

    "Decide it at Phase 5 when sandbx-agent exists to exercise it." It exists, and #128 gave it a caller in the binary. So the async-tool-API question is live rather than speculative — but neither half has been built, and the gate (#106, PR #170) is now a second caller whose correctness depends on the current shape.

    The constraint. approve is asked between BuiltinTool::from_name and spawn_blocking, never concurrently with it, because of this issue: dropping a JoinHandle leaves the blocking task running, so a gate raced against the call would answer "denied" about a write that had already landed. context/decision-approval-gate.md records it. If tool execution moves to tokio::process::Command with a cancellable future, the refusal stops depending on position — but any intermediate shape that keeps spawn_blocking must keep the gate ahead of it.

    The parallelism half gained a reason to stay sequential. answer_calls runs calls in the order the model asked for them, and its doc cites this issue: concurrency needs the ordering semantics of two tools sharing one ExecutionContext settled first. A gate decision per call is unaffected by ordering, so the gate does not make this harder — but a parallel answer_calls would call approve for several calls before any runs, which an operator prompt (#165) would have to render as a batch rather than one at a time. Worth deciding with #165 in view.

    Still not labelled security: unchanged by the gate, which refuses before a call starts and so needs no cancellation to be effective.

  7. removed this from the milestone on Oct 8, 2026
  8. changed the title [-]Tool calls cannot be cancelled or run in parallel[/-] [+]A running tool call cannot be cancelled[/+] on Oct 8, 2026
  9. danczw commented on Oct 8, 2026

    @danczw
    OwnerAuthor

    Triage at aae4bd0. Narrowed to cancellation; the parallelism half is now #242.
    This issue keeps its number because eleven of the twelve in-tree citations of #26 are
    about cancellation — including SECURITY.md:323-325 and a shipped release note, which
    should not be repointed. The one exception, crates/sandbx-agent/src/turn/tools.rs:14-15,
    is about parallelism and wants the new number.

    Title and body rewritten to match. What changed in substance since the last triage:

    The precondition this issue deferred on — "decide it at Phase 5 when sandbx-agent
    exists to exercise it" — is discharged twice over, and the interrupt landed reaching
    nothing.
    #133 shipped the tui interrupt: keys on a detached OS thread, because
    event::read cannot be cancelled (crates/sandbx-tui/src/input.rs:27-29, thread at
    :34-39); ctrl-C or Escape, latched (:78, interrupts at :99-105); raw mode taken
    before the reader or ctrl-C would arrive as a signal
    (crates/sandbx-cli/src/agent/tui.rs:154-156). The keystroke then drops the turn's future
    and stops there (tui.rs:176-179). So the shape to validate against now exists, and the
    gap is no longer "no caller" but "nothing in run_turn for the caller to reach".

    Every line reference in this issue and its comments is stale. run_with_deadline
    command.rs:194 → :279; the try_wait poll :234 → :316-325; killpg :341 →
    :423, inside kill_group at :415. answer_calls and the spawn_blocking site left
    turn.rs entirely — they are crates/sandbx-agent/src/turn/tools.rs:21 and :113, so
    both the :226 and the corrected :364 in the comments above are wrong now. run_turn's
    doc turn.rs:203-208 → :255-257. max_rounds :130 → :191 (default 8 at :215),
    enforced :245 → :297. stream_timeout :144 → :201 (300s at :217), applied
    :274/:279 → :337-343. guide-tools.md:126 → :187-189, and the honest table row
    :134 → :195.

    And a type correction: TurnError::RoundLimit does not exist. It is
    TurnStop::RoundLimit (turn.rs:167-170, constructed at :422-424) — a stop carrying the
    work the turn did, not an error. The 2026-10-05 comment's turn.rs:333 citation is for a
    name that was never there.

    Two things found while verifying, recorded rather than fixed here:

    Confirmed unchanged: the mechanism is still there, the bounds #58 shipped are all still
    present, and nothing has made a running call killable.

  10. 6 remaining items

  11. added this to the v0.1.0-alpha.26 milestone on Nov 12, 2026
  12. changed the title [-]A running tool call cannot be cancelled[/-] [+]An in-process tool call cannot be cancelled[/+] on Nov 12, 2026
  13. removed
    crate:cliargv parsing and the five subcommand bodies
    crate:coresandboxing; the only crate allowed to spawn a subprocess
    on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

crate:toolsthe seven built-in tools, each confined by corecrate:turnthe turn loopenhancementNew feature or request

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions