Skip to content

fix: dispatch's second review — no sandbox without an id, no turn deciding nothing, a way out of a clean-up that cannot finish - #63

Merged
OmarAlex24 merged 2 commits into
dispatch/review-fixesfrom
dispatch/review2-fixes
Nov 17, 2026
Merged

OmarAlex24 merged 2 commits into
dispatch/review-fixesfrom
dispatch/review2-fixes

Conversation

@OmarAlex24

@OmarAlex24 OmarAlex24 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

The second review of #61 and #62 found 12 problems (head 9b3ff2e). Two of them leave a sandbox billing. This PR fixes them, plus what the real-Boat run at 9b3ff2e showed. It stacks on #62; the owner merges.

What changes

Sandboxes that kept billing

  • boat new without an id (finding 1).
    • The run now records, before new is asked, that a sandbox may exist without an id.
    • It stays unsettled until the id is known or a look-up by name says there is none. Until then it holds its epic and a slot, stop exits non-zero, and the watcher stays.
    • A new cut off by its time limit keeps what it printed, so its created line still gives the id.
    • Any create failure without an id looks the sandbox up by name, in every clean-up path.
    • A boat list page that says hasMore and lacks the name counts as "not known yet". boat list --help has no cursor option. An answer of another shape counts the same way.
    • The watcher waits for a sandbox longer than new may take: 16 minutes instead of 2.
  • A turn Boat no longer knows (finding 3).
    • The runner's read <n> now decides it. An exit file gives the verdict. No pid file means the turn never ran, which is stuck. A pid that still runs (the runner now says alive=) is followed.
    • A turn that cannot be followed for 10 minutes in a row makes the run stuck, with the reason.

Smaller fixes

  • Finding 2. A turn whose detach was already on its way when the run was stopped is ended, and the sandbox is stopped again (best effort). This covers both the dispatch's first turn and the watcher's turns.
  • Finding 4.
    • machine_not_running from an exec during clean-up means nothing is left to kill or remove there. It is reported, not counted as a failure, and the stop goes on.
    • New: kontext dispatch stop <epic> --forget, the owner's way out.
      • Only for a run that already ended and is not clean.
      • It stops nothing and tries the token's revoke once more.
      • It marks the run forgotten, which counts as clean.
      • It writes the end entry and lists what kontext did not verify.
  • Finding 5. When the credential's rm fails for another reason, the run records credential_left.
    • stop exits 1 and says to replace the credential (claude setup-token, then kontext dispatch credential set).
    • status shows it, and the stuck question does not suggest boat resume.
    • Docs:
      • Boat's stop keeps /dev/shm. What keeps the credential out of a stopped sandbox is the removal before kontext stops it.
      • A stop Boat makes itself keeps the credential in that sandbox's saved state.
      • kontext clone keeps the (revoked) kontext token in the sandbox's kontext config.
  • Finding 6. A live run with an empty id (written by an older kontext) gets its sandbox stopped, and the run is stuck with "dispatched by an older kontext".
  • Finding 7. status shows the "watcher is not running" line whenever some run is not settled. The overdue refusal names the command that starts it.
  • Finding 8. The watcher leaves only when the project is no longer registered. A project.toml that cannot be read is logged and tried again. An error from done() no longer ends it either.
  • Finding 9. The clean-up lease is now 20 minutes. A whole clean-up with every call at its limit takes 13.
  • Finding 10. The watcher's after() skips a run that holds a lease. The end entry ("its sandbox stopped") is written only once the run is clean, by stop and by the watcher alike.
  • Finding 11. A Failed turn seen on a failed pull waits for a fresh pull, as Ended does. "Every failure is final" stays, and DESIGN says so.
  • Finding 12.
    • credential set refuses a bare prefix and NUL or control bytes.
    • The run of spaces in stop's message is gone.
    • A --json failure before the reservation shows "run": null instead of an older run.
    • Both skills say that dispatch can hold the foreground for about 20 minutes, and that the watcher finishes a killed one.

From the real run

  • A failed start script's reason is one line: the last kontext-dispatch: line, else the last non-empty line. The whole text, redacted, goes in the watcher's log or the --json output's detail.
  • The stuck question points at boat history <id> and ~/.kontext-dispatch/turn-<n>/. boat events printed nothing in the real run.
  • The start script installs gh stack when gh extension list lacks it. A failure is reported on stderr and is not fatal.
  • Docs (DESIGN §12.5, README, skills):
    • Boat's GitHub app must have access to the code repo.
    • A non-GitHub [[repos]] remote is cloned as written.

Found while fixing: a stop's clean-up could clear a pending mark that the watcher's give_up had just set again. Fixed with a looked mark (the clean-up clears only the mark it looked for) and a fresh mark in give_up. No test; checked by reading.

The fake boat now answers as real Boat did in the facts file:

  • any exec on a stopped sandbox exits 1 with machine_not_running;
  • stop on a stopped sandbox exits 0;
  • --kill of an exited process exits 0 with signaled:false;
  • list --json is one {"pageInfo","sandboxes"} document.

New steering files: fail-new-silent, list-more, gone-<id>-<n>, hang-detach. The fake's info now JSON-escapes a multi-line setupError.

Tests

  • Two tests pinned the old behaviour and were rewritten:
    • a_credential_that_cannot_be_removed_still_lets_the_sandbox_stop is now …_stops_the_sandbox_and_says_to_replace_it, and expects exit 1.
    • a_turn_that_cannot_be_followed_decides_nothing is now a_turn_boat_forgot_is_read_from_the_sandbox_and_an_unfollowed_one_ends.
  • a_failed_turn_is_stuck_at_once_with_its_reason also checks the stuck question's hint.
  • The start-script test also checks the gh stack install.

Test plan

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings (fresh target dir)
  • KONTEXT_REQUIRE_SERVER_TESTS=1 cargo test, full suite, fresh target dir: 544 passed, 0 failed
  • This PR's tests/dispatch.rs and tests/fixtures/fake-boat.sh run against 9b3ff2e's code (scratch worktree, own target dir): the 17 new or changed tests fail and the other 29 pass. Each fails on its first assertion that the old code breaks, so a later assertion in the same test was not red-checked: the spaces in stop's message (behind finding 10's assertion), finding 3's "stuck after 10 minutes" and "turn never ran" (behind its first half; the old suite's a_turn_that_cannot_be_followed_decides_nothing pinned the opposite), and the forget test's exit-2 refusal of a live run (old clap exits 2 on the unknown flag; a later assertion fails). The new unit tests (lease length, list_lookup, credential checks, failure_line, the new grace) use symbols 9b3ff2e does not have, so they were not run there.

Run against real Boat at 9b3ff2e, before this PR's changes, by the session that ordered it

  • A refusal without a credential: exit 2, one line.
  • A start script that fails (git clone 403: the sandbox's GitHub login had no access to a new private repo): dispatch exit 1, credential removed, sandbox stopped, token revoked, tmp/ empty, watcher gone.
  • Redispatch after stuck, a whole real run (Opus 5.5): clone from a non-GitHub remote; git author from gh api user; "coordinator started … turn 1", exit 0; the coordinator did both tasks, pushed a branch and closed them in about 90 s; the watcher saw the epic closed and ran exec --kill, exec … rm -f <credential paths>, stop --yes, token revoked. Run closed.
  • Ctrl-C during the setup wait: exit by signal, tmp/ empty; the watcher took the start over, waited for the start script and started turn 1.
  • Resume once, then stuck (Haiku): turn 1 ended → resumed → turn 2 (--resume, same session) ended → stuck, with a question to the pm, credential removed, sandbox stopped, token revoked.
  • Ctrl-C during boat new: the sandbox existed anyway; the watcher found it by name 5 s later and finished the start.
  • dispatch stop during turn 1: exec --kill <pid>, rm, stop, token revoked; exit 0; run stopped.
  • Secrets: no kxt1. or sk-ant- in the argument log, boat history, boat events, the state folder, the dispatch output or the ledger; in the sandbox the credential was only in /dev/shm/kontext-dispatch-agent-credential (0600).
  • A state.json in docs: kontext dispatch in DESIGN, the README and the pm's skills #58's format (one closed run) is read by dispatch status.
  • dispatch credential rm removes the file.

Run against real Boat at 57dffcb (this PR's head), one sandbox, by the session that ordered it

  • A state file written by 9b3ff2e is read by this binary's dispatch status.
  • dispatch of an epic stopped earlier, with a credential that is not valid: boat new through the new create path gives the id; the start script finishes; turn 1 fails; dispatch exits 1 with one line ("the coordinator did not start: Claude Code said: Failed to authenticate. API Error: 401 …"); the credential is removed, the sandbox stopped, the token revoked; the run is stuck with that reason.
  • The start script installed gh stack in the sandbox: after a resume, gh extension list says gh stack github/gh-stack v0.3.0 (so the script's grep gh-stack matches) and gh stack --help works.
  • After that resume there was no credential file in /dev/shm or under $HOME.

Last run against real Boat at 57dffcb, three sandboxes, a real credential and a private GitHub repo

  • Pull requests from a coordinator (Opus 5.5): the start script cloned the private repo and installed gh stack; the coordinator opened two draft pull requests, the second based on the first, in one GitHub stack, and closed the epic; the watcher killed the turn, removed the credential, stopped the sandbox and revoked the token. Checked from outside the sandbox with gh.
  • Resume once, then stuck (Haiku): turn 1 ended, the watcher resumed it once, turn 2 ended, the run went stuck, cleaned up, and the pm's question carries the boat history hint.
  • A sandbox stopped with plain boat stop while turn 1 ran: dispatch got machine_not_running, warned that the credential stays in that sandbox's saved state, stopped (exit 0 from Boat), revoked the token and exited 1; the run is stuck and clean; a later dispatch stop is refused with exit 2.
  • No kxt1. or sk-ant- in boat history, boat events, the argument log, the dispatch output or the state folder.

Not checked against real Boat

  • what boat stop says for an archived or deleted sandbox;
  • whether boat exec reaches a sandbox that is stopping;
  • whether a reboot makes Boat forget a detached pid;
  • whether boat new prints its created line before its time limit cuts it off (finding 1's fix reads it from the partial output; the look-up by name covers the rest);
  • the runner's alive= check through /proc/<pid>/cmdline;
  • a sandbox stopped from outside after dispatch has returned, seen by the watcher (it was seen by dispatch itself, above).

🤖 Generated with Claude Code

OmarAlex24 and others added 2 commits November 16, 2026 20:00
…nnot finish never leave a run deciding nothing

The second review of #61 and #62 (findings 1 to 12) and the real-Boat run at 9b3ff2e:
- A `boat new` that fails or is cut off without an id: what it printed is kept, the
  sandbox is looked up by name, and the run stays unsettled until the id is known or
  Boat's whole list says there is none (a page with hasMore does not). The watcher waits
  longer than `new` may take.
- A turn Boat no longer knows is decided by the runner's `read`; one that cannot be
  followed for 10 minutes makes the run stuck. A failed turn waits for a fresh pull.
- machine_not_running during clean-up is nothing to remove; a credential that cannot be
  removed is recorded and makes `stop` exit 1; `stop --forget` is the owner's way out.
- A turn that started while the run was stopped is ended and the sandbox stopped again.
- The lease outlasts a clean-up, the end entry waits for the stop, the watcher leaves only
  when the project is unregistered, a live run from an older kontext is ended, and
  `status` shows the watcher line while any run is not settled.
- The start script installs gh stack; a failed start script's reason is one line.
- The fake boat answers as real Boat did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st reach, and what --forget does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OmarAlex24
OmarAlex24 added this pull request to stack #59 November 17, 2026 02:00
@OmarAlex24
OmarAlex24 marked this pull request as ready for review November 17, 2026 14:30
@OmarAlex24
OmarAlex24 merged commit 27b7d92 into main Nov 17, 2026
1 check passed
@OmarAlex24 OmarAlex24 mentioned this pull request Nov 17, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant