Keep a fleet of long-running coding agents alive across restarts, upgrades and machines.
Watchbill is a cockpit CLI for people who run several coding agents — Claude
Code, Grok, Codex, OpenCode — as persistent sessions inside a terminal
multiplexer (Herdr, tmux, or cmux on
macOS), on one machine or across a few reached over SSH or Tailscale. It
catalogs what is running where, stands it down cleanly, brings every agent
back into the same pane with its own conversation and configuration, and
wraps multiplexer upgrades, agent CLI upgrades, plugin installs and harness
restarts around that cycle. Every mutating command is a dry-run until you
say --yes.
A multiplexer server owns the PTYs of every agent inside it. The moment you upgrade it, restart it, install a plugin with startup hooks, or reboot the box, every agent dies with it. With one agent that is a nuisance; with eight agents across three machines it is an afternoon:
- you have to remember which pane held which conversation, on which host, in which directory, started with which flags and permission mode;
- each agent CLI resumes differently (
claude --resume <id>,codex resume <id>,opencode --session <id>,grok --resume <id>), and only if you still have the session id; - a server restarted from an SSH shell loses the desktop session, so resumed agents can no longer open a browser or reach the keychain;
- one agent mid-task, or sitting on an approval dialog, must not be typed into; one you forgot about gets killed;
- plugins, hooks, MCP servers and folder trust drift between what the agent had and what it comes back with;
- and it all happens again next week.
Multiplexers each solve a slice of this (Herdr knows its agents, cmux restores its own, tmux knows nothing), none of them across machines, and none of them around an upgrade of themselves.
| Verb | Does | Mutates? |
|---|---|---|
watchbill roll |
catalog every occupant of every session on every host: role, agent kind, status, session id, cwd, flags | no |
watchbill snap |
roll and write a versioned roster (JSON); current.json points at the last good one |
roster only |
watchbill secure park|fold|dismiss|host [targets] |
/exit the agents (only the idle ones), interrupt watchers; optionally close workspaces or stop sessions |
with --yes |
watchbill set [targets] |
start the session if needed, rebuild workspaces and panes by label, resume each agent on its session id with its recorded flags, relaunch allowlisted watchers, send each agent a resume prompt | with --yes |
watchbill relieve <action> |
a maintenance window: secure → action → set, one host at a time, resumable after a failed host | with --yes |
watchbill status |
live roll against the last roster: what appeared, vanished, changed | no |
watchbill doctor |
per host: mux version, install flavor, whether a live handoff is possible, agent CLI versions, mux-specific warnings | no |
The roster is the contract. Each occupant has a durable slot_id and a
human_id of the form host/session/workspace/tab/pane; live pane ids are
hints for one server generation, never keys. Rosters never contain secrets:
credential flags are redacted, and environment variables are recorded by
name only when they look like secrets.
Each agent comes back as the same agent. At park time Watchbill records the agent's flags, the permission mode and model they imply, its config directory, its non-secret seat environment, and a fingerprint of its settings, plugins, hooks, MCP servers, folder trust and CLI version. At resume the flags, working directory and environment are restored exactly, the on-disk configuration is verified before the agent starts, and a change that would break the resume stops that host instead of guessing.
The session comes back in the same environment. A server restarted over SSH is given the user's live desktop session variables (display, bus, desktop), never the SSH connection Watchbill arrived on; on macOS a Herdr or tmux server is started through launchd or the app so the keychain works.
| Multiplexer | Verified on | Agent detection | Agent status | Resume | Session stop / start |
|---|---|---|---|---|---|
| Herdr 0.8.2, 0.9.0 | Linux (Omarchy), macOS | native | native | native session id → each CLI's resume flag | session stop / the host's start (systemd unit, launchd, or detached server) |
| tmux 3.x | Linux, macOS | from the process table (ps -t) |
heuristic: quiet time + approval-prompt patterns | cwd-scoped --continue, refused when two agents of one kind share a directory |
kill-server / new-session -d with the exact layout re-applied |
| cmux 0.64 | macOS | native (cmux agent hooks) | native (hook lifecycle) | native session id; the app resumes its own agents on relaunch and Watchbill waits for it | AppleScript quit / open -a cmux |
| Agent CLI | Park | Resume form | Live-tested |
|---|---|---|---|
| Claude Code | /exit |
claude <flags> --resume <id> |
Herdr, tmux, cmux |
| Grok | /exit |
grok <flags> --resume <id> |
Herdr, tmux |
| Codex | /quit |
codex resume <flags> <id> |
Herdr |
| OpenCode | /exit |
opencode <flags> --session <id> |
Herdr |
| Gemini CLI | /exit |
gemini <flags> --resume <id> |
flag table only |
| Cursor | /exit |
cursor-agent <flags> --resume <id> |
flag table only |
Transports: local, ssh_cli (per-host ssh_options), and an
exec_prefix so the same Watchbill can manage a seat inside a container
(for example distrobox enter <name> --) on this or another machine.
Watchbill itself runs on Linux or macOS with Python 3.11+ and no runtime
dependencies; the hosts need their mux CLI and ssh access with keys.
uv tool install git+https://github.com/gw7523/watchbill # or: pipx install git+…
watchbill --help
# from a clone
git clone https://github.com/gw7523/watchbill && cd watchbill
uv sync --group dev && uv run pytest # no live mux needed for the testsConfiguration lives in ~/.config/watchbill/: hosts.toml (required),
allowlist.txt (watcher command lines allowed to relaunch), pins.toml
(resume prompts per agent), prompts/<role>.txt, config.toml. Rosters go
to ~/.local/share/watchbill/rosters/<fleet>/, the journal to
~/.local/state/watchbill/journal.jsonl. WATCHBILL_HOME=/some/dir keeps
all of it under one directory for a rehearsal.
hosts.toml for a cockpit and three targets:
[fleet]
name = "home"
[[host]] # the machine you run watchbill on
name = "rig2"
cockpit = true
transport = "local"
[[host]] # Herdr on Linux, server owned by a systemd user unit
name = "ser6"
target = "ser6.tail1234.ts.net"
sessions = ["default"]
start = "systemctl --user start herdr.service"
ignore = ["*distrobox enter sfl*"] # catalogued, never touched; a stop may end it
[[host]] # tmux on a Mac
name = "mini-tmux"
target = "mini.tail1234.ts.net"
ssh_options = ["-i", "~/.ssh/id_ed25519_personal"]
mux = "tmux"
sessions = ["work"] # tmux server socket names (-L)
mux_options = { idle_after_s = 30 }
[[host]] # cmux on the same Mac
name = "mini-cmux"
target = "mini.tail1234.ts.net"
ssh_options = ["-i", "~/.ssh/id_ed25519_personal"]
mux = "cmux"
sessions = ["app"]Then:
watchbill doctor # every host reachable? versions, install flavor, warnings
watchbill roll # who is running where
watchbill snap -m manual # write the first roster
watchbill secure park # dry-run: what would be parked, what is refused and why
watchbill secure park --yes # park every idle agent on every target host
watchbill set --yes # bring them all back, each on its own conversation
# (falls in from the roster taken before the park)Targets for secure and set are a slot_id, a full human_id, or a label
suffix (personal-config/1/p1, or just personal-config); --host breaks
ties. A working agent is refused unless --force; an agent on an approval
dialog is never typed into. The cockpit host is left alone unless
--include-local.
Put a resume prompt in pins.toml for agents that should be told what to do
after they come back:
[pins."ser6/default/personal-config/1/p1"]
prompt = "You were relaunched by Watchbill. Re-read TODO.md and continue the open item."relieve runs one action per host, rolling: park the affected agents, stop
the session if the action needs it, run the action, bring the session and
the agents back, verify, journal. A failed host stops the run; --resume
continues with the hosts not yet done. Each action declares its blast
radius:
| action | parks | stops the session | what it runs |
|---|---|---|---|
upgrade-mux (alias upgrade-herdr) |
agents | yes | Herdr: the installer for its flavor (mise, official; pacman refused unless --allow-partial-pacman). tmux: brew upgrade tmux / pacman -S tmux. cmux: brew upgrade --cask cmux (not yet verified live) |
restart-harness / restart-herdr |
agents | yes | nothing but the stop and start: for a binary already replaced, a config that needs a bounce, a plugin with startup hooks |
reload-config |
nothing | no | herdr server reload-config, tmux source-file, cmux reload-config |
upgrade-agents [--kinds claude,codex] |
those agents only | no | mise upgrade / npm install -g / brew upgrade / the CLI's self-update, per install flavor; Herdr integration hooks refreshed |
install-plugin --plugin owner/repo [--ref v1] [--startup-hooks] |
agents if startup hooks | if startup hooks | herdr plugin install; tmux: TPM + source-file; cmux: refused (no plugin system) |
omarchy-update |
agents | yes | omarchy-update -y on an Omarchy host |
custom --cmd '…' [--park agent,watcher] [--session-stop] [--may-reboot] |
as declared | as declared | anything else |
watchbill doctor # flavors: rig2 mise, ser6 official, vps mise
watchbill relieve upgrade-mux # dry-run: the full plan, host by host
watchbill relieve upgrade-mux --yes --expected-version 0.9.0Per host: /exit each idle agent and wait for it to leave, copy the session
file aside, herdr session stop, run the installer, start the server with
the user's desktop environment (start = in hosts.toml, or a detached
herdr --session S server), rebuild any workspace the server did not
restore, herdr agent start … -- claude --resume <id> <flags> for each
agent, wait until each one is idle, send its pinned prompt, and check that
doctor now reports 0.9.0. A host whose agents are working refuses the
window; park them first, wait, or pass --force knowingly. An Omarchy host
with a pacman-owned Herdr is refused: use omarchy-update there, or
--allow-partial-pacman if you accept a partial upgrade.
watchbill relieve upgrade-mux --host mini-tmux --yestmux keeps nothing across kill-server, so Watchbill records the exact
window layouts and every pane's command first, parks the agents, kills the
server, runs brew upgrade tmux, creates the sessions and windows again with
new-session -d / new-window, re-applies each recorded layout, relaunches
the allowlisted watchers, and resumes the agents with their cwd-scoped
--continue form (when two agents of one kind share a directory it refuses
to guess and starts that one fresh, with a note). Start the first tmux
server from a GUI terminal on the Mac, not over SSH, if the agents need the
keychain.
watchbill relieve restart-harness --host mini-cmux --yes # or upgrade-muxcmux's own agent hooks tell Watchbill which surface holds which session and
whether it is idle. The window parks the agents, quits the app through
AppleScript (app.confirmQuit = "never" in ~/.config/cmux/cmux.json,
which doctor checks), relaunches it with open -a cmux, and then waits:
cmux rebuilds its workspaces with the same ids and resumes each agent itself.
Only if an agent does not reappear does Watchbill type the recorded resume
command. The pinned prompt is sent once the agent is idle.
watchbill relieve restart-harness --yes # every target host, one at a time
watchbill relieve restart-harness --yes --resume # after fixing a host that failed
watchbill relieve upgrade-agents --kinds claude --yes # Claude CLIs only; sessions stay up
watchbill relieve install-plugin --plugin acme/herdr-notify --startup-hooks --yes
watchbill relieve reload-config --yes # live, nothing parkedupgrade-agents parks only the agents of the kinds being upgraded, leaves
the session and every other pane running, upgrades each CLI the way it was
installed, and resumes those agents on the new binary with their original
session ids and flags.
watchbill snap -m heartbeat # from cron or a timer
watchbill status # what changed since the last rostercurrent.json is only retargeted when the occupant count is plausible (a
reboot can leave a mux with an empty session file); snap --force overrides
the guard.
- Dry-run by default;
--yesexecutes. The plan you see is the plan that runs. - Only
exec.pymutates anything. Planners are pure functions of the roster and collected facts; transports are the only code that knows about SSH. - Never an unplanned server stop:
herdr server stop, atmux kill-serveroutside a declared window, or a cmux quit outside one needs--force-server-stop. - Never a keystroke into an agent that is working or on an approval dialog;
a Claude with Remote Control connected is refused (it ignores
/exit) unless the host opts into disconnecting it. excludeglobs protect occupants: never parked, never restored, and a window that would stop their session is refused.ignoreglobs mark occupants as out of scope without protecting them.- Exit codes:
0ok ·1partial (a host was down) ·2usage ·3refused unsafe plan ·4transport or action error. A journal records every step.
| key | default | meaning |
|---|---|---|
name |
— | host name used in human_id and --host |
target |
— | ssh destination (user@host or an ssh config alias); omit with transport = "local" |
transport |
ssh_cli |
local or ssh_cli |
cockpit |
false |
the host Watchbill runs on; skipped by mutating verbs unless --include-local |
mux |
herdr |
herdr, tmux, cmux |
sessions |
["default"] |
Herdr session names, tmux socket names, or a label for the cmux app |
start |
per mux | how to start a session: a systemd unit, launchctl kickstart gui/$(id -u)/<label> on macOS, or the built-in detached server start; open -a cmux for cmux |
ssh_options |
[] |
extra ssh arguments, e.g. ["-i", "~/.ssh/id_x"] |
exec_prefix |
[] |
run every command inside an environment on the target, e.g. ["distrobox", "enter", "sfl", "--"] |
exclude / ignore |
[] |
globs against an occupant's command line or human_id |
mux_options |
{} |
tmux: idle_after_s, conf; cmux: socket, idle_after_s; herdr: attach_before_resume, disconnect_remote_control |
- No live handoff of PTYs: a session stop always ends its agents; Watchbill resumes conversations, it cannot keep a process alive across the stop.
- tmux agent status is a heuristic; an
unknownstatus needs--forceto park, like aworkingone. - Gemini CLI and Cursor resume forms come from their
--help, not from a live run.brew upgrade --cask cmuxhas not been run live. - A freshly relaunched cmux agent reads
unknownuntil its first turn. - Herdr
--remoteis not used as a transport (version-string skew in 0.8.2 packages); every host is reached byssh … -- herdr --session S.
docs/architecture.md module map, sequences, pitfalls → tests
docs/mux-backends.md the mux axis, capability matrix, per-mux action table
docs/herdr-0.8.2-facts.md, docs/tmux-3.7-facts.md, docs/cmux-facts.md, docs/macos-facts.md,
docs/agent-cli-facts.md what was verified on live boxes; check before touching any argv
docs/roster.schema.json roster schema 1
examples/tmux/ runnable walkthrough: a throwaway tmux server cycled through a window
skills/watchbill/ SKILL.md for an agent that drives Watchbill
plugin/ herdr-plugin.toml (id agents.watchbill)
AGENTS.md how to work on this repo (CLAUDE.md points here)
Apache-2.0. See CHANGELOG.md and CONTRIBUTING.md.
