Skip to content

No per-host capacity model: bb over-schedules one machine until its daemon dies, and drops the spawn prompt on retry #2433

Description

@brsbl

Summary

bb schedules work onto enrolled machines with no capacity model. The server will start any number of environment provisions and thread turns on one host, so a modest machine can be pushed until its daemon dies — and when it does, in-flight work is lost rather than queued or retried.

Reproduced on a two-machine account: an 8 GB laptop enrolled as a second machine produced three Host is not connected daemon deaths in ~2 hours, one of them while a second spawn ran pnpm install (2000+ packages) concurrently with an active build. Two smaller gaps surfaced alongside it, both of which make multi-machine work lossy.

1. No concurrency guard anywhere

There is no limit on concurrent jobs per host — not in environment-provisioning-internal.ts, not in turn execution. A "job" that saturates a machine is either an environment provision or an active turn, and bb will run arbitrarily many of both on one daemon.

Placement is also manual-only: bb thread spawn --machine <id>, with no notion of which machine is busy. Without --machine, bb silently uses the default source's host regardless of load.

2. A spawn's initial prompt is dropped when provisioning retries

The first turn is requested before provisioning completes. If provisioning fails and recovers, the worktree is restored but the prompt is not re-queued — the thread comes up idle and promptless, and the requested work never runs.

Observed in the event log: client/turn/requested at spawn, then thread_provisioning_failed / "Host is not connected", then a successful re-provision with no turn.

3. --file passes a path, not content

--file / --image on spawn, tell, and fork resolve on the executing host, not where the CLI ran. Attaching a file to a worker on another machine silently hands it a path that does not exist there, so there is no supported way to move a file to a thread on another host.

Why this looks like an oversight rather than a decision

The daemon is already supervised for crash recovery — install-machine.sh installs it with launchd KeepAlive=true, so it reconnects on its own. The intent to survive daemon death exists; what's missing is (a) not overloading the host in the first place and (b) preserving in-flight state across the reconnect that already happens.

Every comparable system bounds this by default: Buildkite agents take one job each and scale by agent count, GitHub Actions runners take exactly one job per runner process, GitLab Runner exposes a single concurrent integer. None ship unbounded.

Reproduction

Over-scheduling (needs a ≤8 GB second machine):

# with a build already running on <host>
bb thread spawn --project <p> --machine <host> --new-environment worktree --prompt 'hello'
# the concurrent pnpm install + build kills the daemon; both threads lose their connection

Prompt loss:

bb thread spawn --project <p> --machine <host> --new-environment worktree --prompt 'echo hello'
# kill the host daemon during provisioning, then let it reconnect
bb thread output <id>   # empty — the thread is idle and the prompt never ran

Cross-machine attachment:

bb thread tell <id-on-other-host> --file /absolute/path/on/this/machine.md "read this"
# worker reports the path does not exist

Suggested fix

Capacity model. Each host daemon reports logical cores, total RAM, and a derived maxConcurrentJobs at connect and heartbeat — RAM-tiered defaults (<12 GB → 1, 12–24 GB → 2, >24 GB → 4), overridable per machine in settings. A job is a provisioning run or an active turn.

Per-host queue. Work exceeding maxConcurrentJobs waits visibly rather than failing or overloading: thread status reads queued (position n on <machine>). Queued spawn prompts are held durably with the thread, not with the provisioning attempt.

Load-aware placement. spawn without --machine places onto the least-loaded connected machine that has the project source registered, and names the chosen machine in its response. --machine stays an explicit pin. When every eligible machine is at capacity, enqueue on the least-loaded one.

Prompt survives provisioning retry. Persist the initial prompt with the thread at spawn time; re-queue it after any provisioning recovery.

Attachments carry content. --file / --image read bytes where the CLI runs, upload to thread storage, and materialize on the executing host.

Thread graduation (bb thread move <id> --machine <target>). Migrate a running thread to another machine — quiesce the turn, WIP-commit and push a dirty worktree, provision or reuse an environment on the target from that branch (subject to its capacity queue), rebind, and retire the old environment only after the new one verifies at the pushed head. Thread identity, history, and attachments are already server-side; process state does not transfer and is recorded in the move message. Manual command only — no auto-migration policy in this pass. Requires the same project source registered on the target; a thread mid-provisioning cannot move.

Acceptance criteria

  • A machine reporting <12 GB RAM never runs two jobs concurrently; the second visibly queues
  • Killing a daemon mid-provisioning and reconnecting never loses the spawn's initial prompt
  • spawn without --machine on a two-machine account places onto the less-loaded host and names it
  • tell --file from machine A delivers file content to a worker on machine B
  • thread move on a thread with a dirty worktree lands on the target at the pushed head with nothing lost; the old environment is retired only after verification
  • Single-machine flows are unchanged (one machine with capacity ≥1 behaves exactly as today)

Out of scope

Cross-machine artifact sync beyond attachments; LAN-direct daemon transport (the relay path stays); moving or replicating the bb server itself — graduation moves threads, not the control plane; autoscaling or cloud machines.

Metadata

Metadata

Assignees

No one assigned

    Labels

    clibb CLIhostHost daemon, process lifecycle, memory, event loop

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions