data-angel · Oct 17, 2026 · Oct 17, 2026 · Oct 17, 2026
@@ -164,6 +164,7 @@ CLI surface evolves; check `lloom help` before documenting a missing command or
 - `spark-node-01` is the common Spark host alias, but re-check SSH, config, active workload, container/image, and memory state.
 - Use Tailscale/private admin reachability where configured. Keep inference/admin authorization distinct.
 - Before changing Jinx's substrate, gracefully drain her Runtime presence or otherwise coordinate the live workload.
+- For shared hardware, take a lease on the owning head: `lloom reserve <resource> --apply --yes`, report work with `lloom job …`, and `lloom release <resource> --apply --yes` when done; check `lloom schedule` before using hardware a lease covers. See `docs/reservations.md`.
 - For temporary hardware takeover, use `lloom suspend <model-or-alias> --apply --yes` and `lloom resume <model-or-alias> --apply --yes` through the owner gateway; see `docs/model-maintenance.md`. Model maintenance gates all aliases and bare model calls, drains active work, verifies unloading, and restores routing only after health. Alias-member suspension alone does not unload or block direct calls. Older deployed gateways without these commands still require the legacy takeover procedure.
 - Sync LLooM credentials from the actual gateway environment; a 401 after a model change may be stale auth rather than model failure.
 - Do not call an alternate lane as a “light role model” if doing so causes disruptive swapping. Co-residency and quality must be demonstrated.

@@ -6,6 +6,8 @@ All notable changes to LLooM will be documented in this file. The format follows
 
 ### Added
 
+- Built-in reservations and a job log (`lloom schedule`, `reserve`, `renew`, `release`, `job`; `/gateway/schedule`). Expiring leases on configured resources run route and suspend actions, possibly on other cluster nodes, and undo them in reverse with health-gated restore. There are queues, renewal and heartbeat-extended leases, stalled-job detection, and an event feed for boards such as Tessera. Schedule refusals now reach the CLI as redacted messages.
+
 - Atlas SparkQwen RC16 two-Spark recipe (`linux-nvidia-dgx-spark-2x-qwen38-flash-next-atlas`, profile `8x262k`) with a fail-closed pin gate and a digest-only image installer, and explicit per-node `${fabricHca}` settings for distributed recipes. Both Atlas recipes pin the published images with exact rail selection; missing settings and ambiguous discovery fail clearly.
 
 - LLooM Hear CPU audio analysis, with structured estimates, optional dashboard images, and opt-in upstream interpretation. File and URL inputs require operator configuration.

@@ -587,6 +587,22 @@ candidates. Resume restores eligibility after guarded loading and health checks.
 Omit `--apply --yes` to preview. See [model maintenance](docs/model-maintenance.md)
 for alias setup, distributed ownership, timeouts, and recovery.
 
+### Share hardware between agents
+
+```sh
+lloom schedule                                         # holders, queues, jobs
+lloom reserve pair --purpose "decode A/B" --wait --apply --yes
+job=$(lloom job start "decode A/B" --on pair)
+lloom job progress $job 2/6 --unit arms
+lloom job done $job "flat vs baseline"
+lloom release pair --apply --yes
+```
+
+A reservation is an expiring lease on a configured resource. Taking it routes
+aliases away and suspends that resource's models, and releasing it restores them
+after a health check. Jobs report progress to one board instead of each agent's
+window. See [reservations and jobs](docs/reservations.md).
+
 For authentication, transport failures, and machine-readable CLI errors, see
 [Gateway diagnostics](docs/cli-diagnostics.md).
 

@@ -93,6 +93,7 @@ import {
 import { createLloomServer } from '../src/server.mjs';
 import { deploymentFailureReport, formatDeploymentReport, runDeploymentCli } from '../src/cluster-deployment-cli.mjs';
 import { runNodeAgentCli } from '../src/node-agent-cli.mjs';
+import { runScheduleCli, SCHEDULE_COMMANDS, scheduleUsage } from '../src/schedule-cli.mjs';
 import { applySetup, createSetupPlan, syncClusterSetupMembers } from '../src/setup.mjs';
 import { createSetupStatus } from '../src/setup-status.mjs';
 import {
@@ -117,6 +118,11 @@ const COMMAND_REGISTRY = [
   { name: 'suspend', aliases: [], tier: 'primary', needsInstalledConfig: true },
   { name: 'resume', aliases: [], tier: 'primary', needsInstalledConfig: true },
   { name: 'route', aliases: ['routing'], tier: 'primary', needsInstalledConfig: true },
+  { name: 'schedule', aliases: [], tier: 'primary', needsInstalledConfig: false },
+  { name: 'reserve', aliases: [], tier: 'primary', needsInstalledConfig: false },
+  { name: 'renew', aliases: [], tier: 'primary', needsInstalledConfig: false },
+  { name: 'release', aliases: [], tier: 'primary', needsInstalledConfig: false },
+  { name: 'job', aliases: ['jobs'], tier: 'primary', needsInstalledConfig: false },
   { name: 'fleet', aliases: ['profiles'], tier: 'primary', needsInstalledConfig: true },
   { name: 'integrate', aliases: [], tier: 'primary', needsInstalledConfig: true },
   { name: 'integrations', aliases: [], tier: 'primary', needsInstalledConfig: true },
@@ -194,6 +200,9 @@ Primary commands:
   lloom models                     List gateway model IDs
   lloom suspend <model> --apply --yes  Drain, unload, and suspend a model
   lloom resume <model> --apply --yes   Load, verify health, and restore routing
+  lloom schedule                   Who holds shared hardware, queues, and agent jobs
+  lloom reserve <resource>         Lease shared hardware (preview; add --apply --yes)
+  lloom job start <title>          Report a long-running job; returns its id
   lloom integrate <client|all>     Write client configs (omp, opencode, codex, …)
   lloom add-model <ref>            Import an ad hoc model (dry-run; add --apply --yes)
   lloom remove-model <model-id>    Remove a model and its dedicated config resources
@@ -205,7 +214,9 @@ Also useful:
   lloom help advanced              Full command catalog
 
 Environment:
-  LLOOM_CONFIG  Path to config JSON
+  LLOOM_CONFIG    Path to config JSON
+  LLOOM_HEAD_URL  Gateway that owns the schedule (key in LLOOM_HEAD_API_KEY)
+  LLOOM_OWNER     Your name on reservations and jobs
 `;
 }
 
@@ -256,6 +267,8 @@ Backends and runtimes:
   lloom runtime-start|runtime-warmup|runtime-stop <runtime-id>
   lloom keep-warm
 
+${scheduleUsage()}
+
 Models and clients:
   lloom route [alias [profile]] [--suspend-member model|--resume-member model] [--apply --yes]
   lloom add-model <hf-url|repo-id|local-path|ollama:tag|lmstudio:id|openai:url#model> [--backend id] [--api-key-env NAME] [--input TYPE] [--capability ID] [--tag ID] [--keep-warm] [--default] [--go|--apply --yes]
@@ -399,7 +412,8 @@ const INSTALLED_CONFIG_COMMANDS = new Set([
   'setup',
   'setup-status',
   'status',
-  'up'
+  'up',
+  ...SCHEDULE_COMMANDS
 ]);
 
 const OPERATIONAL_CONFIG_COMMANDS = new Set([
@@ -3074,6 +3088,11 @@ async function main() {
   handlers['pack-export'] = handlers['recipe-export'];
   handlers['recipe-pack'] = handlers['recipe-import'];
   handlers['pack-submit'] = handlers['recipe-submit'];
+  for (const name of SCHEDULE_COMMANDS) {
+    handlers[name] = async ({ args, config, command: scheduleCommand }) => {
+      process.exitCode = (await runScheduleCli(scheduleCommand, args, { config })) || process.exitCode;
+    };
+  }
   handlers['profiles'] = handlers['fleet'];
   handlers['routing'] = handlers['route'];
   handlers['runtime-status'] = handlers['runtimes'];
@@ -3117,10 +3136,15 @@ async function main() {
 
 // Redact any configured credential value from arbitrary text (including stacks)
 // before it can reach a log, error, or receipt.
+// Keys the CLI may have sent: an explicitly selected one, and a remote head's.
+function sentKeyValues() {
+  return [cliAdminKeyEnvName ? process.env[cliAdminKeyEnvName] : null, process.env.LLOOM_HEAD_API_KEY];
+}
+
 function safeErrorText(text, config = null) {
   return redactGatewaySecrets(text, {
     config: config ?? currentCliConfig ?? undefined,
-    extraValues: cliAdminKeyEnvName ? [process.env[cliAdminKeyEnvName]] : []
+    extraValues: sentKeyValues()
   });
 }
 
@@ -3136,7 +3160,7 @@ function formatCliError(error, config = currentCliConfig) {
 function formatCliErrorJson(error) {
   const options = {
     config: currentCliConfig,
-    extraValues: cliAdminKeyEnvName ? [process.env[cliAdminKeyEnvName]] : []
+    extraValues: sentKeyValues()
   };
   // Receipts describe partial service mutations; retain them while redacting
   // every string value before serializing the envelope.

@@ -0,0 +1,191 @@
+# Reservations and jobs
+
+Several agents (or people) sharing a few machines need two things from
+LLooM. The first is a way to take hardware without colliding with each other or
+with production serving. The second is a single place that answers "how is it
+going?" without anyone having to ask each agent.
+
+The schedule-owning gateway provides both. This is normally the cluster head.
+
+- A **reservation** is a lease on a named resource. Taking it runs that
+  resource's actions: route aliases away, then drain and suspend models.
+  Releasing it undoes them in reverse. A release only reports success once the
+  resumed models pass their health check, and only then is traffic routed back.
+  Leases expire unless they are renewed, so an agent that vanishes cannot hold a
+  machine forever.
+- A **job** is a status record that an agent posts and keeps current: notes,
+  progress, metrics, a receipt path and an implicit heartbeat. A job that stops
+  reporting is marked stalled. A job attached to a reservation keeps that lease
+  alive while it reports.
+
+Both appear on one board: `lloom schedule`, `GET /gateway/schedule`, and
+Tessera's Fleet band.
+
+## Configure resources
+
+Resources live in the owning gateway's config:
+
+```json
+{
+  "reservations": {
+    "resources": {
+      "pair": {
+        "description": "two-node Atlas pair",
+        "hosts": ["gpu-box-1", "gpu-box-2"],
+        "routes": [{ "alias": "omp", "profile": "cloud", "node": "gpu-box-1" }],
+        "suspend": [{ "model": "pair-model", "node": "gpu-box-1" }]
+      },
+      "builds": { "description": "build slots on gpu-box-3", "capacity": 2 },
+      "drafter": { "hosts": ["gpu-box-3", "gpu-box-4"], "allowIndefinite": true }
+    }
+  }
+}
+```
+
+| Field | Meaning |
+|-------|---------|
+| `description`, `hosts` | Shown on the board. |
+| `capacity` | Number of concurrent holders. The default is 1. Resources with actions must have capacity 1. |
+| `routes` | Route-profile switches applied on take, in order (`lloom route <alias> <profile>`). The profile active at take time is recorded and restored on release. |
+| `suspend` | Models to drain and unload on take (`lloom suspend`) and resume on release (`lloom resume`). |
+| `node` | The cluster node whose gateway owns the route or model. Omit it for the local gateway. Remote steps use that node's admin API with its `cluster.nodes.<id>` credential, so the node needs `security.allowRemoteAdmin`. |
+| `ttlMs`, `maxTtlMs` | Default lease length (30 min) and longest requestable lease (8 h). |
+| `allowIndefinite` | Permit `--ttl none`, for standing reservations such as "these hosts belong to the drafter project". |
+| `onExpiry` | `release` (the default) restores on expiry. `keep` only flags the lease as overdue. |
+
+Schedule-wide settings sit beside `resources`: `warnBeforeMs` (5 min),
+`queueTtlMs` (30 min), `staleAfterMs` (15 min), `drainTimeoutMs` (5 min),
+`retentionMs` (7 days) and `eventLimit` (500). State is kept in
+`schedule.json` beside the config, and the durable history is appended to
+`schedule-events.ndjson`. Override these locations with `statePath` and
+`eventsPath`.
+
+Routes run before suspends on take, so traffic has already moved when a model
+drains. Each step records what it found, so release restores that and nothing
+more:
+
+- A route already on the target profile is left alone.
+- A model that was already suspended stays suspended.
+- A route with no active profile has nothing exact to restore, so the take
+  refuses it. Select a profile first with `lloom route <alias> <profile>`.
+- Two resources that suspend the same model, or move the same route, would undo
+  each other's leases. Both are reported as configuration errors until one of
+  them is changed. A step that fails rolls back the steps already done, and the reservation
+ends as `failed`. A restore that fails leaves the reservation `stuck`: the
+resource stays held and traffic stays away from the unhealthy model until a
+`release` succeeds. A gateway restart during an action also leaves the
+reservation `stuck`. Releasing it re-runs the undo, which is idempotent.
+
+## Use it from an agent
+
+Point the CLI at the head once. The key is the head's admin key; it is never
+taken from argv, and the local gateway's key is never sent to a different
+host:
+
+```sh
+export LLOOM_HEAD_URL=http://head.tailnet.example:8100   # private network, or https
+export LLOOM_HEAD_API_KEY=...        # admin key for the head
+export LLOOM_OWNER=buildopt          # who you are on the board
+export LLOOM_SESSION=claude:<session-id>   # optional; lets Tessera open your conversation
+```
+
+Then:
+
+```sh
+lloom schedule                                   # who holds what, queues, jobs
+lloom reserve pair --purpose "decode A/B"        # preview the actions
+lloom reserve pair --purpose "decode A/B" --wait --apply --yes   # queue if busy
+job=$(lloom job start "decode A/B" --on pair --note "building")
+lloom job progress $job 2/6 --unit arms "arm 2 of 6"
+lloom job metric $job tok_s=58.2 image=f399caa
+lloom job done $job "flat vs baseline" --receipt receipts/decode-ab.json
+lloom release pair --apply --yes                 # restore and free
+```
+
+- `reserve` without `--wait` fails straight away when the resource is busy, and
+  names the holder.
+- With `--wait` it queues and polls until the lease is active. It renews its
+  queue slot while it waits, and an abandoned wait drops out after `queueTtlMs`.
+- Taking a resource you already hold renews the lease.
+- `renew` and `release` accept the reservation id or the resource name.
+  `release --force` frees another owner's lease.
+- `--ttl 2h` sets the lease length, and `--ttl none` asks for an indefinite one.
+- Every job update (`note`, `progress`, `metric`, `beat`, `done`) is a heartbeat.
+  Updates carry `LLOOM_OWNER` when it is set, and the head refuses updates to
+  another owner's job.
+- A job attaches only to its owner's lease.
+- Two agents under the same owner are different holders when both name a session.
+- `lloom job start` prints only the id, so scripts can capture it. Add `--json`
+  to any command for the full record.
+
+## HTTP API
+
+These are `/gateway` admin routes, so the same rules apply as for other admin
+endpoints: once `security.adminApiKeys` is set they need an admin key, and a
+non-loopback bind needs `security.allowRemoteAdmin` for writes. They are served
+only by a primary gateway with a file-backed config. An unreadable
+`schedule.json` makes only these routes answer 503 `schedule_unavailable`; the
+gateway still serves inference, and the file is left untouched for inspection.
+
+| Route | Purpose |
+|-------|---------|
+| `GET /gateway/schedule?since=SEQ` | Board: `resources` (with `holders` and `queue`), `reservations`, `jobs` (with derived `health` and `heartbeatAgeMs`) and `events` after `SEQ`. |
+| `POST /gateway/schedule/reservations` | `{resource, owner, purpose?, session?, ttlMs?, wait?}`. Returns a preview unless `apply` and `yes` are both true. |
+| `GET /gateway/schedule/reservations/:id` | One reservation, including `position` when queued and `remainingMs` when active. |
+| `POST /gateway/schedule/reservations/:id/renew` | `{owner, ttlMs?}` |
+| `POST /gateway/schedule/reservations/:id/release` | `{owner, force?, apply, yes}`. Without `apply`, returns the undo plan. |
+| `POST /gateway/schedule/jobs` | `{title, owner, session?, resource? or reservationId?, note?, staleAfterMs?}` |
+| `GET /gateway/schedule/jobs/:id` | One job. |
+| `POST /gateway/schedule/jobs/:id` | `{note?, progress?: {done, total?, unit?}, metrics?, receipt?, state?: done or failed or cancelled}` |
+
+Refusals carry `error.type: "schedule_error"` and a code. For example,
+`schedule_busy` (409) includes `details.holders`.
+
+### Events
+
+Event kinds:
+
+- `reservation.acquiring`, `.queued`, `.active`, `.expiring`, `.overdue`, `.expired`, `.released`, `.withdrawn`, `.failed`, `.stuck`
+- `job.started`, `.stalled`, `.recovered`, `.done`, `.failed`, `.cancelled`
+
+Each event has a `severity`:
+
+- `info`
+- `attention`: expiring or overdue leases, stalled jobs, finished jobs
+- `error`: failed jobs, failed takes, stuck restores
+
+Boards poll with `since` to receive only new events. Job notes are written to
+the durable log but kept out of the event ring.
+
+## Limits and recovery
+
+- At most 500 jobs run at once, and at most 50 agents wait per resource.
+- A running job with no report for 24 hours is failed as abandoned.
+- The event log rotates to `schedule-events.ndjson.1` at 16 MB.
+- While a deployment fence holds, leases neither expire nor start, and they get
+  that time back afterwards.
+- When the gateway shuts down, it waits a few seconds for actions in flight.
+  Anything still unfinished comes back as `stuck` on the next start.
+- `reserve --wait` exits 0 only once the lease is active. It keeps waiting
+  through a briefly unreachable head, and on Ctrl-C it gives its place back
+  (exit 130).
+
+## Security
+
+Agents on other machines need the head's admin key, and the head needs
+`allowRemoteAdmin`. That key opens every admin route, not just the schedule.
+There is no narrower schedule-only credential yet. Reach the head over a
+private network such as Tailscale, or use HTTPS: never send the key in
+cleartext across a network you do not control.
+
+## Scope
+
+LLooM enforces reservations through the actions it performs, by suspending a
+model and routing traffic away from it. It does not stop processes it did not
+start. A build or container launched outside LLooM honours a reservation only
+when its agent checks `lloom schedule` or holds the lease, so standing
+reservations such as `drafter` work by being visible.
+
+The schedule belongs to the gateway that keeps it. Moving the head to another
+machine (`lloom cluster promote-head`) does not move `schedule.json`. Release
+leases first, or copy the file across while both gateways are stopped.