Pool head: reconciler, controller, membership and a fleet simulator

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes to LLooM will be documented in this file. The format follows
 
 ### Added
 
+- Pool reconciler and controller, tested in simulation. `lloom pool simulate <scenario>` runs a fleet scenario through the real planner, reconciler and controller in a deterministic discrete-event model, and compares today's behaviour with the controller's. The reconciler orders make-before-break steps (prefetch, start, route flip, drain and stop), with swaps through interim members, per-step timeouts, and dependents skipped when a step fails. The controller has `off`, `advise` and `auto` modes, urgency, stability, cooldowns, a load budget, failure backoff, and preemption by an urgent new target. Neither is wired to a live gateway yet. See `docs/pool-planning.md`.
+
+- Pool membership: nodes join a head with a one-time enrollment token (`lloom pool token create`, `lloom pool join`) and keep a lease with heartbeats from `serve`. The head lists members (`lloom pool members`), cordons, drains and uncordons them, and `lloom pool plan` uses their liveness, cordons and inventory. Membership is inert until a token is issued, keeps only hashed secrets, and does not grant routing: an enrolled-only node still needs `lloom cluster add-node`. See `docs/pool-planning.md`.
+
 - `lloom pool plan`: a dry-run placement plan across every cluster node. It proposes which lanes run where, which residency they get and how each alias routes, and diffs that against what runs now. It covers every required alias first, then minimises demand-weighted latency and the load cost of changes, within per-domain memory headroom, direct-link tensor parallelism, compute architecture and pins. Nodes report a pool inventory through `/gateway/node?inventory=1`. Recipes may declare `requirements.computeArchs`, `requirements.tensorParallel` and `settings.maxSequences`. `/gateway/metrics` adds `requested` buckets, which give per-alias demand. See `docs/pool-planning.md`.
 
 - LLooM Hear CPU audio analysis, with structured estimates, optional dashboard images, and opt-in upstream interpretation. File and URL inputs require operator configuration.
@@ -33,6 +37,8 @@ All notable changes to LLooM will be documented in this file. The format follows
 
 ### Changed
 
+- Admission keeps the memory safety guard's margin free. The host check now reserves the larger of `reserveMemoryGb` and the guard's margin (`total × (1 − ceiling)`, 12.8 GB on a 128 GB host at the default 0.9), and cluster node budgets are capped at the ceiling. Before, admission could approve a load the guard was certain to abort and then latch until a manual start. A guard abort now records `memorySafetyAbort` (`usedGb`, and `consumedGb`, a lower bound on the load's footprint) and `memorySafetyLatched` in runtime status. Pool planning caps host budgets at the same ceiling and learns from aborts.
+
 - LLooM now requires Node.js 22.19 or newer, the minimum for its undici 8 HTTP client. It already failed to start on Node 20 after that upgrade; the engine range and CI matrix now say so.
 
 - The action view frames only models serving traffic and their live requests. Active cards cluster near the loom; idle models and machine racks no longer pull the camera outward. Manual zoom holds independently of the automatic fit until Reset view restores it.

README.md

@@ -123,6 +123,7 @@ lloom route q38fn --suspend-member qwen3.8-flash-next --apply --yes      # keep
 lloom runtime-stop qwen38-flash-next-cluster                             # stop local TP-2 after suspension
 lloom route q38fn --resume-member qwen3.8-flash-next --apply --yes       # restore preferred-local recovery
 lloom pool plan --cordon spark-node-02                                   # dry-run placement across the cluster
+lloom pool members                                                       # nodes that joined with `lloom pool join`
 lloom integrate omp --apply --yes
 lloom add-model mlx-community/Qwen3.6-27B-OptiQ-4bit --keep-warm --default --go           # Mac: install, download, start
 lloom remove-model mlx-community/Qwen3.6-27B-OptiQ-4bit                                  # preview safe cleanup

bin/lloom.mjs

@@ -36,7 +36,21 @@ import {
   selectedRecipeIdFromCommunityPlan
 } from '../src/community-client.mjs';
 import { loadConfig } from '../src/config.mjs';
-import { POOL_USAGE, createPoolPlanReport, formatPoolPlanReport, poolPlanOptions } from '../src/pool-cli.mjs';
+import {
+  POOL_USAGE,
+  createPoolPlanReport,
+  formatPoolMembers,
+  formatPoolPlanReport,
+  poolPlanOptions,
+  poolTokenOptions
+} from '../src/pool-cli.mjs';
+import { joinPool } from '../src/pool-node.mjs';
+import {
+  POOL_SIMULATE_USAGE,
+  formatPoolSimulation,
+  poolSimulateOptions,
+  runPoolSimulateCommand
+} from '../src/pool-sim-cli.mjs';
 import { mutateConfigSource } from '../src/config-mutation.mjs';
 import { runtimeControlTimeoutMs } from '../src/control-timeout.mjs';
 import { createDoctorReport } from '../src/doctor.mjs';
@@ -231,6 +245,11 @@ Backends and runtimes:
   lloom down
   lloom cluster [status|doctor|discover] [--apply] [--id NAME] [--api-key-env NAME] [--json]
   lloom pool plan [--cordon NODE[,NODE]] [--pin ALIAS=MODEL] [--profile NAME] [--consider-installs] [--headroom-gb N] [--churn-weight N] [--external fallback|allow|never] [--offline] [--json]
+  lloom pool token create [--node-id ID] [--ttl DURATION] [--uses N] [--label KEY=VALUE] [--json]
+  lloom pool members [--json]
+  lloom pool cordon|uncordon|drain|leave <node> [--reason TEXT]
+  lloom pool join <head-url> --token-stdin [--node-id ID] [--endpoint URL] [--apply --yes]
+  lloom pool simulate <scenario.json> [--mode off|advise|auto|compare] [--hours N] [--seed N] [--json]
   lloom deployment plan --plan FILE [--json]
   lloom deployment apply --plan FILE --nodes FILE [--operation-id ID] --apply --yes [--journal FILE] [--json]
   lloom deployment status --operation-id ID [--journal FILE] [--json]
@@ -2502,6 +2521,82 @@ async function main() {
     },
     pool: async ({ args, config }) => {
       const action = positional(args)[1] ?? 'plan';
+      const json = hasFlag(args, '--json');
+      if (action === 'token') {
+        if (positional(args)[2] !== 'create') throw new Error(POOL_USAGE);
+        const result = await gatewayRequest(config, '/gateway/pool/tokens', {
+          args,
+          method: 'POST',
+          body: poolTokenOptions({
+            ttl: argValue(args, '--ttl'),
+            uses: argValue(args, '--uses'),
+            labels: argValues(args, '--label'),
+            nodeId: argValue(args, '--node-id') ?? undefined
+          }),
+          timeoutMs: 10000
+        });
+        console.log(
+          json
+            ? JSON.stringify(result, null, 2)
+            : [
+                `Enrollment token ${result.id} (shown once; expires ${result.expiresAt}):`,
+                result.token,
+                '',
+                'On the joining node:',
+                `  lloom pool join <this-head-url> --token-stdin --endpoint <node-gateway-url> --apply --yes`
+              ].join('\n')
+        );
+        return;
+      }
+      if (action === 'members') {
+        const result = await gatewayRequest(config, '/gateway/pool/members', { args, timeoutMs: 10000 });
+        console.log(json ? JSON.stringify(result, null, 2) : formatPoolMembers(result));
+        return;
+      }
+      if (['cordon', 'uncordon', 'drain', 'leave'].includes(action)) {
+        const nodeId = positional(args)[2];
+        if (!nodeId) throw new Error(POOL_USAGE);
+        const reason = argValue(args, '--reason');
+        const result = await gatewayRequest(config, `/gateway/pool/nodes/${encodeURIComponent(nodeId)}/${action}`, {
+          args,
+          method: 'POST',
+          body: reason == null ? {} : { reason },
+          timeoutMs: 10000
+        });
+        console.log(JSON.stringify(result, null, 2));
+        return;
+      }
+      if (action === 'join') {
+        const headUrl = positional(args)[2];
+        if (!headUrl || !hasFlag(args, '--token-stdin')) throw new Error(POOL_USAGE);
+        const apply = hasFlag(args, '--apply');
+        const result = await joinPool(config, {
+          headUrl,
+          // The token is read only when it will be used.
+          token: apply && hasFlag(args, '--yes') ? await readBoundedStdinCredential(process.stdin) : null,
+          nodeId: argValue(args, '--node-id') ?? undefined,
+          endpoint: argValue(args, '--endpoint'),
+          apply,
+          yes: hasFlag(args, '--yes')
+        });
+        console.log(JSON.stringify(result, null, 2));
+        return;
+      }
+      if (action === 'simulate') {
+        const scenario = positional(args)[2];
+        if (!scenario) throw new Error(POOL_SIMULATE_USAGE);
+        const reports = await runPoolSimulateCommand(
+          scenario,
+          poolSimulateOptions({
+            mode: argValue(args, '--mode') ?? undefined,
+            hours: argValue(args, '--hours'),
+            seed: argValue(args, '--seed')
+          })
+        );
+        console.log(json ? JSON.stringify(reports, null, 2) : formatPoolSimulation(reports));
+        if (Object.values(reports).some((report) => !report.ok)) process.exitCode = 1;
+        return;
+      }
       if (action !== 'plan') throw new Error(POOL_USAGE);
       const options = poolPlanOptions({
         cordon: argValues(args, '--cordon'),

docs/clusters.md

@@ -256,7 +256,7 @@ before contacting any node.
 
 The live dashboard renders node cards between the main LLooM chassis and model cards. Each node card reports architecture, accelerator type, CPU, RAM, GPU compute, GPU memory, temperature, and power when the host can report them. Unsupported fields render as unavailable rather than inventing values; Apple Silicon still reports its profile and shared-memory pressure even when per-GPU utilization is unavailable. Select a node for its complete profile. Replicated models connect to every target node and distributed models connect to every member node.
 
-`GET /gateway/node?inventory=1` and `GET /gateway/cluster?inventory=1` add each node's pool inventory (compute architectures, model-root disk, cached recipe weights, installed images and fabric links). `lloom pool plan` uses it to propose placements across the whole cluster; see [`pool-planning.md`](pool-planning.md).
+`GET /gateway/node?inventory=1` and `GET /gateway/cluster?inventory=1` add each node's pool inventory (compute architectures, model-root disk, cached recipe weights, installed images and fabric links). `lloom pool plan` uses it to propose placements across the whole cluster; see [`pool-planning.md`](pool-planning.md). Nodes can also join a head with an enrollment token and keep a lease (`lloom pool join`); membership adds liveness and cordons to planning but does not replace `cluster add-node` for routing.
 
 Before sending model traffic, use `lloom cluster doctor --runtime ID` to verify the required nodes and serving endpoint. Whole-topology doctor reports optional offline nodes and missing memory telemetry as warnings; they do not make unrelated runtimes unavailable. Then start a small replicated lane, verify requests alternate under equal load, and only then canary the distributed engine.