Pool head: reconciler, controller, membership and a fleet simulator

[data-angel]

Summary

Stacked on #59 (lloom pool plan). This PR adds the rest of the pool head and tests it all in a deterministic fleet simulator. Nothing here is wired into a live gateway's control loop.

• The reconciler and controller run only in the simulator.
• Membership is inert until a token is issued.
• Two changes do touch live paths: admission and the runtime manager. See "Live-path changes" below.

What's in it

• Reconciler (src/pool-reconciler.mjs): turns a plan into ordered make-before-break steps: prefetch → start → route flip → drain and stop.
  • Residency downgrades happen before stops.
  • When the old lane must go first, a swap moves its aliases to an interim member, then back. If there is no interim member, the interruption is reported.
  • When a step fails, everything that depends on it is skipped. A lane that never starts therefore never strands its aliases.
  • Every step has a timeout and can be cancelled.
• Controller (src/pool-controller.mjs): runs in off, advise or auto mode.
  • Changes that restore coverage, or vacate a cordoned node, act at once.
  • Other changes need a stable plan, a minimum gain, a cooldown and room in an hourly load budget.
  • Residency-only changes get slower hysteresis.
  • A failed start is backed off.
  • An urgent plan with different routes preempts the change in progress.
  • Ticks never overlap.
• Membership (src/pool-membership*.mjs, src/pool-node.mjs): enrollment tokens, node leases and heartbeats, and cordon, drain, uncordon and leave. It comes with HTTP routes and CLI commands, and feeds liveness into pool plan.
• Simulator (src/pool-sim.mjs, lloom pool simulate): a discrete-event model of the fleet.
  • It models requests, admission with LRU eviction, keep-warm restore, memory-guard aborts and the latch, node hangs, failures and partitions, membership leases, and operator actions.
  • The real planner, reconciler and controller run against it.
  • Seven scenarios in test/fixtures/pool/scenarios/ compare today's behaviour (off) with the controller (auto).

Live-path changes (please review closely)

• Admission now keeps the memory guard's margin free (src/runtime-policy.mjs).
  • Problem: the simulator found that admission kept only reserveMemoryGb (8 GB) free, while the guard aborts at 90% use (12.8 GB free on a 128 GB Spark). Admission could approve a load the guard was certain to abort, and the guard then latched the runtime until a manual start.
  • Now: the host check reserves max(reserve, total × (1 − ceiling)), and cluster node budgets are capped at the ceiling (never below 0).
  • Effect: admission is more conservative on large hosts. It refuses or evicts earlier, instead of letting a load fail mid-way.
• The runtime manager records guard aborts (src/runtime-manager.mjs). Status now includes memorySafetyAbort (usedGb, plus consumedGb, a lower bound on the load's footprint) and memorySafetyLatched. A healthy start clears it.
• The pool planner caps host budgets at the guard ceiling. On the fixture fleet this exposes a real conflict in the hand-written standalone profile: video cannot load next to a keep-warm 27B on spark-node-03.

Scenario results (seeded simulation, not benchmarks)

Scenario	off (today)	auto
pair-crash	SLO 98.4%, 12 to cloud, 60 cold waits, 1414 GB reloaded	SLO 99.5%, 0 to cloud, 26 cold waits, 1008 GB reloaded, back on the pair after recovery
lease-loss	same as above	acts 2 min after the crash (lease lost at 90 s); a 60 s partition moves nothing
mac-joins	cognition stays on the 27B	prefetches weights, moves cognition to the Mac, 0 failed requests
pair-taken	operator switches profiles by hand	cordon/uncordon only; one apply each way
demand-flap	–	≤ 1 residency change per hour, no extra loads

Reviews

The project rules require a full code and security review before commit, so both ran on this diff, and every verified finding is fixed with a test.

• Security. The one medium finding was a node-id takeover: any token could re-enroll an existing node. Tokens can now be bound with --node-id, and claiming a member or a configured peer requires a bound token. Low findings fixed:

  • Heartbeats are checked for the node-secret format before the body is read, and are rate limited.
  • The per-client rate limiter evicts least-recently-used entries instead of clearing everything.
  • Control characters in node-reported text are dropped.
  • A failed join reports the secret file path and what to set by hand.
  • Joining over plain HTTP prints a warning.

• Correctness. Six bugs fixed:

  • a lane whose model moved to another node was never stopped;
  • a lane already stopping was treated as freed memory;
  • the planner's guard ceiling could exceed the guard's own;
  • a node with unknown capacity got a negative admission budget;
  • abort evidence was never cleared;
  • an interim route was never reverted.

  Hardening items from the same review are in the last commit.

Tests

• node --test test/pool-*.test.mjs: 109 pass.
• Every test file touching admission, the runtime manager or the server, run individually: all pass.
• Also passing: test:cluster, security, admin-browser-security, node-onboarding, cli-diagnostics, smoke, runtime-policy, runtime-memory-safety, check:js, prettier --check, and eslint (pre-existing warnings only).
• Not run: the full npm test.

Not in this PR

• A real actuator that maps reconcile steps onto the runtime manager, routes and residency, and wiring the controller into serve. Advise mode first.
• Capacity-aware planning (replicas for overloaded lanes) and interference between co-located lanes. The simulator models capacity; the planner doesn't use it yet.

🤖 Generated with Claude Code

[data-angel]

Superseded by #74, which integrates the reconciler/controller/membership/simulator work rebased onto current main (package.json and CHANGELOG resolved against main; pool sources/tests added to check:js and test:unit).