LLooM clusters keep one stable OpenAI/Anthropic-compatible client endpoint while each node remains a complete, locally callable LLooM. Every machine has one local config and one sovereign runtime controller. That controller owns its local models, admission decisions, and lifecycle. A federated leader proxies inference through the node's LLooM API and reads the node's advertised model, runtime, and hardware status; it does not start, stop, or connect directly to ordinary runtimes on that node.
Tensor-parallel (TP) runtimes are the deliberate exception. A distributed logical model is callable only through its leader. Its physical members declare delegated lifecycle authority to that leader, allowing the leader to start, stop, restart, and reclaim only those named members. A worker's local admission, keep-warm, reload, watchdog, and stop-all paths cannot operate or evict delegated members. Delegation does not grant the leader control of any unrelated local runtime.
The cluster layer supports two placement modes:
- Replicated/federated: one complete model is hosted by each node's sovereign LLooM. The leader proxies requests to healthy node APIs, while each node retains its own concurrency queue and memory admission.
- Distributed: one leader-owned logical runtime has an ordered list of explicitly delegated physical members across nodes. The leader starts members in order, rolls back in reverse order after a failure, and stops them in reverse order. The backend launcher still owns engine-specific tensor/pipeline parallelism.
There is no two-node assumption. Node maps, replica targets, distributed members, topology cards, and admission plans are all arrays/maps over the recipe's declared nodes.
Expose the joining LLooM only on a trusted private network, require authentication, and then run this on the central gateway:
export LLOOM_CLUSTER_KEY='the key accepted by the joining LLooM'
lloom cluster add-node macbook-local http://macbook-private:8100 --apply --yesThe command reads the joining node's /gateway/node snapshot, records its architecture, accelerators, memory, and directly hosted model catalog, and creates deterministic gateway names such as macbook-local/local/qwen. Imported targets retain the joining gateway's runtime ID as observational metadata, so the central topology and replica selection can reflect remote lifecycle and load without taking ownership of that process. It never copies credentials into config: the node and generated backend retain only apiKeyEnv. Config reload activates the node without restarting the central gateway.
The declarative form is useful for review or hand editing:
{
"cluster": {
"nodeId": "spark-1",
"leaderNode": "spark-1",
"apiKeyEnv": "LLOOM_CLUSTER_KEY",
"nodes": {
"spark-1": { "labels": { "role": "leader", "accelerator": "cuda" } },
"macbook-local": {
"endpoint": "http://macbook-private:8100",
"labels": { "role": "node", "architecture": "darwin-arm64", "accelerator": "metal" },
"resources": { "memoryGb": 64 },
"proxy": {
"models": [{ "id": "local/qwen", "as": "macbook-local/local/qwen", "kind": "chat", "remoteRuntime": "qwen" }]
}
}
}
}
}Set the same as ID on two or more node model entries to create one logical model duplicated across hosts. LLooM chooses a reachable, healthy, least-loaded target, applies target weights to ties, and briefly removes a target after an upstream 5xx response so another replica can take over. The cooldown expires automatically, allowing a restarted gateway to rejoin without operator repair. Use --merge with cluster add-node to merge matching remote model IDs instead of namespacing them.
Only runtime-backed models are imported by default. Pass --include-external when the central gateway should also route the joining node's unmanaged cloud or network backends. A model that was itself federated from another LLooM is never re-exported by cluster add-node, preventing accidental routing loops. Explicit models[].targets remains available when different nodes need different upstream model IDs, weights, or backends.
On a DGX Spark cluster created by NVIDIA Sync, LLooM can discover the peer alias and private fabric directly:
lloom cluster discover --id spark-node-cluster
lloom cluster discover --id spark-node-cluster --api-key-env LLOOM_ADMIN_API_KEY --applyDiscovery groups NVSyncClusterAlias entries into stable node IDs, including duplicate IP aliases and multiple rails per peer. Older Sync entries with a named Host alias still work. The configured local identity and leader remain unchanged. Local observations appear under cluster.discovery.links; they do not prove gateway reachability, bandwidth, or a complete ring.
New peers remain physical inventory under cluster.discovery.nodes. Join a peer with lloom cluster add-node <id> <authenticated-gateway-url> --apply --yes after its gateway is reachable. Discovery does not create a live gateway endpoint from an SSH address.
--apply merges into the raw configuration and validates it before an atomic write. It preserves existing federation nodes, model catalogs, authentication references, custom endpoints, and model placements. An endpoint that names its previous 10.100.* backend host moves to the observed address while retaining its scheme, port, and path. Other endpoints remain unchanged and receive a diagnostic. Discovery does not change gateway listeners or NCCL settings; verify new endpoints before using them.
Physical membership and model placement are separate. In a three-Spark ring, a model can run on one node while a distributed model uses an explicit two-node subset. A three-node member list is also representable, but TP-3 requires a compatible model architecture and backend launcher; discovery does not infer that compatibility or repartition a loaded model. Resource admission applies to the chosen members. Select the NCCL adapters that connect those members: a TP-2 job must not use adapters whose cables lead to a third node outside its placement.
Use private fabric addresses for node-to-node LLooM and raw backend traffic. backendHost is deliberately required when a recipe auto-materializes replicas; LLooM binds the generated backend only to that address, never to every interface.
Distributed recipes can template each member's fabric with ${fabricInterface} (the node's fabricInterface) and ${fabricHca} (the node's fabricHca). Recipes that consume ${fabricHca} require exact, node-local RDMA device names: LLooM expects fabricHca (a comma-separated list of unique device names, no spaces or :port suffixes) and fabricInterface on every participating node, and stops materialization with the node and field named in the error when either is missing or malformed. It never guesses an HCA from the NIC name, because PCI naming and cabling differ by host.
{
"server": { "host": "${LLOOM_GATEWAY_HOST}", "port": 8100 },
"security": {
"allowMissingAuth": false,
"allowNonLoopbackBind": true,
"allowRemoteAdmin": true,
"apiKeys": ["${LLOOM_API_KEY}"],
"adminApiKeys": ["${LLOOM_CLUSTER_KEY}"]
},
"cluster": {
"id": "spark-cluster",
"nodeId": "${LLOOM_NODE_ID}",
"leaderNode": "spark-1",
"apiKeyEnv": "LLOOM_CLUSTER_KEY",
"nodes": {
"spark-1": {
"endpoint": "http://10.10.10.1:8100",
"backendHost": "10.10.10.1",
"labels": { "hardware": "dgx-spark", "fabric": "connectx" },
"resources": { "memoryGb": 128, "maxMemoryUtilization": 0.9 }
},
"spark-2": {
"endpoint": "http://10.10.10.2:8100",
"backendHost": "10.10.10.2",
"labels": { "hardware": "dgx-spark", "fabric": "connectx" },
"resources": { "memoryGb": 128, "maxMemoryUtilization": 0.9 }
}
}
}
}Each Spark keeps its own normal LLooM config. Shared cluster identity and node declarations may be generated from the same reviewed template, but ordinary local model entries belong to the node that serves them. Set the local identity in each node's environment:
# spark-1
export LLOOM_NODE_ID=spark-1 LLOOM_GATEWAY_HOST=10.10.10.1
# spark-2
export LLOOM_NODE_ID=spark-2 LLOOM_GATEWAY_HOST=10.10.10.2Keep LLOOM_CLUSTER_KEY identical on all nodes. A production firewall should allow port 8100 and TP backend ports only on the private fabric. Clients may call a node's complete local models through that node's LLooM URL; TP logical models are advertised and callable only at the leader. Raw backend ports are never client endpoints.
A recipe can materialize the same model on every configured node without naming a fixed cluster size:
{
"model": "unsloth/Qwen3.6-27B-NVFP4",
"gatewayModel": "unsloth/Qwen3.6-27B-NVFP4",
"settings": {
"memoryGb": 48,
"maxActiveRequests": 4,
"placement": { "mode": "replicated" }
}
}Omit placement.nodes to use all cluster.nodes; set it to an explicit node-ID array for a subset, or use the leader, worker, or workers role selectors. During materialization LLooM creates a local backend/runtime target per selected node. For every non-leader replica it also creates a leader-only proxy target aimed at that node's LLooM endpoint. The raw target is visible only on its owner node; the leader routes through the proxy target and never connects to or controls the remote backend directly.
The equivalent explicit config is:
{
"models": [
{
"id": "example/Qwen",
"targets": [
{ "id": "spark-1", "node": "spark-1", "backend": "qwen-spark-1", "runtime": "qwen-spark-1" },
{ "id": "spark-2", "node": "spark-2", "backend": "qwen-spark-2", "runtime": "qwen-spark-2" },
{ "id": "spark-2-proxy", "node": "spark-2", "backend": "lloom-node-spark-2", "remoteRuntime": "qwen-spark-2" }
]
}
]
}On spark-1, the callable targets are spark-1 and spark-2-proxy. On spark-2, only spark-2 is callable. The proxy backend points to spark-2's /v1 API and uses the configured cluster credential.
For a model such as DeepSeek V4 Flash that uses both Sparks, a recipe may declare member runtime templates. LLooM resolves nodeRole selectors, materializes one physical runtime per node, and adds one logical distributed runtime. The member list can contain any number of nodes and roles. Headless Docker workers use healthStrategy: "container"; the head owns the HTTP health check.
The checked-in linux-nvidia-dgx-spark-2x-deepseek-v4-flash-mia-vllm recipe is the concrete two-Spark example. It is based on MiaAI Lab's validated multiprocess vLLM recipe, pins the official 0731 model revision and runtime image digest, packages its reviewed correctness hotfixes, starts the worker before the head, and binds the raw vLLM API only to the private NVIDIA Sync fabric. Hotfix assets are bind-mounted read-only and applied idempotently before vLLM starts, so container recreation preserves the tested runtime contract.
The companion linux-nvidia-dgx-spark-cluster-flux2-klein-4b recipe demonstrates a single worker-pinned service. It keeps the image model on a worker while clients continue to use the leader LLooM endpoint.
Explicit config remains supported:
{
"runtimes": {
"dsv4-ray-head": {
"enabled": true,
"node": "spark-1",
"authority": { "owner": "spark-1", "scope": "distributed-member", "group": "dsv4flash-cluster" },
"command": "/opt/lloom/bin/start-ray-head",
"healthUrl": "http://10.10.10.1:8265/api/version",
"memoryGb": 6
},
"dsv4-ray-worker-2": {
"enabled": true,
"node": "spark-2",
"authority": { "owner": "spark-1", "scope": "distributed-member", "group": "dsv4flash-cluster" },
"command": "/opt/lloom/bin/start-ray-worker",
"healthUrl": "http://10.10.10.2:8266/health",
"memoryGb": 6
},
"dsv4-vllm-server": {
"enabled": true,
"node": "spark-1",
"authority": { "owner": "spark-1", "scope": "distributed-member", "group": "dsv4flash-cluster" },
"command": "/opt/lloom/bin/start-dsv4flash-vllm",
"healthUrl": "http://10.10.10.1:8205/health",
"memoryGb": 54
},
"dsv4flash-cluster": {
"enabled": true,
"authority": { "owner": "spark-1", "scope": "distributed-model", "group": "dsv4flash-cluster" },
"healthUrl": "http://10.10.10.1:8205/health",
"placement": {
"mode": "distributed",
"members": [
{
"node": "spark-1",
"runtime": "dsv4-ray-head",
"role": "head",
"order": 10,
"resources": { "memoryGb": 6 }
},
{
"node": "spark-2",
"runtime": "dsv4-ray-worker-2",
"role": "worker",
"order": 20,
"resources": { "memoryGb": 54 }
},
{
"node": "spark-1",
"runtime": "dsv4-vllm-server",
"role": "server",
"order": 30,
"resources": { "memoryGb": 54 }
}
]
}
}
},
"models": [
{
"id": "deepseek-ai/DSv4Flash",
"backend": "dsv4flash",
"runtime": "dsv4flash-cluster",
"upstreamModel": "deepseek-ai/DSv4Flash"
}
]
}Launcher commands in explicit config are installation-specific. They must remain attached (or manage containers), report real health, bind only to the private fabric, and configure the vLLM/SGLang/Ray parallelism appropriate to the exact model build. LLooM coordinates lifecycle and admission; engine-specific flags live in reviewed recipes rather than being guessed at runtime.
Distributed resource estimates belong on each member. Admission evaluates every node independently using live MemAvailable plus configured limits. A group is admitted only if it fits every node after safe, node-relevant evictions. Recipe materialization writes the authority blocks automatically; explicit configs should include them for auditability. Older distributed configs derive the same leader ownership defensively during loading.
lloom cluster
lloom cluster doctor
lloom runtime-plan dsv4flash-cluster
lloom runtime-start dsv4flash-clusterGET /gateway/node returns the local node's system identity, machine profile, directly advertised models, CPU/RAM telemetry, optional GPU telemetry, and local runtime state. GET /gateway/cluster returns all node snapshots, reachability, labels, and runtime state. /gateway/status includes the same cluster graph alongside logical runtime status.
GET /gateway/cluster/readiness?runtime=ID and GET /gateway/doctor?runtime=ID return the
same scoped readiness contract. lloom doctor --runtime ID and
lloom cluster doctor --runtime ID use those gateway endpoints. Add
--include-federation / includeFederation=1 to collect warnings for optional peers. The
scope contains only the nodes needed to judge that runtime: the effective local or pinned
node, deduplicated explicit distributed member nodes, all member nodes in the owning group, or a
materialized replica node. Unknown, disabled, ambiguous, or unmaterialized scopes fail closed
before contacting any node.
The live dashboard renders node cards between the main LLooM chassis and model cards. Each node card reports architecture, accelerator type, CPU, RAM, GPU compute, GPU memory, temperature, and power when the host can report them. Unsupported fields render as unavailable rather than inventing values; Apple Silicon still reports its profile and shared-memory pressure even when per-GPU utilization is unavailable. Select a node for its complete profile. Replicated models connect to every target node and distributed models connect to every member node.
Before sending model traffic, use lloom cluster doctor --runtime ID to verify the required nodes and serving endpoint. Whole-topology doctor reports optional offline nodes and missing memory telemetry as warnings; they do not make unrelated runtimes unavailable. Then start a small replicated lane, verify requests alternate under equal load, and only then canary the distributed engine.
Host aliases such as spark-node-01 and addresses in configuration examples
are placeholders. Replace them with the names and private addresses used by
your deployment. Historical benchmark and evidence files use anonymized node
labels; their measurements and artifact hashes are preserved.