Dynamic backend provisioning: spawn worlds on demand (Docker, then Kubernetes)

[Pixnop]

Correction to my original framing (thanks @jessica38 for pushing on it): I wrote that the registry already amounts to dynamic service discovery, and that is only half true. Checked against BackendRouter:

• Initial joins are config-driven. The candidate pool comes from [servers] and the try order in nimbus.proxy.toml; the registry snapshot is used to filter that pool for health (stale, maintenance, capacity), never to add to it. A backend that heartbeats but is absent from the config is not a candidate for a player connecting to the proxy.
• Transfers are registry-driven. /server <id>, /nimbus send and intents resolve the target from the registry snapshot, so those can reach a backend that only exists in the registry.

So the explicit Velocity-shaped model (a config file listing servers and a fallback order) is already the primary one, and it should stay that way. What this issue asks for is creating and destroying containers; it must be an opt-in layer on top of that model, not a replacement for it.

Proposal

• A separate provisioner process (not a proxy plugin: cloud/orchestrator credentials stay out of the player-facing process), driving world templates declared in config: image or egg, variables, resource limits, savegame persistence policy.
• Driver 1: Docker. Talks to the Docker socket; spawning a backend is exactly what the egg install already does, containerized. Smallest useful increment.
• Driver 2: Kubernetes operator. A NimbusBackend CRD reconciled into StatefulSet + PVC per world. No k8s Services needed for discovery, heartbeats already do it.
• Commands: nimctl spawn <template> [--name events-3] and nimctl despawn <server> = drain, evacuate (Evacuate: move every player off a backend in one command #41), then delete. Scale-down is only safe because those primitives exist.
• Flagship use case: ephemeral event/minigame worlds that exist only while players are on them, later auto-scale rules (spawn when the hub queue grows, despawn when empty for N minutes).

Open questions: savegame lifecycle for ephemeral worlds (discard vs archive), secret distribution to spawned containers (inject at spawn, never bake into templates), who is allowed to spawn (admin socket permission model already has scopes), and how templates relate to the existing eggs (same install scripts, ideally).

Relates to #41 (evacuate is the safe-despawn primitive) and the panel eggs, which are effectively the manual version of a template.

Scale-to-zero: pause idle worlds

Nimbus has the exact signal needed: heartbeats carry player counts and the proxy sees every session, so "this world is empty" is known with certainty. Tiered lifecycle per template policy:

1. Warm pause (docker pause / SIGSTOP) after N idle minutes: zero CPU, RAM kept, resume in milliseconds. A frozen process's listening socket still accepts connections into its backlog, so wake-on-connect is genuinely feasible: the proxy dials, the provisioner unfreezes, the connection completes.
2. Cold stop (docker stop) after M idle hours: frees the RAM too (the real cost of a VS server); resume is 30-60s since worldgen already happened (the 2-4 min initial generation is paid once).
3. Archive: container deleted, savegame kept. For finished event worlds.

Design point that needs care: a frozen backend stops heartbeating, so the registry would mark it stale and the proxy would refuse to route to it. This needs a provisioner-set Paused state in the registry, distinct from stale/dead, with "routable after wake" semantics: a route request toward a paused world triggers the wake, then routes, instead of rejecting.

[Pixnop]

Going deeper on the parts that decide whether this is buildable, because the shape of the answer changes what phase 1 should even contain. Nothing below is committed; this is the discussion I want before code.

What the registry has to learn

Today a backend is either heartbeating (alive) or not (stale, then dropped). Provisioning needs a third thing the registry does not have: a backend that is known and intended to exist but is not currently answering. Without it, every paused or booting world looks exactly like a crashed one.

Concretely I think the registry gains a small lifecycle field, owned by the provisioner rather than inferred from heartbeats:

State	Meaning	Router behaviour
Running	heartbeating normally	routable (today's only state)
Starting	container up, worldgen or boot in progress	not routable yet; a transfer toward it waits or is refused with "starting"
Paused	deliberately frozen or stopped	routable after a wake, which the router triggers
Stopped	container gone, savegame kept	not routable; a spawn is needed

The important line: Paused must be distinguishable from Stale. Stale means "something is wrong, route around it". Paused means "this is fine, wake it if someone wants it". Conflating them is how you get either a proxy that refuses to route to a healthy paused world, or a proxy that keeps trying to route to a genuinely dead one.

The wake path, and its honest cost

Wake-on-connect only feels good if the wait is short and the player is told what is happening. Measured on the real thing earlier this week: a first VS server boot in a container is 2 to 4 minutes because of worldgen, and a subsequent boot of an already-generated world is 30 to 60 seconds. A docker pause/unpause is milliseconds.

So the tiers are not equivalent from the player's point of view:

• Warm pause: transparent. The transfer just takes a beat longer.
• Cold stop: 30 to 60 seconds of a player staring at something. That needs a real UX answer, not a spinner. Options worth discussing: hold them in the hub with a message, or a queue with a countdown, or simply refuse and say "that world is asleep, try again in a minute".
• Never pay worldgen on a wake: a template's first instance must be pre-generated at spawn time, not at first player.

My instinct is that phase 1 should ship warm pause only, precisely because it is the tier that needs no UX invention, and treat cold stop as a phase 2 with a designed waiting experience.

Who is allowed to spawn

A spawn is an expensive, resource-allocating action, and an evacuate-then-delete is destructive. The admin socket already has scoped permissions (nimbus.command.*), so the mechanism exists, but the policy needs deciding:

• Should a plugin be able to spawn? (An event-manager plugin creating a world per event is the flagship use case, so probably yes, with an explicit permission.)
• Should a player command ever spawn? I would say no for phase 1: "player creates infrastructure" is a bill-shaped footgun.
• Hard caps per template (max instances, max total) belong in the provisioner config, not in the caller's hands.

Secrets

Spawned containers need the network's shared secret, and it must not live in a template file that gets copied around. The provisioner should inject it at spawn time from its own config, which also means a template is safe to share publicly. Related: #40 (generating the secret instead of shipping a guessable default) becomes more important once containers are minted automatically, because a default secret plus automatic spawning is a very fast path to a compromised network.

Prior art worth reading before we design further

Agones (the Kubernetes game-server operator behind several large deployments) solved almost exactly this lifecycle: Scheduled -> Ready -> Allocated -> Shutdown, with a fleet autoscaler and an explicit "allocated servers are never scaled down" rule. Two things I would borrow: the allocation concept (a world with players is untouchable by the autoscaler) and fleet buffers (keep N ready instances warm so a spawn is never on the critical path of a player request). Two things I would not: their CRD surface is enormous, and their model assumes stateless match servers, whereas VS worlds are stateful savegames, which is the whole difficulty here.

Where I would draw phase 1

Smallest thing that is genuinely useful and does not paint us into a corner:

1. Registry lifecycle field (Running/Starting/Paused/Stopped) plus router awareness, because everything else needs it and it is meaningless to add later.
2. Docker driver, templates in config, nimctl spawn / despawn (despawn = drain, evacuate per Evacuate: move every player off a backend in one command #41, delete).
3. Warm pause with wake-on-route, no cold stop yet.
4. Per-template instance caps, secret injected at spawn.

No Kubernetes operator in phase 1. If someone on the team actually runs k8s in production and wants to own that piece, that changes my mind, but committing a small team to an operator's maintenance surface before the Docker path has users seems like the wrong order.

Two questions I would like answered before writing any of it: does anyone here run a panel or orchestrator they would actually point this at (that decides driver priority), and is the ephemeral-event-world case the real driver, or is the goal mainly to cut idle hosting cost? The answers pull the design in different directions: events want fast spawn, cost wants aggressive sleep.

[Pixnop]

@trevorftp @jessica38 @PHoenixOPHury the two questions at the end are the ones that actually change the design, so any opinion helps: is there an orchestrator or panel you would point this at (decides whether the Docker driver or something else comes first), and is the driver ephemeral event worlds or cutting idle hosting cost? Fast spawn and aggressive sleep pull in opposite directions.

[jessica38]

Interesting, I know that were using panels like pelican and pterodactyl to run this service, I think running as a egg is fine I don't really want automatic system to get all local services in thinking like velocity has a config file with the servers and the fallback ones.

I would need to think more on the rest of it but that's my feedback right off the bat.

[Pixnop]

That feedback is worth more than agreement, because it caught an error in how I framed the issue, and I have corrected the issue body.

You are describing what Nimbus already does, and it is the primary model. I checked BackendRouter rather than trusting my memory:

• Initial joins are config-driven, exactly like Velocity. [servers] is the list, try is the fallback order. The registry snapshot only filters that list for health (stale, in maintenance, full); it never adds a server to it. A backend that heartbeats but is missing from nimbus.proxy.toml is simply not a candidate for someone connecting to the proxy.
• Transfers are the part that reads the registry. /server <id> and /nimbus send resolve the target from the snapshot, so those can reach a backend that only exists in the registry.

I had written in the issue that a booting backend "becomes routable without touching the proxy's config", which is true for transfers and wrong for initial joins. Fixed.

Which means your preference stays the default, permanently. I would not want auto-discovery of local services either: a proxy that routes players to whatever announced itself is a proxy where a misconfigured or hostile container silently joins your network. The explicit list is a security property, not just a preference.

So the provisioning idea is strictly an opt-in layer for people who want it, and your feedback sharpens one specific design question I had left open: should a provisioner write into [servers] (staying inside the explicit model, so what you read in the config is still the truth), or should the router grow an opt-in "also accept registry-only backends" switch? Your instinct pushes me toward the first, because the config file remains the single place you look to know what your network is.

And running it as an egg is the plan regardless: the three eggs already exist and were retested against the v0.2.0 release this week. Nothing about provisioning would change how a hand-run egg deployment works.

No rush on the rest, but if you do think about it further, the one answer that would help most: for a network you would actually run, would you ever want a world that does not exist until someone asks for it (an event world, say), or is a fixed set of servers you declare yourself simply the right model full stop? A completely honest "fixed set is fine" is a useful answer, and would keep #47 low priority.