Feature Request: Node Dependencies #374

I'd like to be able to label nodes as having a dependency of other nodes, so that if I do an apply, particularly with the --reboot flag, it will wait for those previous nodes to be completed.

In my specific usecase, my configuration affects systems that are only available via an SSH ProxyJump, but the router that they proxy through is also maintained with Colmena. As such, I have to run the deployment as two separate commands:

$ colmena apply --on @routers --reboot
$ # Wait for them to complete
$ colmena apply --on @webservers --reboot

My proposal is to include a new value in the node config to include things the node depends on, and, if those are in the apply set, wait for them to "finish" before beginning.

For example, with a sample config as shown below:

{
  zone-a-router = {
    deployment.tags = ["routers"];
    services.firewall.nat.enabled = true; 
  };
  zone-a-server = {
    deployment = {
      tags = ["servers"];
      dependsOn = ["@routers"];
    };
  };
  zone-b-server = {
    deployment = {
      tags = ["servers"];
      dependsOn = ["@routers"];
    };
  };
  site-storage = {
    deployment.dependsOn = ["@routers" "zone-a-*"];
  };
}

The example may be a bit contrived, but I wanted to show how I want it to still use all the same matching as the --on <NodeFilter> flag, but in order to create a dependency graph of nodes.

In the example above, I would imagine the following invocations:

$ colmena apply --on "@routers,zone-a-server" --reboot
Apply zone-a-router first, reboot it, apply zone-a-server and reboot that too
$ colmena apply --on "@routers,@servers" --reboot
Apply zone-a-router first, reboot it, apply all the servers in parallel (up to the parallelism limit), and reboot them too
$ colmena apply --on "@routers,site-storage" --reboot
Apply zone-a-router first, reboot it, apply site-storage, reboot that.

A few rules:

• A node only continues when the previous node is "done", which means after the reboot, if the --reboot flag is supplied.
• If a node fails, any subsequent nodes that depend on it will not be run.
• If a node is not explicitly matched with one of the --on tags, anything that depends on it will not "trigger" the dependency.
  • So in example 3, since zone-a-server did not get included in the --on tags, site-storage will not wait for it to be completed (though it will obviously still wait for @routers to finish.
• Cyclic dependencies (i.e., @routers depending on say @servers and @servers depending on @routers would raise a ColmenaError, prior to attempting to do any deployment.

I'd be interested in trying to implement this, if there's maintainer interest in the feature. (Also, if there are other rules that I'm missing, I'd be interested in those too!)