diff --git a/Cargo.lock b/Cargo.lock index e992998..ebc3b5e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1970,7 +1970,7 @@ dependencies = [ [[package]] name = "fleet-mcp" -version = "0.1.9" +version = "0.1.10" dependencies = [ "anyhow", "axum 0.7.9", @@ -1985,7 +1985,7 @@ dependencies = [ [[package]] name = "fleet-node" -version = "0.1.9" +version = "0.1.10" dependencies = [ "anyhow", "clap", @@ -2001,7 +2001,7 @@ dependencies = [ [[package]] name = "fleet-proof-app" -version = "0.1.9" +version = "0.1.10" dependencies = [ "anyhow", "axum 0.7.9", @@ -2021,7 +2021,7 @@ dependencies = [ [[package]] name = "fleet-proto" -version = "0.1.9" +version = "0.1.10" dependencies = [ "anyhow", "prost 0.13.5", @@ -2035,7 +2035,7 @@ dependencies = [ [[package]] name = "fleet-scheduler" -version = "0.1.9" +version = "0.1.10" dependencies = [ "alloy-primitives", "anyhow", @@ -2070,7 +2070,7 @@ dependencies = [ [[package]] name = "fleet-ziren-node" -version = "0.1.9" +version = "0.1.10" dependencies = [ "anyhow", "bincode", @@ -2095,7 +2095,7 @@ dependencies = [ [[package]] name = "fleetctl" -version = "0.1.9" +version = "0.1.10" dependencies = [ "alloy-primitives", "anyhow", diff --git a/Cargo.toml b/Cargo.toml index db5ffa5..3c96db6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,7 @@ default-members = ["crates/fleet-proto", "crates/fleet-scheduler", "crates/fleet resolver = "2" [workspace.package] -version = "0.1.9" +version = "0.1.10" edition = "2021" publish = false diff --git a/README.md b/README.md index 83a8c76..a5068e7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/ProjectZKM/zkm-fleet/actions/workflows/ci.yml/badge.svg)](https://github.com/ProjectZKM/zkm-fleet/actions/workflows/ci.yml) -Fleet: A universal schedules apps and batch jobs on shared GPU/CPU machines. +Fleet schedules apps and batch jobs on shared GPU and CPU machines. - **Apps** keep containers running: a model server, or a Ziren prover with its program. - **Batches** run finite commands and release resources when cleanup finishes. @@ -11,25 +11,23 @@ Fleet: A universal schedules apps and batch jobs on shared GPU/CPU machines. ![Fleet architecture: apps and batches share machine resources](images/diagram.svg) -The control host runs the Rust scheduler, Postgres and an HTTPS gateway. MCP is -optional. Each execution machine runs `fleet-node` and its own local Ray instance. -Agents connect outward and register automatically. Fleet manages admission, -quotas, placement and lifecycle; workers own their application data and pipelines. -The [code model](docs/reference.md#code-names) uses jobs, app deployments and a -workload controller. Shared records come from protobuf; Kubernetes is not required. +The control host runs the Rust scheduler, Postgres, an HTTPS gateway and, optionally, +MCP. Each execution machine runs `fleet-node` and its own local Ray, and connects +outward to register. Fleet handles admission, quotas, placement and lifecycle; apps +own their data and pipelines. The [code model](docs/reference.md#code-names) follows +Kubernetes' jobs, deployments and controllers, without needing Kubernetes. -App owners create apps and manage workload metadata, priority, enable/disable and -scaling with namespace tokens holding the `manage` permission; the scheduler picks -each app's node. Administrators -manage namespace quotas, access, batch mount roots and runtime users, and app -execution settings; owner changes remain subject to namespace and tag limits. +Namespace owners create apps and batches and manage their priority, enabled state and +scaling, within their namespace's and tag's quotas; the scheduler picks each app's +node. Administrators manage namespaces, quotas, access, host storage for batches and +apps' execution settings. -EthProofs prepares witnesses, proves locally and reports from one integrated app -worker. It resumes its saved cursor and proves every block in sequence. +Two integrated app workers prove on their node's own GPU server: -The GOAT proof builder runs as an integrated app worker: it serves Bitcoin node -proof requests, proves its five circuits on its loopback GPU server and keeps its -task database and proofs in app storage. +- **EthProofs** prepares witnesses, proves and reports every block in sequence, + resuming from its saved cursor. +- **The GOAT proof builder** serves Bitcoin node proof requests, proves its five + circuits and keeps its task database and proofs in app storage. ## Get started @@ -44,6 +42,6 @@ task database and proofs in app storage. | [Installation](docs/installation.md) | Dependencies, binaries, control host, execution nodes and registration | | [Operations](docs/operations.md) | Namespace commands, tags, node capacity, app scaling, batches and logs | | [Maintenance](docs/maintenance.md) | Upgrades, Ray capacity changes, self-built programs, liveness and troubleshooting | -| [Reference](docs/reference.md) | Protocols, namespaces, tokens, quotas, APIs, storage and ports | +| [Reference](docs/reference.md) | Architecture, workloads, proofs, access, quotas, APIs, storage and ports | | [EthProofs](apps/ethproofs/README.md) | Pinned source, worker build, sequential proving and durable recovery | | [GOAT](apps/goat/README.md) | Pinned source, program bundle identity, packaging, cutover and recovery | diff --git a/crates/fleet-mcp/src/main.rs b/crates/fleet-mcp/src/main.rs index b366644..0f2c509 100644 --- a/crates/fleet-mcp/src/main.rs +++ b/crates/fleet-mcp/src/main.rs @@ -107,7 +107,7 @@ impl FleetctlClient { #[derive(Deserialize, schemars::JsonSchema)] #[serde(rename_all = "lowercase")] enum JobType { - /// Jobs of apps (proofs). + /// Proofs submitted to a proof app through the scheduler; an app's own work is not a job. App, Batch, } @@ -402,7 +402,7 @@ impl FleetMcp { #[tool( name = "fleet_jobs_list", - description = "List jobs visible to the authenticated token. Results are scoped by the token, namespace, and tag.", + description = "List jobs visible to the authenticated token: batch jobs, and proofs submitted to proof apps through the scheduler. An app's own work in its replicas is not a job. Results are scoped by the token, namespace, and tag.", annotations(read_only_hint = true, destructive_hint = false) )] async fn jobs_list( diff --git a/crates/fleet-proto/src/lib.rs b/crates/fleet-proto/src/lib.rs index d71089f..1371ae6 100644 --- a/crates/fleet-proto/src/lib.rs +++ b/crates/fleet-proto/src/lib.rs @@ -8,6 +8,7 @@ pub mod artifacts; pub mod json; pub mod models; pub mod sdk; +pub mod workload; pub use fleet::model; diff --git a/crates/fleet-proto/src/models.rs b/crates/fleet-proto/src/models.rs index 1a958d8..da28e5c 100644 --- a/crates/fleet-proto/src/models.rs +++ b/crates/fleet-proto/src/models.rs @@ -2,7 +2,7 @@ use crate::fleet::v1::ProofKind; use crate::model::v1::*; use crate::stage::v1::Status as StageStatus; -use serde_json::{json, Value}; +use serde_json::Value; pub fn validate_tag(tag: &str) -> anyhow::Result<()> { let (key, value) = tag @@ -228,35 +228,6 @@ impl serde::Serialize for ProofJob { } } -impl Job { - pub fn value(&self) -> Value { - let mut v = self.status.0.clone(); - for (k, x) in [ - ("type", json!("batch")), - ("id", json!(self.id)), - ("batch", json!(self.batch)), - ("namespace", json!(self.namespace)), - ("tag", json!(self.tag)), - ("owner", json!(self.owner)), - ("priority", self.policy["priority"].clone()), - ("worker", json!(self.worker)), - ("node", json!(self.worker)), - ] { - v[k] = x; - } - v - } - pub fn state(&self) -> &str { - self.status["state"].as_str().unwrap_or("") - } - pub fn gpus(&self) -> f64 { - self.request["gpus"].as_f64().unwrap_or(0.0) - } - pub fn cpus(&self) -> f64 { - self.request["cpus"].as_f64().unwrap_or(1.0) - } -} - impl ProofJobFilter { pub fn matches(&self, j: &ProofJob) -> bool { self.namespace.as_ref().is_none_or(|n| &j.namespace == n) @@ -359,6 +330,7 @@ impl RegisterAppRequest { mod tests { use super::*; use prost::Message; + use serde_json::json; #[test] fn an_app_is_its_container_and_a_prover_also_its_program() { diff --git a/crates/fleet-proto/src/workload.rs b/crates/fleet-proto/src/workload.rs new file mode 100644 index 0000000..a2840b2 --- /dev/null +++ b/crates/fleet-proto/src/workload.rs @@ -0,0 +1,629 @@ +//! The JSON shapes the scheduler and the nodes exchange about workloads. Records keep them as +//! JSON, as a node configured or reported them, so keys the scheduler does not know pass +//! through untouched; these views read and write the keys it does know, each in one place. +use crate::fleet::v1::NodeOperation; +use crate::model::v1::{AppDeployment, CommandRecord, Job, JobReport, NodeReport, ScaleStatus, WorkloadTemplate}; +use serde_json::{json, Value}; + +fn text<'a>(v: &'a Value, key: &str) -> &'a str { + v[key].as_str().unwrap_or("") +} + +/// A node's `apps.yaml` entry for one app, as a deployment, an app's template, a change's +/// target or a start carries it. In a node's report it also says how the app runs there. +#[derive(Clone, Copy, Debug)] +pub struct Entry<'a>(pub &'a Value); + +impl<'a> Entry<'a> { + /// What a scale changes: the GPUs and CPUs of each replica, and how many replicas run. + pub const RESOURCES: [&'static str; 3] = ["gpus", "cpus", "replicas"]; + /// The node's own configuration of an entry: the app it runs, its program and how its + /// worker launches. No change the scheduler sends replaces them. + pub const NODE_OWNED: [&'static str; 8] = ["app", "elf_id", "worker_command", "worker_env_file", + "worker_workdir", "worker_stop_timeout_s", "machine", "proof_storage_root"]; + /// What a report says about an entry beside its configuration, and the namespace reference + /// the scheduler keeps from it. + pub const REPORTED: [&'static str; 4] = ["ready", "logs", "cleaned", "namespace_ref"]; + + /// The entry's own name, unique on its node. + pub fn name(self) -> &'a str { + text(self.0, "name") + } + /// The app the entry runs. + pub fn app(self) -> &'a str { + text(self.0, "app") + } + /// The namespace's id once the scheduler has resolved it. + pub fn namespace(self) -> &'a str { + text(self.0, "namespace") + } + /// How the node's configuration names the namespace, by id or by name. + pub fn namespace_reference(self) -> Option<&'a str> { + self.0["namespace_ref"].as_str().or_else(|| self.0["namespace"].as_str()) + } + pub fn tag(self) -> Option<&'a str> { + self.0["tag"].as_str() + } + /// `app` for a container, `ziren-gpu` or `ziren-cpu` for a prover. + pub fn kind(self) -> &'a str { + text(self.0, "kind") + } + pub fn image(self) -> &'a str { + text(self.0, "image") + } + /// The machine the node's configuration anchors the entry to. + pub fn machine(self) -> &'a str { + text(self.0, "machine") + } + pub fn priority(self) -> i64 { + self.0["priority"].as_i64().unwrap_or(0) + } + pub fn disabled(self) -> bool { + self.0["disabled"].as_bool() == Some(true) + } + /// In a report: every replica runs and is healthy. + pub fn ready(self) -> bool { + self.0["ready"].as_bool() == Some(true) + } + /// In a report: nothing of the app is left running on the node. + pub fn cleaned(self) -> bool { + self.0["cleaned"].as_bool() == Some(true) + } + /// The GPUs of each replica, when the entry gives them. + pub fn gpus(self) -> Option { + self.0["gpus"].as_f64() + } + /// The GPUs of each replica: one where the entry gives none, none for a CPU prover. + pub fn gpus_or_default(self) -> f64 { + self.gpus().unwrap_or(if self.kind() == "ziren-cpu" { 0.0 } else { 1.0 }) + } + /// The CPUs of each replica, one where the entry gives none. + pub fn cpus(self) -> f64 { + self.0["cpus"].as_f64().unwrap_or(1.0) + } + /// How many replicas run, one where the entry gives none. + pub fn replicas(self) -> u64 { + self.0["replicas"].as_u64().unwrap_or(1) + } + fn replica_count(self) -> f64 { + self.0["replicas"].as_f64().unwrap_or(1.0) + } + /// The GPUs and CPUs of all its replicas; a replica counts `gpus` GPUs where the entry + /// gives none. + pub fn total(self, gpus: f64) -> (f64, f64) { + let n = self.replica_count(); + (self.gpus().unwrap_or(gpus) * n, self.cpus() * n) + } + /// Whether `other` gives it the same resources. A missing setting is one, and no GPU when + /// `other` is a CPU prover. + pub fn same_resources(self, other: Entry) -> bool { + let gpus = if other.kind() == "ziren-cpu" { 0.0 } else { 1.0 }; + (self.gpus().unwrap_or(gpus) - other.gpus().unwrap_or(gpus)).abs() < 1e-9 + && (self.cpus() - other.cpus()).abs() < 1e-9 + && (self.replica_count() - other.replica_count()).abs() < 1e-9 + } + /// How long its replicas take to become ready before a start fails. + pub fn ready_timeout_s(self) -> f64 { + self.0["ready_timeout_s"].as_f64().unwrap_or(600.0) + } + pub fn is_container(self) -> bool { + self.kind() == "app" + } + pub fn is_prover(self) -> bool { + self.kind().starts_with("ziren-") + } + /// A GPU prover whose node also runs the app's own worker against it, on one replica. + pub fn integrated_worker(self) -> bool { + self.kind() == "ziren-gpu" && self.0["worker_command"].as_array().is_some_and(|command| !command.is_empty()) + } + /// Whether the deployment holds its namespace's quota for all its replicas: a container, + /// or an integrated worker. A plain prover is charged per proof instead. + pub fn holds_allocation(self) -> bool { + self.is_container() || self.integrated_worker() + } +} + +/// An entry being changed. +pub struct EntryMut<'a>(pub &'a mut Value); + +impl EntryMut<'_> { + pub fn set_disabled(&mut self, disabled: bool) { + self.0["disabled"] = json!(disabled); + } + pub fn set_priority(&mut self, priority: i64) { + self.0["priority"] = json!(priority); + } + /// Gives the entry priority 0 where it has none. + pub fn default_priority(&mut self) { + if self.0["priority"].is_null() { + self.set_priority(0); + } + } + /// Sets the resources a scale gives, keeping the others. + pub fn set_resources(&mut self, gpus: Option, cpus: Option, replicas: Option) { + if let Some(v) = gpus { + self.0["gpus"] = json!(v); + } + if let Some(v) = cpus { + self.0["cpus"] = json!(v); + } + if let Some(v) = replicas { + self.0["replicas"] = json!(v); + } + } + /// Keeps how the node names the namespace, and the namespace's id once it is known. + pub fn set_namespace(&mut self, reference: &str, id: Option<&str>) { + self.0["namespace_ref"] = json!(reference); + if let Some(id) = id { + self.0["namespace"] = json!(id); + } + } + /// Drops what a report said about the entry, keeping its configuration. + pub fn forget_report(&mut self) { + if let Some(fields) = self.0.as_object_mut() { + for key in Entry::REPORTED { + fields.remove(key); + } + } + } +} + +impl AppDeployment { + /// Its entry as the scheduler last configured it. + pub fn entry(&self) -> Entry<'_> { + Entry(&self.spec) + } +} + +impl ScaleStatus { + /// The entry the change ends with. + pub fn target_entry(&self) -> Entry<'_> { + Entry(&self.target) + } +} + +impl NodeReport { + /// The entries the node configures, with how each runs there. + pub fn entries(&self) -> impl Iterator> { + self.apps.iter().map(|v| Entry(v)) + } + /// Its entry named `name`. + pub fn entry(&self, name: &str) -> Option> { + self.entries().find(|e| e.name() == name) + } +} + +impl WorkloadTemplate { + /// An app's entry as the scheduler creates it on a node; a batch's settings. + pub fn entry(&self) -> Entry<'_> { + Entry(&self.spec) + } + /// The tag it belongs to: the one it was created for, or the one its node configures. + pub fn fixed_tag(&self) -> Option<&str> { + self.tag.as_deref().or_else(|| self.spec["tag"].as_str()) + } + pub fn disabled(&self) -> bool { + self.spec["disabled"].as_bool() == Some(true) + } + pub fn set_disabled(&mut self, disabled: bool) { + self.spec["disabled"] = json!(disabled); + } + pub fn priority(&self) -> &Value { + &self.spec["priority"] + } + pub fn set_priority(&mut self, priority: i64) { + self.spec["priority"] = json!(priority); + } +} + +/// A batch's settings, as its template keeps them and each of its jobs keeps them as its policy. +#[derive(Clone, Copy, Debug)] +pub struct Batch<'a>(pub &'a Value); + +impl<'a> Batch<'a> { + /// The most GPUs one job may ask for. + pub fn max_gpus(self) -> f64 { + self.0["max_gpus"].as_f64().unwrap_or(8.0) + } + /// The VRAM limit of a job that gives none. + pub fn default_gpu_mem_gb(self) -> u64 { + self.0["default_gpu_mem_gb"].as_u64().unwrap_or(0) + } + /// The host paths its jobs may mount. + pub fn allow(self) -> &'a [Value] { + self.0["allow"].as_array().map(Vec::as_slice).unwrap_or_default() + } + /// The user its jobs run as. + pub fn user(self) -> Option<&'a str> { + self.0["user"].as_str() + } + pub fn priority(self) -> i64 { + self.0["priority"].as_i64().unwrap_or(0) + } + /// Whether a higher-priority job may stop its running jobs. + pub fn preemptible(self) -> bool { + self.0["preemptible"].as_bool().unwrap_or(true) + } + /// How long a job runs before it may be preempted. + pub fn min_runtime_s(self) -> u64 { + self.0["min_runtime_s"].as_u64().unwrap_or(300) + } +} + +impl WorkloadTemplate { + /// A batch's settings. + pub fn batch(&self) -> Batch<'_> { + Batch(&self.spec) + } +} + +/// A batch job's submission, as its owner sent it and its record keeps it once checked. +#[derive(Clone, Copy, Debug)] +pub struct Submission<'a>(pub &'a Value); + +impl<'a> Submission<'a> { + /// The name its job's id starts with. + pub fn name(self) -> &'a str { + text(self.0, "name") + } + /// The GPUs it asks for: one where it gives none. + pub fn gpus(self) -> f64 { + self.0["gpus"].as_f64().unwrap_or(1.0) + } + /// The CPUs it asks for: eight where it gives none. + pub fn cpus(self) -> u64 { + self.0["cpus"].as_u64().unwrap_or(8) + } + /// The machines it runs on: one where it gives none. + pub fn nodes(self) -> u64 { + self.0["nodes"].as_u64().unwrap_or(1) + } + /// Its VRAM limit, when it gives one. + pub fn gpu_mem(self) -> Option { + self.0["gpu_mem"].as_u64() + } + /// Its command, when it is a non-empty list of strings. + pub fn command(self) -> Option<&'a [Value]> { + self.0["command"].as_array().filter(|cmd| !cmd.is_empty() && cmd.iter().all(Value::is_string)).map(Vec::as_slice) + } + /// The host paths it mounts, each `PATH[:MODE]`. + pub fn mounts(self) -> &'a [Value] { + self.0["mount"].as_array().map(Vec::as_slice).unwrap_or_default() + } + /// The machine it is pinned to. + pub fn machine(self) -> Option<&'a str> { + self.0["machine"].as_array().and_then(|m| m.first()).and_then(Value::as_str) + } + /// The job it waits for. + pub fn after(self) -> Option<&'a str> { + self.0["after"].as_str() + } + /// Whether it names its own log file. + pub fn names_log(self) -> bool { + self.0["log"].is_string() + } +} + +/// A submission being completed with what its batch and the scheduler decide. +pub struct SubmissionMut<'a>(pub &'a mut Value); + +impl SubmissionMut<'_> { + /// Records the resources it gets, on one machine. + pub fn set_resources(&mut self, gpus: f64, cpus: u64, gpu_mem: u64) { + self.0["gpus"] = json!(gpus); + self.0["cpus"] = json!(cpus); + self.0["gpu_mem"] = json!(gpu_mem); + self.0["nodes"] = json!(1); + } +} + +impl Job { + /// The record as listings show it: its status with who and where. + pub fn value(&self) -> Value { + let mut v = self.status.0.clone(); + for (k, x) in [ + ("type", json!("batch")), + ("id", json!(self.id)), + ("batch", json!(self.batch)), + ("namespace", json!(self.namespace)), + ("tag", json!(self.tag)), + ("owner", json!(self.owner)), + ("priority", self.policy["priority"].clone()), + ("worker", json!(self.worker)), + ("node", json!(self.worker)), + ] { + v[k] = x; + } + v + } + pub fn state(&self) -> &str { + self.status["state"].as_str().unwrap_or("") + } + pub fn gpus(&self) -> f64 { + self.request["gpus"].as_f64().unwrap_or(0.0) + } + pub fn cpus(&self) -> f64 { + self.request["cpus"].as_f64().unwrap_or(1.0) + } + /// What it was submitted with. + pub fn submission(&self) -> Submission<'_> { + Submission(&self.request) + } + /// Its batch's settings when it was submitted. + pub fn policy(&self) -> Batch<'_> { + Batch(&self.policy) + } + pub fn submitted_at(&self) -> Option { + self.status["submitted_at"].as_f64() + } + pub fn started_at(&self) -> Option { + self.status["started_at"].as_f64() + } + pub fn finished_at(&self) -> Option { + self.status["finished_at"].as_f64() + } + pub fn set_state(&mut self, state: &str) { + self.status["state"] = json!(state); + } + pub fn set_detail(&mut self, detail: &str) { + self.status["detail"] = json!(detail); + } + /// Ends it in `state` at `at`. + pub fn finish(&mut self, state: &str, at: f64) { + self.set_state(state); + self.status["finished_at"] = json!(at); + } + /// Ends it failed with `error` at `at`. + pub fn fail(&mut self, error: impl serde::Serialize, at: f64) { + self.set_state("failed"); + self.status["error"] = json!(error); + self.status["finished_at"] = json!(at); + } + /// Puts it back in the queue, unfinished. + pub fn requeue(&mut self) { + self.set_state("waiting_quota"); + self.status["finished_at"] = Value::Null; + } +} + +impl JobReport { + pub fn state(&self) -> &str { + text(&self.status, "state") + } + /// The batch of a job the node runs. + pub fn batch(&self) -> &str { + text(&self.spec, "batch") + } + pub fn owner(&self) -> &str { + self.spec["owner"].as_str().unwrap_or("admin") + } + pub fn tag(&self) -> Option<&str> { + self.spec["tag"].as_str() + } + pub fn policy(&self) -> &Value { + &self.spec["policy"] + } + /// The submission of a job the node runs that the scheduler has no record of: its command + /// and resources, on one machine. + pub fn submission(&self) -> Value { + let mut request = json!({"command": self.spec["command"], "gpus": self.spec["gpus"], "cpus": self.spec["cpus"], "nodes": 1}); + if request["gpus"].is_null() { + request["gpus"] = json!(0); + } + request + } +} + +impl CommandRecord { + pub fn is(&self, operation: NodeOperation) -> bool { + self.operation == operation as i32 + } + /// A job command's job. + pub fn job_id(&self) -> Option<&str> { + self.parameters["id"].as_str() + } + /// A job command's execution lease. + pub fn lease(&self) -> &str { + text(&self.parameters, "lease") + } + /// A start or stop's entry. + pub fn entry(&self) -> Entry<'_> { + Entry(&self.parameters["app"]) + } + /// Whether it starts its entry. + pub fn starts(&self) -> bool { + self.parameters["disabled"].as_bool() == Some(false) + } + /// Whether it stops its entry. + pub fn stops(&self) -> bool { + self.parameters["disabled"].as_bool() == Some(true) + } + /// The change a start or stop is a step of. + pub fn scale_id(&self) -> Option<&str> { + self.parameters["scale_id"].as_str() + } + /// Whether a stop cleans up after a failed change. + pub fn scale_cleanup(&self) -> bool { + self.parameters["scale_cleanup"].as_bool() == Some(true) + } + /// Whether it is a start or stop of entry `name` on `worker` that its node has not run yet. + pub fn changes_entry(&self, worker: &str, name: &str) -> bool { + !self.done && self.worker == worker && self.is(NodeOperation::SetApp) && self.entry().name() == name + } + /// The entry an app message creates on its node. + pub fn created(&self) -> Option> { + self.parameters.get("create").map(Entry) + } + /// The entry whose settings an app message changes. + pub fn changed_entry(&self) -> &str { + text(&self.parameters["entry"], "name") + } + /// The namespace and name of the app whose entries an app message removes. + pub fn removal(&self) -> Option<(&str, &str)> { + let app = &self.parameters["app"]; + (self.is(NodeOperation::App) && self.parameters["remove"] == json!(true)) + .then(|| (text(app, "namespace"), text(app, "name"))) + } + /// Whether it is an app message removing the entries that run `app`. + pub fn removes_app(&self, app: &str) -> bool { + self.removal().is_some_and(|(_, name)| name == app) + } + /// In its result: nothing of what it stopped is left running. + pub fn cleaned(&self) -> bool { + self.result["cleaned"].as_bool() == Some(true) + } + /// In an app removal's result: the entries the node removed. + pub fn removed(&self) -> impl Iterator { + self.result["removed"].as_array().into_iter().flatten().filter_map(Value::as_str) + } + /// In an app change's result: whether the node wrote entry `name`. + pub fn updated(&self, name: &str) -> bool { + self.result["updated"].as_array().is_some_and(|names| names.iter().any(|n| n.as_str() == Some(name))) + } +} + +/// The parameters of the commands the scheduler sends a node. +pub mod command { + use super::*; + + /// Starts an entry, or stops it when `disabled` (NODE_OPERATION_SET_APP). + pub fn set_app(entry: &Value, disabled: bool) -> Value { + json!({"app": entry, "disabled": disabled}) + } + /// Stops an entry whose change failed, cleaning up what is left of it. + pub fn clean_up(entry: &Value) -> Value { + json!({"app": entry, "disabled": true, "scale_cleanup": true}) + } + /// A step of change `scale_id`: `phase` is `stop` or `apply`. + pub fn scale_step(entry: &Value, disabled: bool, scale_id: &str, phase: &str) -> Value { + json!({"app": entry, "disabled": disabled, "scale_id": scale_id, "scale_phase": phase}) + } + /// Runs `job`'s execution lease on the node it is assigned to. + pub fn launch(job: &Job) -> Value { + json!({"id": job.id, "lease": job.lease, "namespace": job.namespace, "tag": job.tag, "owner": job.owner, + "batch": job.batch, "policy": job.policy, "request": job.request, "submitted_at": job.status["submitted_at"]}) + } + /// Cancels, preempts or purges a job's execution `lease`. + pub fn job(id: &str, lease: &str) -> Value { + json!({"id": id, "lease": lease}) + } + /// An app message's address: the entries that run app `name`. A node's configuration may + /// name the namespace by id or by name, so both go along. + pub fn app(namespace: &str, namespace_name: &str, name: &str) -> Value { + json!({"app": {"namespace": namespace, "namespace_name": namespace_name, "name": name}}) + } + /// An app message's address: the entry `name`. + pub fn entry(name: &str) -> Value { + json!({"entry": {"name": name}}) + } + /// An app message's address: the batch `name`. + pub fn batch(name: &str) -> Value { + json!({"batch": {"name": name}}) + } + /// Removes what the message addresses. + pub fn remove() -> Value { + json!({"remove": true}) + } + /// Changes these settings of what the message addresses. + pub fn set(settings: impl serde::Serialize) -> Value { + json!({"set": settings}) + } + /// Creates `entry` on the node. + pub fn create(entry: &Value) -> Value { + json!({"create": entry}) + } + /// `address` with `change`: an app message. + pub fn with(mut address: Value, change: &Value) -> Value { + for (key, value) in change.as_object().into_iter().flatten() { + address[key] = value.clone(); + } + address + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn an_entry_without_resources_counts_one_cpu_and_replica_and_the_gpus_its_reader_assumes() { + let bare = json!({"name": "model", "kind": "app"}); + let e = Entry(&bare); + assert_eq!((e.gpus(), e.cpus(), e.replicas()), (None, 1.0, 1)); + assert_eq!(e.total(0.0), (0.0, 1.0), "reserving and admitting count no GPU"); + assert_eq!(e.total(1.0), (1.0, 1.0), "placing counts one"); + assert_eq!(e.gpus_or_default(), 1.0); + assert_eq!(Entry(&json!({"kind": "ziren-cpu"})).gpus_or_default(), 0.0); + let sized = json!({"gpus": 2, "cpus": 8, "replicas": 3}); + assert_eq!(Entry(&sized).total(0.0), (6.0, 24.0)); + } + + #[test] + fn same_resources_takes_the_missing_gpus_from_the_other_entrys_kind() { + let missing = json!({"cpus": 1, "replicas": 1}); + assert!(Entry(&missing).same_resources(Entry(&json!({"gpus": 1, "kind": "app"})))); + assert!(Entry(&missing).same_resources(Entry(&json!({"gpus": 0, "kind": "ziren-cpu"})))); + assert!(!Entry(&missing).same_resources(Entry(&json!({"gpus": 2})))); + } + + #[test] + fn integrated_workers_and_containers_hold_their_allocation() { + let worker = json!({"kind": "ziren-gpu", "worker_command": ["run"]}); + assert!(Entry(&worker).integrated_worker() && Entry(&worker).holds_allocation()); + assert!(!Entry(&json!({"kind": "ziren-gpu", "worker_command": []})).holds_allocation()); + assert!(Entry(&json!({"kind": "app"})).holds_allocation()); + } + + #[test] + fn a_submission_and_a_batch_without_settings_take_the_documented_defaults() { + let bare = json!({"command": ["true"]}); + let sub = Submission(&bare); + assert_eq!((sub.gpus(), sub.cpus(), sub.nodes(), sub.gpu_mem()), (1.0, 8, 1, None)); + assert_eq!((sub.machine(), sub.after(), sub.names_log(), sub.mounts().len()), (None, None, false, 0)); + assert!(sub.command().is_some()); + assert!(Submission(&json!({"command": []})).command().is_none()); + assert!(Submission(&json!({"command": ["a", 1]})).command().is_none()); + let batch = Batch(&json!({})); + assert_eq!((batch.max_gpus(), batch.default_gpu_mem_gb(), batch.priority()), (8.0, 0, 0)); + assert_eq!((batch.preemptible(), batch.min_runtime_s(), batch.user(), batch.allow().len()), (true, 300, None, 0)); + } + + #[test] + fn ending_and_requeueing_a_job_write_its_state_error_and_finish_time() { + let mut job = Job { status: json!({"state": "running", "submitted_at": 1.0}).into(), ..Default::default() }; + job.fail("out of memory", 5.0); + assert_eq!(job.status.0, json!({"state": "failed", "submitted_at": 1.0, "error": "out of memory", "finished_at": 5.0})); + job.requeue(); + assert_eq!((job.state(), job.finished_at()), ("waiting_quota", None)); + job.finish("cancelled", 7.0); + assert_eq!((job.state(), job.finished_at(), job.submitted_at()), ("cancelled", Some(7.0), Some(1.0))); + } + + #[test] + fn commands_read_back_what_their_builders_wrote() { + let entry = json!({"name": "model-a", "app": "model"}); + let record = |operation: NodeOperation, parameters: Value, result: Value| CommandRecord { + id: "c".into(), worker: "gpu".into(), operation: operation as i32, parameters: parameters.into(), + done: false, result: result.into(), error: None, admitted: true, + }; + let start = record(NodeOperation::SetApp, command::scale_step(&entry, false, "s1", "apply"), json!(null)); + assert!(start.starts() && !start.stops() && start.scale_id() == Some("s1") && !start.scale_cleanup()); + assert!(start.changes_entry("gpu", "model-a") && !start.changes_entry("cpu", "model-a")); + let stop = record(NodeOperation::SetApp, command::clean_up(&entry), json!({"cleaned": true})); + assert!(stop.stops() && stop.scale_cleanup() && stop.cleaned()); + let remove = record(NodeOperation::App, command::with(command::app("ns", "team", "model"), &command::remove()), + json!({"removed": ["model-a"]})); + assert_eq!(remove.removal(), Some(("ns", "model"))); + assert!(remove.removes_app("model") && !remove.removes_app("other")); + assert_eq!(remove.removed().collect::>(), ["model-a"]); + let create = record(NodeOperation::App, command::with(command::app("ns", "team", "model"), &command::create(&entry)), json!(null)); + assert_eq!(create.created().map(Entry::name), Some("model-a")); + assert_eq!(create.removal(), None); + let set = record(NodeOperation::App, command::with(command::entry("model-a"), &command::set(json!({"disabled": true}))), + json!({"updated": ["model-a"]})); + assert_eq!(set.changed_entry(), "model-a"); + assert!(set.updated("model-a") && !set.updated("model-b")); + let job = record(NodeOperation::CancelJob, command::job("j1", "l1"), json!(null)); + assert_eq!((job.job_id(), job.lease()), (Some("j1"), "l1")); + } +} diff --git a/crates/fleet-scheduler/src/workloads.rs b/crates/fleet-scheduler/src/workloads.rs index 4d5499f..04882c9 100644 --- a/crates/fleet-scheduler/src/workloads.rs +++ b/crates/fleet-scheduler/src/workloads.rs @@ -65,6 +65,7 @@ fn response_error(e: anyhow::Error) -> Response { } use fleet_proto::model::v1::{WorkloadTemplate, Job, CommandRecord, NodeReport, ScaleStatus, AppDeployment}; +use fleet_proto::workload::{command as message, Batch, Entry, EntryMut, Submission, SubmissionMut}; use records::{Record, Records}; @@ -85,7 +86,7 @@ impl Record for CommandRecord { let job_operation = [NodeOperation::LaunchJob, NodeOperation::CancelJob, NodeOperation::PreemptJob] .iter() .any(|o| self.operation == *o as i32); - job_operation.then(|| self.parameters["id"].as_str()).flatten() + job_operation.then(|| self.job_id()).flatten() } } @@ -112,16 +113,11 @@ pub struct WorkloadController { ops: tokio::sync::Mutex<()>, notify: tokio::sync::Notify, } -pub(crate) fn integrated_worker(entry: &Value) -> bool { - text(entry, "kind") == "ziren-gpu" - && entry["worker_command"].as_array().is_some_and(|command| !command.is_empty()) -} - /// The proof apps an enabled integrated worker feeds: (namespace, app). fn worker_owned(inner: &WorkloadState) -> HashSet<(String, String)> { inner.deployments.values() - .filter(|s| integrated_worker(&s.spec) && !s.disabled) - .map(|s| (text(&s.spec, "namespace").to_owned(), text(&s.spec, "app").to_owned())) + .filter(|s| s.entry().integrated_worker() && !s.disabled) + .map(|s| (s.entry().namespace().to_owned(), s.entry().app().to_owned())) .collect() } @@ -144,7 +140,7 @@ impl WorkloadController { for v in load::(&dir.join("deployments"))? { inner .deployments - .insert(format!("{}-{}", v.worker, text(&v.spec, "name")), v); + .insert(format!("{}-{}", v.worker, v.entry().name()), v); } fleet.set_worker_owned_apps(worker_owned(&inner)); Ok(Self { @@ -169,11 +165,11 @@ impl WorkloadController { /// Mirrors a prover's state onto its proof app. A prover whose app is not registered has /// nothing to mirror: that must not fail the heartbeat or scale pass carrying it, or every /// later node result is rejected with it. - pub(super) async fn sync_app_record(&self, entry: &Value, disabled: bool) -> anyhow::Result<()> { - let (namespace, app) = (text(entry, "namespace"), text(entry, "app")); + pub(super) async fn sync_app_record(&self, entry: Entry<'_>, disabled: bool) -> anyhow::Result<()> { + let (namespace, app) = (entry.namespace(), entry.app()); if !self.fleet.app_known(namespace, app) { - if text(entry, "kind").starts_with("ziren-") { - tracing::warn!(entry = text(entry, "name"), app, "prover's proof app is not registered"); + if entry.is_prover() { + tracing::warn!(entry = entry.name(), app, "prover's proof app is not registered"); } return Ok(()); } @@ -199,15 +195,13 @@ impl WorkloadController { let i = self.inner.lock(); i.deployments.iter() .filter(|(_, intent)| { - let name = text(&intent.spec, "name"); + let name = intent.entry().name(); intent.worker == worker && intent.disabled && !intent.scale.as_ref().is_some_and(ScaleStatus::pending) && !reported.iter().any(|r| r == name) - && !i.commands.open().any(|c| !c.done && c.worker == worker - && c.operation == NodeOperation::SetApp as i32 - && text(&c.parameters["app"], "name") == name) + && !i.commands.open().any(|c| c.changes_entry(worker, name)) }) - .map(|(key, intent)| (key.clone(), text(&intent.spec, "namespace").to_owned())) + .map(|(key, intent)| (key.clone(), intent.entry().namespace().to_owned())) .collect() }; for (key, namespace) in candidates { @@ -218,19 +212,17 @@ impl WorkloadController { Ok(()) } - /// An app-level node message's address for `app` in `namespace`. A node's configuration may name the namespace by id - /// or by name, so both go along; the node finds the entries that run the app. + /// An app-level node message's address for `app` in `namespace`; the node finds the + /// entries that run the app. pub(super) fn app_message(&self, namespace: &str, app: &str) -> Value { let name = self.fleet.namespace(namespace).map(|n| n.name).unwrap_or_default(); - json!({"app":{"namespace":namespace,"namespace_name":name,"name":app}}) + message::app(namespace, &name, app) } - /// Sends an app-level change (`{"remove":true}` or `{"set":{..}}`) for `message`'s app or - /// batch to `workers`. A command waits in the scheduler while its node is offline. - pub(super) fn tell_nodes<'a>(&self, workers: impl IntoIterator, mut message: Value, change: Value) -> anyhow::Result> { - for (key, value) in change.as_object().into_iter().flatten() { - message[key] = value.clone(); - } + /// Sends an app-level change (`message::remove()` or `message::set(..)`) for `address`'s + /// app, entry or batch to `workers`. A command waits in the scheduler while its node is offline. + pub(super) fn tell_nodes<'a>(&self, workers: impl IntoIterator, address: Value, change: Value) -> anyhow::Result> { + let message = message::with(address, &change); workers.into_iter().map(|worker| self.command(worker, NodeOperation::App, message.clone())).collect() } @@ -238,20 +230,21 @@ impl WorkloadController { /// configures for it. pub(crate) fn remove_app(&self, namespace: &str, app: &str) -> anyhow::Result<()> { let workers: Vec = self.inner.lock().workers.keys().cloned().collect(); - self.tell_nodes(&workers, self.app_message(namespace, app), json!({"remove":true}))?; + self.tell_nodes(&workers, self.app_message(namespace, app), message::remove())?; Ok(()) } - fn check_worker_owner(&self, inner: &WorkloadState, worker: &str, entry: &Value, + fn check_worker_owner(&self, inner: &WorkloadState, worker: &str, entry: Entry<'_>, ) -> anyhow::Result<()> { - if !integrated_worker(entry) { return Ok(()); } - let namespace = text(entry, "namespace"); + if !entry.integrated_worker() { return Ok(()); } + let namespace = entry.namespace(); let allocations = self.fleet.allocations(namespace); - let key = format!("{}-{}", worker, text(entry, "name")); + let key = format!("{}-{}", worker, entry.name()); let owner = inner.deployments.iter().find(|(other_key, intent)| { - *other_key != &key && integrated_worker(&intent.spec) - && text(&intent.spec, "namespace") == namespace - && text(&intent.spec, "app") == text(entry, "app") + let other = intent.entry(); + *other_key != &key && other.integrated_worker() + && other.namespace() == namespace + && other.app() == entry.app() && (!intent.disabled || allocations.iter().any(|a| a.id == format!("deployment:{other_key}"))) }); if owner.is_some() { @@ -274,78 +267,69 @@ impl WorkloadController { parameters: Value, ) -> anyhow::Result { let mut inner = self.inner.lock(); - let same = |c: &&CommandRecord| { - (!c.done || operation == NodeOperation::LaunchJob) - && c.worker == worker - && c.operation == operation as i32 - && c.parameters == parameters + let c = CommandRecord { + id: uuid::Uuid::new_v4().to_string(), + worker: worker.into(), + operation: operation as i32, + parameters: parameters.into(), + done: false, + result: Value::Null.into(), + error: None, + admitted: operation != NodeOperation::SetApp, + }; + let same = |o: &&CommandRecord| { + (!o.done || operation == NodeOperation::LaunchJob) + && o.worker == worker + && o.is(operation) + && o.parameters == c.parameters }; - let found = match parameters["id"].as_str() { + let found = match c.job_id() { Some(job) if matches!(operation, NodeOperation::LaunchJob | NodeOperation::CancelJob | NodeOperation::PreemptJob) => { inner.commands.about(job).find(same) } _ => inner.commands.open().find(same), }; - if let Some(c) = found { - return Ok(c.id.clone()); + if let Some(o) = found { + return Ok(o.id.clone()); } - if operation == NodeOperation::SetApp { - if parameters["disabled"].as_bool() == Some(false) { - self.check_worker_owner(&inner, worker, ¶meters["app"])?; + if c.is(NodeOperation::SetApp) { + let entry = c.entry(); + if c.starts() { + self.check_worker_owner(&inner, worker, entry)?; } - if inner.deployments.get(&format!("{}-{}", worker, text(¶meters["app"], "name"))) - .and_then(|s| s.scale.as_ref()).is_some_and(|s| - s.pending() && parameters["scale_id"].as_str() != Some(s.id())) { + let key = format!("{}-{}", worker, entry.name()); + if inner.deployments.get(&key) + .and_then(|s| s.scale.as_ref()).is_some_and(|s| s.pending() && c.scale_id() != Some(s.id())) { return Err(fail(409, "app scaling is already pending")); } - let same_entry = |c: &CommandRecord| { - !c.done - && c.worker == worker - && c.operation == operation as i32 - && text(&c.parameters["app"], "name") == text(¶meters["app"], "name") - }; // An operation still waiting for capacity was never sent to the node, so a newer one // replaces it. Without this an app that cannot be placed could not be disabled or scaled. - let waiting: Vec<_> = inner.commands.open().filter(|c| same_entry(c) && !c.admitted).map(|c| c.id.clone()).collect(); + let waiting: Vec<_> = inner.commands.open().filter(|o| o.changes_entry(worker, entry.name()) && !o.admitted) + .map(|o| o.id.clone()).collect(); for id in waiting { - let replaced = inner.commands.update(&id, |c| { - c.done = true; - c.error = Some("replaced by a newer app operation before it was sent to the node".into()); - c.clone() + let replaced = inner.commands.update(&id, |o| { + o.done = true; + o.error = Some("replaced by a newer app operation before it was sent to the node".into()); + o.clone() }); - if let Some(c) = replaced { - self.save("commands", &id, &c)?; + if let Some(o) = replaced { + self.save("commands", &id, &o)?; } } - if inner.commands.open().any(same_entry) { + if inner.commands.open().any(|o| o.changes_entry(worker, entry.name())) { return Err(fail(409, "an app operation is already pending")); } let intent = AppDeployment { worker: worker.into(), - spec: parameters["app"].clone().into(), - disabled: parameters["disabled"].as_bool().unwrap_or(true), - scale: inner.deployments.get(&format!("{}-{}", worker, text(¶meters["app"], "name"))) - .and_then(|s| { - s.scale.as_ref().filter(|_| { - parameters["scale_id"].is_string() || parameters["scale_cleanup"].as_bool() == Some(true) - }).cloned() - }), + spec: entry.0.clone().into(), + disabled: !c.starts(), + scale: inner.deployments.get(&key) + .and_then(|s| s.scale.as_ref().filter(|_| c.scale_id().is_some() || c.scale_cleanup()).cloned()), }; - let key = format!("{}-{}", worker, text(&intent.spec, "name")); self.save("deployments", &key, &intent)?; inner.deployments.insert(key, intent); self.update_worker_owned_apps(&inner); } - let c = CommandRecord { - id: uuid::Uuid::new_v4().to_string(), - worker: worker.into(), - operation: operation as i32, - parameters: parameters.into(), - done: false, - result: Value::Null.into(), - error: None, - admitted: operation != NodeOperation::SetApp, - }; self.save("commands", &c.id, &c)?; let id = c.id.clone(); inner.commands.insert(id.clone(), c); @@ -362,9 +346,7 @@ impl WorkloadController { .chain(proofs.iter().filter(|n| n.machine == w.report.machine).filter_map(|n| n.busy.clone()), ) .collect::>(); - let apps = w.report.apps.iter().map(|s| { - text(s, "app") - }).collect::>(); + let apps = w.report.entries().map(Entry::app).collect::>(); json!({"id":id,"name":id,"backend":"ray-local", "gpus":w.report.total_gpus,"cpus":w.report.total_cpus,"machine":w.report.machine, "agent_version":w.report.agent_version, @@ -410,7 +392,7 @@ impl WorkloadController { ) -> bool { let selected = principal.and_then(|c| c.tag.as_deref()) .or_else(|| q.get("tag").map(String::as_str)); - let fixed = app.tag.as_deref().or_else(|| app.spec["tag"].as_str()); + let fixed = app.fixed_tag(); self.visible(&app.namespace, selected, principal, q) && fixed.is_none_or(|tag| { if principal.is_some() { selected == Some(tag) } @@ -468,7 +450,7 @@ impl WorkloadController { ) -> anyhow::Result<()> { if let Some(c) = c { let (_, deployment) = self.app_deployment(namespace, app, node)?; - if deployment.spec["tag"].as_str().is_some_and(|tag| c.tag.as_deref() != Some(tag)) { + if deployment.entry().tag().is_some_and(|tag| c.tag.as_deref() != Some(tag)) { return Err(fail(403, "another tag cannot be selected")); } } @@ -511,61 +493,7 @@ impl WorkloadController { q: &BTreeMap, ) -> anyhow::Result { let _ops = self.ops.lock().await; - let body = request - .as_object_mut() - .ok_or_else(|| fail(400, "expected object"))?; - let allowed = [ - "name", - "command", - "gpus", - "cpus", - "gpu_mem", - "vram_wait", - "machine", - "nodes", - "after", - "retry", - "restart_on_oom", - "mount", - "env", - "log", - "workdir", - ]; - if body.keys().any(|k| !allowed.contains(&k.as_str())) { - return Err(fail(400, "unknown or server-side submission fields")); - } - for key in [ - "cpus", - "nodes", - "gpu_mem", - "vram_wait", - "retry", - "restart_on_oom", - ] { - if !request[key].is_null() && request[key].as_u64().is_none() { - return Err(fail(400, format!("{key}: expected non-negative integer"))); - } - } - if !request["gpus"].is_null() && request["gpus"].as_f64().is_none() { - return Err(fail(400, "gpus: expected number")); - } - for key in ["mount", "env", "machine"] { - if !request[key].is_null() - && !request[key] - .as_array() - .is_some_and(|a| a.iter().all(Value::is_string)) - { - return Err(fail(400, format!("{key}: expected string array"))); - } - } - if request["machine"].as_array().is_some_and(|a| a.len() > 1) { - return Err(fail(422, "one machine may be selected for a local job")); - } - for key in ["name", "after", "log", "workdir"] { - if !request[key].is_null() && !request[key].is_string() { - return Err(fail(400, format!("{key}: expected string"))); - } - } + check_submission(&request)?; let name = batch.to_string(); Self::require_grant(c, &name)?; let app = self @@ -576,34 +504,39 @@ impl WorkloadController { .filter(|a| !a.removed) .cloned() .ok_or_else(|| fail(404, "unknown batch"))?; - if app.persistent || app.spec["disabled"].as_bool() == Some(true) { + // Only those who can see a batch submit to it: a batch fixed to a tag takes that tag's + // jobs, which then run with its mounts, user and environment. + if !self.app_visible(&app, c, q) { + return Err(fail(404, "unknown batch")); + } + if app.persistent || app.disabled() { return Err(fail(409, "app does not accept batch jobs")); } let mut selectors = q.clone(); selectors .entry("namespace".into()) .or_insert(app.namespace.clone()); + // An admin's job on a tagged batch is that tag's, so it counts against that tag's quota. + if let Some(tag) = app.fixed_tag() { + selectors.entry("tag".into()).or_insert(tag.to_owned()); + } let (namespace, tag) = self.scope(c, &selectors, Some(&app.namespace))?; - if !request["command"] - .as_array() - .is_some_and(|cmd| !cmd.is_empty() && cmd.iter().all(Value::is_string)) - { + let submission = Submission(&request); + if submission.command().is_none() { return Err(fail(400, "command: a non-empty list of strings")); } - let nodes = request["nodes"].as_u64().unwrap_or(1); - if nodes != 1 { + if submission.nodes() != 1 { return Err(fail( 422, "cross-machine jobs are not supported by local Ray; use one node", )); } - let gpus = request["gpus"].as_f64().unwrap_or(1.0); - let cpus = request["cpus"].as_u64().unwrap_or(8); + let (gpus, cpus) = (submission.gpus(), submission.cpus()); if !gpus.is_finite() || gpus < 0.0 || cpus < 1 || (gpus > 1.0 && gpus.fract() != 0.0) - || gpus > app.spec["max_gpus"].as_f64().unwrap_or(8.0) + || gpus > app.batch().max_gpus() { return Err(fail( 400, @@ -619,31 +552,25 @@ impl WorkloadController { if !n.quota.fits(gpus, cpus as f64) || !tq.fits(gpus, cpus as f64) { return Err(fail(422, "job exceeds namespace/tag quota")); } - if let Some(after) = request["after"].as_str() { + if let Some(after) = submission.after() { self.job(after, c, &selectors)?; } - let gpu_mem = request["gpu_mem"] - .as_u64() - .unwrap_or_else(|| app.spec["default_gpu_mem_gb"].as_u64().unwrap_or(0)); + let gpu_mem = submission.gpu_mem().unwrap_or_else(|| app.batch().default_gpu_mem_gb()); if gpus > 0.0 && gpus < 1.0 && gpu_mem == 0 { return Err(fail(400, "fractional GPUs require a VRAM limit")); } - let mounts = request["mount"].as_array().cloned().unwrap_or_default(); - let roots = app.spec["allow"].as_array().cloned().unwrap_or_default(); - if mounts.iter().any(|m| { + if submission.mounts().iter().any(|m| { m.as_str() - .is_none_or(|m| !path_allowed(m.split(':').next().unwrap_or(""), &roots)) + .is_none_or(|m| !path_allowed(m.split(':').next().unwrap_or(""), app.batch().allow())) }) { return Err(fail(403, "mount outside app allow roots")); } - if c.is_some() && request["log"].is_string() { + if c.is_some() && submission.names_log() { return Err(fail(403, "users must use job log storage")); } - request["gpus"] = json!(gpus); - request["cpus"] = json!(cpus); - request["gpu_mem"] = json!(gpu_mem); - request["nodes"] = json!(1); - let base = text(&request, "name"); + let waits = submission.after().is_some(); + SubmissionMut(&mut request).set_resources(gpus, cpus, gpu_mem); + let base = Submission(&request).name(); let base = if base.is_empty() { name.as_str() } else { base }; let base: String = base .chars() @@ -657,11 +584,7 @@ impl WorkloadController { }) .collect(); let id = format!("{base}-{}", &uuid::Uuid::new_v4().simple().to_string()[..8]); - let state = if request["after"].is_string() { - "waiting_dependency" - } else { - "waiting_quota" - }; + let state = if waits { "waiting_dependency" } else { "waiting_quota" }; let r = Job { id: id.clone(), lease: uuid::Uuid::new_v4().to_string(), @@ -713,32 +636,25 @@ impl WorkloadController { } let mut report = report; for entry in &mut report.apps { - let reference = entry["namespace_ref"] - .as_str() - .or_else(|| entry["namespace"].as_str()) + let reference = Entry(entry).namespace_reference() .unwrap_or(crate::store::SYSTEM_NAMESPACE) .to_owned(); - entry["namespace_ref"] = json!(reference); - if let Some(n) = self.fleet.namespace(&reference) { - entry["namespace"] = json!(n.id); - } + let id = self.fleet.namespace(&reference).map(|n| n.id); + EntryMut(entry).set_namespace(&reference, id.as_deref()); } for spec in report.batches.values().chain(report.apps.iter()) { - if let Some(tag) = spec["tag"].as_str() { + if let Some(tag) = Entry(spec).tag() { crate::fleet::Scheduler::validate_tag(tag).map_err(|e| fail(400, e.to_string()))?; } } let mut worker_streams = HashSet::new(); - for entry in &report.apps { - if integrated_worker(entry) { - if entry["replicas"].as_u64().unwrap_or(1) != 1 { - return Err(fail(409, "integrated workers require exactly one replica")); - } - if entry["disabled"].as_bool() != Some(true) - && !worker_streams.insert((text(entry, "namespace"), text(entry, "app"))) { - return Err(fail(409, "duplicate integrated worker stream in node report", - )); - } + for entry in report.entries().filter(|e| e.integrated_worker()) { + if entry.replicas() != 1 { + return Err(fail(409, "integrated workers require exactly one replica")); + } + if !entry.disabled() && !worker_streams.insert((entry.namespace(), entry.app())) { + return Err(fail(409, "duplicate integrated worker stream in node report", + )); } } let _ops = self.ops.lock().await; @@ -750,10 +666,8 @@ impl WorkloadController { { return Err(fail(409, "another agent session is active for this worker")); } - for entry in &report.apps { - if entry["disabled"].as_bool() != Some(true) && entry["cleaned"].as_bool() != Some(true) { - self.check_worker_owner(&i, worker_id, entry)?; - } + for entry in report.entries().filter(|e| !e.disabled() && !e.cleaned()) { + self.check_worker_owner(&i, worker_id, entry)?; } let draining = i.workers.get(worker_id).is_some_and(|w| w.draining); i.workers.insert( @@ -770,8 +684,8 @@ impl WorkloadController { report .apps .iter() - .filter(|s| text(s, "kind") == "app") - .map(|s| (text(s, "name").into(), s.clone())), + .filter(|s| Entry(s).is_container()) + .map(|s| (Entry(s).name().into(), s.clone())), ) { if !is_safe_id(&name) || !spec.is_object() { continue; @@ -783,10 +697,8 @@ impl WorkloadController { continue; }; spec["namespace"] = json!(namespace.id); - if spec["priority"].is_null() { - spec["priority"] = json!(0); - } - let persistent = text(&spec, "kind") == "app"; + EntryMut(&mut spec).default_priority(); + let persistent = Entry(&spec).is_container(); let existing = self.inner.lock().apps.get(&name).cloned(); // Apps and batches share one name space: a node's batch and app of the same // name would otherwise merge into one record. The one already known keeps the name. @@ -812,15 +724,15 @@ impl WorkloadController { self.save("apps", &name, &app)?; self.inner.lock().apps.insert(name, app); } - for observed in &report.apps { - let name = text(observed, "name"); + for observed in report.entries() { + let name = observed.name(); if !is_safe_id(name) - || !matches!(text(observed, "kind"), "app" | "ziren-gpu" | "ziren-cpu") - || text(observed, "kind") == "app" && self.inner.lock().apps.get(name).is_some_and(|a| !a.removed && !a.persistent) + || !matches!(observed.kind(), "app" | "ziren-gpu" | "ziren-cpu") + || observed.is_container() && self.inner.lock().apps.get(name).is_some_and(|a| !a.removed && !a.persistent) { continue; } - let Some(namespace) = self.fleet.namespace(text(observed, "namespace")) else { + let Some(namespace) = self.fleet.namespace(observed.namespace()) else { continue; }; let key = format!("{worker_id}-{name}"); @@ -828,23 +740,21 @@ impl WorkloadController { let intent = if let Some(mut intent) = existing { // The app an entry runs, its program and its launch settings are the node's own // configuration, not resource-scale overrides. - for field in ["app", "elf_id", "worker_command", "worker_env_file", "worker_workdir", "worker_stop_timeout_s", "machine", "proof_storage_root", - ] { - if observed.get(field).is_some() || intent.spec.get(field).is_some() { - intent.spec[field] = observed[field].clone(); + for field in Entry::NODE_OWNED { + if observed.0.get(field).is_some() || intent.spec.get(field).is_some() { + intent.spec[field] = observed.0[field].clone(); } } self.save("deployments", &key, &intent)?; self.inner.lock().deployments.insert(key.clone(), intent.clone()); intent } else { - let mut spec = observed.clone(); - spec["namespace_ref"] = observed["namespace_ref"].clone(); + let mut spec = observed.0.clone(); spec["namespace"] = json!(namespace.id); let intent = AppDeployment { worker: worker_id.into(), - spec, - disabled: observed["disabled"].as_bool().unwrap_or(false), + spec: spec.into(), + disabled: observed.disabled(), scale: None, }; self.save("deployments", &key, &intent)?; @@ -854,44 +764,30 @@ impl WorkloadController { .insert(key.clone(), intent.clone()); intent }; - if text(observed, "kind") == "app" || integrated_worker(observed) { + if observed.holds_allocation() { let id = format!("deployment:{key}"); - let replicas = observed["replicas"].as_f64().unwrap_or(1.0); - if observed["cleaned"].as_bool() == Some(true) && intent.disabled { + if observed.cleaned() && intent.disabled { self.fleet.release_job(&namespace.id, &id).await?; - } else if observed["ready"].as_bool() == Some(true) { + } else if observed.ready() { + let (gpus, cpus) = observed.total(0.0); self.fleet - .reserve_job( - &namespace.id, - observed["tag"].as_str(), - &id, - observed["gpus"].as_f64().unwrap_or(0.0) * replicas, - observed["cpus"].as_f64().unwrap_or(1.0) * replicas, - true, - ) + .reserve_job(&namespace.id, observed.tag(), &id, gpus, cpus, true) .await?; } } - let mismatch = observed["disabled"].as_bool().unwrap_or(false) != intent.disabled - || if intent.disabled { - observed["cleaned"].as_bool() != Some(true) - } else { - observed["ready"].as_bool() != Some(true) - }; + let mismatch = observed.disabled() != intent.disabled + || if intent.disabled { !observed.cleaned() } else { !observed.ready() }; let pending = { self.inner.lock().commands.open().any(|c| { - !c.done - && c.worker == worker_id - && (c.operation == NodeOperation::SetApp as i32 && text(&c.parameters["app"], "name") == name - || c.operation == NodeOperation::App as i32 && text(&c.parameters["entry"], "name") == name - // A start placed on another node: its own node removes its entry. - || c.operation == NodeOperation::App as i32 && c.parameters["remove"] == json!(true) - && text(&c.parameters["app"], "name") == text(observed, "app")) + c.changes_entry(worker_id, name) + || !c.done && c.worker == worker_id && c.is(NodeOperation::App) && c.changed_entry() == name + // A start placed on another node: its own node removes its entry. + || !c.done && c.worker == worker_id && c.removes_app(observed.app()) }) }; if !pending && !intent.scale.as_ref().is_some_and(ScaleStatus::pending) && !intent.scale.as_ref().is_some_and(ScaleStatus::failed) - && !scaling::same_resources(observed, &intent.spec) + && !observed.same_resources(intent.entry()) { // A node restart or local edit cannot replace the control plane's // desired allocation. Repair drift with the same drain/cleanup fences. @@ -904,14 +800,13 @@ impl WorkloadController { if mismatch && !pending && !intent.scale.as_ref().is_some_and(ScaleStatus::pending) { if intent.scale.as_ref().is_some_and(ScaleStatus::failed) { // After a failed change the app stays down, its failure still shown. - if observed["disabled"].as_bool() != Some(true) { - self.tell_nodes([&worker_id.to_owned()], json!({"entry":{"name":name}}), json!({"set":{"disabled":true}}))?; + if !observed.disabled() { + self.tell_nodes([&worker_id.to_owned()], message::entry(name), message::set(json!({"disabled": true})))?; } - if observed["cleaned"].as_bool() != Some(true) { - self.command(worker_id, NodeOperation::SetApp, - json!({"app":intent.spec,"disabled":true,"scale_cleanup":true}))?; + if !observed.cleaned() { + self.command(worker_id, NodeOperation::SetApp, message::clean_up(&intent.spec))?; } - } else if observed["disabled"].as_bool().unwrap_or(false) != intent.disabled { + } else if observed.disabled() != intent.disabled { // The node's configuration drifted from the desired one: write it back like a scale. let mut repaired = intent.clone(); repaired.scale = Some(scaling::new_scale(self.with_priority(intent.spec.clone()), intent.disabled)); @@ -920,11 +815,11 @@ impl WorkloadController { } else { // Configured as desired but not running as desired: start or stop it. self.command(worker_id, NodeOperation::SetApp, - json!({"app":self.with_priority(intent.spec.clone()),"disabled":intent.disabled}))?; + message::set_app(&self.with_priority(intent.spec.clone()), intent.disabled))?; } } } - self.forget_unreported_deployments(worker_id, &report.apps.iter().map(|s| text(s, "name").to_owned()).collect::>())?; + self.forget_unreported_deployments(worker_id, &report.entries().map(|e| e.name().to_owned()).collect::>())?; self.refresh_worker_owned_apps(); for incoming in &report.jobs { if !is_safe_id(&incoming.id) || !incoming.status.is_object() { @@ -934,23 +829,19 @@ impl WorkloadController { let mut r = if let Some(r) = existing.clone() { r } else { - let batch = text(&incoming.spec, "batch").to_string(); + let batch = incoming.batch().to_string(); let Some(app) = self.inner.lock().apps.get(&batch).filter(|a| !a.removed).cloned() else { continue; }; - let mut request = json!({"command":incoming.spec["command"],"gpus":incoming.spec["gpus"],"cpus":incoming.spec["cpus"],"nodes":1}); - if request["gpus"].is_null() { - request["gpus"] = json!(0); - } Job { id: incoming.id.clone(), lease: incoming.lease.clone(), - owner: incoming.spec["owner"].as_str().unwrap_or("admin").into(), + owner: incoming.owner().into(), namespace: app.namespace, - tag: incoming.spec["tag"].as_str().map(str::to_owned), + tag: incoming.tag().map(str::to_owned), batch, - policy: incoming.spec["policy"].clone().into(), - request: request.into(), + policy: incoming.policy().clone().into(), + request: incoming.submission().into(), worker: Some(worker_id.into()), status: incoming.status.clone(), quota_held: false, @@ -974,7 +865,7 @@ impl WorkloadController { "preempted", "lost", ]; - if !accepted.contains(&text(&incoming.status, "state")) { + if !accepted.contains(&incoming.state()) { continue; } // A reconnect cannot regress terminal state or cancel/preemption intent. @@ -994,11 +885,10 @@ impl WorkloadController { r.lease = uuid::Uuid::new_v4().to_string(); r.worker = None; r.logs_ready = false; - r.status["state"] = json!("waiting_quota"); - r.status["finished_at"] = Value::Null; + r.requeue(); r.stop = None; } else if r.stop.as_deref() == Some("cancel") { - r.status["state"] = json!("cancelled"); + r.set_state("cancelled"); r.stop = None; } } else if !r.quota_held @@ -1036,17 +926,14 @@ impl WorkloadController { c.done = true; c.result = result.result.clone(); c.error = result.error.clone(); - if c.operation == NodeOperation::LaunchJob as i32 && c.error.is_some() { - let id = text(&c.parameters, "id"); - let job = { self.inner.lock().jobs.get(id).cloned() }; + if c.is(NodeOperation::LaunchJob) && c.error.is_some() { + let job = { c.job_id().and_then(|id| self.inner.lock().jobs.get(id).cloned()) }; if let Some(mut r) = job { - if r.lease == text(&c.parameters, "lease") + if r.lease == c.lease() && r.worker.as_deref() == Some(worker_id) { - r.status["state"] = json!("failed"); - r.status["error"] = json!(c.error); - r.status["finished_at"] = json!(now()); - if c.result["cleaned"].as_bool() == Some(true) && r.quota_held { + r.fail(&c.error, now()); + if c.cleaned() && r.quota_held { self.fleet.release_job(&r.namespace, &r.id).await?; r.quota_held = false; r.logs_ready = true; @@ -1060,17 +947,11 @@ impl WorkloadController { NodeOperation::try_from(c.operation), Ok(NodeOperation::CancelJob | NodeOperation::PreemptJob) ) && c.error.is_none() - && c.result["cleaned"].as_bool() == Some(true) + && c.cleaned() { - let existing = { - self.inner - .lock() - .jobs - .get(text(&c.parameters, "id")) - .cloned() - }; + let existing = { c.job_id().and_then(|id| self.inner.lock().jobs.get(id).cloned()) }; if let Some(mut r) = existing { - if r.lease == text(&c.parameters, "lease") + if r.lease == c.lease() && r.worker.as_deref() == Some(worker_id) && r.stop.is_some() { @@ -1081,11 +962,9 @@ impl WorkloadController { if r.stop.as_deref() == Some("preempt") { r.lease = uuid::Uuid::new_v4().to_string(); r.worker = None; - r.status["state"] = json!("waiting_quota"); - r.status["finished_at"] = Value::Null; + r.requeue(); } else { - r.status["state"] = json!("cancelled"); - r.status["finished_at"] = json!(now()); + r.finish("cancelled", now()); } r.stop = None; self.save("jobs", &r.id, &r)?; @@ -1093,26 +972,38 @@ impl WorkloadController { } } } - if c.operation == NodeOperation::SetApp as i32 && c.error.is_none() { - let entry = &c.parameters["app"]; - if (text(entry, "kind") == "app" || integrated_worker(entry)) - && c.parameters["disabled"].as_bool() == Some(true) - && c.result["cleaned"].as_bool() == Some(true) - { + if c.is(NodeOperation::SetApp) && c.error.is_none() { + let entry = c.entry(); + if entry.holds_allocation() && c.stops() && c.cleaned() { self.fleet .release_job( - text(entry, "namespace"), - &format!("deployment:{}-{}", worker_id, text(entry, "name")), + entry.namespace(), + &format!("deployment:{}-{}", worker_id, entry.name()), ) .await?; } - if text(entry, "kind") == "ziren-gpu" - && c.parameters["disabled"].as_bool() == Some(false) - && c.parameters["scale_id"].is_null() - { + if entry.kind() == "ziren-gpu" && c.starts() && c.scale_id().is_none() { self.sync_app_record(entry, false).await?; } } + // A node removes an app's entries only once their containers are gone, so their + // room is free. It reports them no more and a stop sent after finds nothing, so + // nothing else releases it: a start placed on another node would wait on it. + if let Some((namespace, app)) = c.removal().filter(|_| c.error.is_none()) { + if let Some(namespace) = self.fleet.namespace(namespace) { + let keys: Vec = { + let i = self.inner.lock(); + c.removed() + .map(|name| format!("{worker_id}-{name}")) + .filter(|key| i.deployments.get(key).is_some_and(|d| d.worker == worker_id + && d.entry().app() == app && d.entry().holds_allocation())) + .collect() + }; + for key in keys { + self.fleet.release_job(&namespace.id, &format!("deployment:{key}")).await?; + } + } + } self.save("commands", &c.id, &c)?; self.inner.lock().commands.insert(c.id.clone(), c); } @@ -1132,7 +1023,7 @@ impl WorkloadController { .report .jobs .iter() - .filter(|s| !matches!(text(&s.status, "state"), "queued" | "waiting_quota")) + .filter(|s| !matches!(s.state(), "queued" | "waiting_quota")) .map(|s| (s.id.as_str(), s.lease.as_str())) .collect(); let pending = i @@ -1148,20 +1039,10 @@ impl WorkloadController { let starting = i .commands .open() - .filter(|c| { - c.worker == worker - && !c.done - && c.admitted - && c.operation == NodeOperation::SetApp as i32 - && c.parameters["disabled"].as_bool() == Some(false) - }) + .filter(|c| c.worker == worker && !c.done && c.admitted && c.is(NodeOperation::SetApp) && c.starts()) .fold((0.0, 0.0), |(g, c), cmd| { - let s = &cmd.parameters["app"]; - let n = s["replicas"].as_f64().unwrap_or(1.0); - ( - g + s["gpus"].as_f64().unwrap_or(0.0) * n, - c + s["cpus"].as_f64().unwrap_or(1.0) * n, - ) + let (gpus, cpus) = cmd.entry().total(0.0); + (g + gpus, c + cpus) }); Some(( (w.report.free_gpus - pending.0 - starting.0).max(0.0), @@ -1179,7 +1060,7 @@ impl WorkloadController { self.command( worker, NodeOperation::PurgeJob, - json!({"id":r.id,"lease":r.lease}), + message::job(&r.id, &r.lease), )?; } let path = self.dir.join("artifacts").join(&r.id); @@ -1203,11 +1084,10 @@ impl WorkloadController { } else { NodeOperation::CancelJob }, - json!({"id":r.id,"lease":r.lease}), + message::job(&r.id, &r.lease), )?; } else if !preempt { - r.status["state"] = json!("cancelled"); - r.status["finished_at"] = json!(now()); + r.finish("cancelled", now()); r.logs_ready = true; self.save("jobs", &r.id, &r)?; self.inner.lock().jobs.insert(r.id.clone(), r.clone()); @@ -1239,13 +1119,13 @@ impl WorkloadController { r.worker.as_ref().map(|w| PreemptionCandidate { id: r.id.clone(), namespace: r.namespace.clone(), - priority: r.policy["priority"].as_i64().unwrap_or(0) as i32, + priority: r.policy().priority() as i32, node_ip: w.clone(), gpus: r.gpus(), cpus: r.cpus(), - started_at: r.status["started_at"].as_f64(), - preemptible: r.policy["preemptible"].as_bool().unwrap_or(true), - min_runtime_s: r.policy["min_runtime_s"].as_u64().unwrap_or(300), + started_at: r.started_at(), + preemptible: r.policy().preemptible(), + min_runtime_s: r.policy().min_runtime_s(), }) }) .collect::>(); @@ -1326,12 +1206,10 @@ impl WorkloadController { } else { NodeOperation::CancelJob }; - let parameters = json!({"id":r.id,"lease":r.lease}); + let parameters = message::job(&r.id, &r.lease); let recorded = { self.inner.lock().commands.about(&r.id).any(|c| { - c.worker == *r.worker.as_ref().unwrap() - && c.operation == operation as i32 - && c.parameters == parameters + c.worker == *r.worker.as_ref().unwrap() && c.is(operation) && c.parameters == parameters }) }; if !recorded { @@ -1343,25 +1221,23 @@ impl WorkloadController { .lock() .commands .open() - .filter(|c| { - !c.done && !c.admitted && c.operation == NodeOperation::SetApp as i32 - }) + .filter(|c| !c.done && !c.admitted && c.is(NodeOperation::SetApp)) .cloned() .collect::>() }; for mut c in waiting { - let s = &c.parameters["app"]; - let disabled = c.parameters["disabled"].as_bool() == Some(true); + let s = c.entry(); + let disabled = c.stops(); // A start that names a superseded scale is stale: the deployment's current scale // (or its absence) is authoritative. Retire it instead of letting it place the app // on some other node, or evict the placement of the change that replaced it. - if !disabled && c.parameters.get("scale_id").is_some() { + if !disabled && c.scale_id().is_some() { let stale = { let i = self.inner.lock(); i.deployments - .get(&format!("{}-{}", c.worker, text(s, "name"))) + .get(&format!("{}-{}", c.worker, s.name())) .and_then(|d| d.scale.as_ref()) - .is_none_or(|d| c.parameters["scale_id"].as_str() != Some(d.id())) + .is_none_or(|d| c.scale_id() != Some(d.id())) }; if stale { c.done = true; @@ -1374,8 +1250,8 @@ impl WorkloadController { // Drain scheduler-owned executions before stopping an app, regardless of its // runtime. Persisted executions also fence stops while a replica is disconnected. if disabled { - let names = (0..s["replicas"].as_u64().unwrap_or(1)) - .map(|n| format!("{}-{n}", text(s, "name"))) + let names = (0..s.replicas()) + .map(|n| format!("{}-{n}", s.name())) .collect::>(); if self .fleet @@ -1390,8 +1266,8 @@ impl WorkloadController { .fleet .recent_jobs( &crate::store::ProofJobFilter { - namespace: Some(text(s, "namespace").into()), - app: Some(text(s, "app").into()), + namespace: Some(s.namespace().into()), + app: Some(s.app().into()), active: true, ..Default::default() }, @@ -1409,16 +1285,10 @@ impl WorkloadController { let already_ready = { let i = self.inner.lock(); i.workers.get(&c.worker).is_some_and(|w| { - w.report.apps.iter().any(|v| { - text(v, "name") == text(s, "name") && v["ready"].as_bool() == Some(true) - && v["disabled"].as_bool() != Some(true) - && scaling::same_resources(v, s) - }) + w.report.entries().any(|v| v.name() == s.name() && v.ready() && !v.disabled() && v.same_resources(s)) }) }; - let n = s["replicas"].as_f64().unwrap_or(1.0); - let g = s["gpus"].as_f64().unwrap_or(0.0) * n; - let cpu = s["cpus"].as_f64().unwrap_or(1.0) * n; + let (g, cpu) = s.total(0.0); let fits = already_ready || self .capacity(&c.worker) @@ -1429,9 +1299,9 @@ impl WorkloadController { } if !fits { self.preempt_for( - text(s, "namespace"), - s["tag"].as_str(), - s["priority"].as_i64().unwrap_or(0) as i32, + s.namespace(), + s.tag(), + s.priority() as i32, g, cpu, vec![Some(c.worker.clone())], @@ -1440,13 +1310,13 @@ impl WorkloadController { .await?; continue; } - if (text(s, "kind") == "app" || integrated_worker(s)) + if s.holds_allocation() && self .fleet .reserve_job( - text(s, "namespace"), - s["tag"].as_str(), - &format!("deployment:{}-{}", c.worker, text(s, "name")), + s.namespace(), + s.tag(), + &format!("deployment:{}-{}", c.worker, s.name()), g, cpu, false, @@ -1475,16 +1345,8 @@ impl WorkloadController { }; drop(ops); jobs.sort_by(|a, b| { - b.policy["priority"] - .as_i64() - .unwrap_or(0) - .cmp(&a.policy["priority"].as_i64().unwrap_or(0)) - .then( - a.status["submitted_at"] - .as_f64() - .unwrap_or(0.0) - .total_cmp(&b.status["submitted_at"].as_f64().unwrap_or(0.0)), - ) + b.policy().priority().cmp(&a.policy().priority()) + .then(a.submitted_at().unwrap_or(0.0).total_cmp(&b.submitted_at().unwrap_or(0.0))) .then(a.id.cmp(&b.id)) }); for id in jobs.into_iter().map(|r| r.id) { @@ -1504,14 +1366,12 @@ impl WorkloadController { .cloned() .unwrap_or_default(); if !namespace.quota.fits(r.gpus(), r.cpus()) || !tag_quota.fits(r.gpus(), r.cpus()) { - r.status["state"] = json!("failed"); - r.status["error"] = json!("job exceeds current namespace/tag quota"); - r.status["finished_at"] = json!(now()); + r.fail("job exceeds current namespace/tag quota", now()); self.save("jobs", &r.id, &r)?; self.inner.lock().jobs.insert(r.id.clone(), r); continue; } - if let Some(after) = r.request["after"].as_str() { + if let Some(after) = r.submission().after() { let state = self .inner .lock() @@ -1520,9 +1380,7 @@ impl WorkloadController { .map(|r| r.state().to_owned()); if state.as_deref() != Some("succeeded") { if state.as_deref().is_some_and(terminal) { - r.status["state"] = json!("failed"); - r.status["error"] = json!("dependency did not succeed"); - r.status["finished_at"] = json!(now()); + r.fail("dependency did not succeed", now()); self.save("jobs", &r.id, &r)?; self.inner.lock().jobs.insert(r.id.clone(), r); } @@ -1535,10 +1393,7 @@ impl WorkloadController { i.workers .iter() .filter(|(id, w)| { - let pin = r.request["machine"] - .as_array() - .and_then(|m| m.first()) - .and_then(Value::as_str); + let pin = r.submission().machine(); let app_ok = app.is_none_or(|a| a.workers.is_empty() || a.workers.contains(*id)); app_ok && pin.is_none_or(|p| p == id.as_str() || p == w.report.machine) @@ -1560,7 +1415,7 @@ impl WorkloadController { self.preempt_for( &r.namespace, r.tag.as_deref(), - r.policy["priority"].as_i64().unwrap_or(0) as i32, + r.policy().priority() as i32, r.gpus(), r.cpus(), allowed.into_iter().map(Some).take(1).collect(), @@ -1573,14 +1428,14 @@ impl WorkloadController { // the reservation refused would write its record twice on every pass. if !self.fleet.fits_quota(&r.namespace, r.tag.as_deref(), &format!("job:{}", r.id), r.gpus(), r.cpus()) { if r.state() != "waiting_quota" { - r.status["state"] = json!("waiting_quota"); + r.set_state("waiting_quota"); self.save("jobs", &r.id, &r)?; self.inner.lock().jobs.insert(r.id.clone(), r.clone()); } self.preempt_for( &r.namespace, r.tag.as_deref(), - r.policy["priority"].as_i64().unwrap_or(0) as i32, + r.policy().priority() as i32, r.gpus(), r.cpus(), vec![Some(worker)], @@ -1591,8 +1446,8 @@ impl WorkloadController { } r.quota_held = true; r.worker = Some(worker.clone()); - r.status["state"] = json!("queued"); - r.status["detail"] = json!("assigned; awaiting node acknowledgement"); + r.set_state("queued"); + r.set_detail("assigned; awaiting node acknowledgement"); self.save("jobs", &r.id, &r)?; if self .fleet @@ -1609,13 +1464,13 @@ impl WorkloadController { { r.quota_held = false; r.worker = None; - r.status["state"] = json!("waiting_quota"); + r.set_state("waiting_quota"); self.save("jobs", &r.id, &r)?; self.inner.lock().jobs.insert(r.id.clone(), r.clone()); self.preempt_for( &r.namespace, r.tag.as_deref(), - r.policy["priority"].as_i64().unwrap_or(0) as i32, + r.policy().priority() as i32, r.gpus(), r.cpus(), vec![Some(worker)], @@ -1625,7 +1480,7 @@ impl WorkloadController { continue; } self.inner.lock().jobs.insert(r.id.clone(), r.clone()); - self.command(&worker,NodeOperation::LaunchJob,json!({"id":r.id,"lease":r.lease,"namespace":r.namespace,"tag":r.tag,"owner":r.owner,"batch":r.batch,"policy":r.policy,"request":r.request,"submitted_at":r.status["submitted_at"]}))?; + self.command(&worker, NodeOperation::LaunchJob, message::launch(&r))?; } // A durable assigned job whose process died before command persistence still // receives the same lease. Uncertain execution is never moved to another node. @@ -1640,7 +1495,7 @@ impl WorkloadController { .collect::>() }; for r in assigned { - self.command(r.worker.as_deref().unwrap(),NodeOperation::LaunchJob,json!({"id":r.id,"lease":r.lease,"namespace":r.namespace,"tag":r.tag,"owner":r.owner,"batch":r.batch,"policy":r.policy,"request":r.request,"submitted_at":r.status["submitted_at"]}))?; + self.command(r.worker.as_deref().unwrap(), NodeOperation::LaunchJob, message::launch(&r))?; } Ok(()) } @@ -1661,7 +1516,7 @@ fn heartbeat_commands(i: &WorkloadState, worker: &str) -> Vec { const LIMIT: usize = 16; let (mut work, mut purges) = (vec![], vec![]); for c in i.commands.open().filter(|c| c.worker == worker && !c.done && c.admitted) { - if c.operation == NodeOperation::PurgeJob as i32 { + if c.is(NodeOperation::PurgeJob) { if purges.len() < LIMIT { purges.push(c); } @@ -1693,7 +1548,7 @@ impl WorkerService for NodeControlService { let reported = report .jobs .iter() - .filter(|r| r.cleaned && r.logs_ready && terminal(text(&r.status, "state"))) + .filter(|r| r.cleaned && r.logs_ready && terminal(r.state())) .map(|r| (r.id.clone(), r.lease.clone())) .collect::>(); let ack = self @@ -1767,6 +1622,53 @@ fn path_allowed(path: &str, roots: &[Value]) -> bool { }) } +/// What a batch job submission's field takes. +enum Takes { + Count, + Number, + Strings, + /// A list of at most one machine. + Machine, + Text, + /// Checked once the batch is known: `command: a non-empty list of strings`. + Command, +} + +/// A batch job submission's fields, in the order they are checked. +const SUBMISSION: [(&str, Takes); 15] = [ + ("cpus", Takes::Count), ("nodes", Takes::Count), ("gpu_mem", Takes::Count), ("vram_wait", Takes::Count), + ("retry", Takes::Count), ("restart_on_oom", Takes::Count), ("gpus", Takes::Number), + ("mount", Takes::Strings), ("env", Takes::Strings), ("machine", Takes::Machine), + ("name", Takes::Text), ("after", Takes::Text), ("log", Takes::Text), ("workdir", Takes::Text), + ("command", Takes::Command), +]; + +/// A submission has only the fields its owner may give, each of the type it takes. +fn check_submission(request: &Value) -> anyhow::Result<()> { + let body = request.as_object().ok_or_else(|| fail(400, "expected object"))?; + if body.keys().any(|k| !SUBMISSION.iter().any(|(key, _)| key == k)) { + return Err(fail(400, "unknown or server-side submission fields")); + } + for (key, takes) in SUBMISSION { + let value = &request[key]; + if value.is_null() { + continue; + } + let strings = || value.as_array().is_some_and(|a| a.iter().all(Value::is_string)); + match takes { + Takes::Count if value.as_u64().is_none() => return Err(fail(400, format!("{key}: expected non-negative integer"))), + Takes::Number if value.as_f64().is_none() => return Err(fail(400, format!("{key}: expected number"))), + Takes::Strings | Takes::Machine if !strings() => return Err(fail(400, format!("{key}: expected string array"))), + Takes::Machine if value.as_array().is_some_and(|a| a.len() > 1) => { + return Err(fail(422, "one machine may be selected for a local job")); + } + Takes::Text if !value.is_string() => return Err(fail(400, format!("{key}: expected string"))), + _ => {} + } + } + Ok(()) +} + /// The checks every batch's settings pass, on create and on update. fn check_batch_spec(spec: &Value) -> anyhow::Result<()> { if spec["priority"].as_i64().is_none_or(|p| i32::try_from(p).is_err()) @@ -1794,6 +1696,25 @@ fn check_batch_spec(spec: &Value) -> anyhow::Result<()> { Ok(()) } +/// What a new batch's request gives beside the settings that have a default. +const BATCH_IDENTITY: [&str; 4] = ["name", "namespace", "image", "user"]; + +/// A new batch's settings that have a default, with it. +fn batch_defaults() -> [(&'static str, Value); 10] { + [ + ("max_gpus", json!(8)), + ("default_gpu_mem_gb", json!(0)), + ("shm_gb", json!(16)), + ("priority", json!(0)), + ("allow", json!([])), + ("env", json!({})), + ("disabled", json!(false)), + ("preemptible", json!(true)), + ("min_runtime_s", json!(300)), + ("stop_timeout_s", json!(30)), + ] +} + /// Settings an update may change on a batch (its name, namespace, priority and enabled state /// have their own operations). const BATCH_SETTINGS: [&str; 10] = ["image", "user", "max_gpus", "default_gpu_mem_gb", "shm_gb", "allow", "env", @@ -1937,46 +1858,19 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: return Err(fail(400, "image is required")); } WorkloadController::require_grant(c, &name)?; - let allowed = [ - "name", - "namespace", - "image", - "user", - "max_gpus", - "default_gpu_mem_gb", - "shm_gb", - "allow", - "env", - "disabled", - "preemptible", - "min_runtime_s", - "stop_timeout_s", - "priority", - ]; - if body - .as_object() - .is_none_or(|o| o.keys().any(|k| !allowed.contains(&k.as_str()))) - { + let defaults = batch_defaults(); + if body.as_object().is_none_or(|o| { + o.keys().any(|k| !BATCH_IDENTITY.contains(&k.as_str()) && !defaults.iter().any(|(key, _)| key == k)) + }) { return Err(fail(400, "unknown batch fields")); } - for (key, default) in [ - ("max_gpus", json!(8)), - ("default_gpu_mem_gb", json!(0)), - ("shm_gb", json!(16)), - ("priority", json!(0)), - ("allow", json!([])), - ("env", json!({})), - ("disabled", json!(false)), - ("preemptible", json!(true)), - ("min_runtime_s", json!(300)), - ("stop_timeout_s", json!(30)), - ] { + for (key, default) in defaults { if body[key].is_null() { body[key] = default; } } check_batch_spec(&body)?; - if c.is_some() && (!body["allow"].as_array().unwrap().is_empty() || !body["user"].is_null()) { + if c.is_some() && (!Batch(&body).allow().is_empty() || Batch(&body).user().is_some()) { return Err(fail(403, "batch host mount roots and runtime user require administrator credentials")); } body["namespace"] = json!(namespace); @@ -2024,13 +1918,10 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: let _ops = s.ops.lock().await; s.ensure_not_scaling(&app.namespace, name)?; let persistent_ready_to_remove = !app.persistent - || (app.spec["disabled"].as_bool() == Some(true) + || (app.disabled() && app.workers.iter().all(|id| { s.inner.lock().workers.get(id).is_some_and(|w| { - now() - w.seen < 30.0 - && w.report.apps.iter().any(|v| { - text(v, "name") == name && v["cleaned"].as_bool() == Some(true) - }) + now() - w.seen < 30.0 && w.report.entries().any(|v| v.name() == name && v.cleaned()) }) })); if !persistent_ready_to_remove @@ -2087,7 +1978,7 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: .filter(|a| !a.removed && a.persistent != batch) .cloned() .ok_or_else(|| fail(404, "unknown app"))?; - app.spec["priority"] = json!(p); + app.set_priority(p.into()); s.save("apps", name, &app)?; s.inner.lock().apps.insert(name.into(), app.clone()); return Ok(Json(view(&s, &app)).into_response()); @@ -2110,7 +2001,7 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: .filter(|a| !a.removed && a.persistent != batch) .cloned() .ok_or_else(|| fail(404, "unknown app"))?; - app.spec["disabled"] = json!(disabled); + app.set_disabled(disabled); s.save("apps", name, &app)?; s.inner.lock().apps.insert(name.into(), app.clone()); if app.persistent { @@ -2151,12 +2042,7 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: .cloned() .collect::>() }; - jobs.sort_by(|a, b| { - b.status["submitted_at"] - .as_f64() - .unwrap_or(0.0) - .total_cmp(&a.status["submitted_at"].as_f64().unwrap_or(0.0)) - }); + jobs.sort_by(|a, b| b.submitted_at().unwrap_or(0.0).total_cmp(&a.submitted_at().unwrap_or(0.0))); return Ok(Json( jobs.into_iter() .take(limit) @@ -2177,7 +2063,7 @@ async fn handle_result(s: Arc, request: Request) -> anyhow:: && r.batch == batch && terminal(r.state()) && !r.quota_held - && r.status["finished_at"].as_f64().is_some_and(|v| v < cutoff) + && r.finished_at().is_some_and(|v| v < cutoff) && s.visible(&r.namespace, r.tag.as_deref(), c, &q) }; let ids: Vec = s.inner.lock().jobs.values().filter(|r| purgeable(r)).map(|r| r.id.clone()).collect(); @@ -2244,7 +2130,7 @@ impl WorkloadController { /// The keys of `app`'s deployment records in `namespace`. pub(crate) fn app_deployments(&self, namespace: &str, app: &str) -> Vec { self.inner.lock().deployments.iter() - .filter(|(_, d)| text(&d.spec, "app") == app && text(&d.spec, "namespace") == namespace) + .filter(|(_, d)| d.entry().app() == app && d.entry().namespace() == namespace) .map(|(k, _)| k.clone()) .collect() } @@ -2321,8 +2207,15 @@ impl WorkloadController { if changes { if let Some(c) = c { let i = self.inner.lock(); - if i.deployments.values().any(|d| text(&d.spec, "namespace") == namespace && text(&d.spec, "app") == name - && d.spec["tag"].as_str().is_some_and(|tag| c.tag.as_deref() != Some(tag))) { + // So must the tag the app was created for: until the app is placed, no deployment + // records it. + let owner = i.apps.get(&name).filter(|t| !t.removed && t.namespace == namespace) + .and_then(WorkloadTemplate::fixed_tag); + if owner.is_some_and(|tag| c.tag.as_deref() != Some(tag)) { + return Err(fail(403, "another tag owns this app")); + } + if i.deployments.values().map(AppDeployment::entry).any(|d| d.namespace() == namespace && d.app() == name + && d.tag().is_some_and(|tag| c.tag.as_deref() != Some(tag))) { return Err(fail(403, "another tag's deployments run this app")); } } diff --git a/crates/fleet-scheduler/src/workloads_scale.rs b/crates/fleet-scheduler/src/workloads_scale.rs index 4b24ab7..41846fa 100644 --- a/crates/fleet-scheduler/src/workloads_scale.rs +++ b/crates/fleet-scheduler/src/workloads_scale.rs @@ -29,17 +29,6 @@ fn waiting_for_capacity(i: &WorkloadState, intent: &AppDeployment) -> bool { }) } -pub(super) fn same_resources(a: &Value, b: &Value) -> bool { - ["gpus", "cpus", "replicas"].iter().all(|key| { - let default = if *key == "gpus" && text(b, "kind") == "ziren-cpu" { - 0.0 - } else { - 1.0 - }; - (a[*key].as_f64().unwrap_or(default) - b[*key].as_f64().unwrap_or(default)).abs() < 1e-9 - }) -} - impl WorkloadController { pub(crate) async fn app_lifecycle_guard( &self, @@ -53,9 +42,7 @@ impl WorkloadController { pub(super) fn ensure_not_scaling(&self, namespace: &str, app: &str) -> anyhow::Result<()> { if self.inner.lock().deployments.values().any(|s| { - text(&s.spec, "namespace") == namespace - && text(&s.spec, "app") == app - && s.scale.as_ref().is_some_and(ScaleStatus::pending) + s.entry().namespace() == namespace && s.entry().app() == app && s.scale.as_ref().is_some_and(ScaleStatus::pending) }) { return Err(fail(409, "app scaling is already pending")); } @@ -70,11 +57,11 @@ impl WorkloadController { ) -> anyhow::Result<(String, AppDeployment)> { let i = self.inner.lock(); let mut deployments = i.deployments.iter().filter(|(_, intent)| { - let s = &intent.spec; - text(s, "namespace") == namespace + let s = intent.entry(); + s.namespace() == namespace && node.is_none_or(|n| n == intent.worker) - && matches!(text(s, "kind"), "ziren-gpu" | "app") - && text(s, "app") == app + && matches!(s.kind(), "ziren-gpu" | "app") + && s.app() == app }); let first = deployments .next() @@ -88,21 +75,12 @@ impl WorkloadController { fn scale_value(&self, intent: &AppDeployment) -> Value { let i = self.inner.lock(); - let observed = i.workers.get(&intent.worker).and_then(|w| { - w.report - .apps - .iter() - .find(|s| text(s, "name") == text(&intent.spec, "name")) - }); - let target = intent - .scale - .as_ref() - .map(|s| &s.target) - .unwrap_or(&intent.spec); - json!({"node":intent.worker,"entry":text(&intent.spec,"name"), - "namespace":text(&intent.spec,"namespace"), - "gpus":target["gpus"].as_f64().unwrap_or(1.0),"cpus":target["cpus"].as_f64().unwrap_or(1.0), - "replicas":target["replicas"].as_u64().unwrap_or(1), + let observed = i.workers.get(&intent.worker).and_then(|w| w.report.entry(intent.entry().name())).map(|e| e.0); + let target = intent.scale.as_ref().map(ScaleStatus::target_entry).unwrap_or(intent.entry()); + json!({"node":intent.worker,"entry":intent.entry().name(), + "namespace":intent.entry().namespace(), + "gpus":target.gpus().unwrap_or(1.0),"cpus":target.cpus(), + "replicas":target.replicas(), "state":intent.scale.as_ref().map(|s| serde_json::to_value(&s.phase()).unwrap()).unwrap_or(json!("configured")), "disabled":intent.scale.as_ref().map(|s|s.disabled).unwrap_or(intent.disabled), "operation":intent.scale.as_ref().map(|s|s.id.clone()), @@ -129,7 +107,7 @@ impl WorkloadController { let bytes = self .fleet .log_relay - .request(&deployment.worker, text(&deployment.spec, "name"), lines) + .request(&deployment.worker, deployment.entry().name(), lines) .await .map_err(|e| fail(fleet_proto::sdk::http_status_code(e.code()), e.message().to_owned()))?; Ok(bytes.into_response()) @@ -163,13 +141,8 @@ impl WorkloadController { { let i = self.inner.lock(); if intent.scale.as_ref().is_some_and(ScaleStatus::pending) && !waiting_for_capacity(&i, &intent) - || i.commands.open().any(|c| { - !c.done - && c.admitted // one still waiting for capacity is replaced by this request - && c.worker == intent.worker - && c.operation == NodeOperation::SetApp as i32 - && text(&c.parameters["app"], "name") == text(&intent.spec, "name") - }) + // one still waiting for capacity is replaced by this request + || i.commands.open().any(|c| c.admitted && c.changes_entry(&intent.worker, intent.entry().name())) { return Err(fail(409, "an app operation is already pending")); } @@ -198,11 +171,7 @@ impl WorkloadController { let busy = { let i = self.inner.lock(); intent.scale.as_ref().is_some_and(ScaleStatus::pending) && !waiting_for_capacity(&i, &intent) - || i.commands.open().any(|c| { - !c.done && c.admitted && c.worker == intent.worker - && c.operation == NodeOperation::SetApp as i32 - && text(&c.parameters["app"], "name") == text(&intent.spec, "name") - }) + || i.commands.open().any(|c| c.admitted && c.changes_entry(&intent.worker, intent.entry().name())) }; if busy { return Err(fail(409, "an app operation is already pending")); @@ -219,13 +188,13 @@ impl WorkloadController { /// scheduler keeps (a proof app's record, or the app's template). pub(super) fn with_priority(&self, entry: impl Into) -> Value { let mut entry: Value = entry.into(); - let (namespace, app) = (text(&entry, "namespace"), text(&entry, "app")); + let (namespace, app) = (Entry(&entry).namespace(), Entry(&entry).app()); let priority = if self.fleet.app_known(namespace, app) { self.fleet.app_priority(namespace, app).into() } else { - self.inner.lock().apps.get(app).filter(|a| a.persistent).map_or(0, |a| a.spec["priority"].as_i64().unwrap_or(0)) + self.inner.lock().apps.get(app).filter(|a| a.persistent).map_or(0, |a| a.entry().priority()) }; - entry["priority"] = json!(priority); + EntryMut(&mut entry).set_priority(priority); entry } @@ -300,19 +269,10 @@ impl WorkloadController { } let (key, mut intent) = self.deployment_to_change(namespace, app, req.node.as_deref())?; let mut target = self.with_priority(intent.spec.clone()); - if let Some(v) = req.gpus { - target["gpus"] = json!(v); - } - if let Some(v) = req.cpus { - target["cpus"] = json!(v); - } - if let Some(v) = req.replicas { - target["replicas"] = json!(v); - } - let g = target["gpus"].as_f64().unwrap_or(1.0); - let cpu = target["cpus"].as_f64().unwrap_or(1.0); - let replicas = target["replicas"].as_u64().unwrap_or(1); - if super::integrated_worker(&target) && replicas != 1 { + EntryMut(&mut target).set_resources(req.gpus, req.cpus, req.replicas); + let t = Entry(&target); + let (g, cpu, replicas) = (t.gpus().unwrap_or(1.0), t.cpus(), t.replicas()); + if t.integrated_worker() && replicas != 1 { return Err(fail(400, "integrated app workers require one machine-pinned replica", )); } @@ -325,7 +285,7 @@ impl WorkloadController { { return Err(fail(400,"gpus must be nonnegative, cpus positive, and replicas between 1 and 64; use app disable to stop all replicas")); } - if text(&target, "kind") == "ziren-gpu" && (g < 1.0 || g.fract() != 0.0) { + if t.kind() == "ziren-gpu" && (g < 1.0 || g.fract() != 0.0) { return Err(fail( 400, "a proof server requires a positive integer GPU count", @@ -365,14 +325,14 @@ impl WorkloadController { } // Container replicas reserve the whole app. Proof quota is charged // per assigned proof; extra replicas can remain idle while that quota is occupied. - let (quota_g, quota_cpu) = if text(&target, "kind") == "app" || super::integrated_worker(&target) { + let (quota_g, quota_cpu) = if t.holds_allocation() { (total_g, total_cpu) } else { (g, cpu) }; let tag_quota = ns .tag_quotas - .get(target["tag"].as_str().unwrap_or("")) + .get(t.tag().unwrap_or("")) .cloned() .unwrap_or_default(); if !ns.quota.fits(quota_g, quota_cpu) || !tag_quota.fits(quota_g, quota_cpu) { @@ -381,7 +341,7 @@ impl WorkloadController { "requested allocation exceeds namespace or tag GPU/CPU quota", )); } - if same_resources(&intent.spec, &target) + if intent.entry().same_resources(t) && intent .scale .as_ref() @@ -410,8 +370,7 @@ impl WorkloadController { disabled: bool, phase: &str, ) -> anyhow::Result { - let parameters = json!({"app":spec,"disabled":disabled, - "scale_id":intent.scale.as_ref().unwrap().id,"scale_phase":phase}); + let parameters = message::scale_step(spec, disabled, &intent.scale.as_ref().unwrap().id, phase); // Completed commands are also reused after a crash between command and // phase persistence; the worker's command journal fences duplicate execution. if let Some(id) = self @@ -420,9 +379,7 @@ impl WorkloadController { .commands .values() .find(|c| { - c.worker == intent.worker - && c.operation == NodeOperation::SetApp as i32 - && c.parameters == parameters + c.worker == intent.worker && c.is(NodeOperation::SetApp) && c.parameters == parameters }) .map(|c| c.id.clone()) { @@ -463,10 +420,10 @@ impl WorkloadController { let mut scale = intent.scale.clone().unwrap(); if scale.phase() == Phase::Draining { // A proof app refuses jobs while its prover is down; an app's template records it. - self.sync_app_record(&intent.spec, true).await?; - let template = self.inner.lock().apps.get(text(&intent.spec, "app")).filter(|a| a.persistent).cloned(); + self.sync_app_record(intent.entry(), true).await?; + let template = self.inner.lock().apps.get(intent.entry().app()).filter(|a| a.persistent).cloned(); if let Some(mut app) = template { - app.spec["disabled"] = json!(true); + app.set_disabled(true); self.save("apps", &app.name, &app)?; self.inner.lock().apps.insert(app.name.clone(), app); } @@ -485,7 +442,7 @@ impl WorkloadController { if !command.done { return Ok(()); } - if command.error.is_some() || command.result["cleaned"].as_bool() != Some(true) { + if command.error.is_some() || !command.cleaned() { scale.phase = Phase::Failed as i32; scale.error = Some( command @@ -499,14 +456,13 @@ impl WorkloadController { { if scale.update.is_none() { let mut set = scale.set.as_ref().map(|s| s.0.clone()).unwrap_or_else(|| json!({})); - for k in ["gpus", "cpus", "replicas"] { + for k in Entry::RESOURCES { if !scale.target[k].is_null() { set[k] = scale.target[k].clone(); } } - set["disabled"] = json!(scale.disabled); - let message = json!({"entry":{"name":text(&intent.spec, "name")}}); - let sent = self.tell_nodes([&intent.worker], message, json!({"set": set}))?; + EntryMut(&mut set).set_disabled(scale.disabled); + let sent = self.tell_nodes([&intent.worker], message::entry(intent.entry().name()), message::set(set))?; scale.update = sent.into_iter().next(); self.save_scale(&key, scale.clone())?; } @@ -514,9 +470,7 @@ impl WorkloadController { if !command.done { return Ok(()); } - let written = command.result["updated"].as_array() - .is_some_and(|names| names.iter().any(|n| n.as_str() == Some(text(&intent.spec, "name")))); - if command.error.is_some() || !written { + if command.error.is_some() || !command.updated(intent.entry().name()) { scale.phase = Phase::Failed as i32; scale.error = Some(command.error.unwrap_or_else(|| "node did not write the new configuration".into())); self.save_scale(&key, scale)?; @@ -543,11 +497,7 @@ impl WorkloadController { } let first_ack = scale.ready_since.is_none(); let ready_since = *scale.ready_since.get_or_insert_with(now); - let timeout = scale.target["ready_timeout_s"] - .as_f64() - .unwrap_or(600.0) - .clamp(1.0, 3600.0) - + 30.0; + let timeout = scale.target_entry().ready_timeout_s().clamp(1.0, 3600.0) + 30.0; let (connected, applied) = { let i = self.inner.lock(); let worker = i @@ -557,12 +507,12 @@ impl WorkloadController { ( worker.is_some(), worker.is_some_and(|w| { - w.report.apps.iter().any(|s| { - text(s, "name") == text(&scale.target, "name") - && same_resources(s, &scale.target) - && s["disabled"].as_bool().unwrap_or(false) == scale.disabled - && s[if scale.disabled { "cleaned" } else { "ready" }].as_bool() - == Some(true) + let target = scale.target_entry(); + w.report.entries().any(|s| { + s.name() == target.name() + && s.same_resources(target) + && s.disabled() == scale.disabled + && if scale.disabled { s.cleaned() } else { s.ready() } }) }), ) @@ -584,22 +534,22 @@ impl WorkloadController { return Ok(()); } if !scale.disabled { - self.sync_app_record(&scale.target, false).await?; + self.sync_app_record(scale.target_entry(), false).await?; } let app = self .inner .lock() .apps - .get(text(&intent.spec, "app")) + .get(intent.entry().app()) .cloned(); if let Some(mut app) = app.filter(|a| a.persistent) { - for k in ["gpus", "cpus", "replicas"] { + for k in Entry::RESOURCES { app.spec[k] = scale.target[k].clone(); } for (k, v) in scale.set.as_ref().and_then(|set| set.as_object()).into_iter().flatten() { app.spec[k.as_str()] = v.clone(); } - app.spec["disabled"] = json!(scale.disabled); + app.set_disabled(scale.disabled); self.save("apps", &app.name, &app)?; self.inner.lock().apps.insert(app.name.clone(), app); } @@ -628,7 +578,7 @@ impl WorkloadController { { let i = self.inner.lock(); if i.apps.get(&name).is_some_and(|a| !a.removed) - || i.deployments.values().any(|d| text(&d.spec, "name") == name) + || i.deployments.values().any(|d| d.entry().name() == name) || self.fleet.app_known(&namespace, &name) { return Err(fail(409, format!("an app or batch named {name} exists"))); @@ -686,15 +636,14 @@ impl WorkloadController { return Ok(()); }; let pending = self.inner.lock().commands.open() - .find(|c| !c.done && c.operation == NodeOperation::App as i32 && text(&c.parameters["create"], "name") == app) + .find(|c| !c.done && c.is(NodeOperation::App) && c.created().is_some_and(|e| e.name() == app)) .map(|c| (c.worker.clone(), c.id.clone())); match pending { Some(pending) => pending, None => { - let (node, _) = self.place_app(&template.spec, &[])?; - let mut message = self.app_message(namespace, app); - message["create"] = template.spec.0.clone(); - let create = self.command(&node, NodeOperation::App, message)?; + let (node, _) = self.place_app(template.entry(), &[])?; + let create = message::with(self.app_message(namespace, app), &message::create(&template.spec)); + let create = self.command(&node, NodeOperation::App, create)?; template.workers = vec![node.clone()]; self.save("apps", app, &template)?; self.inner.lock().apps.insert(app.into(), template); @@ -737,10 +686,8 @@ impl WorkloadController { /// fewest free GPUs among those, as a batch-class proof app is placed, so larger requests /// keep the larger gaps; otherwise the one with the most free GPUs, where the app waits for /// room. Whether it has the room now comes along. `except` are nodes not to pick. - fn place_app(&self, entry: &Value, except: &[String]) -> anyhow::Result<(String, bool)> { - let replicas = entry["replicas"].as_f64().unwrap_or(1.0); - let gpus = entry["gpus"].as_f64().unwrap_or(1.0) * replicas; - let cpus = entry["cpus"].as_f64().unwrap_or(1.0) * replicas; + fn place_app(&self, entry: Entry<'_>, except: &[String]) -> anyhow::Result<(String, bool)> { + let (gpus, cpus) = entry.total(1.0); let large: Vec = self.inner.lock().workers.iter() .filter(|(id, w)| !except.contains(id) && w.report.total_gpus + 1e-9 >= gpus && w.report.total_cpus + 1e-9 >= cpus) .map(|(id, _)| id.clone()).collect(); @@ -760,24 +707,33 @@ impl WorkloadController { /// reports it (`start_on`); a node whose create fails is not picked again. With no room /// anywhere the caller preempts on the start's own node, and the start keeps the room that /// frees. Nothing of what the app keeps on a node's disks is copied: that is the app's own. + /// An entry whose node configures its `machine` is anchored to that machine and stays. /// Returns whether the start runs on another node, so it is not sent to its own. The caller /// holds the operation lock. pub(super) fn place_start(&self, start: &CommandRecord, fits: bool) -> anyhow::Result { - let s = &start.parameters["app"]; - let (namespace, app, name) = (text(s, "namespace"), text(s, "app"), text(s, "name")); + let s = start.entry(); + let (namespace, app, name) = (s.namespace(), s.app(), s.name()); let placed = self.inner.lock().apps.get(app).is_some_and(|a| a.persistent && !a.removed && a.namespace == namespace); - if !placed { + if !placed || !s.machine().is_empty() { return Ok(false); } let creates: Vec = self.inner.lock().commands.values() - .filter(|c| c.worker != start.worker && c.operation == NodeOperation::App as i32 - && text(&c.parameters["create"], "name") == name) + .filter(|c| c.worker != start.worker && c.is(NodeOperation::App) && c.created().is_some_and(|e| e.name() == name)) .cloned().collect(); if creates.iter().any(|c| !c.done) { return Ok(true); } - let created_on = creates.iter().filter(|c| c.error.is_none()).map(|c| c.worker.clone()) - .find(|w| self.inner.lock().deployments.contains_key(&format!("{w}-{name}"))); + // A node the app was created on before and has since moved off keeps a disabled record + // until it reports again; a start sent there would find no entry. So the start follows + // only a create whose node still reports the entry and is not told to remove the app. + let created_on = { + let i = self.inner.lock(); + creates.iter().filter(|c| c.error.is_none()).map(|c| c.worker.clone()).find(|w| { + i.deployments.contains_key(&format!("{w}-{name}")) + && i.workers.get(w).is_some_and(|node| node.report.entry(name).is_some()) + && !i.commands.open().any(|c| &c.worker == w && c.removes_app(app)) + }) + }; if let Some(target) = created_on { self.start_on(start, &target)?; return Ok(true); @@ -793,7 +749,7 @@ impl WorkloadController { let skip: Vec = { let i = self.inner.lock(); i.workers.iter() - .filter(|(id, w)| **id == start.worker || w.report.apps.iter().any(|a| text(a, "name") == name)) + .filter(|(id, w)| **id == start.worker || w.report.entry(name).is_some()) .map(|(id, _)| id.clone()) .chain(creates.iter().filter(|c| c.error.is_some()).map(|c| c.worker.clone())) .collect() @@ -801,20 +757,11 @@ impl WorkloadController { let Ok((target, true)) = self.place_app(s, &skip) else { return Ok(false); }; - let mut entry = s.clone(); - if let Some(fields) = entry.as_object_mut() { - // What the old node reported about the entry, not its configuration. - for key in ["ready", "logs", "cleaned", "namespace_ref"] { - fields.remove(key); - } - } - entry["disabled"] = json!(true); - if !entry["machine"].is_null() { - entry["machine"] = json!(self.inner.lock().workers[&target].report.machine); - } - let mut message = self.app_message(namespace, app); - message["create"] = entry; - self.command(&target, NodeOperation::App, message)?; + // What the old node reported about the entry goes, its configuration comes along. + let mut entry = s.0.clone(); + EntryMut(&mut entry).forget_report(); + EntryMut(&mut entry).set_disabled(true); + self.command(&target, NodeOperation::App, message::with(self.app_message(namespace, app), &message::create(&entry)))?; tracing::info!(app, node = target, "placing an app's start on a node with room for it"); Ok(true) } @@ -824,8 +771,8 @@ impl WorkloadController { /// starts on `target`. Removing the entry stops what is left of the app there; its data /// stays where the app keeps it, which may be storage `target` shares. fn start_on(&self, start: &CommandRecord, target: &str) -> anyhow::Result<()> { - let s = &start.parameters["app"]; - let (namespace, app, name) = (text(s, "namespace"), text(s, "app"), text(s, "name")); + let s = start.entry(); + let (namespace, app, name) = (s.namespace(), s.app(), s.name()); let old_key = format!("{}-{name}", start.worker); let old = self.inner.lock().deployments.get(&old_key).cloned(); if let Some(mut old) = old { @@ -841,7 +788,7 @@ impl WorkloadController { unsent.error = Some(format!("the app starts on {target}, which has room for it")); self.save("commands", &unsent.id, &unsent)?; self.inner.lock().commands.insert(unsent.id.clone(), unsent); - self.tell_nodes([&start.worker], self.app_message(namespace, app), json!({"remove": true}))?; + self.tell_nodes([&start.worker], self.app_message(namespace, app), message::remove())?; let template = self.inner.lock().apps.get(app).cloned(); if let Some(mut template) = template { template.workers = vec![target.into()]; diff --git a/crates/fleet-scheduler/src/workloads_tests.rs b/crates/fleet-scheduler/src/workloads_tests.rs index 8d441a7..4ada661 100644 --- a/crates/fleet-scheduler/src/workloads_tests.rs +++ b/crates/fleet-scheduler/src/workloads_tests.rs @@ -1046,6 +1046,44 @@ async fn user_token(c: &WorkloadController, ns: &str, seed: u8, tag: &str, apps: secret } +/// A submission's checks run in a fixed order, so a body with several faults gets the first +/// one's status and message; a batch's request takes only its own settings. +#[tokio::test] +async fn submissions_and_new_batches_are_checked_field_by_field() { + let (_dir, c, ns) = setup().await; + let created = scale_call(c.clone(), "admin", "POST", &format!("/batches?namespace={ns}"), json!({"name":"training","image":"python:3"})).await; + assert_eq!(created.status(), StatusCode::CREATED); + let unknown = scale_call(c.clone(), "admin", "POST", &format!("/batches?namespace={ns}"), + json!({"name":"other","image":"python:3","gpus":1})).await; + assert_eq!(unknown.status(), StatusCode::BAD_REQUEST); + assert_eq!(to_bytes(unknown.into_body(), 1 << 20).await.unwrap(), "unknown batch fields"); + let path = format!("/batches/training/jobs?namespace={ns}"); + for (body, status, message) in [ + (json!([]), 400, "expected object"), + (json!({"command":["true"],"priority":1}), 400, "unknown or server-side submission fields"), + (json!({"cpus":-1,"gpus":"one"}), 400, "cpus: expected non-negative integer"), + (json!({"retry":1.5}), 400, "retry: expected non-negative integer"), + (json!({"gpus":"one","mount":[1]}), 400, "gpus: expected number"), + (json!({"env":"A=1"}), 400, "env: expected string array"), + (json!({"machine":"gpu"}), 400, "machine: expected string array"), + (json!({"machine":["a","b"],"name":3}), 422, "one machine may be selected for a local job"), + (json!({"name":3,"after":4}), 400, "name: expected string"), + (json!({"workdir":["/"]}), 400, "workdir: expected string"), + (json!({"command":[]}), 400, "command: a non-empty list of strings"), + (json!({"command":["true"],"nodes":2}), 422, "cross-machine jobs are not supported by local Ray; use one node"), + (json!({"command":["true"],"gpus":0.5,"cpus":1}), 400, "fractional GPUs require a VRAM limit"), + ] { + let response = scale_call(c.clone(), "admin", "POST", &path, body.clone()).await; + assert_eq!(response.status().as_u16(), status, "{body}"); + assert_eq!(to_bytes(response.into_body(), 1 << 20).await.unwrap(), message, "{body}"); + } + let response = scale_call(c.clone(), "admin", "POST", &path, json!({"command":["true"],"gpus":0,"cpus":1,"machine":["gpu"]})).await; + assert_eq!(response.status(), StatusCode::CREATED); + let job: Value = serde_json::from_slice(&to_bytes(response.into_body(), 1 << 20).await.unwrap()).unwrap(); + let request = c.job(text(&job, "id"), None, &BTreeMap::new()).unwrap().request.0; + assert_eq!(request, json!({"command":["true"],"gpus":0.0,"cpus":1,"machine":["gpu"],"gpu_mem":0,"nodes":1})); +} + #[tokio::test] async fn scoped_managers_cannot_change_app_execution_or_batch_host_access() { for proof in [false, true] { @@ -1138,6 +1176,22 @@ async fn batch_metadata_belongs_to_scoped_owners_and_job_policy_is_inherited() { assert_eq!(response.status(), StatusCode::CREATED); let job: Value = serde_json::from_slice(&to_bytes(response.into_body(), 1 << 20).await.unwrap()).unwrap(); assert_eq!(job["priority"], 17); + // Only those who can see the batch submit to it; another tag is told what another namespace is. + let submit = json!({"gpus":0,"cpus":1,"mount":["/data/models:ro"],"command":["env"]}); + for token in [&foreign_tag, &foreign_ns] { + let status = scale_call(c.clone(), token, "POST", "/batches/training/jobs", submit.clone()).await.status(); + assert_eq!(status, StatusCode::NOT_FOUND); + } + assert_eq!(scale_call(c.clone(), &restricted, "POST", "/batches/training/jobs", submit.clone()).await.status(), StatusCode::FORBIDDEN); + assert_eq!(scale_call(c.clone(), "admin", "POST", &format!("/batches/training/jobs?namespace={ns}&tag=scope=research"), + submit.clone()).await.status(), StatusCode::NOT_FOUND); + // An admin's job on it is the batch's tag's, which its owner then sees and cancels. + let response = scale_call(c.clone(), "admin", "POST", &format!("/batches/training/jobs?namespace={ns}"), submit).await; + assert_eq!(response.status(), StatusCode::CREATED); + let admin_job: Value = serde_json::from_slice(&to_bytes(response.into_body(), 1 << 20).await.unwrap()).unwrap(); + assert_eq!(c.job(text(&admin_job, "id"), None, &BTreeMap::new()).unwrap().tag.as_deref(), Some("scope=default")); + assert_eq!(scale_call(c.clone(), &owner, "POST", &format!("/batches/training/jobs/{}/cancel", text(&admin_job, "id")), + Value::Null).await.status(), StatusCode::OK); assert_eq!(scale_call(c.clone(), &owner, "PUT", "/batches/training/priority", json!({"priority":31})).await.status(), StatusCode::OK); assert_eq!(c.job(text(&job, "id"), None, &BTreeMap::new()).unwrap().policy["priority"], 17); assert_eq!(c.inner.lock().apps["training"].spec["allow"], json!(["/data/models"])); @@ -1147,6 +1201,33 @@ async fn batch_metadata_belongs_to_scoped_owners_and_job_policy_is_inherited() { assert_eq!(scale_call(c, &owner, "DELETE", "/batches/training", Value::Null).await.status(), StatusCode::OK); } +/// A proof app is changed only by the tag it was created for, also before it is placed, when no +/// deployment records that tag yet. +#[tokio::test] +async fn an_unplaced_proof_app_is_changed_only_by_its_own_tag() { + let (_dir, c, ns) = setup().await; + let owner = user_token(&c, &ns, 1, "scope=default", None, true).await; + let foreign_tag = user_token(&c, &ns, 4, "scope=research", None, true).await; + let body = json!({"name":"eth","namespace":ns,"spec":{"image":"ziren-gpu:1","gpus":1}, + "proof":{"elf_id":"AB".repeat(32),"class":"latency"}}); + assert_eq!(scale_call(c.clone(), &owner, "POST", "/apps", body).await.status(), StatusCode::CREATED); + for (method, path, body) in [ + ("PUT", "/apps/eth/priority", json!({"priority":31})), + ("DELETE", "/apps/eth/disabled", Value::Null), + ("PUT", "/apps/eth/disabled", Value::Null), + ("DELETE", "/apps/eth", Value::Null), + ] { + let status = scale_call(c.clone(), &foreign_tag, method, path, body).await.status(); + assert_eq!(status, StatusCode::FORBIDDEN, "{method} {path}"); + } + assert!(c.fleet.app_known(&ns, "eth") && !c.inner.lock().apps["eth"].removed, "the app is still there"); + assert_eq!(c.fleet.app(&ns, "eth").unwrap().priority, 0); + assert_eq!(scale_call(c.clone(), &owner, "PUT", "/apps/eth/priority", json!({"priority":31})).await.status(), StatusCode::OK); + assert_eq!(c.fleet.app(&ns, "eth").unwrap().priority, 31); + assert_eq!(scale_call(c.clone(), &owner, "DELETE", "/apps/eth", Value::Null).await.status(), StatusCode::OK); + assert!(!c.fleet.app_known(&ns, "eth")); +} + #[tokio::test] async fn app_owners_manage_apps_and_scale_without_widening_grants_or_tags() { for proof in [false, true] { @@ -2323,10 +2404,14 @@ async fn nodes_run_commands(c: &WorkloadController, nodes: &mut BTreeMap = report.apps.iter().filter(|a| a["app"] == p["app"]["name"]).map(|a| a["name"].clone()).collect(); report.apps.retain(|a| a["app"] != p["app"]["name"]); - json!({"removed":[p["app"]["name"]]}) + json!({"removed":removed}) } else { for a in report.apps.iter_mut().filter(|a| a["name"] == p["entry"]["name"]) { for (k, v) in p["set"].as_object().unwrap() { @@ -2412,6 +2499,88 @@ async fn an_app_start_is_placed_on_a_node_with_room() { assert!(!i.deployments.contains_key("gpu-model") && !i.deployments["big-model"].disabled); } +/// App `model` runs on node `gpu`, its entry anchored to `machine` when given; a job then has +/// the node's other card, and node `big` has room. +async fn running_on_a_full_node(machine: Option<&str>) -> (tempfile::TempDir, Arc, BTreeMap) { + let (dir, c, ns) = setup().await; + c.report("gpu", "session", inventory(&ns)).await.unwrap(); + let body = json!({"name":"model","namespace":ns,"spec":{"image":"serve:1","gpus":1,"cpus":2,"port_base":50100}}); + assert_eq!(scale_call(c.clone(), "admin", "POST", "/apps", body).await.status(), StatusCode::CREATED); + assert_eq!(deployed(&c, &ns, "model").await.unwrap().0, "gpu"); + let mut nodes = BTreeMap::from([("gpu".to_string(), c.inner.lock().workers["gpu"].report.clone())]); + if let Some(machine) = machine { + nodes.get_mut("gpu").unwrap().apps[0]["machine"] = json!(machine); + } + c.report("gpu", "session", nodes["gpu"].clone()).await.unwrap(); + c.begin_toggle(&["gpu-model".into()], false).unwrap(); + nodes_run_commands(&c, &mut nodes, &[]).await; + assert_eq!(nodes["gpu"].apps[0]["ready"], json!(true), "{:?}", nodes["gpu"].apps); + nodes.get_mut("gpu").unwrap().free_gpus = 0.0; + let mut big = inventory(&ns); + big.machine = "big".into(); + nodes.insert("big".into(), big); + (dir, c, nodes) +} + +/// The nodes ever sent a `create` of an app. +fn created_on(c: &WorkloadController) -> Vec { + c.inner.lock().commands.values().filter(|v| !v.parameters["create"].is_null()).map(|v| v.worker.clone()).collect() +} + +/// An app's replica stops and is cleaned up while a job fills its node, so its start moves to a +/// node with room. The node it leaves frees its room once it removed the app, and the start +/// runs where it was placed: it is not sent back to the node it left, where it no longer is. +#[tokio::test] +async fn an_app_moved_to_a_node_with_room_frees_the_old_one_and_runs_there() { + let (_dir, c, mut nodes) = running_on_a_full_node(None).await; + let ns = c.inner.lock().apps["model"].namespace.clone(); + let held = |c: &WorkloadController| { + let mut ids: Vec = c.fleet.allocations(&ns).iter().map(|a| a.id.clone()).collect(); + ids.sort(); + ids + }; + assert_eq!(held(&c), ["deployment:gpu-model"]); + let gpu = nodes.get_mut("gpu").unwrap(); + (gpu.apps[0]["ready"], gpu.apps[0]["cleaned"]) = (json!(false), json!(true)); + for (id, report) in &nodes { + c.report(id, "session", report.clone()).await.unwrap(); + } + nodes_run_commands(&c, &mut nodes, &[]).await; + assert!(nodes["gpu"].apps.is_empty(), "the node it left removed it"); + let running = nodes["big"].apps.iter().any(|a| a["name"] == "model" && a["ready"] == json!(true) && a["disabled"] == json!(false)); + assert!(running, "it runs on the node that had room: {:?}", nodes["big"].apps); + assert_eq!(c.inner.lock().apps["model"].workers, ["big"]); + assert_eq!(held(&c), ["deployment:big-model"], "the node it left freed its room"); + // That node's record goes with its next report. + c.report("gpu", "session", nodes["gpu"].clone()).await.unwrap(); + assert!(!c.inner.lock().deployments.contains_key("gpu-model")); +} + +/// An entry its node anchors to the machine (`machine`) is never moved. Once its replica is +/// gone and its node has no room, its start waits there, and runs once room frees. +#[tokio::test] +async fn an_app_anchored_to_its_machine_restarts_only_there() { + let (_dir, c, mut nodes) = running_on_a_full_node(Some("machine")).await; + let gpu = nodes.get_mut("gpu").unwrap(); + (gpu.apps[0]["ready"], gpu.apps[0]["cleaned"]) = (json!(false), json!(true)); + for (id, report) in &nodes { + c.report(id, "session", report.clone()).await.unwrap(); + } + nodes_run_commands(&c, &mut nodes, &[]).await; + assert_eq!(created_on(&c), ["gpu"], "no other node is sent it"); + let starts: Vec<(String, bool)> = c.inner.lock().commands.values() + .filter(|v| !v.done && v.operation == NodeOperation::SetApp as i32).map(|v| (v.worker.clone(), v.admitted)).collect(); + assert_eq!(starts, [("gpu".to_string(), false)], "its start waits for room on its node"); + + nodes.get_mut("gpu").unwrap().free_gpus = 1.0; + c.report("gpu", "session", nodes["gpu"].clone()).await.unwrap(); + nodes_run_commands(&c, &mut nodes, &[]).await; + assert_eq!(nodes["gpu"].apps[0]["ready"], json!(true), "{:?}", nodes["gpu"].apps); + assert_eq!(nodes["gpu"].apps[0]["machine"], json!("machine")); + assert!(nodes["big"].apps.is_empty()); + assert_eq!(c.inner.lock().apps["model"].workers, ["gpu"]); +} + #[tokio::test] async fn an_app_a_node_refuses_to_create_is_placed_again_on_the_next_enable() { let (_dir, c, ns) = setup().await; diff --git a/crates/fleet-scheduler/src/workloads_views.rs b/crates/fleet-scheduler/src/workloads_views.rs index 58563b1..921ec77 100644 --- a/crates/fleet-scheduler/src/workloads_views.rs +++ b/crates/fleet-scheduler/src/workloads_views.rs @@ -30,10 +30,10 @@ impl WorkloadController { .deployments .values() .filter(|intent| { - let entry = &intent.spec; - text(entry, "kind").starts_with("ziren-") + let entry = intent.entry(); + entry.is_prover() && replica - .strip_prefix(&format!("{}-", text(entry, "name"))) + .strip_prefix(&format!("{}-", entry.name())) .is_some_and(|index| index.parse::().is_ok()) }) .map(|intent| intent.worker.clone()) @@ -64,11 +64,7 @@ impl WorkloadController { let deployments: Vec<_> = i .deployments .values() - .filter(|intent| { - let s = &intent.spec; - text(s, "namespace") == namespace - && text(s, "app") == app - }) + .filter(|intent| intent.entry().namespace() == namespace && intent.entry().app() == app) .collect(); let mut gpus = 0.0; let mut cpus = 0.0; @@ -83,29 +79,19 @@ impl WorkloadController { .scale .as_ref() .filter(|s| s.pending()) - .map(|s| s.target()) - .unwrap_or(&intent.spec); - let count = s["replicas"].as_u64().unwrap_or(1); + .map(|s| s.target_entry()) + .unwrap_or(intent.entry()); + let count = s.replicas(); replicas += count; - gpus += s["gpus"] - .as_f64() - .unwrap_or(if text(s, "kind") == "ziren-cpu" { - 0.0 - } else { - 1.0 - }) - * count as f64; - cpus += s["cpus"].as_f64().unwrap_or(1.0) * count as f64; + gpus += s.gpus_or_default() * count as f64; + cpus += s.cpus() * count as f64; nodes.insert(intent.worker.clone()); - tags.insert(s["tag"].as_str().unwrap_or("").to_owned()); - if let Some(image) = s["image"].as_str().filter(|i| !i.is_empty()) { - images.insert(image.to_owned()); + tags.insert(s.tag().unwrap_or("").to_owned()); + if !s.image().is_empty() { + images.insert(s.image().to_owned()); } let worker = i.workers.get(&intent.worker).filter(|w| now() - w.seen < 30.0); - let observed = worker.and_then(|w| { - w.report.apps.iter() - .find(|s| text(s, "name") == text(&intent.spec, "name")) - }); + let observed = worker.and_then(|w| w.report.entry(intent.entry().name())); let state = if intent.scale.as_ref().is_some_and(|s| s.failed()) { "failed" } else if worker.is_none() { @@ -116,10 +102,8 @@ impl WorkloadController { } scale.phase().name() } else if intent.disabled { - if observed.is_some_and(|s| s["cleaned"].as_bool() == Some(true)) { "disabled" } else { "draining" } - } else if worker.is_some_and(|w| w.report.ready) && observed.is_some_and(|s| { - s["ready"].as_bool() == Some(true) && s["disabled"].as_bool() != Some(true) - }) { + if observed.is_some_and(Entry::cleaned) { "disabled" } else { "draining" } + } else if worker.is_some_and(|w| w.report.ready) && observed.is_some_and(|s| s.ready() && !s.disabled()) { "running" } else { reasons.push(Self::starting_reason(&i.commands, intent, worker)); @@ -142,18 +126,10 @@ impl WorkloadController { /// Why an enabled app is not running yet, for `fleetctl app`'s DETAIL column. fn starting_reason(commands: &super::records::Records, intent: &AppDeployment, worker: Option<&NodeSession>) -> String { - let name = text(&intent.spec, "name"); - let pending = commands.open().find(|c| { - !c.done - && c.worker == intent.worker - && c.operation == NodeOperation::SetApp as i32 - && text(&c.parameters["app"], "name") == name - }); + let pending = commands.open().find(|c| c.changes_entry(&intent.worker, intent.entry().name())); match pending { Some(c) if !c.admitted => { - let s = &c.parameters["app"]; - let n = s["replicas"].as_f64().unwrap_or(1.0); - let (g, cpu) = (s["gpus"].as_f64().unwrap_or(0.0) * n, s["cpus"].as_f64().unwrap_or(1.0) * n); + let (g, cpu) = c.entry().total(0.0); match worker { Some(w) => format!("waiting for capacity: needs {g} GPU and {cpu} CPU, {} has {} GPU and {} CPU free", intent.worker, w.report.free_gpus, w.report.free_cpus), @@ -216,7 +192,7 @@ impl WorkloadController { "active": count(&|r| !terminal(r.state())), "succeeded": count(&|r| r.state() == "succeeded"), "failed": count(&|r| matches!(r.state(), "failed" | "lost")), - "last_at": jobs.iter().filter_map(|r| r.status["submitted_at"].as_f64()).max_by(f64::total_cmp), + "last_at": jobs.iter().filter_map(|r| r.submitted_at()).max_by(f64::total_cmp), }) } @@ -232,9 +208,9 @@ impl WorkloadController { } o.insert("workers".into(), json!(app.workers)); } - let disabled = app.spec["disabled"].as_bool() == Some(true); - let fixed_tag = app.tag.as_deref().or_else(|| app.spec["tag"].as_str()); - let mut v = self.app_view(&app.namespace, &app.name, fixed_tag, app.spec["priority"].clone(), disabled, + let disabled = app.disabled(); + let fixed_tag = app.fixed_tag(); + let mut v = self.app_view(&app.namespace, &app.name, fixed_tag, app.priority().clone(), disabled, jobs, tag, vec![("batch", settings)]); // A batch runs no app of its own: its state is whether it takes jobs, and its tag // is the one it was registered with. @@ -259,8 +235,8 @@ impl WorkloadController { /// An app that proves no program, in the app shape. pub(crate) fn app_value(&self, app: &WorkloadTemplate, tag: Option<&str>) -> Value { let jobs = self.template_jobs(app, tag); - self.app_view(&app.namespace, &app.name, app.tag.as_deref().or_else(|| app.spec["tag"].as_str()), - app.spec["priority"].clone(), app.spec["disabled"].as_bool() == Some(true), jobs, tag, + self.app_view(&app.namespace, &app.name, app.fixed_tag(), + app.priority().clone(), app.disabled(), jobs, tag, vec![("spec", Self::spec_value(app))]) } diff --git a/crates/fleetctl/src/main.rs b/crates/fleetctl/src/main.rs index 2388413..c115279 100644 --- a/crates/fleetctl/src/main.rs +++ b/crates/fleetctl/src/main.rs @@ -57,7 +57,7 @@ enum Cmd { App(AppArgs), /// Finite batch workloads and their jobs. Batch(BatchArgs), - /// Execution records: list or inspect app and batch jobs; fetch proof outputs. + /// Execution records: proofs submitted to proof apps, and batch jobs; fetch proof outputs. Job(JobArgs), /// Machine nodes: list their capacity, or drain / undrain a machine. Node(NodeArgs), @@ -162,8 +162,10 @@ struct ListArgs { #[derive(Clone, Copy, PartialEq, Eq, clap::ValueEnum)] enum JobType { - /// Jobs of apps: the proofs an app runs. + /// Proofs submitted to a proof app through the scheduler. An app's own work in its replicas + /// is not a job: `fleetctl app` and `fleetctl app logs` show it. App, + /// Commands submitted to a batch. Batch, } diff --git a/docs/maintenance.md b/docs/maintenance.md index e44b912..e297a02 100644 --- a/docs/maintenance.md +++ b/docs/maintenance.md @@ -1,17 +1,16 @@ # Maintenance -Routine work in one place: upgrading, changing a node's capacity, liveness and -troubleshooting. One-time moves between specific older and newer releases are in -[one-time migrations](#one-time-migrations) at the end of this file; ordinary +Upgrading, changing a node's capacity, liveness and troubleshooting. Moves between +specific older releases are in [one-time migrations](#one-time-migrations); routine upgrades need none of them. ## Upgrading -`deploy/fleet-ctl.sh` (in the release package, or in a checkout of that revision) -upgrades the control host and systemd-supervised nodes. Cron-supervised nodes use -the manual procedure below. Every scripted upgrade copies what it replaces to `backup/` first, then waits -for the scheduler's health check or the node's reconnection; the matching `rollback` -restores that backup. +`deploy/fleet-ctl.sh`, from the release package or a checkout of that revision, +upgrades the control host and systemd-supervised nodes; cron-supervised nodes use the +[manual procedure](#cron-supervised-nodes). Each scripted upgrade first copies what it +replaces to `backup/`, then waits for the scheduler's health check or the node's +reconnection, and the matching `rollback` restores that copy. ```bash export FLEET_PREFIX=/path/to/control/install FLEET_NODE_SSH=gpu-1 # or ~/.config/zkm-fleet/ctl.env @@ -21,27 +20,26 @@ deploy/fleet-ctl.sh -n scheduler upgrade v0.1.6 # -n prints every step and chan ### Control host -The scheduler and MCP run from the release's GHCR image -(`ghcr.io/projectzkm/zkm-fleet-scheduler:VERSION`) in Docker Compose; only `fleetctl` -is a host binary. +The scheduler and MCP run in Docker Compose from the release's GHCR image +(`ghcr.io/projectzkm/zkm-fleet-scheduler:VERSION`); only `fleetctl` is a host binary. ```bash deploy/fleet-ctl.sh scheduler upgrade v0.1.6 ``` -It downloads the release package (SHA-256 verified), pulls the image first (a missing +It downloads the release package and checks its SHA-256, pulls the image (a missing image changes nothing), backs up `fleetctl`, the Compose files and `compose.env`, -records the new image, recreates `scheduler` — and `mcp`, when enabled — and waits -for `/healthz`; the recreation takes ~90s while the scheduler finishes in-flight -calls. `scheduler rollback` restores the backup. Site changes to the release's -Compose file, such as another gateway or an existing Postgres volume, stay in +records the new image, recreates `scheduler` (and `mcp` when enabled) and waits for +`/healthz`. Recreating takes about 90 seconds while the scheduler finishes in-flight +calls. `scheduler rollback` restores the backup. Keep site changes to the Compose +file, such as another gateway or an existing Postgres volume, in `deploy/docker-compose.override.yml`: an upgrade replaces only `docker-compose.yml`, and every Compose call applies the override after it. -Verify the gateway's `/healthz`, `fleetctl node` and `fleetctl app`; when MCP is -enabled, check its initialization and one authenticated tool call. For a -permissions-only update, exercise the changed grant with a temporary workload and a -token without the grant, then remove the workload and revoke the temporary tokens. +Then check the gateway's `/healthz`, `fleetctl node` and `fleetctl app`, and with MCP +enabled, its initialization and one authenticated tool call. After a permissions +change, try the changed grant with a temporary workload and a token without the +grant, then remove both. ### Execution node @@ -52,27 +50,25 @@ deploy/fleet-ctl.sh node upgrade target/release/fleet-node bridge \ target/release/fleet-proof-app deploy ``` -It installs the release's `fleet-node`, bridge, `fleet-proof-app` and the three -launch scripts (`node-start.sh`, `ray-start.sh`, `fleet-watchdog.sh`) around a short -stop/start of the agent's unit, checking that the new bridge imports in the node's -own venv before the agent is touched, and waits for the node to reconnect. -`node rollback` restores everything the last upgrade replaced, launch scripts -included. The last two local-file arguments are optional: the proof app (its -endpoint units, named in `FLEET_PROOF_APP_UNITS`, restart too) and the launch +It installs the release's `fleet-node`, bridge, `fleet-proof-app` and launch scripts +(`node-start.sh`, `ray-start.sh`, `fleet-watchdog.sh`) around a short stop and start +of the agent's unit, after checking that the new bridge imports in the node's venv, +and waits for the node to reconnect. `node rollback` restores everything the last +upgrade replaced. In the local form, the last two arguments are optional: the proof +app, whose endpoint units in `FLEET_PROOF_APP_UNITS` restart too, and the launch scripts' directory. -A program update does not require restarting Ray or changing its pool; running apps -and batches keep going through the agent restart. The upgrade touches neither -`etc/` (watchdog.env, apps.yaml, batches.yaml, tokens) nor `work/`; keep independent -app endpoints running when their programs are unchanged. +Ray keeps running, and so do apps and batches, through the agent restart. The upgrade +touches neither `etc/` (watchdog.env, apps.yaml, batches.yaml, tokens) nor `work/`; +leave app endpoints running when their programs are unchanged. #### Cron-supervised nodes -`fleet-ctl.sh node upgrade` and `node rollback` require a systemd unit. For a -cron-supervised node, first comment out its watchdog line with `crontab -e`, send -SIGTERM to that installation's `fleet-node` process, and wait for it to exit. -Keep Ray running. With `PACKAGE` set to an unpacked, checksum-verified release -directory (or a local package from the tested revision), replace the programs: +`node upgrade` and `node rollback` need a systemd unit. On a cron-supervised node, +comment out the watchdog line with `crontab -e`, send SIGTERM to that installation's +`fleet-node` and wait for it to exit, leaving Ray running. With `PACKAGE` set to an +unpacked, checksum-verified release directory (or a local package from the tested +revision), replace the programs: ```bash FLEET_PREFIX="$HOME/app/fleet" @@ -90,73 +86,66 @@ done FLEET_ENV="$FLEET_PREFIX/etc/watchdog.env" "$FLEET_PREFIX/deploy/fleet-watchdog.sh" ``` -Restore the same cron line, then verify a fresh heartbeat and workload progress. -If verification fails, pause cron and stop the agent again, restore `fleet-node`, -`bridge/` and the launch scripts from `BACKUP`, and restart the watchdog before -restoring cron. Preserve `etc/` and `work/`. If also updating `fleet-proof-app`, -back up and replace that binary while its endpoint supervisor is stopped, then -restart that supervisor; keep its previous binary for rollback. +Restore the cron line, then check for a fresh heartbeat and workload progress. To roll +back, pause cron and stop the agent again, restore `fleet-node`, `bridge/` and the +launch scripts from `BACKUP`, and restart the watchdog before restoring cron. To +update `fleet-proof-app` too, back up and replace it while its endpoint's supervisor +is stopped, then restart that supervisor. ### Control host and nodes together -Upgrade the control host and the nodes from the same release when the release -changes the shared model or execution protocol: the scheduler, agent and Python -bridge must agree on the machine report (currently version 3). A change limited to -scheduler permissions, the CLI or MCP needs only a control-host update. Read the -release's changes before choosing; the one-time migrations below record the past -cross-version moves. +Upgrade both from the same release when it changes the shared model or execution +protocol: the scheduler, agent and Python bridge must agree on the machine report +(version 3). A change to scheduler permissions, the CLI or MCP needs only the control +host. The release notes say which applies. ## Self-built node programs -CI builds on Ubuntu 24.04 (glibc 2.39). Nodes on older glibc (22.04 is 2.35) run -such binaries with a loader warning, and production node programs are usually cut in -a matching build image instead. Build from the release's revision, naming the -release so each node's agent reports it in the VERSION column of `fleetctl node` -(a build that names nothing shows `-`): +CI builds on Ubuntu 24.04 (glibc 2.39). Older nodes (22.04 has 2.35) run those +binaries with a loader warning, so production node programs are usually built in a +matching image instead. Build from the release's revision and name the release, so +the agent reports it in `fleetctl node`'s VERSION column (`-` otherwise): ```bash FLEET_VERSION=0.1.6 cargo build --locked --release \ -p fleet-scheduler -p fleetctl -p fleet-mcp -p fleet-node -p fleet-proof-app ``` -For systemd nodes, install with the local-file form of `node upgrade` above; -for cron nodes, follow the manual procedure with the tested local package. +Install it with the local form of `node upgrade` on systemd nodes, and the manual +procedure on cron nodes. ## Changing a node's capacity -Only this procedure restarts Ray. Use administrator credentials for node commands on -the control host and the app owner's token for workload actions. +This is the only procedure that restarts Ray. Node commands need administrator +credentials, workload commands the app owner's token. -**Pause execution.** Drain the node, then disable each enabled app on it: +**Pause.** Drain the node, then disable each enabled app on it: ```bash fleetctl node drain NODE fleetctl --token-file owner.token --namespace NAMESPACE app disable APP ``` -Wait for active proof/batch jobs to finish and for app cleanup to complete before -restarting Ray. Integrated workers own their own input streams, so draining the node -alone does not stop them. Disabling EthProofs preserves completed outputs and -retries any unfinished block after resume; its -[app guide](../apps/ethproofs/README.md#operate-and-recover) describes finishing a -current proof first. Record the enabled apps, their allocations and each app's saved -progress. Back up `etc/` and app-owned durable directories under `work/apps/`. Do -not change app checkpoints or remove pending reports. +Wait for active proofs and batch jobs to finish and for app cleanup to complete. +Draining alone doesn't stop integrated workers, which run their own input streams. +Disabling EthProofs keeps its finished outputs and retries any unfinished block on +resume; its [guide](../apps/ethproofs/README.md#operate-and-recover) describes +finishing the current proof first. Note the enabled apps, their allocations and each +app's saved progress, and back up `etc/` and the apps' durable directories under +`work/apps/`. Don't change app checkpoints or remove pending reports. -**Change the capacity.** On the node, edit `~/app/fleet/etc/watchdog.env`: +**Change.** On the node, edit `~/app/fleet/etc/watchdog.env`: ```bash RAY_NUM_CPUS=128 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ``` -Choose capacity within the machine's hardware; an empty `RAY_NUM_CPUS` enables -automatic CPU detection. These values are total node capacity, shared by every -allocation. Stop the agent with its existing supervisor (`sudo systemctl stop -zkm-fleet-node.service`, or, for cron installations, comment out the watchdog line -and stop `fleet-node` with SIGTERM, waiting for it to exit — one supervisor per -agent). Then restart local Ray while holding the watchdog lock, so the minute -watchdog cannot race the restart: +These are the node's total capacity, shared by every allocation, within its hardware; +an empty `RAY_NUM_CPUS` detects CPUs automatically. Stop the agent with its +supervisor, of which each agent has exactly one: `sudo systemctl stop zkm-fleet-node.service`, or on cron, comment out the +watchdog line and send `fleet-node` SIGTERM, waiting for it to exit. Then restart Ray +while holding the watchdog lock, so the minute watchdog can't race the restart: ```bash bash <<'SH' @@ -172,12 +161,11 @@ ray stop SH ``` -**Restart the agent.** For systemd, run -`sudo systemctl start zkm-fleet-node.service`. For cron, restore the watchdog line -and run +**Restart the agent.** On systemd, `sudo systemctl start zkm-fleet-node.service`. On +cron, restore the watchdog line and run `FLEET_ENV="$HOME/app/fleet/etc/watchdog.env" ~/app/fleet/deploy/fleet-watchdog.sh`. -**Verify and resume.** On the control host: +**Resume.** On the control host: ```bash fleetctl node NODE @@ -186,152 +174,180 @@ fleetctl --token-file owner.token --namespace NAMESPACE app scale APP --gpus GPU fleetctl --token-file owner.token --namespace NAMESPACE app enable APP ``` -Wait for the node to be ready with the expected total capacity before scaling, and -for scaling to complete before enabling a disabled app. Restore only the apps that -were enabled before maintenance. Confirm the observed allocations, application -progress and output/report delivery in the normal logs. EthProofs must resume at its -saved next block and continue without gaps; keep the prior state backup until that -check passes. On failure, disable the affected app, inspect its error and restore -the prior configuration; keep the current durable cursor and reports. +Wait for the node to be ready with the expected capacity before scaling, and for +scaling to complete before enabling. Re-enable only the apps that were enabled +before. Check the observed allocations, the apps' progress and their output and +report delivery in the logs. EthProofs must resume at its saved next block with no +gaps; keep the state backup until it does. If something fails, disable the app, +read its error and restore the previous configuration, keeping its current cursor +and reports. ## Liveness -Nothing here monitors itself; watch these on a schedule (a five-minute timer or -cron), alerting on state change rather than repeating: +Nothing here monitors itself. Check these on a schedule, such as a five-minute timer, +and alert when they change: -- Control host: the gateway's `/healthz`; the scheduler, MCP, Postgres and gateway - containers running; the scheduler log without repeated - `job dispatch retained durable intents` lines (a stalled scale conflict — see - below); the root disk under 90% (upgrades stage ~400 MB in /tmp). -- Each node: in `fleetctl node`, READY with a fresh heartbeat and the expected - VERSION; its disk under 90%; `work/logs/watchdog.log` quiet. -- Workloads: `fleetctl app` — nothing sitting `failed`, or `starting` beyond its - ready timeout, and `fleetctl job` — nothing stuck `waiting_quota` for hours. +- **Control host:** the gateway's `/healthz`; the scheduler, MCP, Postgres and gateway + containers running; no repeated `job dispatch retained durable intents` in the + scheduler log (a [blocked scale](#an-app-is-stuck-starting-failed-or-a-job-waits-on-quota)); + the root disk under 90%, since upgrades stage about 400 MB in /tmp. +- **Each node:** READY in `fleetctl node`, with a fresh heartbeat and the expected + VERSION; its disk under 90%; a quiet `work/logs/watchdog.log`. +- **Workloads:** no app in `fleetctl app` sitting `failed` or `starting` beyond its + ready timeout, and no job in `fleetctl job` stuck `waiting_quota` for hours. ## Troubleshooting ### A node is offline or not READY -Check its supervisor (the unit, or the cron watchdog), `work/logs/ray.log` and -`watchdog.log`, the gateway URL, cluster token and CA trust, then the node-local +Check its supervisor (the unit or the cron watchdog), `work/logs/ray.log` and +`watchdog.log`, the gateway URL, the cluster token and CA trust. Then the node's own traps: -- The Ray temp link `/tmp/zkf-ray-` must resolve. A dead link (a reboot, or a - stray `ray-start.sh` run against a cleaned-up temp workspace) aborts every agent - restart with an opaque log-path error; the watchdog repairs it each minute from - v0.1.7, and by hand with - `ln -sfn ~/app/fleet/work/ray /tmp/zkf-ray-$(id -u)`. -- A full disk: bridge writes, app starts and even the agent's log fail on it. +- **The Ray temp link** `/tmp/zkf-ray-` must resolve. A dead link, after a reboot + or a stray `ray-start.sh` run against a cleaned-up temp workspace, aborts every + agent restart with an opaque log-path error. The watchdog repairs it every minute + from v0.1.7; by hand, `ln -sfn ~/app/fleet/work/ray /tmp/zkf-ray-$(id -u)`. +- **A full disk** fails bridge writes, app starts and even the agent's log. ### An app is stuck starting, failed, or a job waits on quota -`fleetctl app` DETAIL says why: `waiting for capacity` (the start is held until it -fits; `app disable` or a smaller `app scale` replaces the held start), `starting on -NODE`, or `waiting for NODE to report it ready`. Read the app log (`fleetctl app -logs APP`) and the node's `work/logs/-0.node.log`. If the scheduler log -repeats `job dispatch retained durable intents`, a scale is blocked — typically by -a stale deployment of the same app on another node (its node no longer configures -it). Since v0.1.7 the blocked scale defers while the rest of the fleet keeps -working, and stale starts are retired; on older releases, stop the scheduler, back -up and remove the stale `data/workloads/deployments/-.json` (and any -undone command for it under `data/workloads/commands/`), then start it again. +`fleetctl app` DETAIL says why ([operations](operations.md#logs-and-recovery)). Read +the app's log with `fleetctl app logs APP`, and the node's +`work/logs/-0.node.log`. + +If the scheduler log repeats `job dispatch retained durable intents`, a scale is +blocked, usually by a stale deployment of the same app on a node that no longer +configures it. From v0.1.7 the blocked scale waits while the rest of the fleet keeps +working, and stale starts are retired. On older releases, stop the scheduler, back up +and remove the stale `data/workloads/deployments/-.json` and any undone +command for it under `data/workloads/commands/`, then start it again. + +A start whose node has no room moves to a node that has, unless its entry anchors it +with `machine`; an anchored start waits on its node, preempting lower-priority batch +jobs of its namespace when it can. Every integrated worker (an app whose node runs +its prover as a Fleet worker, such as goat's proof builder) is anchored, since the +bridge requires `machine` for one. Up to v0.1.9: + +- an anchored start moved too, and the move rewrote the entry's `machine` to the new + node's. Pin such an app again by restoring its node's entry; the bridge keeps the + old `apps.yaml` as `apps.yaml.removed-`. +- the scheduler never freed a moved app's room on the node it left, so its quota + stayed held, and a start could be sent back to that node, leaving the app on no + node until `fleetctl app enable` placed it again. ### Every node's heartbeat times out while jobs queue The nodes log `node heartbeat rejected; local execution continues ... Timeout -expired` and the gateway logs heartbeat 499s, yet they look idle while `fleetctl -job` lists jobs `queued` (assigned; awaiting node acknowledgement). The scheduler sits -at a full core with sqlx `slow_acquire` warnings although Postgres is idle. Up to -v0.1.7 each dispatch pass scanned every job and command record ever kept, once per -waiting job, while holding the lock node reports wait for. With tens of thousands of -records and about a hundred jobs in flight a pass outlasts the nodes' 10 s heartbeat -timeout, and no launch reaches them. Later releases scan only what is in flight and -let reports in between jobs. On v0.1.7 and older, keep fewer jobs in flight: below -about 50 the passes stay short. - -A bulk purge (`fleetctl job purge --older-than DAYS`) of a busy batch's history has the -same symptom on v0.1.8 and older: it holds that lock while it writes a record and a -node command for every job, minutes for tens of thousands. Its node commands then -crowd out launches for as long again, at 16 per heartbeat. Later releases purge in -chunks and send a node's purges after its other commands. - -The scheduler logs one `api request` line for each API call that can change state: -method, path with query, status, duration, and the caller's token id, label, -namespace and tag (never the token). `docker logs zkm-fleet-scheduler-1 | grep 'api -request'` shows who submitted, cancelled, purged or reconfigured what. +expired` and the gateway logs heartbeat 499s. The nodes look idle while `fleetctl job` +lists jobs `queued` (assigned, awaiting node acknowledgement), and the scheduler sits +at a full core with sqlx `slow_acquire` warnings although Postgres is idle. + +- **Up to v0.1.7**, each dispatch pass scanned every job and command record ever kept, + once per waiting job, while holding the lock node reports wait for. With tens of + thousands of records and about a hundred jobs in flight, a pass outlasts the nodes' + 10-second heartbeat timeout and no launch gets through. Keep fewer than about 50 + jobs in flight. Later releases scan only what's in flight and let reports in between + jobs. +- **Up to v0.1.8**, a bulk purge of a busy batch's history (`fleetctl job purge + --older-than DAYS`) does the same: it holds that lock while it writes a record and a + node command for every job, minutes for tens of thousands, and its node commands + then crowd out launches for as long again, at 16 per heartbeat. Later releases purge + in chunks and send a node's purges after its other commands. + +### Who changed what + +The scheduler logs an `api request` line for each API call that can change state: +method, path and query, status, duration, and the caller's token ID, label, namespace +and tag (never the token itself). +`docker logs zkm-fleet-scheduler-1 | grep 'api request'` shows who submitted, +cancelled, purged or reconfigured what. ## Installed files and backups -Keep source/build files separate from the installed runtime. A node installation -uses these directories: +Keep source and build files apart from the installed runtime. A node installation +has: | Directory | Contents | |---|---| -| `bin/`, `bridge/`, `venv/`, `deploy/` | Installed Fleet programs and launch scripts | +| `bin/`, `bridge/`, `venv/`, `deploy/` | Fleet's programs and launch scripts | | `apps/` | Packaged application binaries and guest ELFs | | `etc/` | Machine configuration and private credentials | | `work/jobs/` | Local batch job records and output workspaces (`FLEET_JOBS_DIR`) | | `work/apps/` | App-owned checkpoints, inputs, outputs and reports | | `work/logs/`, `work/ray/`, `work/spill/` | Process logs, Ray runtime and spilled objects | | `work/.maintenance/` | Deployment and resource provisioning receipts | -| `backup/` | Temporary rollback package; remove after successful verification | +| `backup/` | The last upgrade's rollback copy; remove it once the upgrade checks out | + +Back up the control host's Postgres and `data_dir` together, since batch records and +app intents are under `data_dir/workloads/`, and keep tokens private. Across any +update, keep `etc/`, Postgres, the control `data_dir` and each node's `work/`, and back +them up first: rollbacks restore programs, not data. Delete old build packages and +logs only after checking running processes and rollback needs; proof retention is the +app's own policy. + +Pin validated GPU images by digest. Update EthProofs to its committed submodule +revision with `git submodule update --init --recursive`, never a moving branch. +Package the GOAT worker from its committed `third_party/bitvm-node` revision with +`apps/goat/build.sh`; a guest change is a new app identity (see the +[GOAT guide](../apps/goat/README.md)). Storage paths and ports are in the +[reference](reference.md#data). -Back up the control host's Postgres and its `data_dir` together: batch records and -app intents live under `data_dir/workloads/`. Keep tokens private. Delete obsolete -build packages and old logs only after checking running processes and rollback -needs; keep proof retention an explicit app policy. +## One-time migrations -Pin validated GPU images by digest. Update EthProofs using its committed submodule -revision with `git submodule update --init --recursive`; never deploy a moving -submodule branch. Package the GOAT worker from its committed `third_party/bitvm-node` -revision with `apps/goat/build.sh`; a guest change is a new app identity (see the -[GOAT guide](../apps/goat/README.md)). See [data and ports](reference.md#data) for -storage and network details. +Each applies only to the versions it names. -## One-time migrations +### Owner-controlled runtime settings -Each section names the versions it applies to; a routine upgrade needs none of them. +Releases that moved app execution settings, batch mount roots and runtime users to +administrators don't rewrite existing configuration or cancel jobs an earlier +scheduler accepted. Review existing apps' settings, batches' mount roots and runtime +users, and queued jobs before giving untrusted owners access again. ### Machine report version 2 to 3 -Upgrading from report version 2 changes the control host and every node at once. -Let in-flight commands finish, stop the scheduler and every node, and migrate their -stored state before starting the new binaries. The scheduler's per-node records move -from `workloads/services` to `workloads/deployments` (`service` becomes `spec`), -stored commands and `service:` reservations take the new names, and each registered -proof app needs its app record. A node's `etc/services.yaml` becomes -`etc/apps.yaml` (`services:` becomes `apps:`) and its `node-start.sh` passes -`--apps`. Back up `data_dir`, Postgres and the node's `etc/` first. +This changes the control host and every node at once. Let in-flight commands finish, +stop the scheduler and every node, back up `data_dir`, Postgres and each node's +`etc/`, and migrate their stored state before starting the new binaries: + +- The scheduler's per-node records move from `workloads/services` to + `workloads/deployments` (`service` becomes `spec`); stored commands and `service:` + reservations take the new names; each registered proof app needs its app record. +- A node's `etc/services.yaml` becomes `etc/apps.yaml` (`services:` becomes `apps:`), + and its `node-start.sh` passes `--apps`. ### The scheduler's SDK stage API (retired) The release that moved to the app model retired the scheduler's SDK stage API (`GenerateProof`/`GetStatus` on `stage_addr`): SDK clients prove through their -[app-owned proof endpoint](reference.md#app-owned-proof-endpoint). Remove -`stage_addr`, `tls`, `public_url`, `long_poll_ms` and `proof_output_dir` from -`fleet.yaml` first; the scheduler refuses to start with them. Let proof jobs -submitted through the stage API finish: one still queued or running fails when the -new scheduler starts. Remove the gateway's `/stage.v1.StageService/` route, which -has nothing behind it. The control host's `data_dir/elf`, `data_dir/jobs` and -`data_dir/proofs` held that API's payloads; delete them once rollback is no longer -needed. A node ignores `proof_output_dir` in `apps.yaml`. +[app-owned proof endpoint](reference.md#app-owned-proof-endpoint). + +1. Remove `stage_addr`, `tls`, `public_url`, `long_poll_ms` and `proof_output_dir` + from `fleet.yaml`; the scheduler refuses to start with them. A node ignores + `proof_output_dir` in `apps.yaml`. +2. Let proofs submitted through the stage API finish: any still queued or running + fails when the new scheduler starts. +3. Remove the gateway's `/stage.v1.StageService/` route, which has nothing behind it. +4. Once rollback is no longer needed, delete the control host's `data_dir/elf`, + `data_dir/jobs` and `data_dir/proofs`, which held that API's payloads. ### Database schema 14 or 15 to 16 -Schema 16 installations retain namespaces and tokens across ordinary updates. A -schema 14 or 15 database migrates automatically when the scheduler starts, in one -transaction: each app takes a name (its tenant's when that tenant had one app in -the namespace, else `TENANT-`) and the default -placement policy, jobs name their app, `TENANT/ELF` grants become app names, and -namespace-less tenant signer tokens are removed. Update each prover's `apps.yaml` -entry to `app: NAME` plus `elf_id`. Remove `tenants`, `default_tenant`, -`require_signature` and `admin_token_file` from `fleet.yaml` first; the scheduler -refuses to start with them. Issue an administrator token with -`fleet-scheduler -c fleet.yaml issue-admin --label NAME --out FILE`, since the old -`admin_token_file` no longer applies. Only another schema requires an explicit -metadata reset. A reset invalidates namespace access, so recreate and redistribute -scoped credentials before resuming clients. - -Across any update, keep the existing `etc/`, Postgres database, control `data_dir` -and node `work/`; back them up first — the rollback procedures above restore -programs, not data. +A schema 14 or 15 database migrates automatically when the scheduler starts, in one +transaction: each app takes a name (its tenant's when that tenant had one app in the +namespace, else `TENANT-`) and the default placement +policy, jobs name their app, `TENANT/ELF` grants become app names, and namespace-less +tenant signer tokens are removed. + +- Update each prover's `apps.yaml` entry to `app: NAME` plus `elf_id`. +- Remove `tenants`, `default_tenant`, `require_signature` and `admin_token_file` from + `fleet.yaml` first; the scheduler refuses to start with them and names what replaced + each. +- Issue an administrator token with + `fleet-scheduler -c fleet.yaml issue-admin --label NAME --out FILE`, since + `admin_token_file` no longer applies. + +Any other schema needs an explicit metadata reset, which removes job history and +access registrations. Recreate namespaces and tokens, redistribute the scoped +credentials before clients resume, and keep the apps' own proofs and checkpoints. +Schema 16 keeps namespaces and tokens across ordinary updates. diff --git a/docs/operations.md b/docs/operations.md index cb79f5d..c69b2f5 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,151 +1,96 @@ # Operations -Follow the [installation guide](installation.md) to set up the control host and -nodes. These commands use the control-plane API; no SSH connection to a node is -needed for app scaling. - -Run these on the control host, where `fleetctl` reads the administrator's -`etc/admin.token` by default. Remote operators set `FLEET_URL` to the HTTPS -gateway and `FLEET_TOKEN_FILE` to their private administrator token file. -Namespace users set `FLEET_TOKEN_FILE` or `--token-file` to their own token. -Tokens with the `manage` permission change app and batch metadata; tokens with -`read` and `submit` submit and inspect permitted jobs. Scoped tokens can list and -inspect their own namespace, including its shared quota and their tag's quota and -usage. Node administration and namespace writes require an administrator token. -Add `--json` for structured output. +Day-to-day work with `fleetctl`, once the [installation](installation.md) is done. +Nothing here needs SSH to a node. -## Daily commands +On the control host, `fleetctl` uses the administrator's `etc/admin.token`. +Elsewhere, set `FLEET_URL` to the HTTPS gateway and `FLEET_TOKEN_FILE` (or +`--token-file`) to your token: an administrator's, or a namespace user's own. A +user token works within its namespace and tag, with what its permissions allow +([reference](reference.md#permissions)). Add `--json` for structured output. -The short list; everything else is in the sections below or in `fleetctl --help`. +## Daily commands ```bash fleetctl node # machines: capacity, READY, heartbeats, the agent's VERSION fleetctl node NAME # one machine's record fleetctl app # every app with its state, allocation and node fleetctl app logs APP -n 200 # an app's last log lines, read from where it runs -fleetctl job -a # queued and running jobs -fleetctl batch # batch workloads with their newest jobs +fleetctl job -a # queued and running batch jobs and submitted proofs +fleetctl batch # batches with their newest jobs fleetctl node drain NAME # hold new placements on a machine (undrain to release) ``` +`fleetctl --help` and each subcommand's `--help` list the rest. + ## Namespaces and access -A namespace owns apps and batches and shares one GPU/CPU quota across their -allocations. Scoped users can read their own namespace; administrators manage -ownership, access and quotas: +A namespace owns apps and batches and shares one GPU/CPU quota across them. -| Command | Purpose | Access | +| Command | Does | Who | |---|---|---| -| `fleetctl namespace` or `namespace list` | List namespace identities; JSON also includes configured quotas. | Scoped users: own namespace; administrators: all namespaces. | -| `namespace create NAME --endpoint URL --out NAME.env` | Create the namespace and its owner bundle for `scope=default`. | Administrator | -| `namespace get NAME` | Inspect shared quota and usage, plus tag caps and usage. | Scoped users: own namespace/tag; administrators: any namespace/tag. | -| `namespace quota NAME --gpus N --cpus N` | Replace the namespace's shared limits. | Administrator | -| `namespace quota NAME --tag team=research --gpus N --cpus N` | Replace that tag's limits within the namespace. | Administrator | -| `namespace suspend NAME` | Block tokens and new submissions; accepted work continues. | Administrator | -| `namespace resume NAME` | Restore access; individually revoked tokens remain revoked. | Administrator | - -Read your namespace with the same token used for app and batch commands: - -```bash -fleetctl --token-file alice.token namespace -fleetctl --token-file alice.token namespace get alice -``` - -Names and immutable IDs are accepted. Scoped reads include namespace totals and -the token's tag; another namespace or tag selector is refused. Other tags' quota -settings and job allocation records are not exposed to scoped users. - -For example: +| `fleetctl namespace` (`namespace list`) | List namespaces; JSON adds their quotas | Users see their own | +| `namespace create NAME --endpoint URL --out NAME.env` | Create it and its owner bundle for `scope=default` | Administrator | +| `namespace get NAME [--tag TAG]` | Show the quota and use, and the tags' caps and use | Users see their own namespace and tag | +| `namespace quota NAME [--tag TAG] --gpus N --cpus N` | Replace the namespace's or a tag's caps | Administrator | +| `namespace suspend NAME` / `resume NAME` | Block or restore its tokens; accepted work continues, and revoked tokens stay revoked | Administrator | ```bash fleetctl namespace create alice --gpus 6 --cpus 64 \ --endpoint http://127.0.0.1:51012 --out alice.env -fleetctl namespace get alice fleetctl namespace quota alice --tag team=research --gpus 2 --cpus 16 -fleetctl namespace get alice --tag team=research +fleetctl --token-file alice.token namespace get alice ``` -Creation writes `alice.env`, `alice.token` and, when `--server-ca` is supplied, -`alice.ca.pem` beside the environment file. Give that bundle to the owner securely. -`--endpoint` is the bundle's SDK `ENDPOINT`, the owner's -[app-owned proof endpoint](reference.md#app-owned-proof-endpoint); `--url` selects Fleet's control API. -Use an existing CA certificate here; `init tls` belongs to control-host installation. +Creating writes `alice.env`, `alice.token` and, with `--server-ca`, `alice.ca.pem` +beside it; give the bundle to the owner securely. `--endpoint` is the bundle's SDK +`ENDPOINT`, the owner's [app-owned proof endpoint](reference.md#app-owned-proof-endpoint), +and `--url` is Fleet's control API. Pass an existing CA certificate: `init tls` is for +installing the control host. -Each `quota` command replaces both limits: an omitted limit becomes unlimited, -and `namespace quota NAME` clears both. Zero blocks additional resource allocation. -Quotas limit use of the node pool; [app scaling](#configure-resources) chooses an -app's allocation, and [node maintenance](maintenance.md) changes the pool itself. +Each `quota` command replaces both caps: an omitted cap is unlimited, zero allows +none, and `namespace quota NAME` alone clears both. Quotas limit a namespace's share +of the node pool; [scaling](#configure-resources) sets an app's allocation, and +[maintenance](maintenance.md#changing-a-nodes-capacity) changes the pool. Tokens, +tags and quotas in full are in the [reference](reference.md#namespaces-tags-and-tokens). ## Configure resources -There are two limits: - | Setting | Controls | Change it with | |---|---|---| -| Node capacity | Total pool Fleet may allocate on that machine | Installer `--cuda` / `--cpus`, recorded in `etc/watchdog.env` | -| App allocation | GPUs and CPUs reserved per replica | `fleetctl app scale` | +| Node capacity | The total pool Fleet may allocate on a machine | Installer `--cuda` / `--cpus`, kept in `etc/watchdog.env` | +| App allocation | GPUs and CPUs per replica, and the replica count | `fleetctl app scale` | -CPU units are logical CPUs. Ray's configured pool can be smaller than the hardware. -`fleetctl node` shows total and currently free GPUs/CPUs. App listings show -configured allocations, including disabled apps; they do not show live utilization. -Namespace and optional tag quotas further limit allocation. - -For example, set the EthProofs worker's allocation to five GPUs and 96 CPUs using -the namespace `zkm` owner's token: +CPU units are logical CPUs, and Ray's pool can be smaller than the hardware. +`fleetctl node` shows total and free GPUs and CPUs; app listings show configured +allocations, disabled apps included, not live use. Namespace and tag quotas cap +allocations further. ```bash fleetctl --token-file zkm-owner.token --namespace zkm app scale ethproofs-z4 \ - --gpus 5 --cpus 96 --replicas 1 --node ant-5090-2 - -# Read desired/observed allocation and operation status. -fleetctl --token-file zkm-owner.token --namespace zkm app scale ethproofs-z4 -# Node inspection uses administrator credentials. -fleetctl node ant-5090-2 + --gpus 5 --cpus 96 --replicas 1 --node gpu-1 +fleetctl --token-file zkm-owner.token --namespace zkm app scale ethproofs-z4 # desired, observed, status ``` -Scaling requires the `manage` permission and an online, ready node. -The token must cover the app's namespace, tag and app grant; for a proof app its -tag must cover every node entry running it. Namespace and tag quotas remain -administrator-managed. Requests are asynchronous; inspect until -the state is `complete` or `failed`. Omitted resource flags retain their current -values. Disabled apps stay disabled. Use `--node` to -select a machine when the app has multiple deployments. - -Integrated EthProofs uses exactly one replica. Other apps may use multiple -replicas; resources are per replica. The current -`ant-5090-2` pool is 8 GPUs and 128 CPUs. With only EthProofs enabled at 5 GPUs -and 96 CPUs, 3 GPUs and 32 CPUs remain for other allocations. Inspect current -allocations with `fleetctl app scale`; these resource settings can change. -Existing allocations are accounted for when replacing an app's own allocation. - -To change the node pool, edit `RAY_NUM_CPUS` and `CUDA_VISIBLE_DEVICES` in -`etc/watchdog.env`, then restart local Ray after pausing the node's workloads. -An empty `RAY_NUM_CPUS` enables automatic detection, and an empty -`CUDA_VISIBLE_DEVICES` gives the node no GPUs. Local Ray listens on ports 6379-6381, -52365, 8080 (metrics) and 10002-19999 (workers). On a machine where another service holds -one, set `RAY_METRICS_EXPORT_PORT`, `RAY_MIN_WORKER_PORT` and `RAY_MAX_WORKER_PORT` in -`etc/watchdog.env`. Scaling an app does not increase node capacity. Follow the -[node maintenance procedure](maintenance.md). - -## Apps and batches +Scaling needs `manage` on the app and an online, ready node. It runs in the +background: check it until it's `complete` or `failed`. Omitted flags keep their +values, a disabled app stays disabled, and `--node` picks the deployment when an app +has several. Integrated EthProofs runs exactly one replica; other apps may run more, +with resources per replica. The app's current allocation counts toward the room for +its new one. Scaling never adds node capacity. -Apps and jobs show `TAG` and `STATE`. App state reports replica readiness; -job state reports an execution's progress. App details retain replica settings -and job counts. Tags use `key=value`, for example `scope=default` or `team=research`. -The same namespace/tag selectors work for apps, batches and jobs. +A node's pool is `RAY_NUM_CPUS` and `CUDA_VISIBLE_DEVICES` in its `etc/watchdog.env`; +empty means detect CPUs automatically, and no GPUs. Changing it restarts Ray, so +follow the [maintenance procedure](maintenance.md#changing-a-nodes-capacity). Local +Ray listens on ports 6379–6381, 52365, 8080 (metrics) and 10002–19999 (workers); +where another service holds one, set `RAY_METRICS_EXPORT_PORT`, +`RAY_MIN_WORKER_PORT` and `RAY_MAX_WORKER_PORT` there. -Owner tokens can create and remove workloads, set priority, enable/disable them -and scale apps within their grants. Batch owners set image, environment, resource -policy when registering a batch. Administrators set batch mount roots and runtime -users. App image, environment, command, user, workdir and volume changes also -require an administrator; owners can change resource sizes and timeouts. Batch disabling -blocks new submissions; accepted batch jobs continue. App disabling drains the -app's replicas before releasing their reservations. +## Apps and batches -When upgrading from owner-controlled runtime settings, review existing app -configuration, batch mount roots and runtime users, and queued jobs before -restoring untrusted owner access. The new authorization checks do not rewrite -configuration or cancel jobs accepted by an earlier scheduler. +App and job tables both show `TAG` and `STATE`: an app's state is its replicas' +readiness, a job's its progress. The same `--namespace` and `--tag` selectors work +for apps, batches and jobs. ```bash fleetctl --namespace zkm --tag scope=default app @@ -153,101 +98,91 @@ fleetctl --namespace pi --tag team=research batch fleetctl --namespace pi --tag team=research job ``` -```bash -fleetctl node -fleetctl --namespace zkm app -fleetctl --namespace zkm job -fleetctl --token-file zkm-owner.token --namespace zkm app disable ethproofs-z4 -fleetctl --token-file zkm-owner.token --namespace zkm app enable ethproofs-z4 -``` +### Apps Create an app from a JSON file of its container settings, the keys of its -`apps.yaml` entry (`image`, `replicas`, `gpus`, `cpus`, `env`, and for a container app -`command` and `port_base`). A proof app is an app whose container is a Ziren prover: give -its program and proof policy too, and its spec's `kind` defaults to `ziren-gpu` -(`ziren-cpu` for a CPU prover): +`apps.yaml` entry: `image`, `replicas`, `gpus`, `cpus`, `env`, and for a container +`command` and `port_base`. A proof app's container is a Ziren prover: give its +program and proof policy too, and its `kind` defaults to `ziren-gpu` (or set +`ziren-cpu`). The [reference](reference.md#apps) lists every setting. ```bash fleetctl --token-file zkm-owner.token --namespace zkm app add model --spec model.json fleetctl --token-file zkm-owner.token --namespace zkm app add eth --spec prover.json \ --elf /path/to/program.elf --class latency --anchor exclusive --backend cuda fleetctl --token-file zkm-owner.token --namespace zkm app enable model +fleetctl --token-file zkm-owner.token --namespace zkm app disable model ``` -An app starts disabled and on no node. Its first `app enable` places it: the scheduler -picks a ready node big enough for all its replicas, preferring one where they fit now, -has that node write the entry into its `apps.yaml`, then starts it. Users do not choose the node. App names are unique across -the fleet. The [reference](reference.md) lists every setting. +An app starts disabled and on no node. Its first `app enable` places it: the +scheduler picks a ready node big enough for all its replicas, preferring one where +they fit now, has that node add the entry to its `apps.yaml`, and starts it. Users +don't choose the node, and app names are unique across the fleet. + +Disabling drains the replicas and releases their resources after cleanup; the app's +durable data stays. An interrupted EthProofs block is retried from its saved cursor, +and its [shutdown rules](../apps/ethproofs/README.md#operate-and-recover) say how to +finish an in-flight proof first. `node drain` stops new proof and batch placements, +but not an integrated app's own block stream. -Disabling releases the app's execution resources after cleanup. Durable app data -remains. An interrupted EthProofs block is retried from its saved cursor; -see its [shutdown rules](../apps/ethproofs/README.md#operate-and-recover) to finish an -in-flight proof before stopping. `node drain` stops new Fleet proof/batch -placements; it does not pause an integrated app's own block stream. +### Batches -For a batch, first register reusable configuration, then submit and inspect jobs: +Register a batch's image and policy once, then submit jobs to it: ```bash fleetctl --token-file pi-owner.token --namespace pi batch add gpu-test \ --image nvidia/cuda:12.8.0-base-ubuntu22.04 --priority 10 -# An administrator approves host storage and its runtime user before jobs mount it. +# An administrator approves host storage and a runtime user before jobs mount it: fleetctl --namespace pi batch update gpu-test --allow /data/models --user 1000:1000 -fleetctl --token-file pi-owner.token batch # the batches, with active jobs and the last job -fleetctl --token-file pi-owner.token batch gpu-test # one batch and its newest jobs -fleetctl --token-file pi-owner.token job --type batch -a # the queued and running batch jobs -fleetctl --token-file pi-owner.token batch priority gpu-test 20 fleetctl --token-file pi-owner.token batch submit gpu-test --gpus 1 --cpus 2 \ --mount /data/models:ro -- nvidia-smi +fleetctl --token-file pi-owner.token batch gpu-test # the batch and its newest jobs +fleetctl --token-file pi-owner.token job --type batch -a # queued and running batch jobs fleetctl --token-file pi-owner.token batch logs JOB_ID -f fleetctl --token-file pi-owner.token batch cancel JOB_ID +fleetctl --token-file pi-owner.token batch priority gpu-test 20 ``` -The namespace must already exist. See the [reference](reference.md#administrator-workflow) -for namespace creation, token issuance and quotas. Use `fleetctl --help` and -subcommand `--help` for the complete command list. SDK clients use their app's -endpoint; Fleet CLI and MCP use the control-plane URL. - -Batches are configured in node `etc/batches.yaml`; apps are configured in -`etc/apps.yaml`, where the scheduler also writes the apps it places. Both share namespace, tag, priority, -image and resource policy. Each batch job gets a workspace under -`FLEET_JOBS_DIR/JOB_ID/`, normally `~/app/fleet/work/jobs/JOB_ID/`. -The command's output directory is exposed as `FLEET_OUT`; retrieve artifacts -from application-owned storage. Fleet supplies status and logs. +The namespace must exist first ([administrator workflow](reference.md#administrator-workflow)). +Each job gets a workspace in its node's `FLEET_JOBS_DIR/JOB_ID/`, normally +`~/app/fleet/work/jobs/JOB_ID/`, with `FLEET_OUT` as its output directory; Fleet +supplies status and logs, and the job's results live in the owner's storage. Nodes +can also configure batches in `etc/batches.yaml`, as apps in `etc/apps.yaml`, where +the scheduler writes the apps it places. SDK clients use their app's endpoint; +`fleetctl` and MCP use the control-plane URL. ## Logs and recovery Read an app's log from anywhere with control-plane access. The scheduler asks the -node that runs the app through its next heartbeat, so nothing is stored centrally and -the answer takes a moment: +app's node in its next heartbeat, so nothing is stored centrally and the answer takes +a moment: ```bash fleetctl app logs ethproofs-z4 -n 200 # the last 200 lines (default 100, at most 2000) fleetctl app logs APP --node NODE # when the app runs on several machines ``` -It needs the same access as `fleetctl app scale APP` without changing anything: the app -must be granted to the token and within its tag. An offline node, or one that is busy for -longer than 15 seconds, answers with an error. Each reply carries at most -256 KiB, from the first replica's `-0.node.log` file in the node's log directory, -named after the app's `apps.yaml` entry. -MCP clients use `fleet_app_logs`. A node must run this release for the command to work. +It needs the same access as `fleetctl app scale APP` and changes nothing. An offline +node, or one busy for more than 15 seconds, answers with an error. A reply carries at +most 256 KiB, from the first replica's `-0.node.log` in the node's log +directory. MCP clients use `fleet_app_logs`. + +When `fleetctl app` shows an app `starting`, DETAIL says why: -When `fleetctl app` shows an app as `starting`, DETAIL says why: `waiting for capacity` -with what it needs and what its node has free (the start is held until it fits; `app -disable` or a smaller `app scale` replaces the held start), `starting on NODE` once the node -has the request, or `waiting for NODE to report it ready`. +- `waiting for capacity`, with what it needs and what its node has free: the start + is held until it fits, and `app disable` or a smaller `app scale` replaces it; +- `starting on NODE`, once the node has the request; +- `waiting for NODE to report it ready`. -Inspect ordinary logs on the node: +On the node itself: ```bash tail -f ~/app/fleet/work/logs/node.log -# With a systemd supervisor: -journalctl -u zkm-fleet-node.service -f -# EthProofs worker: -tail -f ~/app/fleet/work/logs/ethproofs-v5-gpu-0.node.log +journalctl -u zkm-fleet-node.service -f # with a systemd supervisor +tail -f ~/app/fleet/work/logs/ethproofs-v5-gpu-0.node.log # the EthProofs worker ``` -For an offline node, check its supervisor, `ray.log`, `watchdog.log`, gateway URL, -cluster token and CA trust. For a stalled app, inspect its worker/container logs -and reporting errors. EthProofs retries failed blocks; diagnose the error instead -of changing the cursor to skip ahead. +For an offline node, check its supervisor, `ray.log`, `watchdog.log`, the gateway +URL, the cluster token and CA trust. For a stalled app, read its worker or container +logs and reporting errors. EthProofs retries failed blocks: diagnose the error rather +than move the cursor ahead. [Maintenance](maintenance.md#troubleshooting) has more. diff --git a/docs/reference.md b/docs/reference.md index 1ae71dd..8276b1e 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -1,549 +1,512 @@ # Fleet reference -See [installation](installation.md), [operations](operations.md) and -[node maintenance](maintenance.md) for practical instructions. - -## Code names - -Fleet follows Kubernetes workload terminology while keeping its own small API: -a `Job` finishes a command, an `AppDeployment` holds the desired state of one app -entry on one node, and `WorkloadController` reconciles that state with node reports. -`JobSpec` describes execution; `JobStatus` records what happened. `ProofJob` -is the proof-specific record. Shared records and requests are generated from -[model.proto](../crates/fleet-proto/proto/fleet/model/v1/model.proto). - -| Name | Responsibility | -| --- | --- | -| `Scheduler` | Places proofs on registered provers | -| `WorkloadController` | Admits and reconciles apps and batch jobs | -| `NodeExecutor` | Applies authorized commands through local Ray | -| `ProofFetcher` | Coordinates bounded proof downloads from storage nodes | -| `FleetClient` / `FleetctlClient` | HTTP client / MCP's CLI subprocess client | - -These names follow Kubernetes' [workload](https://kubernetes.io/docs/concepts/workloads/controllers/) -and [controller](https://kubernetes.io/docs/concepts/architecture/controller/) concepts. -Fleet keeps `fleetctl app` and `fleetctl batch`; it needs no Kubernetes runtime. -Node execution configuration is `apps.yaml`; each entry runs an app's replicas -on that node. Batches use `batches.yaml`, and local job records use -`FLEET_JOBS_DIR`. This release replaces the previous execution names and paths -without compatibility aliases. - -## Execution and proof protocols - -Each execution machine runs an **independent local Ray instance** and `fleet-node`. -It initiates authenticated HTTPS gRPC heartbeats to the control plane. Responses -carry typed launch, stop and app commands; later heartbeats acknowledge commands -and report capacity and job state. Proof actors use their existing outbound proof -heartbeat. The scheduler makes admission, quota, priority and preemption decisions; -a small Python adapter performs local Ray/container operations. The control host -requires no Ray installation. Adding a remote machine requires the control-plane URL -and cluster token; machines do not join a shared Ray head or use SSH channels. - -SDK-driven apps preserve the `GenerateProof`/`GetStatus` protocol: - -1. The client submits to its **app-owned endpoint**, which stores the inputs and - sends Fleet only references, SHA-256 hashes and job parameters. -2. Fleet admits and assigns the job through **node-initiated unary heartbeats**. - The assigned proof actor reads inputs directly from app storage and proves them - using its dedicated GPU server. -3. The node commits outputs to app storage and reports references, hashes, cycles - and `prove_ms`. Fleet records the storage machine from the authenticated node. -4. The client polls its app endpoint, which validates completion references and - reads the local proof and public values directly. Remote CLI/MCP fetches use - an authorized ticket and the storage node's outbound HTTPS upload through Fleet. - Fleet does not persist reference-job proof bytes. - -There is no bidirectional RPC stream or control-plane connection to an inbound -node API. Integrated EthProofs owns the entire block pipeline: head polling, -witness preparation, loopback GPU proving, durable outputs and API reporting. -It needs no separate EthProofs client or SDK proof endpoint. Its saved cursor -advances one block at a time after output and report records are committed. - -Each proof replica owns one GPU server and runs one proof at a time. Fleet places -jobs across eligible replicas; a single proof does not span servers. Every eligible -replica and the app endpoint must access the same app storage path. The initial -implementation uses filesystem references (`file://`), including owner-managed -shared mounts for replicas on different machines. Fleet retains admission, quotas, -placement, retries and execution leases. - -Batch records and app policy are durable on the control host. Nodes retain their local -execution records and sync logs separately over HTTPS. App owners prepare batch -inputs and receive outputs in their storage; the batch command reads and writes there directly. -Lost connections retain -reservations; replay uses the same execution lease and confirmed cleanup releases -resources. Current batch jobs execute on one machine (`--nodes 1`); cross-machine batch jobs -are refused until separate coordination is implemented. - -Persistent app ports remain private application endpoints. Fleet controls their -lifecycle and resource reservations; an authenticated application proxy is required -for external access to an app's own protocol. - -**Priority and preemption.** Jobs inherit their app’s priority at submission (default `0`, higher first). A waiting batch job or starting app can preempt eligible lower-priority batch jobs only within its own namespace. Stopped batch jobs queue again after confirmed cleanup, spending no retry. Equal-priority jobs do not preempt each other; pending jobs with equal priority use FIFO order. - -The Python adapter performs local placement, container startup, retries and cleanup -for commands authorized by Rust. It holds the cluster token for proof heartbeats; -it has no administrator API client, standalone submission CLI or policy editor. -Rust resolves dependencies, enforces quotas, selects preemption victims and drains -apps before sending node commands. Prepare code, datasets and models in the -workload image or owner-managed storage before submission. - -The implementation keeps these boundaries: - -| Component | Responsibility | Deployment dependencies | +What Fleet runs and how, who may do what, its APIs and where it keeps data. +[Installation](installation.md), [operations](operations.md) and +[maintenance](maintenance.md) have the procedures. + +## Architecture + +Each execution machine runs its own local Ray instance and `fleet-node`. The agent +sends authenticated HTTPS gRPC heartbeats to the control plane: replies carry typed +launch, stop and app commands, and later heartbeats acknowledge them and report +capacity and job state. Nothing connects in to a node: there is no streaming RPC, +inbound node API, shared Ray head or SSH channel. Adding a machine needs only the +control-plane URL and the cluster token, and the control host runs no Ray. + +The Rust scheduler decides admission, quotas, priority, dependencies and preemption, +and drains apps before it sends node commands. A small Python adapter on each node +carries out what Rust authorizes: local placement, container startup, retries and +cleanup. It holds the cluster token for proof heartbeats and has no administrator +client, submission CLI, policy editor or HTTP listener. + +| Component | Does | Depends on | |---|---|---| | `fleet-scheduler` | Admission, namespace policy, quotas, placement and durable control records | Postgres; no Ray or prover SDK | -| `fleet-node` | Outbound machine heartbeats, command delivery and log synchronization | Local Python adapter; no database, signer recovery or prover SDK | -| `bridge/fleet_bridge` | Local resource grants, actors, containers and confirmed cleanup | Ray and PyYAML | -| `fleet-proof-app` | Optional SDK protocol adapter beside app-owned storage | Fleet HTTPS API; no database or prover SDK | -| `fleet-ziren-node` | Optional SDK-driven proof execution and retained artifact fetching | Ziren runtime; installed only for workloads that use it | -| App worker | Application preparation, proving calls, outputs, checkpoints and reporting | Dependencies packaged by its app owner | -| `fleetctl` / `fleet-mcp` | Clients of the same authenticated Fleet API | MCP is optional and invokes `fleetctl` | - -Protocol types, artifact validation and SDK step mapping are shared in -`fleet-proto`. Fleet health checks process and container availability; it does -not interpret an app's checkpoint format. Inspect application progress and -failures through its logs. Apps and batches share ownership and container -configuration, while retaining their distinct persistent and finite lifecycles. - -Shared data records and requests are defined in -[`model.proto`](../crates/fleet-proto/proto/fleet/model/v1/model.proto), which generates -the Rust types for jobs, apps, namespaces, tokens, resources, scaling and node reports. -The scheduler and machine agent exchange a typed protobuf report. App-owned settings -and execution details use lossless JSON extension fields; the local Python adapter -needs no protobuf package. Rust owns runtime handles and scheduling behavior. -The report (`NodeReport`) carries the node's `apps.yaml` entries, each with its observed -state, in `apps` (field 9). Machine report version 3 requires updating the scheduler, -agent and adapter together; version 2 named that field `services`. The agent reports the -release version it was built for in `agent_version` (field 12, optional): the build sets -`FLEET_VERSION`, for example `0.1.5`. Builds that name no version, like agents that -predate the field, omit it, and `fleetctl node` shows `-` there. - -Further simplification should move batch and app command persistence behind -the same store interface as proof records, and separate retained artifact fetching -from the native prover package. Both require recovery and compatibility checks; -the current storage formats and GOAT execution path are preserved. - -**Resources in the code.** Namespaces, workloads, execution records and tokens describe ownership, access and work: - -| Resource | Purpose | +| `fleet-node` | Outbound heartbeats, command delivery and log sync | The local Python adapter; no database, signer recovery or prover SDK | +| `bridge/fleet_bridge` | Local resource grants, actors, containers and confirmed cleanup | Ray, PyYAML | +| `fleet-proof-app` | Optional SDK endpoint beside app-owned storage | Fleet HTTPS API; no database or prover SDK | +| `fleet-ziren-node` | Optional SDK-driven proving and retained-artifact fetching | Ziren runtime, installed only where used | +| App worker | The app's own preparation, proving calls, outputs, checkpoints and reporting | Whatever its owner packages | +| `fleetctl` / `fleet-mcp` | Clients of the same authenticated API | MCP is optional and runs `fleetctl` | + +Fleet checks that processes and containers are up; it doesn't read an app's +checkpoints, so follow an app's progress in its logs. Prepare code, datasets and +models in the workload image or the owner's storage before submitting. + +### Code names + +Fleet borrows Kubernetes' [workload](https://kubernetes.io/docs/concepts/workloads/controllers/) +and [controller](https://kubernetes.io/docs/concepts/architecture/controller/) terms +without needing a Kubernetes runtime: + +- `Job`: one run of a command or a proof. `JobSpec` says how to run it, `JobStatus` + what happened, and `ProofJob` is a proof's record. +- `AppDeployment`: the desired state of one app entry on one node. +- `WorkloadController`: admits apps and batch jobs and reconciles deployments with + node reports. +- `Scheduler`: places proofs on registered provers. +- `NodeExecutor`: applies authorized commands through local Ray. +- `ProofFetcher`: coordinates bounded proof downloads from storage nodes. +- `FleetClient` / `FleetctlClient`: the HTTP client, and MCP's client that runs + `fleetctl`. + +A node's apps are entries in its `apps.yaml`, each running one app's replicas there; +its batches are in `batches.yaml`, and its job records in `FLEET_JOBS_DIR`. + +### Records and the node report + +Shared records and requests are generated from +[`model.proto`](../crates/fleet-proto/proto/fleet/model/v1/model.proto): jobs, apps, +namespaces, tokens, resources, scaling and node reports. `fleet-proto` also holds the +protocol types, artifact validation and SDK step mapping. App-owned settings and +execution details travel as lossless JSON fields, so the Python adapter needs no +protobuf package. + +A node's `NodeReport` carries its `apps.yaml` entries, with their observed state, in +`apps` (field 9). The report is version 3; changing it means updating the scheduler, +agent and adapter together. `agent_version` (field 12, optional) is the release the +agent was built for, set by `FLEET_VERSION` at build time (for example `0.1.5`); +`fleetctl node` shows `-` when it's missing. + +### Node commands + +The scheduler owns each app's configuration and sends it with one operation, +`NODE_OPERATION_APP` (7). It addresses an app (`{"app": {namespace, namespace_name, +name}}`: the node finds the `apps.yaml` entries that run it), a batch +(`{"batch": {name}}`) or the one entry a change stopped (`{"entry": {name}}`). It then +removes it (`remove`), writes its settings into `apps.yaml` or `batches.yaml` +(`set`: enabled state, resources and settings), or adds an app being placed, disabled +(`create`, with the entry). Enabling, disabling, scaling and updating an app run the +same way: its replicas drain and stop, the node writes the new configuration, and +they start again unless the app is now disabled. + +`NODE_OPERATION_SET_APP` (4), with `{"app": entry, "disabled", "scale_id", +"scale_phase"}` (or `"scale_cleanup"`), only starts or stops an entry's replicas as +configured: in that sequence, and when a node's replicas stop matching it. Each entry +on each node is an `AppDeployment` keyed `{node}-{entry name}`, holding the +reservation `deployment:{key}`. `fleetctl app logs` asks for an entry's log with a +`LogRequest` naming that `entry`. + +### Durability + +Proof jobs, namespaces, tokens and quota reservations are in Postgres. Batch, app, +deployment and command records are durable files in the control host's `data_dir`; +back that up with Postgres. The scheduler holds a Postgres advisory lock while it +runs, so only one process dispatches from a database. A new database is created at +schema 16, and the scheduler checks the version at startup; +[maintenance](maintenance.md#database-schema-14-or-15-to-16) covers older ones. + +Through a scheduler restart or a lost connection, nodes keep running locally, +reconnect through heartbeats and retry completion metadata until it's acknowledged. +A lost connection keeps its reservations: a replay uses the same execution lease, an +interrupted launch recovers without starting a second actor, and only confirmed +cleanup releases resources. Heartbeats import a node's existing execution records +without relaunching them, and finished jobs stay readable while their node is +offline. + +## Workloads + +Every app and batch belongs to a [namespace](#namespaces-tags-and-tokens). They share +one settings model (name, namespace, image, user, environment, mounts, priority and +enabled state) but not a lifecycle: an app keeps its replicas running, and a batch +runs one command per submission, for minutes or for days. App and batch names are +unique across the fleet and shared between them (409 when taken); a node entry named +like a batch is ignored. + +### Apps + +An app is a long-running container, such as a model server, ComfyUI or a Ziren +prover. A proof app is an app whose container is a Ziren prover of a named program. +`fleetctl app` lists every app (with `--json`, one array of records with `type: app`), +and each is managed by name. + +**Creating.** `fleetctl --namespace NS app add NAME --spec FILE.json` posts a +`RegisterAppRequest` to `POST /apps`: `name`, optional `namespace` and `priority` +(default 0), the file as `spec`, and a prover's `proof`. The spec holds the keys of +the app's `apps.yaml` entry; unknown or mistyped settings are refused. + +| Keys | For | |---|---| -| **Namespace** | Administrator-created ownership and billing boundary. | -| **App** | Long-running container in one namespace; a proof app's container is a Ziren prover of its named program. | -| **Batch** | Reusable finite workload configuration in one namespace; each run is a batch job. | -| **Job** | Execution record for a proof request or batch command, with namespace and tag ownership. | -| **Token** | Belongs to one namespace, with permissions, an optional tag and app grants; an administrator's token has none. | - -Nodes are infrastructure; allocations are internal reservation records. A proof app's `proof` settings set proof scheduling, quotas configure namespace limits, and tags are optional attributes for isolating work and applying additional limits. There is no Group resource. See [namespaces, access and quotas](#namespaces-access-and-quotas) for details. - -**Apps, batches and jobs.** `fleetctl app` lists all long-running workloads together. `fleetctl --json app` returns one array of app records, each with `type: app`. Inspect and manage an app by name, without a kind selector. An app is a container; a proof app is an app whose container is a Ziren prover of a named program. Create either with `fleetctl --namespace NS app add NAME --spec FILE.json`, which posts a `RegisterAppRequest` (`model.proto`) to `POST /apps`: `name`, optional `namespace` and `priority` (default 0), the file as `spec`, and a prover's `proof`. The spec holds the keys of the app's `apps.yaml` entry: `kind` (`app` by default, `ziren-gpu` by default for a prover, or `ziren-cpu`), `image` (required), `replicas`, `gpus`, `cpus`, `shm_gb`, `memory_gb`, `env`, `volumes`, `ready_timeout_s` and `stop_timeout_s`; a `kind: app` container also `command`, `port_base` (required; replica i serves on `port_base` + i), `health_path`, `workdir` and `user`; a prover also `proof_storage_root`, `backend` and `rust_log`. Unknown or mistyped settings are refused. A prover also takes its program, `--elf-id ID` or `--elf FILE`, and its placement policy, sent as `proof` (`elf_id`, `class`, `anchor`, `backends`, `zkm_version`, `max_queued`): `--class latency` places its jobs on the biggest free prover first, `batch` (the default) on the smallest; `--anchor` is `none` (default), `shared` or `exclusive`; repeat `--backend` (`cuda`, `cpu`, `mock`; default any); `--zkm-version` and `--max-queued` (default 64) are optional. Only a prover takes a program, and proof policy without one is refused. A proof runs only on a prover bound to its app. - -An app is created disabled and on no node; the first `app enable` places it, and users never choose the node. Candidates are live, ready nodes that are not draining and whose total capacity holds the GPUs and CPUs of all its replicas. The scheduler prefers a node where they are free now and, among those, the one with the fewest free GPUs; otherwise the one with the most free GPUs, where the app waits for room. With no node big enough, enabling answers 503. The scheduler sends that node a `create` (`NODE_OPERATION_APP`), which writes the entry into its `apps.yaml` disabled, waits up to 120 seconds for the node to report it, then enables it and admits it against the namespace's quota; the control plane then manages enable/disable, priority and scaling. A failed create clears the placement, so the next enable places the app again; if the node has not reported the entry within the wait, enabling answers 503, and enabling again waits for the same create. An app that a node's `apps.yaml` already runs, such as an integrated worker's prover bound with `app: NAME`, is not placed again. Each start of an app the scheduler placed is placed the same way when it leaves the queue: on its own node when that has the room, else on the node that fits it best now, which gets a `create` with the app's entry, disabled, its `machine` set to that node's. Once that node reports the entry, the start's own node removes its entry (its configuration and what is left of its containers and processes, never the app's data) and the app starts on the new node. A node whose create fails is not picked again for that app. With no room anywhere, the start preempts lower-priority work on its own node and keeps the room that frees. The scheduler copies nothing of what an app keeps on a node's disks: the app takes care of its own storage. A namespace token with manage permission creates the namespace's apps; `user`, `volumes` and `proof_storage_root` (the host's user ids and filesystem) require administrator credentials, as `user` and `allow` do for batches. App names are unique across the fleet and shared with batches (409 when taken): a node's entry named like a batch is ignored. - -App and job tables both show `TAG` and `STATE`. An app's state describes its replicas: -`configured`, `starting`, `running`, `draining`, `disabled`, `offline` or `failed`; -a proof app without provers is `registered`. -The app table shows total configured GPUs and CPUs without a replica column; -replica settings and job counts remain in app details. An app with node entries using -multiple tags shows their tags together. Batches show `enabled` or -`disabled`; their execution states appear in the job table. - -`fleetctl batch` lists the batches, as `fleetctl app` lists apps, with each batch's active jobs and last job; `fleetctl batch NAME` shows one batch and its newest jobs, and `fleetctl job --type batch` lists the finite executions themselves. Register a reusable image and resource policy with `fleetctl --namespace NAMESPACE batch add NAME --image IMAGE`, and submit commands with `fleetctl batch submit NAME --gpus N --cpus N -- COMMAND`. Job actions take the job ID: `fleetctl batch logs`, `cancel`, `preempt` and `purge` find its batch, and `--batch NAME` skips that lookup. Batches can also be initially advertised from node `batches.yaml`. A batch may run for hours or days; its finite command lifecycle distinguishes it from an app. - -Apps and batch workloads share the scheduler’s configuration and lifecycle model: name, namespace, image, user, environment, mount policy, priority, and enable/disable state. A batch is that reusable configuration; each submission creates a batch job. Apps keep replicas running; batch workloads execute a command per submission. `batch submit --name` continues to name an individual job. - -Every app and batch belongs to an administrator-created namespace. Proof requests require a registered proof app, which the scheduler finds by the job's namespace and ELF ID; batch submissions require an existing batch. Jobs inherit their workload's namespace, tag and priority. `fleetctl job` remains a combined view of proof and batch execution records, including proof output fetching; `--app NAME` selects one app's or batch's jobs and `--type app|batch` one kind. - -App listings contain registered workloads. Rejected proof requests remain job -records and do not create an app entry. - -Restarting the scheduler preserves proof jobs, namespaces and quota reservations -in Postgres. Batch records, app deployment records and commands are durable files in -the control host's `data_dir`; back up that directory together with Postgres. -Nodes continue locally, reconnect through heartbeats, and retry -completion metadata until acknowledged. App endpoints and the machine storage handler remain independent of -prover replicas. Disabling a registered app releases its GPU server while retained -proofs remain fetchable through the machine agent. -App owners control retention of their input and output files. - -`fleetctl job fetch JOB_ID --out DIR` downloads proof JSON and public values (the -destination must not exist). Fleet authorizes the request, -delivers a short-lived fetch ticket in the machine's next ordinary heartbeat, -and relays the node's HTTPS response in bounded memory. `fleet-ziren-node fetch-artifact` -verifies committed files against the job, execution lease and -hashes; it initializes no prover or GPU. The control host saves no proof files. -Nodes need only outbound connections. Fetching waits for a heartbeat; an offline -storage machine returns a retryable error. -Batch logs remain available through Fleet. Each node retains its batch workspace in -its own `FLEET_JOBS_DIR`; no shared mount is required for one-machine batch jobs. - -### App-owned proof endpoint - -Run an endpoint beside the app's storage, independently of its GPU replicas: - -```bash -fleet-proof-app --scheduler https://scheduler.example.com:8905 \ - --namespace NAMESPACE_ID --elf-id FULL_ELF_SHA256 \ - --storage-root /srv/goat --elf-file /srv/goat/program.elf \ - --token-file /etc/my-app/user.token \ - --stage-addr 127.0.0.1:51012 --http-addr 127.0.0.1:51013 -``` - -Set `proof_storage_root: /srv/goat` on each matching `ziren-gpu` prover, in its app's -spec or its `apps.yaml` entry. Point the SDK's `ENDPOINT` to `http://127.0.0.1:51012` for a local -client. Keep its signing key; older SDKs without bearer headers can use the -endpoint's configured namespace user `--token-file`. The endpoint reads that file -on each request, so token rotation and revocation still apply. Supplied authorization -is checked directly, without falling back to the configured token. This binary binds -loopback only; external access requires an authenticated HTTPS app gateway. -Repeat `--elf-id` and `--elf-file` for multiple allowlisted programs in the same -namespace. Each program is a separate Fleet app; the endpoint finds its name from -`GET /apps` with the caller's token and submits to `/apps/NAME/jobs`. -GOAT uses its app-owned SDK endpoint on loopback ports `51012`/`51013`. -Integrated EthProofs calls its local GPU server directly and requires no SDK endpoint. -For other SDK-driven apps, use a separate storage root, scoped token and pair of -loopback ports for each endpoint. Set the pipeline’s SDK `ENDPOINT` to its local -stage port and its prover's `proof_storage_root` to the matching directory. -Changing the endpoint does not enable a disabled Fleet app. -Successful SDK status calls read the proof directly from this app's local output -storage using Fleet's authenticated completion references. Public values use the -same local path. Both reads check the file size, SHA-256 hash, and confinement to -the configured output directory; no output fetch heartbeat or control-host relay -is needed. The proof worker wakes its control loop immediately when execution -finishes so completion metadata does not wait for the periodic heartbeat. -Fleet CLI/MCP clients keep their control-plane URL. -Pass `--scheduler-ca-file ca.pem` when Fleet uses an operator-issued CA. - -On the machine agent, configure `--proof-node-binary /path/to/fleet-ziren-node` -and repeat `--proof-storage-root /srv/goat` for approved storage roots. -For `deploy/node-start.sh`, set `FLEET_PROOF_STORAGE_ROOTS_FILE` to an administrator-owned -file with one absolute root per line; `deploy/install.sh node --proof-storage-roots FILE` -installs this configuration. Keep these roots configured when GPU actors stop. -The machine agent executes storage-only fetches separately from its heartbeats and Ray operations. - -Fleet stores no proof inputs or outputs. A remote fetch through Fleet transfers its -requested bytes through the control host. The app endpoint logs input storage and metadata admission time -separately from the node's `prove_ms`. - -**Proving time is execution time.** For GPU proofs, `prove_ms` comes from the -GPU server; input/output transfers, queueing, setup and result fetching are -excluded. CPU/mock provers measure the local proof call. Wall time is recorded -separately for diagnostics and must not be reported as proving time. A GPU server -that omits execution timing fails the job instead of substituting RPC elapsed time. -The SDK response carries this value in `GetStatus.proving_time`. Reporting clients -must preserve it (for example, through `take_prove_ms()`); an SDK that only logs -it needs a timing accessor. Integrated EthProofs takes execution timing directly -from its GPU response and excludes preparation, queueing and reporting. - -## Namespaces, access and quotas - -A namespace is a resource and the accounting and billing unit. A token names -one namespace and may optionally have a tag. Tokens with the same tag share -proofs and batch jobs; different tags isolate that work. Untagged tokens -share the namespace's untagged work. Refreshing preserves the namespace and optional -tag, including access to earlier work. Each proof app's registration defines its scheduling policy. -The workload types are `app` and `batch`. - -### Token access and permissions - -An administrator creates a namespace with an immutable ID and unique name such -as `alice`. Names start with a lowercase letter, contain lowercase letters, -digits or hyphens, and have at most 63 characters; `system` and `admin` are -reserved. Renaming and deleting namespaces are not currently supported. - -A token belongs to a namespace, with an optional tag. Several tokens -can share a tag, for example teammates, laptop keys and automation keys. -New tags use `KEY=VALUE`, such as `scope=default` or `team=research`. The key -starts with a letter; each part contains up to 63 letters, digits, dots, -underscores or hyphens. Tags match exactly: `team=research` and `team=production` -are separate scopes. Apps and batches use the same tag rules. Bare tags are rejected. -Members can access that tag's existing jobs; another tag in the -same namespace cannot. A namespace token holds some of the permissions `read`, -`submit` and `manage` (default `read` and `submit`); a token without `manage` -needs explicit `--app` grants, by app or batch name. An administrator's token -holds `admin` alone, has no namespace and may do everything in every namespace. -Only administrators create namespaces, issue tokens or change access. - -Proof status checks the app grant before waiting or revealing any job state. -The SDK can still poll a rejection recorded for its own signer, including a -submission refused because the app was not granted; other signers cannot read -that rejection without the app grant. - -User requests infer their namespace from the verified token. A -`--namespace` selector must match that namespace; it cannot authorize access to -another tag or namespace. Administrators can select `--namespace NAME --tag TAG` -(`?namespace=NAME&tag=TAG` in the API) when listing jobs or looking up a client's proof ID. Fleet-wide records include -namespace IDs and tags and retain the submitting token for audit. - -Revoking one token blocks that key. Suspending a namespace blocks every -token in it. Resuming a namespace does not restore individually revoked -keys. Neither action cancels accepted work; an administrator stops jobs explicitly. -Each proof app's `max_queued` and each batch's resource limits also apply. Each namespace has a shared -GPU/CPU quota, described below. - -### Workload ownership - -Every app belongs to an existing namespace. An administrator creates the namespace -and issues its owner token for the built-in `scope=default` tag. The owner can then -create apps and batches, manage their metadata and submit jobs -inside that namespace and tag: +| `kind`: `app` by default, `ziren-gpu` by default for a prover, or `ziren-cpu` | Every app | +| `image` (required), `replicas`, `gpus`, `cpus`, `shm_gb`, `memory_gb`, `env`, `volumes`, `ready_timeout_s`, `stop_timeout_s` | Every app | +| `command`, `port_base` (required; replica i serves on `port_base` + i), `health_path`, `workdir`, `user` | A `kind: app` container | +| `proof_storage_root`, `backend`, `rust_log` | A prover | + +A prover also takes its program, `--elf-id ID` or `--elf FILE`, and a placement +policy, sent as `proof`: + +- `--class latency` places its jobs on the biggest free prover first, and `batch` + (the default) on the smallest. +- `--anchor` is `none` (default), `shared` or `exclusive`. +- `--backend` is `cuda`, `cpu` or `mock`, repeatable (default any). +- `--zkm-version` is optional. +- `--max-queued` (default 64) caps the app's queued and running proofs; more are + refused. + +Only a prover takes a program, and proof policy without one is refused. A proof runs +only on a prover bound to its app. A namespace has one proof app per program, so +namespaces prove the same program independently, and disabling or deleting one leaves +the others and their jobs intact. In the API: +`POST /apps?namespace=alice` with +`{"name":"eth","spec":{"image":"IMAGE","gpus":1},"proof":{"elf_id":"FULL_ELF_SHA256"}}`. + +**Placement.** An app is created disabled and on no node. Its first `app enable` +places it; users never choose the node. Candidates are live, ready, undrained nodes +whose total capacity holds the GPUs and CPUs of all its replicas. The scheduler +prefers a node where they're free now, and among those the one with the fewest free +GPUs; otherwise it takes the one with the most free GPUs, where the app waits for +room. With no node big enough, enabling answers 503. + +The chosen node gets a `create`, which writes the entry into its `apps.yaml`, +disabled. Once the node reports it, within 120 seconds, the scheduler enables it and +admits it against the namespace quota. A failed create clears the placement, so the +next enable places the app again. If the node hasn't reported the entry in time, +enabling answers 503, and enabling again waits for the same create. An app that a +node's `apps.yaml` already runs, such as an integrated worker's prover bound with +`app: NAME`, isn't placed again. + +**Moves.** Each start of an app the scheduler placed is placed again when it leaves +the queue: on its own node when that has room, else on the node that fits it best +now. That node gets a `create` with the app's entry, disabled. Once it reports the +entry, the old node removes its own (its configuration and leftover containers and +processes, never the app's data), which frees the room there, and the app starts on +the new node. A node whose create fails isn't picked again for that app. With no room +anywhere, the start preempts lower-priority work on its own node and keeps the room +that frees. A start whose entry is anchored with `machine` never moves; every +integrated worker's entry is. It waits for room on its node, preempting +lower-priority batch jobs of its namespace when it can. Fleet copies nothing an app +keeps on a node's disks: the app looks after its own storage. + +**States.** An app is `configured`, `starting`, `running`, `draining`, `disabled`, +`offline` or `failed`; a proof app without provers is `registered`. The table shows +total configured GPUs and CPUs, and every tag when an app's entries use several; the +details show replica settings and job counts. + +**Enabling and disabling.** Enabling starts the replicas before the app takes work. +Disabling drains and stops them, then releases their reservation; durable app data +stays. Disabling a proof app blocks new submissions, drains its active proofs and +stops its GPU server; queued proofs wait for the next enable, and its retained proofs +stay fetchable. + +**Changing.** `fleetctl app scale` sets an app's GPUs, CPUs and replicas within its +quotas ([operations](operations.md#configure-resources)); an active proof can't be +resized ([issue #3](https://github.com/ProjectZKM/zkm-fleet/issues/3)). +`fleetctl app update NAME` (`PUT /apps/NAME/settings`) changes the image, +environment, command, user, workdir, volumes, shm and memory sizes, health path or +timeouts. The image, environment, command, user, workdir and volumes need an +administrator, since an app can reach host networking, storage and prover +credentials; an update mixing them with other fields is refused whole. Changing +priority alone doesn't restart replicas. + +**Removing.** A proof app can't be removed while any of its proofs is queued or +running. Removing it deletes its finished proof records and control-host artifacts; +the app's own storage stays with its owner. + +**Node entries.** Every `apps.yaml` entry has an `app`. A `ziren-gpu` entry binds its +server to the one app it serves with `namespace`, `app: NAME` and +`elf_id: FULL_ELF_SHA256`; a `ziren-cpu` entry does the same, with `elf_id` optional. +Don't bind one Ziren server to several apps. An entry the scheduler creates takes the +app's name. Any other entry (`kind: app`) is its own app, takes no `elf_id`, and its +`app` defaults to the entry name. Provers and workers get `FLEET_APP`, +`FLEET_APP_NAMESPACE` and `FLEET_APP_ELF_ID`; `fleet-ziren-node` takes `--app` / +`FLEET_APP`. An app's ports are private: reaching its own protocol from outside needs +an authenticated proxy. + +**Records.** An app record has `name`, `namespace`, `tag`, `tags`, `state`, `detail`, +`priority`, `disabled`, `resources`, `namespace_quota`, `tag_quota`, `jobs` (`total`, +`active`, `succeeded`, `failed`, `last_at`) and `spec`: its container settings, +including `kind`, and `workers`, the nodes it runs on. A record has no top-level +`kind`. A proof app also has `proof`: `elf_id`, its placement policy, +`mean_prove_ms`, `anchored_on`, `created_at`, and `image`, its provers' image (a list +while they differ). A proof app created before `spec` existed, and proved by provers +its nodes configure, has only `proof`. A refused proof request stays a job record and +creates no app. + +### Batches + +A batch is a reusable image and resource policy; each submission runs one batch job. ```bash -fleetctl namespace create alice --gpus 2 --cpus 16 \ - --endpoint http://127.0.0.1:51012 --out alice.env -fleetctl --token-file alice.token --namespace alice --tag scope=default batch add training --image IMAGE -fleetctl --token-file alice.token --namespace alice --tag scope=default app add eth --spec prover.json --elf-id FULL_ELF_SHA256 +fleetctl --namespace NAMESPACE batch add NAME --image IMAGE +fleetctl batch submit NAME --gpus N --cpus N -- COMMAND +fleetctl batch # the batches, with their active and last jobs +fleetctl batch NAME # one batch and its newest jobs ``` -Owner tokens hold `read`, `submit` and `manage`. Metadata changes still enforce -the token's namespace, tag and any explicit app grants; changing a proof app also -needs the token's tag to cover every node entry running it. Tokens without `manage` -can submit and inspect permitted jobs and read granted scaling status; they -cannot change workload metadata. +`fleetctl batch logs`, `cancel`, `preempt` and `purge` take a job ID and find its +batch; `--batch NAME` skips that lookup. `batch submit --name` names the job. Nodes +can also advertise batches in their `batches.yaml`. + +**Settings.** A new batch takes `name`, `namespace`, `image` (required) and `user`, +plus these settings, shown with their defaults: `max_gpus` 8, `default_gpu_mem_gb` 0, +`shm_gb` 16, `priority` 0, `allow` `[]`, `env` `{}`, `disabled` false, `preemptible` +true, `min_runtime_s` 300 and `stop_timeout_s` 30. Other fields are refused. +`fleetctl batch update NAME --image ...` (`PUT /batches/NAME/settings`) changes them, +except `priority` and `disabled`, which have their own commands; +accepted jobs keep the settings they were submitted with. Only administrators set +`allow` (the host paths jobs may mount) and `user`, so an owner's batch starts with no +host mounts and the image's default user. + +**Records.** A batch record (`type: batch`) has an app record's envelope. Its `state` +is `enabled`, `disabled` or `deleted`, its `jobs` count the batch's job records, and +its policy is under `batch`: `image`, `user`, `max_gpus`, `default_gpu_mem_gb`, +`shm_gb`, `allow`, `env`, `preemptible`, `min_runtime_s`, `stop_timeout_s` and +`workers`. + +In the API, `POST /batches?namespace=alice` takes the batch JSON, or the namespace as +its `namespace` field; a name or an ID is accepted, and the ID is stored. A batch or +`apps.yaml` entry configured on a node may name `namespace: alice`, which is checked +before launch. Disabling a batch blocks new submissions and lets accepted jobs +finish. A deleted batch stays listed, as `deleted`, and readable while it has job +records; its jobs keep their own lifecycle until they're purged. + +**Submissions.** A job takes `command`, a non-empty list of strings, and optionally: -| Action | Required access | -|---|---| -| Namespace creation, token issuance, namespace/tag quotas and node administration | `admin` | -| Namespace listing and quota/usage inspection | `read` for its own namespace and tag; administrators can inspect all | -| App creation, removal, priority and enable/disable | `manage` in the namespace, with any explicit app grants | -| Batch creation, priority, enable/disable and removal | `manage` for the namespace, tag and batch | -| Batch host mount roots and runtime user | `admin` | -| App `user`, `volumes` and `proof_storage_root` at creation | `admin` | -| App image, environment, command, user, workdir and volumes changes | `admin` | -| App GPU/CPU and replica changes | `manage` for the app; namespace/tag quotas and node capacity still apply | -| Job and log inspection | `read` within its scope | -| Job submission and batch job cancellation | `submit` within its scope | -| Batch job preemption | `admin` | - -Batch image, environment and resource policy are supplied when -registering a batch and changed with `fleetctl batch update NAME --image ...` -(`PUT /batches/NAME/settings`). Accepted jobs keep the settings they were -submitted with; new jobs use the new ones. Only administrators may set batch -`allow` roots or `user`, at creation or on update. Owner-created batches start -without host mounts and use the image's default user until an administrator -configures one. Apps start from their creation spec or a node's -`apps.yaml`; owners manage their lifecycle, priority, allocation and -settings through the control plane. `fleetctl app update NAME --image ... --env K=V` -(`PUT /apps/NAME/settings`) changes an app's server or prover image, -environment, command, user, workdir, volumes, shm/memory size, health path or -timeouts. Changing image, environment, command, user, workdir or volumes requires -administrator credentials because apps can reach host networking, storage -and prover credentials. Scoped managers retain resource-size, health-path and -timeout updates. Mixed updates containing an administrator-only field are -rejected without applying any fields. - -Execution nodes independently reject batch mounts overlapping Fleet state, -device, kernel, runtime or container-daemon directories, and reject direct -socket, FIFO and device mounts, even if an older accepted policy allows them. -Administrators must approve storage dedicated to the namespace; an approved -directory grants access to its contents. - -The scheduler is the source of truth for an app's configuration and tells each -node with one app-level node operation, `NODE_OPERATION_APP` (7): it addresses an app -(`{"app": {namespace, namespace_name, name}}`: its namespace by id and name, plus the -app's name; the node finds its `apps.yaml` entries that run it), a batch -(`{"batch": {name}}`) or the one entry a change of an app stopped (`{"entry": {name}}`), -and either removes it (`remove`), writes its configuration into `apps.yaml` or -`batches.yaml` (`set`): enabled state, resources and settings, or, for an app being -placed, adds its entry disabled (`create`, with the entry). Enabling, -disabling, scaling and updating an app all run the same way: its replicas drain -and stop, its node writes the new configuration, and they start again unless the app -is now disabled. `NODE_OPERATION_SET_APP` (4), with `{"app": entry, "disabled", -"scale_id", "scale_phase"}` (or `"scale_cleanup"`), only starts or stops an entry's -replicas as configured, for that sequence and when a node's replicas stop -matching it. The scheduler records each entry on each node as an `AppDeployment` -keyed `{node}-{entry name}`; its reservation's allocation id is `deployment:{key}`. -`fleetctl app logs` asks the node for an entry's log with a `LogRequest` naming that -`entry`. Priority is a scheduling parameter within the namespace and stays in -the scheduler. - -Owner metadata routes use the same authenticated API as the CLI. Every app has -the same routes: - -| Workload | Routes | +| Field | Meaning | |---|---| -| Apps | `GET /apps`, `POST /apps` (create an app from its container `spec`, a prover also with its `proof`), `GET` / `DELETE /apps/NAME`, `PUT /apps/NAME/priority` with `{"priority":N}`, `PUT` / `DELETE /apps/NAME/disabled`, `PUT /apps/NAME/settings`, `GET /apps/NAME/logs` | -| Allocation and status | `PUT` / `GET /apps/NAME/scale` | -| Proof app jobs | `GET /apps/NAME/jobs?limit=N&active=true`, `POST /apps/NAME/jobs` (app-storage references), `GET /apps/NAME/jobs/ID`, `GET /apps/NAME/jobs/ID/output/proof.json` and `.../output/public_values.bin` | -| Batches | `GET` / `POST /batches`, `GET` / `DELETE /batches/NAME`, `PUT /batches/NAME/priority`, `PUT /batches/NAME/settings`, `PUT` / `DELETE /batches/NAME/disabled` | -| Batch jobs | `GET` / `POST /batches/NAME/jobs`, `DELETE /batches/NAME/jobs?older_than=DAYS`, `GET` / `DELETE /batches/NAME/jobs/ID`, `POST /batches/NAME/jobs/ID/cancel`, `POST /batches/NAME/jobs/ID/preempt`, `GET /batches/NAME/jobs/ID/logs` | - -Every API route needs a token permission. `GET /namespaces[/NAME]` (own -namespace), `GET /apps...` and `GET /batches...` need `read`. `POST /apps/NAME/jobs`, -`POST /batches/NAME/jobs` and `POST /batches/NAME/jobs/ID/cancel` need `submit`; -`.../preempt` needs `admin`. Any other `/apps` or `/batches` change (create, -delete, priority, disabled, scale, settings, purge) needs `manage`; the settings -handler additionally requires `admin` for the execution fields listed above. Everything -else (status, namespace creation, quotas and suspension, allocations, the -preemption plan, tokens and node drain) needs `admin`. `/healthz` and -`/{proof_id}/wrap/public_values.bin` need no token, `/access` takes any valid -token and `/internal/*` the cluster token. An administrator selects a namespace -with `?namespace=`. Disabling a batch blocks new submissions and lets -accepted jobs continue; disabling an app drains its replicas before releasing -their reservations. A deleted batch stays listed with state `deleted`, and readable, -while it still has job records; its jobs keep their own lifecycle until purged. - -App and batch creation infer the namespace from a verified user token; -administrator requests must supply it. An unknown namespace is not automatically -created. App names are unique across the fleet; each program has one proof app -per namespace, so namespaces can prove the same program independently. Disabling or deleting one -app leaves the others and their jobs intact. Batch names remain unique within the control plane; each workload has one namespace. -App and batch grants are explicit; a grant from another namespace cannot authorize submission. Admin batch submissions -also inherit the app's namespace and participate in its quota. - -The app creation API uses `POST /apps?namespace=alice` with -`{"name":"eth","spec":{"image":"IMAGE","gpus":1},"proof":{"elf_id":"FULL_ELF_SHA256"}}`, -optional `priority`, and in `proof` optional `class`, `anchor`, `backends`, `zkm_version` -and `max_queued`; a container app has no `proof` and a `spec` with `port_base`. -The control-plane batch API accepts `POST /batches?namespace=alice` with the batch JSON, or a -`namespace` field in the JSON. Names and immutable namespace IDs are accepted; -registrations persist the ID. A configured batch or `apps.yaml` -entry can also specify `namespace: alice`; its namespace is verified before launch. -The owner token is bound to `scope=default`; jobs inherit that tag. Batch owners -prepare inputs in storage and include the code and storage client in their image. -Administrators configure host mount roots and runtime users for batches. Submit batch jobs through -`fleetctl batch submit NAME` or `POST /batches/NAME/jobs`; job records name their -`batch`, and their reported type is `batch`. `GET /batches/NAME/jobs/ID` (`fleetctl job ID`) -also returns `request`, the submission as accepted: its name, command, gpus, cpus, -gpu_mem, mounts, env and nodes. Listings leave it out. - -### GPU and CPU quotas - -A namespace quota caps **concurrent GPU/CPU units** held by proof jobs, one-shot -batch jobs and app replicas assigned to that namespace. Reservations from -all tags and tokens share this cap. An optional tag can cap its own reservations. -Admission must fit both caps; tag caps do not -reserve a guaranteed share of fleet capacity. Neither quota has a duration, expiry, -hourly budget or periodic reset. Jobs and enabled apps can keep their reservations -indefinitely; normal job lifecycle rules and existing application timeouts still apply. +| `gpus`, `cpus` | Default 1 and 8. GPUs are at most the batch's `max_gpus`, and whole when more than one; CPUs are at least 1. | +| `gpu_mem` | GiB of free VRAM each granted card must have before the job starts. Defaults to the batch's `default_gpu_mem_gb`; 0 means the whole card, so a fractional GPU needs one. | +| `nodes` | Must be 1: a batch job runs on one machine | +| `machine` | A list with at most one machine to run on | +| `mount` | `PATH[:MODE]` entries, inside the batch's `allow` | +| `env`, `workdir`, `name` | Its environment, working directory and job name | +| `after` | A job that must succeed first; if it doesn't, this one fails | +| `retry` | How many times to rerun it after a failure (default 0) | +| `restart_on_oom` | How many times to rerun it after it runs out of memory, spent before `retry` (default 0) | +| `vram_wait` | Seconds to wait for its VRAM to be free (default 1800) | +| `log` | A log file of its own, for administrators only | + +Other fields are refused. `GET /batches/NAME/jobs/ID` (`fleetctl job ID`) also +returns `request`, the submission as accepted; listings leave it out. + +**Storage.** A batch job's inputs, outputs and checkpoints live in its owner's shared +or object storage. The command reads and writes them directly, with its own client or +a mount the batch allows; Fleet passes its arguments and environment and moves no +data. Each job gets a workspace on its node, `FLEET_JOBS_DIR/JOB_ID/`, with +`FLEET_OUT` as its output directory, and its logs sync to the control host, so a +one-machine job needs no shared mount. Job IDs and directories are flat; namespace +names never become host paths. ```bash -fleetctl namespace create alice --gpus 2 --cpus 16 \ - --endpoint http://127.0.0.1:51012 --out alice.env -fleetctl namespace quota alice --gpus 4 --cpus 32 -fleetctl namespace get alice # quota and currently reserved units -fleetctl namespace quota alice # clear both caps: unlimited (a write operation) +fleetctl --token-file alice.token batch submit training --gpus 1 \ + --env INPUT_URI=s3://alice-inputs/dataset \ + --env OUTPUT_URI=s3://alice-results/job-output -- python /app/train.py ``` -Each `quota` command replaces both caps. An omitted cap is unlimited; zero -prohibits allocation of that resource. For example `--gpus 0 --cpus 8` permits -CPU-only batch jobs. Fractional GPU units are supported and retain the existing -per-batch VRAM requirements for shared cards. New namespaces -start with unlimited caps unless the administrator sets them. - -Use the publicly reachable TLS endpoint in bundles shared with users; `127.0.0.1` -would refer to the user's own machine. Set `--server-ca` only when that endpoint -uses a private CA. - -Batch jobs requesting more than either cap receive HTTP 422 before creation. The Rust -scheduler selects a ready machine with sufficient reported capacity, reserves -namespace and tag units, then sends a launch command. The node creates the local -Ray placement. If capacity or either quota is occupied, the batch job stays -`waiting_quota` without a reservation. Dependencies also wait without occupying -quota. Once assigned, a batch job retains its reservation until execution cleanup is -confirmed, including when the machine disconnects before acknowledging launch. - -The Rust scheduler retries admission while capacity and quota are occupied. -Durable commands and execution leases recover an interrupted launch without -starting a second actor. Node heartbeats import existing local execution records -without relaunching them. Completed jobs remain readable through the control plane -when a node is disconnected. - -An admitted batch job reserves its requested units throughout execution, VRAM waits -and retries. Finishing or cancelling releases its reservation after actors stop and the actual container is -confirmed stopped or absent. An unreachable machine or failed Docker inspection -keeps the reservation and history; cleanup is retried when the machine returns. Preemption releases it and requires quota admission again before restarting. A lost batch job retains its -reservation until cleanup is confirmed so an orphaned container cannot silently free the -namespace's allowance. Cancelling an unreachable batch job does not release its quota. -Actor cleanup may take a few seconds after a terminal status appears. - -Proof placement charges the selected proof replica's GPU and CPU units. -A proof stays queued until an eligible replica fits both remaining quotas. A timed -out or disconnected proof keeps its reservation and rejoins its original prover; -it cannot fail over to another prover while the original execution is uncertain. -The replica's entire allocation must fit; an active proof keeps its allocation. -App owners can use `fleetctl app scale` to adjust the app's GPU/CPU units -between proofs to fit user quotas. App replicas reserve their configured GPU/CPU units -against the namespace and tag they are assigned to. Shared infrastructure -configured in the `system` namespace is accounted there. - -Reservations are stored in Postgres and recovered after a scheduler restart; -refreshing tokens never changes them. If quota admission cannot be recorded, -new batch jobs remain waiting. Reducing a cap lets already-admitted work -finish, including work recovered during an upgrade. Further admissions wait -until usage fits; a waiting batch job whose request exceeds a newly lowered cap fails -with an explanation. Operators can inspect reservations, including each tag, through -`GET /namespaces/alice/allocations`. - -### App priority and preemption - -App owners set workload priority with their owner token. Individual job -submissions inherit this policy and cannot supply a separate priority: +Nodes refuse batch mounts that overlap Fleet state or device, kernel, runtime or +container-daemon directories, and direct socket, FIFO and device mounts, even under +an older accepted policy. An approved directory grants access to everything in it, +so approve only storage dedicated to the namespace. + +### Jobs + +`fleetctl job` lists what the scheduler queues: proofs submitted to a proof app +(`--type app`) and batch jobs (`--type batch`); `--app NAME` selects one app's or +batch's jobs. An app's own work in its replicas isn't a job, so a plain app, or a +proof app whose worker proves on its own GPU server, has none: `fleetctl app` and +`fleetctl app logs APP` show that work. A job takes its workload's namespace, tag and +priority, and its dependencies, logs and changes check both namespace and tag. + +### Priority and preemption + +A job takes its app's or batch's priority when it's submitted (default 0, higher +first) and keeps it through retries, preemption and restarts. A submission can't set +its own, and a workload's new priority applies to new jobs only. Equal priorities run +in submission order (creation time, then ID). Priority doesn't bypass quotas, grants, +dependencies, placement or anchoring, and a blocked job can be skipped so that work +that fits goes ahead. + +A waiting batch job or a starting app may preempt batch jobs of its own namespace +that have a lower priority, belong to a batch that allows it, and have run for their +`min_runtime_s`. The scheduler takes the lowest priorities first, and plans enough +capacity and namespace and tag quota before it stops anything. Equal and higher +priorities, other namespaces, active proofs and running apps are never preempted; +proof priority only orders queued proofs. Every priority defaults to 0, so preemption +needs priorities set explicitly. + +A stop sends SIGTERM, then SIGKILL after `stop_timeout_s`. Reservations last until +every container is confirmed stopped or gone; an unreachable node keeps them, and the +requeue survives a scheduler restart. A preempted job queues again with its ID, +output and priority, without spending a retry. + +Ray provides the placement groups and actor reservations, in a local Ray namespace on +each machine. Fleet enforces namespaces, tags, quotas and preemption, which a Ray +namespace alone doesn't. + +## Proofs + +### SDK protocol + +SDK-driven apps keep the `GenerateProof`/`GetStatus` protocol: + +1. The client submits to its app's own endpoint, which stores the inputs and sends + Fleet only their references, SHA-256 hashes and job parameters. +2. Fleet admits the job and assigns it through node heartbeats. The proof actor reads + the inputs from app storage and proves them on its own GPU server. +3. The node writes the outputs to app storage and reports their references, hashes, + cycles and `prove_ms`. Fleet records the storage machine from the authenticated + node. +4. The client polls its endpoint, which checks the completion references and reads + the proof and public values locally. + +Each proof replica owns one GPU server and runs one proof at a time; a proof never +spans servers. Every eligible replica and the app endpoint must reach the same app +storage path: references are `file://`, with owner-managed shared mounts for +replicas on different machines. Fleet keeps admission, quotas, placement, retries and +execution leases, and stores no proof inputs or outputs. + +Integrated EthProofs owns its whole pipeline (head polling, witness preparation, +loopback GPU proving, durable outputs and reporting) and needs no separate EthProofs +client or SDK endpoint. Its +cursor advances one block at a time, once the block's output and report are +committed. -```bash -fleetctl --token-file alice.token --namespace alice batch add training \ - --image python:3 --allow /srv/training --priority 10 -fleetctl --token-file alice.token batch priority training 20 -fleetctl --token-file alice.token --namespace alice app priority eth 100 -``` +### App-owned proof endpoint -Proof and batch jobs retain the priority inherited when submitted, including across -retries, preemption and restart. Changing the app affects new submissions. Higher -priority jobs are considered first; equal priority pending jobs use FIFO order -(creation time, then ID). Priority does not bypass quotas, app grants, dependencies, -placement constraints or app anchoring. A blocked job can be skipped so work -that fits can proceed. - -Automatic preemption selects only lower-priority batch jobs in the requesting namespace -whose batch permits preemption and whose minimum runtime has elapsed. It prefers -the lowest priority victims and plans enough physical capacity and namespace/tag -quota before stopping any batch job. The Rust scheduler makes this plan from Ray-reported -capacity and batch candidates; the Python adapter executes the selected stop, and -the Rust scheduler requeues the job after confirmed cleanup. Equal or higher priority batch jobs, batch jobs in another -namespace and active proofs are protected. Proof priority orders queued proofs; -it does not resize active proofs or evict running provers. Persistent apps -are also protected from automatic eviction. Their app priority controls which batch jobs they may displace when starting; -changing priority alone does not recreate actors. An app's creation spec or -`apps.yaml` entry provides the initial execution settings. - -Stopping uses SIGTERM followed by SIGKILL after `stop_timeout_s`. Placement and -quota reservations remain until every container is confirmed stopped or absent. -An unreachable worker keeps its reservations, and the durable requeue intent is -recovered after restart. Batch jobs resume with their original ID, output and inherited -priority, without spending a retry. All priorities default to `0`, so automatic -preemption requires explicitly assigning different priorities. - -Ray provides placement groups and actor resource reservations. Fleet enforces -namespace ownership, optional tag isolation, quotas and preemption policy; a Ray -namespace alone does not enforce these limits. Actors on each machine use a local Ray -control namespace, while Fleet namespace IDs scope policy and accounting. - -### Optional tags +Run an endpoint beside the app's storage, apart from its GPU replicas: ```bash -fleetctl namespace create team --gpus 8 --cpus 64 \ - --endpoint http://127.0.0.1:51012 --out team.env -fleetctl namespace quota team --tag scope=research --gpus 2 --cpus 16 -fleetctl namespace quota team --tag scope=production --gpus 6 --cpus 48 -fleetctl namespace get team # namespace quota, usage and tags -fleetctl namespace get team --tag scope=research # namespace totals and tag quota/usage -fleetctl token issue --namespace team --tag scope=research \ - --app training --endpoint http://127.0.0.1:51012 --out alice.env -fleetctl --namespace team --tag scope=research job +fleet-proof-app --scheduler https://scheduler.example.com:8905 \ + --namespace NAMESPACE_ID --elf-id FULL_ELF_SHA256 \ + --storage-root /srv/goat --elf-file /srv/goat/program.elf \ + --token-file /etc/my-app/user.token \ + --stage-addr 127.0.0.1:51012 --http-addr 127.0.0.1:51013 ``` -Both research and production contribute to the team's shared quota. If production -occupies six GPU units and research occupies two, all eight units are in use and -neither can admit another GPU job. Jobs can retain admitted units indefinitely. Lowering -either quota lets admitted work finish and blocks further admissions as needed. +Set `proof_storage_root: /srv/goat` on each matching `ziren-gpu` prover, in its app's +spec or its `apps.yaml` entry, and point the SDK's `ENDPOINT` at the stage port, +`http://127.0.0.1:51012`. + +- **Tokens.** The SDK keeps its signing key. An older SDK that sends no bearer header + uses the endpoint's `--token-file`, which is reread on every request, so rotation + and revocation still apply. A request's own authorization is checked as given, + never replaced by the configured token. +- **Programs.** Repeat `--elf-id` and `--elf-file` for more programs in the + namespace. Each is its own Fleet app: the endpoint finds its name through + `GET /apps` with the caller's token and submits to `/apps/NAME/jobs`. +- **One per app.** Give each SDK-driven app its own storage root, scoped token and + pair of loopback ports. GOAT uses `51012` and `51013`. +- **Results.** A successful status call reads the proof and public values from the + app's local output storage, checking their size, SHA-256 hash and that they're + inside the output directory, with no fetch through Fleet. The prover wakes its + control loop as soon as a proof finishes, so completion doesn't wait for a + heartbeat. +- **Network.** The endpoint binds loopback only; outside access needs an + authenticated HTTPS gateway. Pass `--scheduler-ca-file ca.pem` for an + operator-issued CA. Fleet CLI and MCP clients keep the control-plane URL. +- **Logs.** It logs input storage and admission time separately from the node's + `prove_ms`. Changing the endpoint doesn't enable a disabled app. + +On the node, give the agent `--proof-node-binary /path/to/fleet-ziren-node` and a +`--proof-storage-root /srv/goat` for each approved root. With `deploy/node-start.sh`, +set `FLEET_PROOF_STORAGE_ROOTS_FILE` to an administrator-owned file with one absolute +root per line; `deploy/install.sh node --proof-storage-roots FILE` installs it. Keep +the roots configured while GPU actors are stopped. The agent runs storage-only +fetches apart from its heartbeats and Ray work. + +### Fetching proofs + +`fleetctl job fetch JOB_ID --out DIR` downloads a proof's JSON and public values into +a new directory. Fleet authorizes the request, hands the storage node a short-lived +ticket in its next heartbeat, and relays the node's HTTPS upload in bounded memory; +the control host keeps no proof files. `fleet-ziren-node fetch-artifact` checks the +files against the job, its execution lease and their hashes, without starting a +prover or GPU. A fetch waits for a heartbeat, and an offline storage node returns a +retryable error. Proofs stay fetchable after their prover is disabled; the app owner +decides how long to keep them. + +### Proving time + +`prove_ms` is execution time. For a GPU proof it comes from the GPU server and leaves +out input and output transfers, queueing, setup and fetching; CPU and mock provers +time the local proof call. Wall time is kept separately for diagnostics and isn't +proving time. A GPU server that doesn't report execution time fails the job rather +than fall back to RPC time. The SDK returns the value in `GetStatus.proving_time`, +and reporting clients must keep it, for example with `take_prove_ms()`. EthProofs +takes it from its GPU response, leaving out preparation, queueing and reporting. + +### Proof jobs + +A proof job comes from a namespace token (`Authorization: Bearer`) with `submit`, and +is signed by that token's key: the signature must recover to the token's address. The +scheduler finds the job's app by namespace and ELF ID. `GetStatus` needs a token of +that namespace with `read` or `submit`, and checks the app grant before it waits or +shows anything; the SDK can still poll a refusal of its own signer, including one for +an app it wasn't granted. + +The SDK keeps its own proof ID. The scheduler maps `(namespace_id, tag, +client_proof_id)` to a unique internal ID, a SHA-256 of the three for new jobs, which +nodes, retries and deduplication use; different tags can reuse a client ID. +Resubmitting within a tag returns the existing job without replacing its input, and a +resubmission with different inputs or parameters is refused. -Tags use the same character and length rules as namespace names and are scoped -to their namespace. The `scope=default` tag is created with each namespace. Additional -tags are introduced when an administrator assigns one to a token or sets its -optional cap. A newly used tag has no extra cap unless the administrator sets one. +| Field | Meaning | +|---|---| +| `id` | Internal execution and storage identifier | +| `owner` | Internal proof signer audit identifier, kept when the token is refreshed | +| `namespace` | Immutable namespace ID, for accounting and access | +| `tag` | `key=value` label; `""` for untagged work | +| `client_id` | The SDK's proof ID, unique within namespace and tag | +| `app`, `elf_id`, `kind`, `created_ms` | Submission metadata, kept through retries; `app` is empty for a request refused before its app was known | + +`node` and `started_ms` can be absent while a job is queued, and `finished_ms` until +it ends. Counters start at zero and the failure history empty. A refused request from +an authenticated owner keeps that identity; one without is refused without a record. + +Public-values URLs carry the issuing token's ID and a key, and are bound to that +token: revoking it or suspending its namespace invalidates them, and a replacement +token can poll the job for a fresh URL. + +## Namespaces, tags and tokens + +**Namespaces.** A namespace is the unit of ownership, accounting and billing. An +administrator creates it with an immutable ID and a unique name such as `alice`: a +lowercase letter, then lowercase letters, digits or hyphens, at most 63 characters. +`system` and `admin` are reserved, and namespaces can't yet be renamed or deleted. +Operator-managed apps use `system`, as do a node's configured batches and app entries +unless they name another; no user token can be issued in it. Registrations need an +existing namespace: a missing one isn't created. + +**Tags.** A tag is `KEY=VALUE`, such as `scope=default` or `team=research`. The key +starts with a letter, and each part has up to 63 letters, digits, dots, underscores or +hyphens. Bare tags are refused, and tags match exactly. A tag belongs to its +namespace: `scope=default` comes with each namespace, and others appear when an +administrator assigns one to a token or gives it a cap. Tokens with the same tag share +proofs and batch jobs, which other tags in the namespace can't see; untagged tokens +share the untagged work. A workload created with a tag belongs to it: only that tag's +tokens change it, and only they see or submit to a tagged batch. An administrator's +job on a tagged batch takes the batch's tag. A record's tag is `null` when absent, +except a proof record's, which is `""`. There is no group resource. + +**Tokens.** A namespace token has an optional tag and some of the permissions `read`, +`submit` and `manage` (default `read` and `submit`). A token without `manage` also +needs `--app` grants, by app or batch name. Grants are explicit, and one from another +namespace authorizes nothing. An administrator's token has only `admin`, no namespace, +and may do everything. A token's namespace and tag never change: refreshing copies +them, with its permissions, grants and label, to a new key, which keeps access to +earlier work. Requests take their namespace and tag from the token: a selector must +match them, and a submission body can't set them. An administrator's request names +its namespace when it creates an app or batch. Administrators select with `--namespace NAME --tag TAG` +(`?namespace=NAME&tag=TAG`). Records keep namespace IDs, tags and the submitting +token for audit. Token IDs are opaque, and proof signer bindings aren't exposed. + +Revoking a token blocks that key, and suspending a namespace blocks all of its +tokens; resuming it doesn't restore individually revoked ones. Neither cancels +accepted work: an administrator stops jobs explicitly. + +### Permissions + +| Action | Needs | +|---|---| +| Read the own namespace, apps, batches, jobs, logs and scaling status | `read`, within the token's scope | +| Submit proofs and batch jobs; cancel batch jobs | `submit`, within the token's scope | +| Create, remove, enable, disable, prioritize, scale or update apps and batches; purge batch jobs | `manage`, with any app grants. Removing, enabling, disabling or prioritizing a proof app also needs the tag it was created for and the tag of every node entry running it; scaling and updating it need the tag of the entry they change. | +| Batch `allow` and `user`; app `user`, `volumes` and `proof_storage_root`; changing an app's image, environment, command, user, workdir or volumes | `admin` | +| Preempt batch jobs | `admin` | +| Namespaces, tokens, quotas, suspension, allocations, the preemption plan, nodes and status | `admin` | -Membership cannot change for an existing key. Refresh copies the original tag, -independently of CLI selectors. Namespace owner bundles use `FLEET_TAG=scope=default`. -Users cannot select another tag or spoof it in a submission body. +`/healthz` and `/{proof_id}/wrap/public_values.bin` need no token, `/access` takes any +valid token, and `/internal/*` takes the cluster token. ### Administrator workflow @@ -552,244 +515,189 @@ fleetctl namespace create alice --gpus 2 --cpus 16 \ --endpoint http://127.0.0.1:51012 --out alice.env fleetctl namespace list fleetctl namespace get alice -# Give alice.env, alice.token, and alice.ca.pem (if generated) to the owner securely. -FLEET_TOKEN_FILE=alice.token FLEET_NAMESPACE=alice FLEET_TAG=scope=default \ - fleetctl batch add training --image APPROVED_IMAGE - -# Copies the namespace, optional tag, permissions, app grants and label to a new key/token. -fleetctl token refresh OLD_TOKEN_ID \ - --endpoint http://127.0.0.1:51012 --out alice-next.env -# After the user verifies the replacement: +# Give alice.env, alice.token and alice.ca.pem (if written) to the owner securely. + +# A token for another tag, limited to one batch: +fleetctl token issue --namespace alice --tag team=research --app training \ + --endpoint http://127.0.0.1:51012 --out research.env + +# Move a user to a new key, then revoke the old one once they have switched: +fleetctl token refresh OLD_TOKEN_ID --endpoint http://127.0.0.1:51012 --out alice-next.env fleetctl token revoke OLD_TOKEN_ID fleetctl namespace suspend alice fleetctl namespace resume alice ``` -Namespace creation takes the SDK endpoint the owner's clients use, their -[app-owned proof endpoint](#app-owned-proof-endpoint), for the bundle's `ENDPOINT`. `refresh` does not revoke -the old key automatically or overwrite an existing bundle. +The owner bundle is for `scope=default` (`FLEET_TAG=scope=default`) with `read`, +`submit` and `manage`. `--endpoint` is the bundle's SDK `ENDPOINT`, the owner's +[app-owned proof endpoint](#app-owned-proof-endpoint). The bundle's Fleet URL must be +the public TLS address, since `127.0.0.1` would be the user's own machine; add +`--server-ca` only when it uses a private CA. Issuing needs an existing, active +namespace, so a typo can't create an owner. `refresh` doesn't revoke the old key or +overwrite an existing bundle. The owner then creates workloads: -Issuance requires an existing, active namespace. A typo cannot silently create a -new owner. Token namespace and optional tag cannot be changed after issuance. +```bash +fleetctl --token-file alice.token --namespace alice --tag scope=default batch add training --image IMAGE +fleetctl --token-file alice.token --namespace alice --tag scope=default app add eth --spec prover.json --elf-id FULL_ELF_SHA256 +``` -Namespace and access HTTP endpoints, on the existing private scheduler listener. -Writes require administrator credentials. Scoped tokens can read their own -namespace by name or immutable ID; lists return only that namespace. Shared -namespace quota and usage are visible, with tag caps and usage restricted to the -token's tag. Another namespace or tag selector is refused: +## Quotas -| Method and path | Action | -|---|---| -| `POST /namespaces` with `{"name":"alice"}` | Create | -| `GET /tokens` | List opaque token IDs, permissions and access scopes | -| `POST /tokens` | Issue an access token (administrator only) | -| `DELETE /tokens/TOKEN_ID` | Revoke a token | -| `GET /namespaces` | Scoped users: own namespace; administrators: all | -| `GET /namespaces/alice` | Scoped users: own namespace quota, total usage and own tag; administrators: all tags | -| `PUT /namespaces/alice/quota` with `{"gpus":2,"cpus":16}` | Replace caps | -| `GET /namespaces/alice/allocations` | Administrator-only: inspect reservations across tags | -| `GET /namespaces/alice?tag=scope=research` | Inspect namespace totals and tag caps/usage; scoped tokens must match that namespace/tag | -| `PUT /namespaces/alice/quota?tag=scope=research` with `{"gpus":2,"cpus":16}` | Replace tag caps, recording a new tag if needed | -| `PUT /namespaces/alice/suspended` | Suspend | -| `DELETE /namespaces/alice/suspended` | Resume | - -Token registration accepts a namespace name or ID, an optional `tag` and its `permissions`. Tokens have opaque IDs; proof signer bindings are internal and are not exposed through the token API. -Job/app lists and job reads accept `?namespace=alice&tag=scope=research`; the batch API supports the same selectors. Refresh -is a CLI operation using administrator token lookup and registration. -No additional listener or public port is required. - -### Job fields and storage - -Every proof job comes from a namespace token (`Authorization: Bearer`) holding -`submit` and is signed by that token's key; the signature must recover to the -token's address. The scheduler finds the job's app by namespace and ELF ID. -`GetStatus` needs a token of that namespace with `read` or `submit`. -The SDK signs and receives its original proof ID. The scheduler resolves -`(namespace_id, optional_tag, client_proof_id)` to a unique internal ID, derived from their -SHA-256 hash for new jobs. Nodes, retries and deduplication use that -internal ID. Different tags can submit the same client proof ID independently. -Resubmission within a tag returns the existing job without replacing input. Reference submissions with conflicting inputs or proof parameters are rejected. - -Batch job IDs and directories retain their existing flat layout, with namespace -and tag ownership recorded in each batch spec. Namespace names are never host paths. -App owners prepare batch inputs in shared or object storage and manage access to -that storage. The app reads inputs directly using its own storage client or a -mount permitted by the owner's batch. Fleet passes command arguments -and environment variables to the app. The app writes outputs and checkpoints directly to its owner's -storage, including any files staged in its local `{out}` workspace. Fleet does not -transfer batch inputs or outputs. Dependencies, logs and mutations check both namespace -and tag membership. -A verified replacement token can read accepted batch jobs from earlier keys. - -For a batch command that uses `INPUT_URI` and `OUTPUT_URI` from its environment: +A namespace quota caps the **concurrent** GPU and CPU units held by its proofs, +batch jobs and app replicas, across all its tags and tokens; a tag can have its own +cap as well, and admission must fit both. A tag cap reserves no share of the fleet, +and no quota expires, resets or counts time: work keeps its reservation for as long +as it runs. ```bash -fleetctl --token-file alice.token batch submit training --gpus 1 \ - --env INPUT_URI=s3://alice-inputs/dataset \ - --env OUTPUT_URI=s3://alice-results/job-output -- python /app/train.py +fleetctl namespace quota alice --gpus 4 --cpus 32 # replace both caps +fleetctl namespace quota alice --tag team=research --gpus 2 --cpus 16 +fleetctl namespace get alice # caps and current use +fleetctl namespace get alice --tag team=research +fleetctl namespace quota alice # clear both caps ``` -Proof job identity is required and immutable: +Each `quota` command replaces both caps: an omitted cap is unlimited, and zero allows +none (`--gpus 0 --cpus 8` permits CPU-only jobs). New namespaces start unlimited. +Fractional GPUs are supported, with each job's VRAM requirement (`gpu_mem`) for shared +cards. With a +team cap of 8 GPUs, production holding 6 and research 2, neither can admit another +GPU job. + +**Batch jobs.** A job asking for more than either cap gets 422 before it's created. +Otherwise the scheduler picks a ready node with enough reported capacity, reserves +namespace and tag units and sends the launch, and the node creates the Ray placement. +While capacity or quota is short, the job waits as `waiting_quota` without a +reservation, as does a job waiting on a dependency. Once assigned, a job holds its +reservation through execution, VRAM waits and retries until its cleanup is confirmed, +even if the node disconnects before acknowledging the launch. Ending or cancelling +releases it once the actors and container are confirmed stopped or gone, which can +take a few seconds after the final status. An unreachable node or a failed Docker +inspection keeps it until cleanup succeeds, so an orphaned container can't free quota; +cancelling a job on an unreachable node doesn't release it either. Preemption releases +it, and the job needs admission again to restart. + +**Proofs.** A proof is charged its prover replica's whole GPU and CPU allocation, and +stays queued until a replica fits both caps. A timed-out or disconnected proof keeps +its reservation and rejoins the same prover; it can't fail over while that execution +is uncertain. Owners can `fleetctl app scale` a prover between proofs to fit their +quota. + +**Apps.** A replica reserves its configured units against its namespace and tag for +as long as it's enabled; disabling or removing it releases them once it stops. Shared +infrastructure in the `system` namespace is counted there. + +Reservations are in Postgres and survive restarts; refreshing tokens doesn't touch +them. If an admission can't be recorded, new batch jobs wait. Lowering a cap lets +admitted work finish, including work recovered during an upgrade, and holds new +admissions until use fits; a waiting batch job +that no longer fits fails with a reason. Administrators see the reservations, per tag, +at `GET /namespaces/alice/allocations`. + +## HTTP API + +`fleetctl` and MCP use the same authenticated API, served on the scheduler's private +listener behind the HTTPS gateway. Administrators select a namespace with +`?namespace=`, and listings and job reads also take `&tag=`. -| Field | Meaning | +| Workload | Routes | |---|---| -| `id` | Internal execution and storage identifier. | -| `owner` | Internal proof signer audit identifier, retained when an access token is refreshed. | -| `namespace` | Immutable namespace ID controlling resource accounting and access. | -| `tag` | `key=value` isolation label; proof records use `""` for untagged work. | -| `client_id` | Original SDK proof ID, unique within namespace and tag. | -| `app`, `elf_id`, `kind`, `created_ms` | Submission metadata retained throughout retries; `app` is empty for a request refused before its app was known. | - -`node` and `started_ms` can be absent while queued. `finished_ms` is absent -until the job succeeds or fails. Counters start at zero and failure history starts -empty. Rejected authenticated proof requests retain their submitting identity; -requests without an authenticated owner are refused without creating a job. -Token tags and batch tags remain optional (`null` when absent). - -Public-values URLs carry the issuing token's ID and a key, and are bound to that token. -Revoking that key or suspending its namespace invalidates its URLs. A replacement -token can poll the same job to obtain a fresh URL. - -Postgres creates schema 16 directly and checks its version at startup. A schema 14 -or 15 database migrates automatically at start, in one transaction: each app takes a -name (its tenant's when that tenant had one app in the namespace, else -`TENANT-`) and the default placement policy; jobs name -their app; `TENANT/ELF` grants become app names; namespace-less tenant signer -tokens are removed. Other schemas require an explicit metadata reset. -Tags must use `key=value`. A reset removes job history and access registrations; -recreate namespaces and tokens, and preserve app-owned proofs and checkpoints. -Operator-managed apps use the reserved `system` namespace. User tokens -cannot be issued in `system`. - -Configured batches and persistent app entries default to `system`. -Administrators assign a user app's namespace explicitly; API registrations require -an existing namespace. Namespace quotas govern proof and batch jobs as well as -app replica reservations. Disabling an app releases its reservation -after the actor stops. - -### MCP clients - -Share these connection details with external MCP client users: +| Apps | `GET` / `POST /apps`, `GET` / `DELETE /apps/NAME`, `PUT /apps/NAME/priority` with `{"priority":N}`, `PUT` / `DELETE /apps/NAME/disabled`, `PUT /apps/NAME/settings`, `GET /apps/NAME/logs` | +| Scaling | `PUT` / `GET /apps/NAME/scale` | +| Proof app jobs | `GET /apps/NAME/jobs?limit=N&active=true`, `POST /apps/NAME/jobs` (app-storage references), `GET /apps/NAME/jobs/ID`, `GET /apps/NAME/jobs/ID/output/proof.json` and `.../output/public_values.bin` | +| Batches | `GET` / `POST /batches`, `GET` / `DELETE /batches/NAME`, `PUT /batches/NAME/priority`, `PUT /batches/NAME/settings`, `PUT` / `DELETE /batches/NAME/disabled` | +| Batch jobs | `GET` / `POST /batches/NAME/jobs`, `DELETE /batches/NAME/jobs?older_than=DAYS`, `GET` / `DELETE /batches/NAME/jobs/ID`, `POST /batches/NAME/jobs/ID/cancel`, `POST /batches/NAME/jobs/ID/preempt`, `GET /batches/NAME/jobs/ID/logs` | + +| Namespaces and tokens | Action | +|---|---| +| `GET /namespaces` | A scoped token's own namespace; every namespace for an administrator | +| `GET /namespaces/alice[?tag=TAG]` | Quota and use, with each tag's caps and use, or one tag's with `?tag=` | +| `POST /namespaces` with `{"name":"alice"}` | Create | +| `PUT /namespaces/alice/quota[?tag=TAG]` with `{"gpus":2,"cpus":16}` | Replace the namespace's or a tag's caps, recording a new tag if needed | +| `PUT` / `DELETE /namespaces/alice/suspended` | Suspend / resume | +| `GET /namespaces/alice/allocations` | Reservations across tags | +| `GET` / `POST /tokens`, `DELETE /tokens/TOKEN_ID` | List token IDs, permissions and scopes; issue; revoke | + +A scoped token reads only its own namespace, by name or ID, and only its own tag's +caps and use; another namespace or tag selector is refused. Token registration takes +a namespace name or ID, an optional `tag` and its `permissions`; refreshing is a CLI +operation built on lookup and registration. + +## MCP + +Give external MCP users these connection details: - **Server:** ZKM Fleet - **Transport:** Streamable HTTP over HTTPS - **URL:** `https://34.215.238.232:8905/mcp` -- **Authentication:** HTTP Bearer token in the `Authorization` header - -For clients with a custom-header field, set `Authorization` to `Bearer `. -An administrator issues each user a namespace/tag-scoped token and delivers it -securely. Do not give external users the administrator token. - -Initialization and `tools/list` are available without a token, so clients can -discover the available tools and input schemas before adding a token. `tools/call` -requires a current user token; the scheduler applies its namespace, tag and -app grants. Proof submissions must also be signed by the token's key. MCP has no -administrator token. Clients must support remote -Streamable HTTP and allow the Bearer header to be configured. Clients that only -support local `stdio` servers cannot connect directly to this endpoint. - -The server lists long-running apps with `fleet_apps_list` and `fleet_app_get`, and -`fleet_app_create` creates one as `fleetctl app add` does: a `spec` object and optional -`priority`, plus a prover's `elf_id`, `class`, `anchor`, `backends`, `zkm_version` and -`max_queued`. -Batch tools are `fleet_batch_create`, `fleet_batches_list`, -`fleet_batch_get`, `fleet_batch_jobs_list`, `fleet_batch_job_get`, -`fleet_batch_job_submit`, `fleet_batch_job_logs` and `fleet_batch_job_cancel`. `fleet_app_logs` reads a visible app's log from its node. Batch job submission -uses a `batch` name and a command; the jobs list, logs and cancel tools take an optional `batch`. `fleet_jobs_list` remains a combined execution -view with `kind: "batch"` or `kind: "app"`. Batch creation requires a -workload-management grant. Its `priority` is owner-managed; nonempty `allow` -roots require an administrator. -`fleet_batches_list` and `fleet_batch_get` use `fleetctl batch` and `fleetctl batch NAME`; `fleet_batch_jobs_list` and `fleet_batch_job_get` use `fleetctl job --type batch` and `fleetctl job ID`. -`fleet_batch_job_submit` accepts a `mounts` array such as `["/data/models:ro"]`; -each path must be inside the batch's allowed roots. Existing metadata changes, -enabling and app scaling use the CLI or HTTP API; MCP currently exposes app and batch -creation and job operations. Proof jobs remain submitted through the Ziren network SDK; -MCP can list and inspect their records. - -Tokens with `manage` manage app and batch -metadata within their namespace, tag and app grants: -creation, priority, enable/disable and removal. Owners can -scale their apps within namespace and tag quotas, and submit -batch jobs in their tag. SDK proof submissions are signed by the token's key. -Tokens without `manage` can use their granted apps but cannot change their metadata. -Administrators manage namespace and tag quotas, namespace access, token issuance, -batch host storage and runtime users, and app execution settings. -App removal refuses while any proof is queued or running and deletes that app's -terminal proof records and control-host artifacts. App-owned storage retention remains with the app owner. Administrators can also issue tokens limited -to approved apps and a selected namespace/tag. Token refresh preserves access to jobs. -Namespaces are the resource accounting and billing unit; namespace and optional tag quotas -cap concurrent GPU/CPU units across proofs, batch jobs, and app replicas, with no quota time limit. An app replica reserves its configured resources for as long as it is enabled; disabling or removing it releases that reservation after the actor stops. - - -Model servers, ComfyUI and provers all appear as `type: app` in `fleetctl app`; a record has no `kind`. An app record carries `name`, `namespace`, `tag`, `tags`, `state`, `detail`, `priority`, `disabled`, `resources`, `namespace_quota`, `tag_quota`, `jobs` (`total`, `active`, `succeeded`, `failed`, `last_at`), `spec` (its container settings, including `kind`, and `workers`: the nodes it runs on), and for a proof app `proof` (`elf_id`, its placement policy, `mean_prove_ms`, `anchored_on`, `created_at`, and `image`: the container image of the provers proving for it, a list while they differ). A proof app created before `spec` existed, proved by provers its nodes configure, has only `proof`. A batch record (`fleetctl batch`, `type: batch`) has the same envelope: its `state` is `enabled`, `disabled` or `deleted`, its `jobs` count the batch's job records, and its policy sits under `batch` (`image`, `user`, `max_gpus`, `default_gpu_mem_gb`, `shm_gb`, `allow`, `env`, `preemptible`, `min_runtime_s`, `stop_timeout_s`, `workers`). Batch jobs appear only under `fleetctl batch` and `fleetctl job --type batch`. -`fleetctl app disable NAME` stops a persistent app and releases its GPUs. A proof -app has its own dedicated Ziren GPU server: disable blocks new submissions, -drains its active proofs, then stops that server and releases its GPUs. Queued -proofs wait until the app is enabled again. Enabling an app starts its -replicas before accepting work. Disabling a batch only blocks new -batch jobs; accepted batch jobs finish. - -Every entry in `apps.yaml` has an `app`. Each `ziren-gpu` entry must bind -its server to the one app it is dedicated to using `namespace`, `app: NAME` and -`elf_id: FULL_ELF_SHA256`; a `ziren-cpu` entry names its app the same way, with -`elf_id` optional. An entry the scheduler creates for an app takes the app's name. -Any other entry (`kind: app`) is its own app, takes no `elf_id`, and its -`app` defaults to the entry name. Do not configure a Ziren server for multiple -apps. Provers and workers receive `FLEET_APP`, `FLEET_APP_NAMESPACE` and -`FLEET_APP_ELF_ID`; `fleet-ziren-node` takes `--app` / `FLEET_APP`. -The spec or `apps.yaml` entry supplies initial resource settings; app owners then manage -allocations through the control plane with `fleetctl app scale`. Batch jobs request resources with -`fleetctl batch submit NAME --gpus N --cpus N`. Active proofs cannot resize -their GPU allocation. Resizing a proof already in progress is tracked in -[issue #3](https://github.com/ProjectZKM/zkm-fleet/issues/3). +- **Authentication:** `Authorization: Bearer ` + +An administrator issues each user a namespace- and tag-scoped token and delivers it +securely; MCP takes no administrator token. Initialization and `tools/list` need no +token, so clients can discover the tools and their schemas first. `tools/call` needs a +current user token, and the scheduler applies its namespace, tag and grants. A client +must support remote Streamable HTTP and a configurable Bearer header; one that runs +only local `stdio` servers can't connect. + +| Tool | Does | +|---|---| +| `fleet_apps_list`, `fleet_app_get` | `fleetctl app`, `fleetctl app NAME` | +| `fleet_app_create` | `fleetctl app add`: a `spec` and optional `priority`, plus a prover's `elf_id`, `class`, `anchor`, `backends`, `zkm_version` and `max_queued` | +| `fleet_app_logs` | `fleetctl app logs` | +| `fleet_batch_create` | `fleetctl batch add`; non-empty `allow` needs an administrator | +| `fleet_batches_list`, `fleet_batch_get` | `fleetctl batch`, `fleetctl batch NAME` | +| `fleet_batch_jobs_list`, `fleet_batch_job_get` | `fleetctl job --type batch` (optional `batch`), `fleetctl job ID` | +| `fleet_batch_job_submit` | `fleetctl batch submit`: a `batch`, a command and optional `mounts` such as `["/data/models:ro"]` | +| `fleet_batch_job_logs`, `fleet_batch_job_cancel` | `fleetctl batch logs`, `cancel`; `batch` is optional | +| `fleet_jobs_list` | `fleetctl job`, with `kind: "batch"` or `kind: "app"` | + +Changing existing workloads, enabling and scaling go through `fleetctl` or the API. +Proofs are submitted through the Ziren network SDK; MCP lists and inspects them. ## Data -| what | where | +| What | Where | +|---|---| +| Registered apps, proof jobs, tokens | Postgres: `fleet_apps` (each namespace's programs and placement policy, with when each was disabled), `fleet_jobs` (state, kind and a failed job's stage status by name, with CHECK constraints), `fleet_tokens`, `fleet_namespaces` (namespace and tag quotas), `fleet_namespace_allocations`, `fleet_schema` (the schema version) | +| SDK proof ELFs and inputs | The app's `proof_storage_root`: `elf/` and `inputs/`; Fleet keeps references, hashes and parameters | +| SDK proof outputs | The app's `outputs///`: proof, public values and a committed manifest | +| Integrated EthProofs data | Its storage root: `checkpoint.json`, `inputs/`, `outputs//`, `outbox/` and `reported/`, in the app's own format | +| Apps, deployments, batches, jobs and commands | `data_dir/workloads/{apps,deployments,jobs,commands}` on the control host: durable JSON records, including lease and cleanup fences | +| Batch inputs, outputs and checkpoints | The owner's shared or object storage | +| Batch logs | `data_dir/workloads/artifacts///logs` on the control host; nodes keep execution files in `FLEET_JOBS_DIR//`, normally `work/jobs/` | + +## Ports + +| Port | Serves | |---|---| -| registered apps, proof jobs, tokens | Postgres (compose): `fleet_apps` (each namespace's named programs and their placement policy, with when each was disabled), `fleet_jobs` (state, kind and a failed job's stage status by name, with CHECK constraints), `fleet_tokens`, `fleet_namespaces` (namespace and optional tag quota settings), `fleet_namespace_allocations`, `fleet_schema` (the schema version) | -| SDK proof ELFs and inputs | App-owned `proof_storage_root`: `elf/` and `inputs/`. Fleet stores references, hashes and parameters. | -| SDK proof output | App-owned `outputs///`: proof, public values and a committed manifest. The local app endpoint reads them directly; remote Fleet fetches use an authorized heartbeat ticket and HTTPS relay. Disabling the GPU actor preserves fetching. | -| Integrated EthProofs data | Its app-owned storage root: `checkpoint.json`, `inputs/`, `outputs//`, `outbox/` and `reported/`; the app manages their format and recovery. | -| apps, app deployments, batches, jobs and command intents | `data_dir/workloads/{apps,deployments,jobs,commands}` on the control host; durable JSON records, including lease and cleanup fences | -| batch inputs, outputs and checkpoints | owner-managed shared or object storage, read and written directly by the batch command | -| batch logs | `data_dir/workloads/artifacts///logs` on the control host; nodes retain execution files in `FLEET_JOBS_DIR//` (normally `work/jobs/`) | +| 8905 | The HTTPS gateway: the API, MCP and node heartbeats. Nodes need outbound HTTPS (HTTP/2) to it; the scheduler's own listeners stay private behind it. | +| `127.0.0.1:51012`, `51013` | GOAT's proof endpoint: SDK stage and HTTP, loopback only | +| 6379–6381, 52365, 8080, 10002–19999 | Each node's local Ray; the metrics port (8080) and the worker range (10002–19999) [move](operations.md#configure-resources) in `etc/watchdog.env` | -## Tokens +## Tokens and TLS -| what | held by | issued by | +| Token | Held by | Issued by | |---|---|---| -| user token (`FLEET_TOKEN_FILE`) | the namespace owner/client | namespace/token issuance; authorizes proof RPCs and HTTP APIs within its scope; revocable | -| SDK signing key (`ZKM_PRIVATE_KEY`) | the client | generated in the token bundle and bound internally to its token | -| server certificate + CA | gateway and control API; clients get `CA_CERT_PATH` when supplied | `install.sh` (`fleetctl init tls`) | -| cluster token | scheduler and every node; authenticates node-initiated requests | `install.sh` (`fleetctl init token`) | -| admin token | operators | `install.sh` (`fleet-scheduler issue-admin`); more with `fleetctl token admin` | - -Administrator tokens live in the database. On the scheduler host, -`fleet-scheduler -c fleet.yaml issue-admin --label NAME --out FILE` writes an -administrator token's record into the database and its secret to FILE (0600, -never overwritten); a running scheduler reloads tokens every `heartbeat_secs`. -`install.sh` issues one into `etc/admin.token` after Compose is up. Issue more -with `fleetctl token admin --label NAME --out FILE`. An in-memory scheduler (no -`database_url`, for development and tests) writes its own to `data_dir/admin.token` -at start. `fleet.yaml` no longer takes `tenants`, `default_tenant`, -`require_signature` or `admin_token_file`; the scheduler refuses to start with -any of them and names what replaced it. - -Proof clients use bundled SDK signing material and bearer authorization at their -app endpoint. Loopback clients use local HTTP; external clients require an authenticated HTTPS app gateway. The token API exposes only opaque IDs and access -scopes. Tokens with `manage` can create apps and manage their metadata; -tokens without it can use only granted apps and batches. Revocation applies to both proofs and -batch jobs. - -The scheduler’s backend listeners stay private behind the HTTPS gateway. Node control gRPC uses TLS with the scheduler -certificate and cluster bearer token; use an HTTPS `scheduler` URL in `apps.yaml`. -For a private certificate authority, set `scheduler_ca_file` so prover containers trust it. -The installer sets this to its generated public CA certificate; distribute that CA to -each execution machine. -The scheduler holds a PostgreSQL advisory lock for its lifetime, so only one process can -dispatch from a database at a time. Fleet does not use SSH channels between machines; -MCP exposes only user-scoped tools through the existing HTTPS gateway. Users and operators both use `FLEET_TOKEN_FILE` -(`--token-file`); without it `fleetctl` reads `admin.token` from `FLEET_ETC`, -`~/.config/zkm-fleet` or `/etc/zkm-fleet`. Batch API handlers are part of the Rust scheduler; there is no separate Python HTTP listener. +| User token (`FLEET_TOKEN_FILE`) | The namespace owner or client; authorizes proof RPCs and the API within its scope | Namespace creation or token issuance; revocable | +| SDK signing key (`ZKM_PRIVATE_KEY`) | The client | Generated in the token bundle and bound to its token | +| Server certificate and CA | The gateway and control API; clients get `CA_CERT_PATH` when supplied | `install.sh` (`fleetctl init tls`) | +| Cluster token | The scheduler and every node; authenticates node requests | `install.sh` (`fleetctl init token`) | +| Administrator token | Operators | `install.sh` (`fleet-scheduler issue-admin`); more with `fleetctl token admin` | + +Administrator tokens live in the database. +`fleet-scheduler -c fleet.yaml issue-admin --label NAME --out FILE`, on the scheduler +host, records one and writes its secret to FILE (mode 0600, never overwritten); a +running scheduler reloads tokens every `heartbeat_secs`. `install.sh` issues the first +into `etc/admin.token` once Compose is up, and `fleetctl token admin --label NAME +--out FILE` issues more. A scheduler without `database_url`, for development and +tests, writes its own to `data_dir/admin.token` at start. `fleetctl` reads +`FLEET_TOKEN_FILE` (`--token-file`), and otherwise `admin.token` from `FLEET_ETC`, +`~/.config/zkm-fleet` or `/etc/zkm-fleet`. + +Node control gRPC uses TLS with the scheduler's certificate and the cluster token, so +`apps.yaml` names an HTTPS `scheduler` URL. With a private CA, set +`scheduler_ca_file` so prover containers trust it; the installer sets it to its +generated CA, which each execution machine needs a copy of. Proof clients use their +bundled signing key and bearer token at their app endpoint: plain HTTP on loopback, +an authenticated HTTPS gateway from outside.