Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ default-members = ["crates/fleet-proto", "crates/fleet-scheduler", "crates/fleet
resolver = "2"

[workspace.package]
version = "0.1.9"
version = "0.1.10"
edition = "2021"
publish = false

Expand Down
34 changes: 16 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![CI](https://github.com/ProjectZKM/zkm-fleet/actions/workflows/ci.yml/badge.svg)](https://github.com/ProjectZKM/zkm-fleet/actions/workflows/ci.yml)

Fleet: A universal schedules apps and batch jobs on shared GPU/CPU machines.
Fleet schedules apps and batch jobs on shared GPU and CPU machines.

- **Apps** keep containers running: a model server, or a Ziren prover with its program.
- **Batches** run finite commands and release resources when cleanup finishes.
Expand All @@ -11,25 +11,23 @@ Fleet: A universal schedules apps and batch jobs on shared GPU/CPU machines.

![Fleet architecture: apps and batches share machine resources](images/diagram.svg)

The control host runs the Rust scheduler, Postgres and an HTTPS gateway. MCP is
optional. Each execution machine runs `fleet-node` and its own local Ray instance.
Agents connect outward and register automatically. Fleet manages admission,
quotas, placement and lifecycle; workers own their application data and pipelines.
The [code model](docs/reference.md#code-names) uses jobs, app deployments and a
workload controller. Shared records come from protobuf; Kubernetes is not required.
The control host runs the Rust scheduler, Postgres, an HTTPS gateway and, optionally,
MCP. Each execution machine runs `fleet-node` and its own local Ray, and connects
outward to register. Fleet handles admission, quotas, placement and lifecycle; apps
own their data and pipelines. The [code model](docs/reference.md#code-names) follows
Kubernetes' jobs, deployments and controllers, without needing Kubernetes.

App owners create apps and manage workload metadata, priority, enable/disable and
scaling with namespace tokens holding the `manage` permission; the scheduler picks
each app's node. Administrators
manage namespace quotas, access, batch mount roots and runtime users, and app
execution settings; owner changes remain subject to namespace and tag limits.
Namespace owners create apps and batches and manage their priority, enabled state and
scaling, within their namespace's and tag's quotas; the scheduler picks each app's
node. Administrators manage namespaces, quotas, access, host storage for batches and
apps' execution settings.

EthProofs prepares witnesses, proves locally and reports from one integrated app
worker. It resumes its saved cursor and proves every block in sequence.
Two integrated app workers prove on their node's own GPU server:

The GOAT proof builder runs as an integrated app worker: it serves Bitcoin node
proof requests, proves its five circuits on its loopback GPU server and keeps its
task database and proofs in app storage.
- **EthProofs** prepares witnesses, proves and reports every block in sequence,
resuming from its saved cursor.
- **The GOAT proof builder** serves Bitcoin node proof requests, proves its five
circuits and keeps its task database and proofs in app storage.

## Get started

Expand All @@ -44,6 +42,6 @@ task database and proofs in app storage.
| [Installation](docs/installation.md) | Dependencies, binaries, control host, execution nodes and registration |
| [Operations](docs/operations.md) | Namespace commands, tags, node capacity, app scaling, batches and logs |
| [Maintenance](docs/maintenance.md) | Upgrades, Ray capacity changes, self-built programs, liveness and troubleshooting |
| [Reference](docs/reference.md) | Protocols, namespaces, tokens, quotas, APIs, storage and ports |
| [Reference](docs/reference.md) | Architecture, workloads, proofs, access, quotas, APIs, storage and ports |
| [EthProofs](apps/ethproofs/README.md) | Pinned source, worker build, sequential proving and durable recovery |
| [GOAT](apps/goat/README.md) | Pinned source, program bundle identity, packaging, cutover and recovery |
4 changes: 2 additions & 2 deletions crates/fleet-mcp/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ impl FleetctlClient {
#[derive(Deserialize, schemars::JsonSchema)]
#[serde(rename_all = "lowercase")]
enum JobType {
/// Jobs of apps (proofs).
/// Proofs submitted to a proof app through the scheduler; an app's own work is not a job.
App,
Batch,
}
Expand Down Expand Up @@ -402,7 +402,7 @@ impl FleetMcp {

#[tool(
name = "fleet_jobs_list",
description = "List jobs visible to the authenticated token. Results are scoped by the token, namespace, and tag.",
description = "List jobs visible to the authenticated token: batch jobs, and proofs submitted to proof apps through the scheduler. An app's own work in its replicas is not a job. Results are scoped by the token, namespace, and tag.",
annotations(read_only_hint = true, destructive_hint = false)
)]
async fn jobs_list(
Expand Down
1 change: 1 addition & 0 deletions crates/fleet-proto/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ pub mod artifacts;
pub mod json;
pub mod models;
pub mod sdk;
pub mod workload;

pub use fleet::model;

Expand Down
32 changes: 2 additions & 30 deletions crates/fleet-proto/src/models.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
use crate::fleet::v1::ProofKind;
use crate::model::v1::*;
use crate::stage::v1::Status as StageStatus;
use serde_json::{json, Value};
use serde_json::Value;

pub fn validate_tag(tag: &str) -> anyhow::Result<()> {
let (key, value) = tag
Expand Down Expand Up @@ -228,35 +228,6 @@ impl serde::Serialize for ProofJob {
}
}

impl Job {
pub fn value(&self) -> Value {
let mut v = self.status.0.clone();
for (k, x) in [
("type", json!("batch")),
("id", json!(self.id)),
("batch", json!(self.batch)),
("namespace", json!(self.namespace)),
("tag", json!(self.tag)),
("owner", json!(self.owner)),
("priority", self.policy["priority"].clone()),
("worker", json!(self.worker)),
("node", json!(self.worker)),
] {
v[k] = x;
}
v
}
pub fn state(&self) -> &str {
self.status["state"].as_str().unwrap_or("")
}
pub fn gpus(&self) -> f64 {
self.request["gpus"].as_f64().unwrap_or(0.0)
}
pub fn cpus(&self) -> f64 {
self.request["cpus"].as_f64().unwrap_or(1.0)
}
}

impl ProofJobFilter {
pub fn matches(&self, j: &ProofJob) -> bool {
self.namespace.as_ref().is_none_or(|n| &j.namespace == n)
Expand Down Expand Up @@ -359,6 +330,7 @@ impl RegisterAppRequest {
mod tests {
use super::*;
use prost::Message;
use serde_json::json;

#[test]
fn an_app_is_its_container_and_a_prover_also_its_program() {
Expand Down
Loading
Loading