Repository navigation
Keep anchored apps on their node, free a moved app's room, keep tags to their own work, and read workload fields through typed views - #88
Merged
Conversation
When an app whose node runs its prover as a Fleet worker had to restart and
its own node had no room, the scheduler moved the start to a node with room
and told the old node to remove the app. Two things then went wrong
(goat-proof-builder-v5, 2026-11-08):
- The old node's allocation was never released. A removed entry is no longer
reported, and the stop sent after it fails with "not configured on this
node", so neither release path fired. The start on the new node waited on
the old deployment as the app's owner ("this app already has an integrated
worker") for 4.5 hours. A successful remove now releases the allocation of
each entry the node removed; the bridge removes an entry only once its
containers are gone.
- Once that allocation was released by hand, the start was sent back to the
old node, matched by the app's original create there and the old node's
lingering record. That node no longer had the entry, so both nodes dropped
the app. A start now follows only a create whose node still reports the
entry and is not being told to remove the app.
The test simulator now fails a start or stop for an entry the node does not
have, and names the entries a remove removed, as the bridge does.
An app's restart was checked for room as if it needed a second set: room its own replicas still held on its node did not count toward it. So when a batch job had just been placed in a node's last free room, the restart of an app that was coming back up there was moved to another node, and the move removed the copy that was recovering. The move also ignored an entry anchored to its machine by the node's configuration and rewrote its `machine` to the new node's (goat-proof-builder-v5, 2026-11-08). - A start fits in the room its replicas hold when the node reports them running or not yet cleaned up, at the same size. - A start whose entry names its `machine` never moves: it waits for room on its node, preempting lower-priority batch jobs of its namespace when it can. The move no longer rewrites `machine`, since nothing that moves carries one.
The previous commit let a start fit in the room its replicas hold when the node reported them not cleaned up. That is wrong: Ray frees a crashed actor's resources while it restarts, so a batch job can take the card, and the start then neither preempts nor moves while the app stays down behind the job. The start's room check is back to what it was; the machine anchor stays. The bridge requires `machine` on every integrated worker, so with the anchor none of them moves. The move test now uses a plain app, which still can, and still fails without the room freed on removal and without the check that keeps a start from going back to the node it left. Freeing that room is limited to entries that hold a deployment allocation (plain apps and integrated workers). The docs say when a start moves and what v0.1.9 did instead.
The hand release of a moved app's room named a scheduler API path. No move can need it any more: integrated workers are anchored and never move, and the scheduler frees a moved app's room once its old node removes it.
Two routes let one tag in a namespace reach another tag's work: - Submitting to a batch checked the grant and namespace but not the batch's tag. A token of another tag, which cannot even see the batch, could submit jobs that ran with the batch's admin-set mounts, user and environment. Submitting now requires seeing the batch (404 otherwise, as for reading it), and an administrator's job on a tagged batch gets the batch's tag, so it counts against that tag's quota. - Changing a proof app checked only the tags of its deployments, so before it was placed another tag could change its priority, enable, disable or delete it. The tag it was created for now counts too.
An app job is a proof submitted to a proof app through the scheduler. The help said "the proofs an app runs", which reads as if a model server's requests, or a worker that proves on its own GPU server, would be listed; they never reach the scheduler. The help, the MCP tool and the docs now say so, and point to fleetctl app and app logs for an app's own work.
The scheduler read an app's apps.yaml entry, a node's report of it and a command's parameters by string key at about 220 places, each with its own default. fleet_proto::workload now holds those shapes once: - Entry reads an entry (a deployment's spec, an app template, a change's target, a start's entry, a report's entry), and EntryMut writes the few settings the scheduler changes. - AppDeployment, WorkloadTemplate, ScaleStatus and NodeReport hand out their entries. - CommandRecord reads a command's parameters and result, and workload::command builds the parameters it reads. The records keep their JSON, so keys the scheduler does not know still pass through, and nothing on the wire or on disk changes. Where call sites used different defaults (a missing gpus counted as no GPU when reserving and admitting, but one GPU when placing a node), the call names the default, and tests pin each one.
A batch job's record, its submission and its batch's settings were read by string key across the report, dispatch, preemption and listing paths, and the submission's fields were listed three times: as the allowed keys, then in four type-check loops. - Batch reads a batch's settings, from its template or from a job's copy of them. - Submission reads what a job was submitted with, and SubmissionMut records the resources it gets. - Job gains its times and the writes that end, fail or requeue it; JobReport reads what a node says about a job it runs. - workload::command::launch builds a job's launch. - A submission's fields and what each takes are one table, checked in the order they were, so each fault keeps its status and message. A new batch's allowed fields come from its defaults table. Checks of a value's type (a batch's settings, a new batch's name and image) still read the raw JSON, since a view would hide a wrong type behind a default. A new test pins each submission fault's status and message, and passes on the code before this change.
The reference stated the permission rules four times, priority and preemption three times, and app placement, disabling and quotas twice each, among release history and a roadmap. Each fact now has one place: - reference.md: architecture, workloads (apps, batches, jobs, priority), proofs, namespaces and tokens with one permissions table, quotas, the API as two route tables, MCP's tools as a table, data, ports, and tokens and TLS. A batch's settings with their defaults and a batch job's submission fields are now listed, from the code. - operations.md: keeps the commands and links to the reference for the rules; examples use gpu-1 instead of a real host and its pool size. - maintenance.md: the same procedures and commands in shorter prose; the owner-settings upgrade note moves to the one-time migrations. - README.md: fixes the opening sentence. Every link and anchor other files use still resolves.
A check of the rewrite against the old docs and the code found: - gpu_mem is the free VRAM each granted card must have before a job starts, not a limit; 0 means the whole card. - batch update changes neither priority nor disabled, which have their own commands. - Scaling or updating a proof app needs only the tag of the entry it changes; the created-for and every-entry tag rule is for removing, enabling, disabling and prioritizing it. - Dropped facts are back: what max_queued caps, that after needs its job to succeed, that restart_on_oom is spent before retry, a deleted batch's jobs, one proof app's change leaving the others intact, the namespace an administrator's create names, the schema check at startup, MCP's optional batch on the jobs list, EthProofs needing no separate client, work recovered during an upgrade under a lowered cap, no group resource, and one supervisor per agent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
On 2026-11-08, goat-proof-builder-v5 (an app whose node runs its prover as a Fleet worker) was down from 02:14 to 06:51 UTC:
machine), as every integrated worker's entry is, but the move ignored that, rewrotemachineto the new node's, and removed the copy that was coming back up. Nothing preempted the app: preemption only ever stops batch jobs.App moves
workloads_scale.rs): a start whose entry names itsmachinestays on its node and waits for room there, preempting lower-priority batch jobs of its namespace when it can. The bridge requiresmachineon every integrated worker, so integrated workers no longer move. The move no longer rewritesmachine.workloads.rs): a successful appremovereleases the allocation of each entry the node reports it removed. This applies to plain apps and integrated workers, the entries that hold one, and only to deployments the scheduler has on record for that node and app. It's safe because the bridge removes an entry only once its containers are confirmed gone. Without it, a moved app's quota and old record were kept for good.workloads_scale.rs):place_startfollows an earlier create only when that node still reports the entry and isn't being told to remove the app.workloads_tests.rs): a start or stop for an entry the node doesn't have now fails with "not configured on this node", and a remove names the entries it removed. The old simulator treated the failed stop as a success, which hid the leak.docs/reference.md,docs/maintenance.md): when a start moves and when it stays, and what v0.1.9 did instead.Tag scope checks
Two routes let one tag in a namespace reach another tag's work (
workloads.rs):Another tag can still read a proof app and submit proofs to it, which is unchanged.
App jobs in help and docs
fleetctl job --type applists only proofs submitted to a proof app through the scheduler. Its help said "the proofs an app runs", which reads as if a model server's requests, or a worker that proves on its own GPU server, would be listed. Neither ever reaches the scheduler. Thefleetctlhelp, the MCPfleet_jobs_listtool,docs/reference.mdanddocs/operations.mdnow say so, and point tofleetctl appandfleetctl app logsfor an app's own work.docs/maintenance.mdnames no API path any more: the hand release of a moved app's room is gone, since no move can need it now.Shorter docs
docs/reference.mdstated the permission rules four times, priority and preemption three times, and app placement, disabling and quotas twice each, mixed with release history and a roadmap. Each fact now has one place, and the reference is a quarter shorter:A check of the rewrite against the old docs and the code restored the facts it had dropped, and corrected the proof-app tag rule, which the old docs had also overstated: scaling or updating a proof app needs only the tag of the entry it changes. Every link and anchor that other files use still resolves.
Typed field reads
The scheduler read app entries, node reports, node commands, batch jobs and submissions by string key, at about 350 places, each with its own default.
fleet_proto::workloadnow holds each shape once, so each key is spelled in one place:EntryandEntryMutread and write an app's entry. Deployments, app templates, scale targets and node reports hand out their entries.CommandRecordreads a command's parameters and result, andworkload::commandbuilds the parameters it reads.Batch,SubmissionandJobReportread a batch's settings, a job's submission and a node's report of a job.Jobgains its times and the writes that end, fail or requeue it.Behaviour is unchanged. Records keep their JSON, so nothing stored or sent changes, and keys the scheduler doesn't know still pass through. Where call sites used different defaults, the call names its default: a missing
gpuscounts as no GPU when reserving and admitting, but as one GPU when placing. Checks of a value's type still read the raw JSON, since a view would hide a wrong type behind a default. Key reads inworkloads.rs,workloads_scale.rsandworkloads_views.rsfall from 353 to 49. What's left is those type checks and the code that builds responses.Tests
Each new test of a fix was checked to fail without the change it covers:
an_app_anchored_to_its_machine_restarts_only_therean_app_moved_to_a_node_with_room_frees_the_old_one_and_runs_therebatch_metadata_belongs_to_scoped_owners_and_job_policy_is_inherited(extended)an_unplaced_proof_app_is_changed_only_by_its_own_tagFor the typed reads,
submissions_and_new_batches_are_checked_field_by_fieldpins each submission fault's status and message and a stored submission's fields. It passes both before and after the change. Unit tests infleet_proto::workloadpin each view's defaults.cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcppasses. The bridge is unchanged.Not changed
Rollout
Bumps the workspace version to 0.1.10 (separate commit). A control-host update (scheduler and MCP) plus the new
fleetctlbinary is enough; nodes stay on their current version.