Skip to content

Keep anchored apps on their node, free a moved app's room, keep tags to their own work, and read workload fields through typed views - #88

Merged
eigmax merged 11 commits into
mainfrom
fix/integrated-worker-move
Nov 8, 2026
Merged

eigmax merged 11 commits into
mainfrom
fix/integrated-worker-move

Conversation

@eigmax

@eigmax eigmax commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

What happened

On 2026-11-08, goat-proof-builder-v5 (an app whose node runs its prover as a Fleet worker) was down from 02:14 to 06:51 UTC:

  1. Its prover kept failing on its own node ("task business error") and was being restarted there. A batch job had just been placed in the node's free room, so the restart's room check found none. The scheduler then moved the start to another node that had room. The entry was anchored to its machine by the node's configuration (machine), as every integrated worker's entry is, but the move ignored that, rewrote machine to the new node's, and removed the copy that was coming back up. Nothing preempted the app: preemption only ever stops batch jobs.
  2. The old node removed the app successfully, but the scheduler never released its allocation there:
    • A removed entry is no longer reported.
    • The stop sent afterwards fails with "not configured on this node".
    • So neither of the existing release paths ran.
    • The start on the new node then waited on the old deployment as the app's owner ("this app already has an integrated worker; clean up its owner before moving it"), about 16,000 times over 4.5 hours.
  3. Someone deleted the allocation by hand at 06:47. The start was then sent back to the old node, matched by the app's original create there and that node's lingering record. That node no longer had the entry, so both nodes dropped the app, and it ran nowhere until it was re-enabled at 06:49.

App moves

  • Never move an app anchored to its machine (workloads_scale.rs): a start whose entry names its machine stays on its node and waits for room there, preempting lower-priority batch jobs of its namespace when it can. The bridge requires machine on every integrated worker, so integrated workers no longer move. The move no longer rewrites machine.
  • Free the room on removal (workloads.rs): a successful app remove releases the allocation of each entry the node reports it removed. This applies to plain apps and integrated workers, the entries that hold one, and only to deployments the scheduler has on record for that node and app. It's safe because the bridge removes an entry only once its containers are confirmed gone. Without it, a moved app's quota and old record were kept for good.
  • Don't send a start back to a node the app left (workloads_scale.rs): place_start follows an earlier create only when that node still reports the entry and isn't being told to remove the app.
  • Test simulator matches the bridge (workloads_tests.rs): a start or stop for an entry the node doesn't have now fails with "not configured on this node", and a remove names the entries it removed. The old simulator treated the failed stop as a success, which hid the leak.
  • Docs (docs/reference.md, docs/maintenance.md): when a start moves and when it stays, and what v0.1.9 did instead.

Tag scope checks

Two routes let one tag in a namespace reach another tag's work (workloads.rs):

  • Submitting to a batch checked the grant and the namespace, but not the batch's tag. A token of another tag, which can't even see the batch, could submit jobs that ran with the batch's admin-set mounts, user and environment. Submitting now requires seeing the batch, and otherwise gets 404, as reading it does. An administrator's job on a tagged batch now gets the batch's tag, so it counts against that tag's quota.
  • Changing a proof app checked only the tags of its deployments. Before the app was placed, another tag could change its priority, enable or disable it, or delete it. The tag it was created for now counts too, and another tag gets 403.

Another tag can still read a proof app and submit proofs to it, which is unchanged.

App jobs in help and docs

fleetctl job --type app lists only proofs submitted to a proof app through the scheduler. Its help said "the proofs an app runs", which reads as if a model server's requests, or a worker that proves on its own GPU server, would be listed. Neither ever reaches the scheduler. The fleetctl help, the MCP fleet_jobs_list tool, docs/reference.md and docs/operations.md now say so, and point to fleetctl app and fleetctl app logs for an app's own work. docs/maintenance.md names no API path any more: the hand release of a moved app's room is gone, since no move can need it now.

Shorter docs

docs/reference.md stated the permission rules four times, priority and preemption three times, and app placement, disabling and quotas twice each, mixed with release history and a roadmap. Each fact now has one place, and the reference is a quarter shorter:

  • Reference: one permissions table, one section each for priority and preemption, quotas, and app placement and moves, the API as two route tables and MCP's tools as a table. A batch's settings with their defaults, a batch job's submission fields and the ports are now listed, from the code.
  • Operations: keeps the commands and links to the reference for the rules.
  • Maintenance: the same procedures in shorter prose.
  • README: a fixed opening sentence.

A check of the rewrite against the old docs and the code restored the facts it had dropped, and corrected the proof-app tag rule, which the old docs had also overstated: scaling or updating a proof app needs only the tag of the entry it changes. Every link and anchor that other files use still resolves.

Typed field reads

The scheduler read app entries, node reports, node commands, batch jobs and submissions by string key, at about 350 places, each with its own default. fleet_proto::workload now holds each shape once, so each key is spelled in one place:

  • Entry and EntryMut read and write an app's entry. Deployments, app templates, scale targets and node reports hand out their entries.
  • CommandRecord reads a command's parameters and result, and workload::command builds the parameters it reads.
  • Batch, Submission and JobReport read a batch's settings, a job's submission and a node's report of a job. Job gains its times and the writes that end, fail or requeue it.
  • A submission's fields and what each takes are one table, checked in the order they were. A new batch's allowed fields come from its defaults table.

Behaviour is unchanged. Records keep their JSON, so nothing stored or sent changes, and keys the scheduler doesn't know still pass through. Where call sites used different defaults, the call names its default: a missing gpus counts as no GPU when reserving and admitting, but as one GPU when placing. Checks of a value's type still read the raw JSON, since a view would hide a wrong type behind a default. Key reads in workloads.rs, workloads_scale.rs and workloads_views.rs fall from 353 to 49. What's left is those type checks and the code that builds responses.

Tests

Each new test of a fix was checked to fail without the change it covers:

Test Without its change
an_app_anchored_to_its_machine_restarts_only_there The anchored app is sent to the node with room
an_app_moved_to_a_node_with_room_frees_the_old_one_and_runs_there Without freeing the room: the old node's allocation stays held. Without the no-send-back check: the app ends up on neither node
batch_metadata_belongs_to_scoped_owners_and_job_policy_is_inherited (extended) Another tag's token submits to the batch (201 instead of 404)
an_unplaced_proof_app_is_changed_only_by_its_own_tag Another tag changes the unplaced proof app's priority (200 instead of 403)

For the typed reads, submissions_and_new_batches_are_checked_field_by_field pins each submission fault's status and message and a stored submission's fields. It passes both before and after the change. Unit tests in fleet_proto::workload pin each view's defaults.

cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcp passes. The bridge is unchanged.

Not changed

  • A restarting app's room isn't reserved against batch jobs. Ray frees a crashed actor's resources while it restarts, so a job can take them, and an app that isn't anchored can then move. Holding that room needs the scheduler to reserve it on the node, which is a separate change. An earlier commit here credited the room instead, and it is reverted.
  • A remove that fails partway, for an app with several entries on one node, still leaves the room of the entries it did remove held.

Rollout

Bumps the workspace version to 0.1.10 (separate commit). A control-host update (scheduler and MCP) plus the new fleetctl binary is enough; nodes stay on their current version.

When an app whose node runs its prover as a Fleet worker had to restart and
its own node had no room, the scheduler moved the start to a node with room
and told the old node to remove the app. Two things then went wrong
(goat-proof-builder-v5, 2026-11-08):

- The old node's allocation was never released. A removed entry is no longer
  reported, and the stop sent after it fails with "not configured on this
  node", so neither release path fired. The start on the new node waited on
  the old deployment as the app's owner ("this app already has an integrated
  worker") for 4.5 hours. A successful remove now releases the allocation of
  each entry the node removed; the bridge removes an entry only once its
  containers are gone.
- Once that allocation was released by hand, the start was sent back to the
  old node, matched by the app's original create there and the old node's
  lingering record. That node no longer had the entry, so both nodes dropped
  the app. A start now follows only a create whose node still reports the
  entry and is not being told to remove the app.

The test simulator now fails a start or stop for an entry the node does not
have, and names the entries a remove removed, as the bridge does.
An app's restart was checked for room as if it needed a second set: room
its own replicas still held on its node did not count toward it. So when
a batch job had just been placed in a node's last free room, the restart
of an app that was coming back up there was moved to another node, and
the move removed the copy that was recovering. The move also ignored an
entry anchored to its machine by the node's configuration and rewrote its
`machine` to the new node's (goat-proof-builder-v5, 2026-11-08).

- A start fits in the room its replicas hold when the node reports them
  running or not yet cleaned up, at the same size.
- A start whose entry names its `machine` never moves: it waits for room
  on its node, preempting lower-priority batch jobs of its namespace when
  it can. The move no longer rewrites `machine`, since nothing that moves
  carries one.
@eigmax eigmax changed the title Let an integrated worker restart on the node it was moved to Keep restarting and anchored apps on their node, and let a moved app start Nov 8, 2026
The previous commit let a start fit in the room its replicas hold when
the node reported them not cleaned up. That is wrong: Ray frees a
crashed actor's resources while it restarts, so a batch job can take the
card, and the start then neither preempts nor moves while the app stays
down behind the job. The start's room check is back to what it was; the
machine anchor stays.

The bridge requires `machine` on every integrated worker, so with the
anchor none of them moves. The move test now uses a plain app, which
still can, and still fails without the room freed on removal and without
the check that keeps a start from going back to the node it left. Freeing
that room is limited to entries that hold a deployment allocation (plain
apps and integrated workers). The docs say when a start moves and what
v0.1.9 did instead.
@eigmax eigmax changed the title Keep restarting and anchored apps on their node, and let a moved app start Keep machine-anchored apps on their node, and let a moved app start Nov 8, 2026
The hand release of a moved app's room named a scheduler API path. No
move can need it any more: integrated workers are anchored and never
move, and the scheduler frees a moved app's room once its old node
removes it.
Two routes let one tag in a namespace reach another tag's work:

- Submitting to a batch checked the grant and namespace but not the
  batch's tag. A token of another tag, which cannot even see the batch,
  could submit jobs that ran with the batch's admin-set mounts, user and
  environment. Submitting now requires seeing the batch (404 otherwise,
  as for reading it), and an administrator's job on a tagged batch gets
  the batch's tag, so it counts against that tag's quota.
- Changing a proof app checked only the tags of its deployments, so
  before it was placed another tag could change its priority, enable,
  disable or delete it. The tag it was created for now counts too.
An app job is a proof submitted to a proof app through the scheduler.
The help said "the proofs an app runs", which reads as if a model
server's requests, or a worker that proves on its own GPU server, would
be listed; they never reach the scheduler. The help, the MCP tool and
the docs now say so, and point to fleetctl app and app logs for an
app's own work.
@eigmax eigmax changed the title Keep machine-anchored apps on their node, and let a moved app start Keep anchored apps on their node, free a moved app's room, and keep tags to their own work Nov 8, 2026
The scheduler read an app's apps.yaml entry, a node's report of it and
a command's parameters by string key at about 220 places, each with
its own default. fleet_proto::workload now holds those shapes once:

- Entry reads an entry (a deployment's spec, an app template, a
  change's target, a start's entry, a report's entry), and EntryMut
  writes the few settings the scheduler changes.
- AppDeployment, WorkloadTemplate, ScaleStatus and NodeReport hand
  out their entries.
- CommandRecord reads a command's parameters and result, and
  workload::command builds the parameters it reads.

The records keep their JSON, so keys the scheduler does not know still
pass through, and nothing on the wire or on disk changes. Where call
sites used different defaults (a missing gpus counted as no GPU when
reserving and admitting, but one GPU when placing a node), the call
names the default, and tests pin each one.
A batch job's record, its submission and its batch's settings were
read by string key across the report, dispatch, preemption and listing
paths, and the submission's fields were listed three times: as the
allowed keys, then in four type-check loops.

- Batch reads a batch's settings, from its template or from a job's
  copy of them.
- Submission reads what a job was submitted with, and SubmissionMut
  records the resources it gets.
- Job gains its times and the writes that end, fail or requeue it;
  JobReport reads what a node says about a job it runs.
- workload::command::launch builds a job's launch.
- A submission's fields and what each takes are one table, checked in
  the order they were, so each fault keeps its status and message. A
  new batch's allowed fields come from its defaults table.

Checks of a value's type (a batch's settings, a new batch's name and
image) still read the raw JSON, since a view would hide a wrong type
behind a default. A new test pins each submission fault's status and
message, and passes on the code before this change.
@eigmax eigmax changed the title Keep anchored apps on their node, free a moved app's room, and keep tags to their own work Keep anchored apps on their node, free a moved app's room, keep tags to their own work, and read workload fields through typed views Nov 8, 2026
The reference stated the permission rules four times, priority and
preemption three times, and app placement, disabling and quotas twice
each, among release history and a roadmap. Each fact now has one place:

- reference.md: architecture, workloads (apps, batches, jobs, priority),
  proofs, namespaces and tokens with one permissions table, quotas, the
  API as two route tables, MCP's tools as a table, data, ports, and
  tokens and TLS. A batch's settings with their defaults and a batch
  job's submission fields are now listed, from the code.
- operations.md: keeps the commands and links to the reference for the
  rules; examples use gpu-1 instead of a real host and its pool size.
- maintenance.md: the same procedures and commands in shorter prose;
  the owner-settings upgrade note moves to the one-time migrations.
- README.md: fixes the opening sentence.

Every link and anchor other files use still resolves.
A check of the rewrite against the old docs and the code found:

- gpu_mem is the free VRAM each granted card must have before a job
  starts, not a limit; 0 means the whole card.
- batch update changes neither priority nor disabled, which have
  their own commands.
- Scaling or updating a proof app needs only the tag of the entry it
  changes; the created-for and every-entry tag rule is for removing,
  enabling, disabling and prioritizing it.
- Dropped facts are back: what max_queued caps, that after needs its
  job to succeed, that restart_on_oom is spent before retry, a deleted
  batch's jobs, one proof app's change leaving the others intact, the
  namespace an administrator's create names, the schema check at
  startup, MCP's optional batch on the jobs list, EthProofs needing no
  separate client, work recovered during an upgrade under a lowered
  cap, no group resource, and one supervisor per agent.
@eigmax
eigmax merged commit e92b07e into main Nov 8, 2026
4 checks passed
@eigmax
eigmax deleted the fix/integrated-worker-move branch November 8, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant