Skip to content

Admit a restarted node agent at once, fix the remote upgrade dry run, and delete apps and batches with all their data - #91

Merged
blake-pro merged 4 commits into
mainfrom
fix/node-upgrade-handover
Nov 13, 2026
Merged

blake-pro merged 4 commits into
mainfrom
fix/node-upgrade-handover

Conversation

@eigmax

@eigmax eigmax commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Two fixes found while upgrading both production nodes to v0.1.11, and one rework of deletion.

1. A remote node upgrade's dry run shows the upgrade it is

fleet-ctl.sh -n node upgrade FLEET_NODE BRIDGE_DIR, with FLEET_NODE_SSH set, ships nothing to the node, so the node is handed paths to files that aren't there. It took the first path for a release tag and printed:

  • a release download from a URL built out of that path;
  • installs of fleet-proof-app and the launch scripts, which the real run never touches.

A release tag is a bare name, so a path now always names local files, whether or not they were shipped.

Test: test_a_dry_run_on_the_node_takes_the_files_it_never_shipped_for_local_files runs the node's part of a dry run with unshipped paths. It checks that nothing is downloaded, that only the agent and the bridge are installed, and that nothing changes. It fails without the fix.

2. A restarted node agent is admitted at once

One agent session at a time drives a node. The scheduler refused a new session until the old one had been quiet for 30 seconds. So every agent restart, upgrades included, left the node unheard for about 30 s ("node heartbeat rejected ... another agent session is active for this worker"). That is right at the threshold where the node counts as offline. Production saw this on both nodes during the upgrade.

  • A new WorkerService.EndSession(worker_id, session) call lets an agent end its own session. The agent sends it when it stops on SIGTERM or Ctrl-C, after it has stopped its bridge, with a 5 s limit.
  • The scheduler then admits the next session at once. The node counts as connected in the meantime, as before.
  • Only the session itself can be ended, and the call needs the cluster token, like the heartbeat. Both gateways route /fleet.v1.WorkerService/ by prefix, so the new method passes through them.
  • Unchanged cases: an agent that crashed, one that can't reach the scheduler as it stops, and one talking to a scheduler without the call all leave the session to go quiet, as before.
  • Mixed versions are safe in both directions. An old agent never sends the call. A new agent talking to an old scheduler logs a warning and waits, as today. docs/maintenance.md says to upgrade the control host first.

Tests:

  • an_agent_that_ends_its_session_hands_its_worker_to_the_next_at_once (flow, over the real control listener):

    • a second session is refused while the first is active;
    • ending the wrong session, or calling without the token, changes nothing;
    • once the first session ends, the second is admitted, and the first can't come back.

    It fails without the scheduler change.

  • a_stopping_agent_ends_its_own_session_with_its_token and an_agent_stops_promptly_when_the_control_plane_is_out_of_reach (agent, against an in-process stub).

3. Apps and batches are deleted the same way, with everything Fleet holds of them

Deleting an app and deleting a batch are now one operation. A delete is refused while any of the app's or batch's work runs or may run:

  • An app's work is its replicas: it has to be disabled first, and every node that configures it has to have stopped them.
  • A batch's work is its queued or running jobs.
  • A proof app's work also includes its queued or running proofs.

Otherwise the app or batch goes with everything Fleet holds of it, and that is now the only way job history goes:

  • Control host: its job records with their logs, and the commands about them and it. A proof app's job rows and anchor go from Postgres.

  • Nodes: its configuration, and what the node holds of it:

    • a batch's jobs lose their directories (record, logs, out/);
    • an app's replicas' logs and its workers' directories (worker_workdir) go;
    • so do its proof jobs' files in its proof storage: each job's output directory, and the inputs and ELF that no other job or app uses.
  • Kept:

    • volumes, batch mounts, env files, storage roots and images;
    • whatever holds or lies in a path another app or batch uses on any node (the scheduler sends the other nodes' paths along);
    • the filesystem's top levels, home, symlinks and Fleet's own directories.

    The node reports what it kept and why, and the scheduler logs it. A container's workdir is the container's own: what it keeps on the host is in its volumes.

Only a deletion's removal, marked "delete", deletes files. A scale that takes an app off one node still keeps its data there, since the app runs on elsewhere.

The record stays, hidden, until every node has removed it. So a node report sent before then can't bring it or its jobs back, and the name can't be reused until then (409, saying so). A node that is offline removes it once it's back, and deleting again retries a node that failed. Deleted records that earlier releases left behind go at start once nothing of them is left. A batch deleted earlier with job records stays listed as deleted until it is deleted again, which removes them.

The purge subcommand, its API routes (DELETE /batches/NAME/jobs?older_than=DAYS, DELETE /batches/NAME/jobs/ID) and the purge ordering of node commands are gone. The node agent no longer runs operation 5.

Before, deleting an app and deleting a batch also differed in their details:

Before Now
Refusal message Both: "disable the app or finish/cancel jobs before deleting", which also told a batch owner to disable an app Says what still runs and how to stop it, e.g. batch training still runs: 1 job(s) queued or running: cancel them or let them finish, app model still runs: it is enabled: disable it first (fleetctl app disable model), its replicas on node gpu are still stopping, node gpu is offline, so it can't confirm its replicas stopped
A connected node in the app's list that no longer configures it Blocked the delete forever Holds nothing of the app, so it doesn't block
An offline node Blocked silently Still blocks, since it can't confirm its replicas stopped, and the refusal names it
What goes The record; data and job history stayed, history until purged Everything above
fleetctl batch delete without --yes "accepted jobs must finish first" Says what it would delete and what stays, as app delete does
fleetctl batch delete --yes Printed "updated batch NAME" "deleted batch NAME and its N job(s); its nodes remove what they hold of it: NODES"
Help text batch delete: "Delete a batch after all accepted jobs finish" (it doesn't wait; it refuses) Both commands describe the shared rule and what goes

docs/reference.md describes all of this in one place.

Tests:

  • deleting_an_app_or_a_batch_is_refused_while_any_of_it_runs covers each refusal, the node that no longer configures the app, and the deletes going through once nothing runs.
  • deleting_a_batch_deletes_its_jobs_and_goes_once_its_node_removed_it covers:
    • the records, logs and commands removed;
    • the node told which job directories to remove;
    • a late report importing nothing;
    • 404, and 409 on the name, until the node's result arrives, after which the name is free.
  • a_deleted_batch_a_node_failed_to_remove_is_deleted_again covers the retry.
  • a_batch_an_earlier_release_deleted_goes_with_its_jobs restarts the scheduler over an earlier release's deleted records.
  • deleting_a_proof_app_tells_its_nodes_the_files_its_jobs_left covers ELF and input sharing within and across namespaces.
  • Postgres: referenced finds the inputs and ELFs that jobs still use.
  • Bridge:
    • test_deleting_an_app_deletes_what_its_entries_held
    • test_deleting_an_app_keeps_what_another_app_or_batch_uses, which covers each refusal
    • test_deleting_a_batch_deletes_its_jobs_directories
    • test_removing_an_app_from_one_node_keeps_its_data
  • fleetctl: app_delete_says_what_goes_until_confirmed and batch_delete_says_what_goes_until_confirmed_like_app_delete.

Checks

  • cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcp: all pass.
  • cargo test --locked --no-default-features --lib -p fleet-ziren-node: passes.
  • python -m pytest -q bridge/tests deploy/tests: 97 passed.
  • cargo test -p fleet-scheduler --test pg against a throwaway local Postgres: 8 passed.

The session handover takes effect once both the scheduler and the node agents run this release. Deploy the scheduler first, then the nodes: a node with an older bridge removes only a deleted app's or batch's configuration, and keeps its files.

A remote dry run ships nothing to the node, so the node got paths to files
that aren't there. It took the first one for a release tag and printed a
release download, plus installs of the proof app and launch scripts the
upgrade would never touch. A release tag is a bare name: a path now always
names local files, shipped or not.
One agent session at a time drives a node, and the scheduler refused a new
session until the old one had been quiet for 30 seconds. Every agent
restart, an upgrade included, left the node unheard for that long, right at
the threshold where it counts as offline.

A stopping agent now ends its own session (WorkerService.EndSession), after
it has stopped its bridge, and the scheduler admits the next session at
once. The node counts as connected meanwhile, as before. Only the session
itself can end it, with the cluster token. An agent that crashed, or that
can't reach the scheduler as it stops, leaves its session to go quiet as
before, and so does an agent talking to a scheduler without the call.
Deleting an app and deleting a batch are one operation: refused while any of
its work runs or may run, then the record goes, every node drops it from
its configuration, and its history stays. A batch's work is its queued or
running jobs. An app's work is its replicas: it is disabled first, and every
node that configures it has stopped them.

A refusal now says what still runs and how to stop it. A batch was refused
with "disable the app or finish/cancel jobs before deleting", which told a
batch owner to disable an app. A node that no longer configures an app no
longer holds up its delete forever. An offline node still does, since it
can't confirm its replicas stopped.

fleetctl batch delete now says what it would delete without --yes, as app
delete does. After deleting it says what was deleted and what stays,
instead of "updated batch". The help of both commands and the reference
describe the shared rule.
Deleting an app or a batch now removes it with its data, on the control host and on
its nodes, and is the only way job history goes:

- Control host: the batch's job records with their logs, and the commands about them
  and the app or batch. A proof app's job rows and anchor go from Postgres.
- Nodes: the configuration, and what the node holds of it. A batch's jobs lose their
  directories (record, logs, out/). An app's replicas' logs and workers' directories
  go, with its proof jobs' files in its proof storage: each job's output directory,
  and the inputs and ELF no other job or app uses.
- Kept: volumes, batch mounts, env files, storage roots, images, and whatever holds or
  lies in a path another app or batch uses on any node, the filesystem's top level
  and Fleet's own directories. The node reports what it kept and why.

Only a deletion's removal, marked with "delete", deletes files; a scale that takes an
app off one node still keeps its data there. The record stays hidden until its nodes
have removed it, so a late node report can't bring it back and the name can't be
reused before then; deleting again retries a node that failed. Deleted records left by
earlier releases go at start once nothing of them is left, and a batch deleted earlier
with job records stays listed as deleted until it is deleted again.

The purge subcommand and its API routes are gone, with the purge ordering of node
commands.
@eigmax eigmax changed the title Admit a restarted node agent at once, fix the remote upgrade dry run, and delete apps and batches the same way Admit a restarted node agent at once, fix the remote upgrade dry run, and delete apps and batches with all their data Nov 12, 2026
@blake-pro
blake-pro merged commit 9225429 into main Nov 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants