Repository navigation
Admit a restarted node agent at once, fix the remote upgrade dry run, and delete apps and batches with all their data - #91
Merged
Conversation
A remote dry run ships nothing to the node, so the node got paths to files that aren't there. It took the first one for a release tag and printed a release download, plus installs of the proof app and launch scripts the upgrade would never touch. A release tag is a bare name: a path now always names local files, shipped or not.
One agent session at a time drives a node, and the scheduler refused a new session until the old one had been quiet for 30 seconds. Every agent restart, an upgrade included, left the node unheard for that long, right at the threshold where it counts as offline. A stopping agent now ends its own session (WorkerService.EndSession), after it has stopped its bridge, and the scheduler admits the next session at once. The node counts as connected meanwhile, as before. Only the session itself can end it, with the cluster token. An agent that crashed, or that can't reach the scheduler as it stops, leaves its session to go quiet as before, and so does an agent talking to a scheduler without the call.
Deleting an app and deleting a batch are one operation: refused while any of its work runs or may run, then the record goes, every node drops it from its configuration, and its history stays. A batch's work is its queued or running jobs. An app's work is its replicas: it is disabled first, and every node that configures it has stopped them. A refusal now says what still runs and how to stop it. A batch was refused with "disable the app or finish/cancel jobs before deleting", which told a batch owner to disable an app. A node that no longer configures an app no longer holds up its delete forever. An offline node still does, since it can't confirm its replicas stopped. fleetctl batch delete now says what it would delete without --yes, as app delete does. After deleting it says what was deleted and what stays, instead of "updated batch". The help of both commands and the reference describe the shared rule.
Deleting an app or a batch now removes it with its data, on the control host and on its nodes, and is the only way job history goes: - Control host: the batch's job records with their logs, and the commands about them and the app or batch. A proof app's job rows and anchor go from Postgres. - Nodes: the configuration, and what the node holds of it. A batch's jobs lose their directories (record, logs, out/). An app's replicas' logs and workers' directories go, with its proof jobs' files in its proof storage: each job's output directory, and the inputs and ELF no other job or app uses. - Kept: volumes, batch mounts, env files, storage roots, images, and whatever holds or lies in a path another app or batch uses on any node, the filesystem's top level and Fleet's own directories. The node reports what it kept and why. Only a deletion's removal, marked with "delete", deletes files; a scale that takes an app off one node still keeps its data there. The record stays hidden until its nodes have removed it, so a late node report can't bring it back and the name can't be reused before then; deleting again retries a node that failed. Deleted records left by earlier releases go at start once nothing of them is left, and a batch deleted earlier with job records stays listed as deleted until it is deleted again. The purge subcommand and its API routes are gone, with the purge ordering of node commands.
blake-pro
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes found while upgrading both production nodes to v0.1.11, and one rework of deletion.
1. A remote node upgrade's dry run shows the upgrade it is
fleet-ctl.sh -n node upgrade FLEET_NODE BRIDGE_DIR, withFLEET_NODE_SSHset, ships nothing to the node, so the node is handed paths to files that aren't there. It took the first path for a release tag and printed:fleet-proof-appand the launch scripts, which the real run never touches.A release tag is a bare name, so a path now always names local files, whether or not they were shipped.
Test:
test_a_dry_run_on_the_node_takes_the_files_it_never_shipped_for_local_filesruns the node's part of a dry run with unshipped paths. It checks that nothing is downloaded, that only the agent and the bridge are installed, and that nothing changes. It fails without the fix.2. A restarted node agent is admitted at once
One agent session at a time drives a node. The scheduler refused a new session until the old one had been quiet for 30 seconds. So every agent restart, upgrades included, left the node unheard for about 30 s ("node heartbeat rejected ... another agent session is active for this worker"). That is right at the threshold where the node counts as offline. Production saw this on both nodes during the upgrade.
WorkerService.EndSession(worker_id, session)call lets an agent end its own session. The agent sends it when it stops on SIGTERM or Ctrl-C, after it has stopped its bridge, with a 5 s limit./fleet.v1.WorkerService/by prefix, so the new method passes through them.docs/maintenance.mdsays to upgrade the control host first.Tests:
an_agent_that_ends_its_session_hands_its_worker_to_the_next_at_once(flow, over the real control listener):It fails without the scheduler change.
a_stopping_agent_ends_its_own_session_with_its_tokenandan_agent_stops_promptly_when_the_control_plane_is_out_of_reach(agent, against an in-process stub).3. Apps and batches are deleted the same way, with everything Fleet holds of them
Deleting an app and deleting a batch are now one operation. A delete is refused while any of the app's or batch's work runs or may run:
Otherwise the app or batch goes with everything Fleet holds of it, and that is now the only way job history goes:
Control host: its job records with their logs, and the commands about them and it. A proof app's job rows and anchor go from Postgres.
Nodes: its configuration, and what the node holds of it:
out/);worker_workdir) go;Kept:
The node reports what it kept and why, and the scheduler logs it. A container's
workdiris the container's own: what it keeps on the host is in its volumes.Only a deletion's removal, marked
"delete", deletes files. A scale that takes an app off one node still keeps its data there, since the app runs on elsewhere.The record stays, hidden, until every node has removed it. So a node report sent before then can't bring it or its jobs back, and the name can't be reused until then (409, saying so). A node that is offline removes it once it's back, and deleting again retries a node that failed. Deleted records that earlier releases left behind go at start once nothing of them is left. A batch deleted earlier with job records stays listed as
deleteduntil it is deleted again, which removes them.The
purgesubcommand, its API routes (DELETE /batches/NAME/jobs?older_than=DAYS,DELETE /batches/NAME/jobs/ID) and the purge ordering of node commands are gone. The node agent no longer runs operation 5.Before, deleting an app and deleting a batch also differed in their details:
batch training still runs: 1 job(s) queued or running: cancel them or let them finish,app model still runs: it is enabled: disable it first (fleetctl app disable model),its replicas on node gpu are still stopping,node gpu is offline, so it can't confirm its replicas stoppedfleetctl batch deletewithout--yesapp deletedoesfleetctl batch delete --yesbatch delete: "Delete a batch after all accepted jobs finish" (it doesn't wait; it refuses)docs/reference.mddescribes all of this in one place.Tests:
deleting_an_app_or_a_batch_is_refused_while_any_of_it_runscovers each refusal, the node that no longer configures the app, and the deletes going through once nothing runs.deleting_a_batch_deletes_its_jobs_and_goes_once_its_node_removed_itcovers:a_deleted_batch_a_node_failed_to_remove_is_deleted_againcovers the retry.a_batch_an_earlier_release_deleted_goes_with_its_jobsrestarts the scheduler over an earlier release's deleted records.deleting_a_proof_app_tells_its_nodes_the_files_its_jobs_leftcovers ELF and input sharing within and across namespaces.referencedfinds the inputs and ELFs that jobs still use.test_deleting_an_app_deletes_what_its_entries_heldtest_deleting_an_app_keeps_what_another_app_or_batch_uses, which covers each refusaltest_deleting_a_batch_deletes_its_jobs_directoriestest_removing_an_app_from_one_node_keeps_its_dataapp_delete_says_what_goes_until_confirmedandbatch_delete_says_what_goes_until_confirmed_like_app_delete.Checks
cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcp: all pass.cargo test --locked --no-default-features --lib -p fleet-ziren-node: passes.python -m pytest -q bridge/tests deploy/tests: 97 passed.cargo test -p fleet-scheduler --test pgagainst a throwaway local Postgres: 8 passed.The session handover takes effect once both the scheduler and the node agents run this release. Deploy the scheduler first, then the nodes: a node with an older bridge removes only a deleted app's or batch's configuration, and keeps its files.