Repository navigation
Keep appended apps.yaml entries inside the apps list, and make an app's replicas the app's - #90
Merged
Merged
Conversation
append_entry inserts a new entry after the list's last line of content. When
that last entry was a one-line flow entry, which append_entry itself writes,
its content ended before the newline, and the "no trailing newline" branch
appended the new entry at the end of the file instead. A key after the list
(such as scheduler_admin) then came before the new "- {...}" line, and the
file no longer parsed: the second app the scheduler placed on such a node
broke the node's configuration.
The new entry now goes after the rest of the last entry's line, as
remove_entry already finds an entry's end; a newline is added only when that
line ends the file.
An app's replica count, GPUs and CPUs are its own settings, wherever the scheduler places its replicas. A scale names no node: the count is split over the app's deployments, those running first. Each keeps the replicas it has, up to the count, and new ones join the first. A deployment left with none is retired: it is disabled, and its node stops its replicas and removes the app's entries, which frees its room. Its record goes with the node's next report. The app's data stays on the node. A node that comes back with a copy of an app whose replicas all run on other nodes no longer starts it: the copy is retired the same way. An entry its node anchors to the machine still starts there. A scale with a node is refused. Checking a scale without one shows the app's replicas over all its nodes, and a node shows that node's part. A deployment being retired no longer counts toward the app, and enabling or disabling the app leaves it alone. A forgotten deployment's node leaves the app's node list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR has two fixes: appended
apps.yamlentries now stay inside theapps:list, and an app's replica count belongs to the app, not to any one node.1. Keep appended apps.yaml entries inside the apps list
Problem
append_entry(bridgenode.py) adds an entry to a node'sapps.yamlwhen the scheduler places an app there. It inserts the new entry after the list's last line of content. If that last entry is a one-line flow entry such as- {"name": ...}, whichappend_entryitself writes, its content ends before the newline. The code then took its "file has no trailing newline" branch and appended the new entry at the end of the file.When another key follows the
apps:list (for examplescheduler_admin:), the new- {...}line lands after that key, and the file no longer parses. In production, the second entry appended to such a node'sapps.yamlbroke it, and the node reported not ready until the line was removed by hand.Fix
The new entry now goes after the rest of the last entry's line, the same way
remove_entryalready finds where an entry ends. A newline is added only when that line ends the file.Tests
test_appended_entries_stay_in_the_list_before_the_keys_after_itappends two entries to four layouts:It checks that both new entries are in the list, that the other keys are unchanged, and that every original line is kept. Three of the four cases fail on the old code.
python -m pytest -q bridge/tests deploy/tests: 94 passed.Nodes get the fix with their next bridge upgrade. Until then, a node whose
apps.yamlhas keys after theapps:list can still hit this.2. Make an app's replicas the app's, not a node's
Problem
app scale --replicas Nset the replicas of one deployment, meaning one node's entry, and needed--nodewhen an app ran on several nodes. An app could run more copies than its count, and there was no way to scale away the copy on one node:--replicas 0 --node NODEwas refused ("replicas between 1 and 64").That happened in production. While its node was away, an app got a second copy on another node. When the first node came back, the scheduler started its old copy again, so the app ran twice.
Change
PUT /apps/NAME/scaletakes no node (a scale with one is refused with 400).replicasis the app's total. GPUs and CPUs apply to every replica. The count is split over the app's deployments, those running first: each keeps the replicas it has up to the count, and new ones join the first.machine) is the node's own and still starts there, just as it is never moved.GET /apps/NAME/scale) shows the app's replicas over all its nodes, with each node's part.?node=NODEshows one node's part, as before.fleetctl app scalerefuses--nodewith resource flags, keeps--nodefor inspection, and prints the app's nodes. Its help anddocs/operations.md/docs/reference.mdare updated.Not in this change: spreading one app's replicas over several nodes when no single node has room for all of them. New replicas still go to one node.
Tests
an_apps_replicas_are_its_own_and_a_node_its_scale_leaves_none_removes_it: an app runs one replica on each of two nodes.a_node_back_with_a_copy_of_an_app_that_runs_its_replicas_elsewhere_removes_it: a node returns with a stopped copy of an app that now runs on another node. Its node removes the copy, and no start is sent. When the copy is anchored to the machine, it stays and its start waits for room. This test fails with the new start check turned off.app_scale_takes_no_node_and_inspects_one_nodes_part(fleetctl):--nodewith--replicasis refused, and--nodealone sendsGET ...?node=.cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcp: all pass (111 scheduler tests, 11 fleetctl app tests).No app in production runs on more than one node today, so deploying this retires nothing.