Skip to content

Keep appended apps.yaml entries inside the apps list, and make an app's replicas the app's - #90

Merged
eigmax merged 2 commits into
mainfrom
fix/apps-yaml-append-after-flow-entry
Nov 12, 2026
Merged

eigmax merged 2 commits into
mainfrom
fix/apps-yaml-append-after-flow-entry

Conversation

@eigmax

@eigmax eigmax commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

This PR has two fixes: appended apps.yaml entries now stay inside the apps: list, and an app's replica count belongs to the app, not to any one node.

1. Keep appended apps.yaml entries inside the apps list

Problem

append_entry (bridge node.py) adds an entry to a node's apps.yaml when the scheduler places an app there. It inserts the new entry after the list's last line of content. If that last entry is a one-line flow entry such as - {"name": ...}, which append_entry itself writes, its content ends before the newline. The code then took its "file has no trailing newline" branch and appended the new entry at the end of the file.

When another key follows the apps: list (for example scheduler_admin:), the new - {...} line lands after that key, and the file no longer parses. In production, the second entry appended to such a node's apps.yaml broke it, and the node reported not ready until the line was removed by hand.

Fix

The new entry now goes after the rest of the last entry's line, the same way remove_entry already finds where an entry ends. A newline is added only when that line ends the file.

Tests

test_appended_entries_stay_in_the_list_before_the_keys_after_it appends two entries to four layouts:

  • a block entry followed by a key
  • a flow entry followed by a key
  • an indented flow entry with a trailing comment, then a comment and a key
  • a list at the end of a file with no trailing newline

It checks that both new entries are in the list, that the other keys are unchanged, and that every original line is kept. Three of the four cases fail on the old code. python -m pytest -q bridge/tests deploy/tests: 94 passed.

Nodes get the fix with their next bridge upgrade. Until then, a node whose apps.yaml has keys after the apps: list can still hit this.

2. Make an app's replicas the app's, not a node's

Problem

app scale --replicas N set the replicas of one deployment, meaning one node's entry, and needed --node when an app ran on several nodes. An app could run more copies than its count, and there was no way to scale away the copy on one node: --replicas 0 --node NODE was refused ("replicas between 1 and 64").

That happened in production. While its node was away, an app got a second copy on another node. When the first node came back, the scheduler started its old copy again, so the app ran twice.

Change

  • A scale is the app's. PUT /apps/NAME/scale takes no node (a scale with one is refused with 400). replicas is the app's total. GPUs and CPUs apply to every replica. The count is split over the app's deployments, those running first: each keeps the replicas it has up to the count, and new ones join the first.
  • A node left with none is retired. Its deployment is disabled, and its node stops the replicas and removes the app's entries, which frees its room (its allocation is released). The record goes with the node's next report, and the node leaves the app's node list. The app's data stays on the node. A node that is offline does this once it is back.
  • A returning copy doesn't start. When a start would run an app beyond its count, because its other deployments already run all its replicas on connected nodes, that deployment is retired instead. An entry its node anchors to the machine (machine) is the node's own and still starts there, just as it is never moved.
  • Checking a scale (GET /apps/NAME/scale) shows the app's replicas over all its nodes, with each node's part. ?node=NODE shows one node's part, as before.
  • The app record's replica count is the app's total.
  • A deployment being retired no longer counts toward the app's resources, and enabling or disabling the app leaves it alone.
  • fleetctl app scale refuses --node with resource flags, keeps --node for inspection, and prints the app's nodes. Its help and docs/operations.md / docs/reference.md are updated.

Not in this change: spreading one app's replicas over several nodes when no single node has room for all of them. New replicas still go to one node.

Tests

  • an_apps_replicas_are_its_own_and_a_node_its_scale_leaves_none_removes_it: an app runs one replica on each of two nodes.
    • Its scale shows 2 replicas on both nodes.
    • A scale with a node is refused, and 0 or 65 replicas are refused.
    • Scaling to 1 leaves the running copy unchanged. The other node removes the app, its allocation is released, and its record and its place in the app's node list go with its next report.
    • Scaling back to 2 adds the replica on the node that runs it.
  • a_node_back_with_a_copy_of_an_app_that_runs_its_replicas_elsewhere_removes_it: a node returns with a stopped copy of an app that now runs on another node. Its node removes the copy, and no start is sent. When the copy is anchored to the machine, it stays and its start waits for room. This test fails with the new start check turned off.
  • app_scale_takes_no_node_and_inspects_one_nodes_part (fleetctl): --node with --replicas is refused, and --node alone sends GET ...?node=.
  • cargo test --locked -p fleet-proto -p fleet-scheduler -p fleet-node -p fleet-proof-app -p fleetctl -p fleet-mcp: all pass (111 scheduler tests, 11 fleetctl app tests).

No app in production runs on more than one node today, so deploying this retires nothing.

append_entry inserts a new entry after the list's last line of content. When
that last entry was a one-line flow entry, which append_entry itself writes,
its content ended before the newline, and the "no trailing newline" branch
appended the new entry at the end of the file instead. A key after the list
(such as scheduler_admin) then came before the new "- {...}" line, and the
file no longer parsed: the second app the scheduler placed on such a node
broke the node's configuration.

The new entry now goes after the rest of the last entry's line, as
remove_entry already finds an entry's end; a newline is added only when that
line ends the file.
An app's replica count, GPUs and CPUs are its own settings, wherever the
scheduler places its replicas. A scale names no node: the count is split over
the app's deployments, those running first. Each keeps the replicas it has, up
to the count, and new ones join the first. A deployment left with none is
retired: it is disabled, and its node stops its replicas and removes the app's
entries, which frees its room. Its record goes with the node's next report.
The app's data stays on the node.

A node that comes back with a copy of an app whose replicas all run on other
nodes no longer starts it: the copy is retired the same way. An entry its node
anchors to the machine still starts there.

A scale with a node is refused. Checking a scale without one shows the app's
replicas over all its nodes, and a node shows that node's part. A deployment
being retired no longer counts toward the app, and enabling or disabling the
app leaves it alone. A forgotten deployment's node leaves the app's node list.
@eigmax eigmax changed the title Keep appended apps.yaml entries inside the apps list Keep appended apps.yaml entries inside the apps list, and make an app's replicas the app's Nov 12, 2026
@eigmax
eigmax merged commit ffda327 into main Nov 12, 2026
4 checks passed
@eigmax
eigmax deleted the fix/apps-yaml-append-after-flow-entry branch November 12, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant