Skip to content

Report a create only with its entry, and write an app's list settings right - #92

Merged
eigmax merged 5 commits into
mainfrom
fix/app-placement-host-paths
Nov 19, 2026
Merged

eigmax merged 5 commits into
mainfrom
fix/app-placement-host-paths

Conversation

@eigmax

@eigmax eigmax commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

In production, an app that runs from files on one node's own disk was moved to another node and failed to start there: its volume wasn't on that node, and Docker mounted a new, empty directory in its place. Such an app is to be anchored to its node with machine in its entry in that node's apps.yaml, which the README and the reference now say. Sorting it out exposed two bugs in the node bridge, fixed here.

1. A create is never reported done before its entry

A node's report read apps.yaml first and its command results last. So a create that finished in between was reported done one report before the node reported the entry it wrote. The scheduler, seeing a successful create whose node didn't configure the app, placed the app on the same node again a second later, and that create failed with ... is already configured on this node. The report now reads the results first: a command finished by then has already changed the configuration read after it.

2. A settings update writes an app's lists right, and never a configuration that doesn't read back

A settings update (PUT /apps/NAME/settings, fleetctl app update) has the node rewrite each changed value in place in apps.yaml, in flow style. For a block list or map, the old value starts at its first item on the next line, so the new [...] was written there, at that line's indentation. When a list's - items sit level with the entry's keys, as in hand-written entries, the new line reads as a stray key.

On another node, an update of an app's workdir, volumes and command failed with while scanning a simple key ... could not find expected ':'. It failed only because re-reading the text for the next field raised. An update of just volumes or just command would have written the broken file and left the node unable to read its configuration.

  • A new value for a block list or map goes on its key's own line, after the colon. A comment on that line is kept.
  • The file is replaced only when the new text parses and the entry reads back with the values asked for. Otherwise the update fails, and the file stays as it was.

Tests

  • test_a_report_carries_a_create_done_only_with_its_entry runs a create while a report is read.
  • test_settings_update_replaces_block_lists_and_maps_however_indented, for lists both level with the keys and indented: it updates volumes, command and env alone and together, and checks that the other entry and the comment are kept.
  • test_settings_update_never_writes_a_file_that_does_not_read_back: the writer produces broken text, and the file is unchanged.

All three fail on main, the second only for lists level with the keys. python -m pytest -q bridge/tests deploy/tests: 101 passed. No Rust code changes.

Both fixes are in the bridge, and take effect once the nodes run it.

…red to its machine

When an app's start found no room on its node, the scheduler moved the app to a node
with room. That node took it although a volume the app runs from wasn't there: Docker
mounted a new, empty directory owned by root in its place, and the app failed to start
there, while its files stayed on the node it left.

- A node refuses to create an app when a host path of its volumes isn't there (a link
  to nothing counts as missing). The scheduler then doesn't pick that node for the app
  again, and the start stays on its own node. A first placement whose create was refused
  goes to another node when one can take it, and back to the same one only when none can.
- A container app can be anchored to a machine: with `machine` in its spec, it is placed
  on that machine's node only; `fleetctl app update NAME --machine MACHINE`
  (`{"machine": ...}` in PUT /apps/NAME/settings) anchors a placed app to the machine it
  runs on, and "" lifts the anchor. Only the node's configuration changes: the app is
  neither drained nor restarted. An anchored app's start waits for room on its node and
  is never moved.
- A node's report reads its command results before its configuration, so a create is
  never reported done before the entry it wrote. The scheduler placed an app a second
  time on the same node when that happened, and the second create failed.
…d never write a node configuration that doesn't read back

A settings update writes each new value in flow style where the old one was. For a
block list or map, the old value starts at its first item on the next line, so the
new list went there, at that line's indentation. With a list's "- " items level with
the entry's keys (valid YAML, and how hand-written entries often look), the new
line read as a stray key: the update of an app's volumes, command and workdir failed,
and an update of just its volumes or command would have written a broken apps.yaml.

A new block list or map value now goes on its key's own line, after the colon, keeping
a comment there. And the file is replaced only when the new text parses and holds
the values asked for; otherwise the update fails and the file stays as it was.
@eigmax eigmax changed the title Keep an app with the node that has its files, and let an app be anchored to its machine Keep an app with the node that has its files, let an app be anchored to its machine, and write an app's list settings right Nov 14, 2026
…ored to its machine

A node no longer refuses to create an app whose volume paths it doesn't have, and a
first placement whose create was refused goes to the best-fitting node again, as
before, not first to another one. An app that runs from or keeps data on a node's
disks is anchored to that node's machine instead, with `machine` in its spec or with
`fleetctl app update NAME --machine MACHINE`.
@eigmax eigmax changed the title Keep an app with the node that has its files, let an app be anchored to its machine, and write an app's list settings right Let an app be anchored to its machine, report a create only with its entry, and write an app's list settings right Nov 14, 2026
…ode's apps.yaml

Neither `machine` in a new app's spec nor `fleetctl app update NAME --machine` anchors
an app any more. As before, an app is anchored by `machine` in its entry in its node's
apps.yaml; the reference now says so where it explains that Fleet copies nothing an
app keeps on a node's disks.
@eigmax eigmax changed the title Let an app be anchored to its machine, report a create only with its entry, and write an app's list settings right Report a create only with its entry, and write an app's list settings right Nov 14, 2026
@eigmax
eigmax merged commit c9892a7 into main Nov 19, 2026
4 checks passed
@eigmax
eigmax deleted the fix/app-placement-host-paths branch November 19, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants