diff --git a/AGENTS.md b/AGENTS.md index 7e2dc9d..64393d9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -527,6 +527,17 @@ scope. Add yours. under 2 packets per second (about 0.2 KiB/s, pings only) after 150 s. → Measure at two settling times, publish the later one, and state the time. +- **A build given the timeout of a test run.** liboqs takes about forty + minutes to compile on four cores; it was started under `timeout 900`, killed + two thirds of the way, and the half hour it had run was lost. → Before + bounding a command, estimate how long it takes (count its steps); bound it to + that, not to a habit. +- **A test double that answered two shapes with one test.** A fake Engine + returned "the container" when its table was a `dict` — and a table of + containers is a `dict` too, so every lookup returned the table and the test + failed for a reason that looked like the code's. → A fake that serves one + thing or many decides by a field of the thing, not by its type. + ### Running the tests (read this before trusting "green") `pyproject.toml` sets: diff --git a/BUGSVULNS.MD b/BUGSVULNS.MD index 256151f..599120e 100644 --- a/BUGSVULNS.MD +++ b/BUGSVULNS.MD @@ -4345,3 +4345,103 @@ mounted socket owned by the host's `docker` group. Dropping privileges in the entrypoint (take ownership of `/data`, then `setpriv` to an `nmesh` account with the socket's group) has to move all three without breaking a node that updates. Trivy's rule DS-0002 is set aside in `.trivyignore` until it does. + +## 117 — high — Fleet never recognised a Portainer stack, and updating one through Portainer erased its environment + +> **Status — fixed.** Tests: `tests/test_fleet_stacks.py::TestFindingStacks` +> (`test_portainer_is_known_by_the_directory_it_deploys_from`), +> `tests/test_fleet_docker.py::TestPortainerCalls` +> (`test_a_redeploy_hands_the_stacks_environment_back`, +> `test_a_git_stack_keeps_the_credentials_portainer_stored`). + +**Where** — `src/apps/fleet_docker.py` (`group_stacks`, the +`io.portainer.stack.name` label), `src/apps/fleet_portainer.py` +(`Portainer.redeploy`). + +**What happens** — three defects on one path. + +1. A Portainer stack was recognised by an `io.portainer.stack.name` label that + Portainer does not write (checked on Portainer CE 2.45.2: its stack + containers carry compose's labels and no `io.portainer.*` one). Every + Portainer stack was shown as a compose project, and its *Update* ran + `docker compose -f /data/compose//docker-compose.yml` on the host, where + that path does not exist — it is inside Portainer's container. A Portainer + running on the machine was not detected either: only an address and token + typed by hand made it visible. +2. With a Portainer configured, `redeploy` sent `"Env": []`. Portainer replaces + a stack's environment with the one a redeploy carries, so every variable of + the stack was erased and its containers recreated without them. Reproduced + on a scratch Portainer: a stack with `GREETING=hello` came back with + `GREETING=` after one redeploy in the old shape. Database passwords and + similar are typically held there: the stack restarts broken, and the + values are gone from Portainer. +3. A git stack was redeployed with `RepositoryAuthentication: false`, which per + Portainer's API makes it forget the repository credentials it stored; the + next pull of a private repository then fails. Not reproduced here (no git + server in the test environment) — the fix follows the API's documented + behaviour and sends the stored username with an empty password, which asks + Portainer to use its own. + +**Fixed** — a Portainer stack is known by its directory, `/data/compose/`, +and matched to Portainer's record by that id; a Portainer server or agent on the +machine is found from its container with no credential. A redeploy reads the +stack's environment and git authentication from `GET /api/stacks/` and hands +them back. Without a connected Portainer, a Portainer stack is updated through +the Engine (each image pulled, each container whose image moved recreated with +its own settings), never through compose on a copy of Portainer's directory — +see `gotchas.md`, "Docker: who owns a stack". + +## 118 — medium — A stack update was reported failed after 25 seconds while it went on running + +> **Status — fixed.** Tests: `tests/test_fleet.py::TestDockerPlane` +> (`test_which_operations_are_slow`, `test_a_long_update_leaves_room_to_read`). + +**Where** — `src/apps/fleet_web.py` (`_DOCKER_SLOW`), `src/apps/fleet.py` +(`MAX_DOCKER_CALLS`). + +**What happens** — the console waited 25 s for every docker operation except +four that pull images; a stack update (`op: stack, action: update`) was not one +of the four. Any update longer than 25 s — every one that pulls an image — +answered the page "that node did not answer" and the toast said it failed, while +the update carried on and finished later. Two operations at once were allowed +per operator, so one update and one page refresh exhausted them: the panel could +not read the machine whose stack it was updating. + +**Fixed** — the update is slow: started, then asked about (`op: answer`) until +the node replies. The far side counts slow operations apart +(`MAX_DOCKER_SLOW = 2`) within `MAX_DOCKER_CALLS = 4`, and one stack is changed +by one act at a time. + +## 119 — low — The Docker and Logs panels did not load when opened + +> **Status — fixed.** No unit test: the page script has no harness; checked in a +> browser against a live node. + +**Where** — `src/webassets/fleet.py`, `enter()`. + +**What happens** — `enter()` called `ROUTER.start(() => {})`. `start` keeps the +handler it is given, so it replaced the `ROUTER.onChange` the page had set beside +its panels — the one that reads a machine's docker when the Docker panel is +opened and refreshes the Logs panel. Opening either showed whatever an earlier +action had left, or nothing until *Refresh* was pressed. + +**Fixed** — `ROUTER.start()` with no argument. + +## 120 — low — A Portainer token could be configured over plain HTTP to any host, and compose was assumed from a `docker` binary + +> **Status — fixed.** Tests: `tests/test_fleet_docker.py::TestPortainerConfiguration` +> (`test_a_token_travels_in_the_clear_only_to_this_machine`), +> `tests/test_fleet_stacks.py::TestCompose`. + +**Where** — `src/apps/fleet_portainer.py` (`clean_url`), +`src/apps/fleet_docker.py` (`compose_available`). + +**What happens** — `http://` was accepted for any Portainer address, so the +access token (Portainer's administrator API key, typically) crossed the network +in the clear on every call. Separately, "this machine has docker compose" was +answered by `shutil.which("docker")`; the CLI is very often installed without +the compose plugin, so stacks were offered a compose update that then failed. + +**Fixed** — plain HTTP only to a loopback address; anything else is HTTPS with +its chain verified or its certificate pinned. Compose is asked for its version +(`docker compose`, then `docker-compose`) and the answer kept ten minutes. diff --git a/Docs/Apps/fleet b/Docs/Apps/fleet index 46453ea..f274df9 100644 --- a/Docs/Apps/fleet +++ b/Docs/Apps/fleet @@ -798,6 +798,173 @@ manages becomes a way to reach the nodes *it* manages. `relation` may travel because it answers about one node the caller already named; these answer about all of them. +## Docker: stacks, and bringing them up to date (Fleet → Docker) + +The `docker` right, on one machine at a time. Everything is read from what the +machine runs — there is no database of stacks anywhere, ours or anybody's — and +the code is split along that line: `fleet_docker.py` speaks the Engine API over +the machine's socket, `fleet_stacks.py` turns its containers into stacks and +updates them, `fleet_portainer.py` speaks Portainer's API. + +### Finding the stacks, and who owns each one + +A **stack** is a compose project. Compose labels every container it creates with +its project, its service, the files it read and the directory it read them from, +so one `GET /containers/json` is enough to rebuild the list. Each stack then has +an **owner**: + +| Owner | Recognised by | Its file lives | +|---|---|---| +| **portainer** | its directory is `/data/compose/` | in Portainer's volume and database | +| **compose** | anything else | on the machine, where the label says | + +Portainer **labels nothing of its own** on a stack's containers (measured on +Portainer CE 2.45): it runs compose with its stack directory, which is a path +inside its own container. That path is what gives it away, and the number in it +is Portainer's stack id — used to match the stack to Portainer's record exactly +rather than by a name that another environment may reuse. + +A **Portainer server** on the machine is found from its container +(`io.portainer.server=true`, or the `portainer/portainer[-ce|-ee]` image), with +the address it publishes as seen from the machine itself (`https://127.0.0.1:9443` +for a wildcard bind). A `portainer/agent` container is noted too: that machine's +stacks are deployed by a Portainer elsewhere. None of this needs a credential. + +`docker compose` is *asked* (`docker compose version`, then the old +`docker-compose`), never assumed from a `docker` binary on the PATH — the CLI is +very often installed without the plugin. The answer is believed for ten minutes. + +### How *Update* brings a stack up to date + +Each stack says, before anything is done, which of three acts its Update is and +why (`method`, `why` in the `stacks` answer): + +1. **Portainer** — owner Portainer, and a Portainer connected on that node. Its + API does what its own *Pull and redeploy* does: a git stack is redeployed + from its reference, a file stack with the file Portainer holds, both with + `PullImage`. The stack's **environment is read and handed straight back**: + Portainer *replaces* a stack's environment with what a redeploy carries, so + an empty list there erases it (BUGSVULNS 117). A git stack whose credentials + Portainer stored is asked to use them (authentication on, empty password); + saying "no authentication" makes Portainer forget them. Recent Portainer + answers a redeploy while it is still deploying (status 3) and recreates the + containers afterwards, so the update is done only once Portainer says so + (polled every `SETTLE_POLL`, 2 s); a failed deployment is reported with + Portainer's own message. +2. **Compose** — owner compose, its files readable by the node's account, and + compose installed: `compose pull` (`--ignore-buildable` from 2.15), then + `compose up -d`, with `--project-name`, `--project-directory` and every `-f` + the labels recorded. **No `--remove-orphans`**: a service dropped from the + file is somebody's decision about their data, not a side effect of an update. +3. **Engine** — everything else: a Portainer stack with no Portainer connected, + a file the node cannot read, no compose. Each image the stack's containers + were created from is pulled **once**, and each container whose image moved is + recreated (below). A service that cannot be updated costs only itself; the + update as a whole then fails, naming each one and why. + +A Portainer stack is **never** updated by re-running compose on a copy of +Portainer's directory, though that directory can be read through the Engine: a +relative bind (`./data`, which Portainer resolves to `/data/compose//data` on +the *host*) and an `env_file: stack.env` resolve against a different directory, +and the containers come back configured differently — the compose hash of such a +service changes, measured. The Engine way keeps exactly what runs. + +### Recreating a container through the Engine + +`fleet_stacks.recreate`: the container's own configuration (`GET +/containers//json`), less what the **old image** supplied, becomes the body of +its replacement — so the new image's environment, command, labels, ports and +volumes apply as they would to a container created fresh, and what the container +was *given* is kept. In detail: + +- environment entries, labels, exposed ports and volumes equal to the old + image's are dropped; `Entrypoint` likewise, and `Cmd` only when the entrypoint + is the image's too (`--entrypoint` clears the image's command, so a command + beside a custom entrypoint was given); +- the hostname is dropped when it is the old container's id (the daemon's + default), kept when somebody chose it; +- **anonymous volumes are carried over by name** — as a bind for one the image + declared, as the missing `Source` of one compose declared in `HostConfig.Mounts` + — or the replacement would start with an empty one; +- network aliases are kept without the old short id; static addresses + (`IPAMConfig`) are kept; the primary network is attached at create time and the + others joined after (an Engine before API 1.44 attaches one at create); +- `com.docker.compose.image` is set to the new image id, so compose does not read + the container as diverged and recreate it again. + +The sequence is stop, rename the old one aside, create, join networks, start, and +check the new one is **still running after `SETTLE` (5 s)**. If anything fails, or +the new one stopped, it is deleted and the old one renamed back and restarted: +a bad image costs seconds of downtime, not the service. Only then is the old one +deleted — without `v=1`, so its volumes stay. A container named by digest, or by +image id, is `pinned` and left alone; one that another container shares a network +with (`network_mode: container:…`) is refused, because recreating it alone would +strand its partner. + +### Checking for updates + +`check` asks, for each image reference the machine's containers use (at most +`MAX_CHECKS`, six at a time), what the registry serves under it now — through the +daemon's `GET /distribution//json`, which reads a manifest and downloads +nothing — and compares it with the image's `RepoDigests`. Each container is then +`current`, `outdated`, `pulled` (a newer image is already here and the container +still runs the old one), `pinned`, `local` (never came from a registry) or +`unknown` (the registry would not say, with its reason). A container listed by +image id is looked up by the name it was created from: that is exactly the one +whose tag has moved on. + +A private registry is reached with the node account's own `docker login` +(`~/.docker/config.json`, `auths`), sent to the local daemon as `X-Registry-Auth` +and nowhere else. Credentials kept by a credential helper are not in that file, +so such a registry is asked anonymously and says so in its refusal. + +### Connecting a Portainer + +Optional, per node, and only useful for stacks Portainer owns. The panel offers +the address the Portainer container publishes; **Read its certificate** (`op: +portainer_probe`) connects without a token, reads the version Portainer reports +at `/api/system/status` and the SHA-256 of the certificate it presents, and the +operator pins that fingerprint — the self-signed certificate of a Portainer on a +private network is accepted only as an anchor somebody chose. Plain HTTP is +accepted only to the machine's own loopback. The token is then proved (an +`/api/endpoints` call) before it is stored in the encrypted drawer, and is never +read back over the mesh: what comes back is `{connected, url, pinned}`. + +A stopped Portainer stack has no container to be found by; with Portainer +connected, those of **this machine's** environment are listed too (Portainer +manages other machines through their agents, and their stacks are theirs). + +### Start, stop, restart, the file, a new stack + +Start, stop and restart follow the same owner: Portainer's API (which has no +restart, so a stop and a start), else compose with the file, else container by +container with dependencies (`com.docker.compose.depends_on`) started first and +stopped last. *Down* needs the compose file and is refused for a Portainer stack. + +**File** shows the compose file a stack came from: Portainer's through its API, or +read out of the Portainer container through the Engine's archive route when none +is connected (the volume's files are root's on the host; the node's account is +not), or the file on disk. Only the compose file: the `stack.env` beside it holds +the stack's variables, and values do not travel — names do. + +**Deploy a stack** writes the file under the node's own state +(`/stacks//docker-compose.yml`, mode 600) and runs `compose up -d`; +or, with Portainer connected, hands it to Portainer on this machine's environment, +which then owns it. + +### On the page + +One machine at a time, five views: **Stacks** (a card per stack — owner, state, +what Update will do, the services with their image status, and the switch that +adds it to the machine's *Update*), **Containers** (all, or only those outside a +stack; a standalone container has its own Update), **Images**, **Volumes & +networks**, and **Portainer**. The summary above them says which engine and which +compose run there, whether a Portainer is found or connected, and how many +containers are behind after a check. A long act — an update, a check, a pull, a +deploy — shows on its card with its elapsed time and is asked about every two +seconds (`op: answer`) until the node replies; an update's result stays on the +card, with its output. + ## Groups: pointing at several machines at once A group is **this console's own name** for a set of machines. It says nothing to @@ -1217,11 +1384,19 @@ per container verb: that would be twenty entries in the dispatch table for one capability, and the capability is the gate. An operation that is not in the table is refused by name. -The reads a Docker panel needs beyond the container list are operations too: +The operations: `overview`, `containers`, `container`, `logs`, `act`, `images`, `stats` (CPU share and memory of up to `MAX_STATS` running containers, sampled -side by side and shown when they arrive), `volumes`, `networks`, and -`remove_image` (a name that is not an image, or that contains `..` because it -becomes part of a request path, never reaches the socket). +side by side and shown when they arrive), `volumes`, `networks`, `remove_image` +(a name that is not an image, or that contains `..` because it becomes part of a +request path, never reaches the socket), `pull`, `deploy` (one container), +`stacks`, `stack` (`start`, `stop`, `restart`, `down`, `update`), `stack_file`, +`check`, `recreate`, `deploy_stack`, `portainer` (connect or forget) and +`portainer_probe`. What each does is the Docker section above. + +`pull`, `deploy`, `deploy_stack`, `check`, `recreate` and `stack` with +`update` are **slow**: counted against `MAX_DOCKER_SLOW` as well as +`MAX_DOCKER_CALLS`, so a page can still read a machine whose stacks are being +updated. One stack is changed by one act at a time (`_stacks_busy`). **No new mesh packet type**: everything travels in `DATA`, section `builtin_id("fleet")`. The node's management plane is untouched. @@ -1253,7 +1428,10 @@ accepted only if they answer a `rid` **we** issued, for that kind of action, | `MAX_FILE_CALLS` | 8 file operations hosted at once, per peer | | `MAX_UPLOADS` / `UPLOAD_IDLE` | 2 uploads per operator / reaped after 300 s | | `MAX_GRANT_SESSIONS` | 4 passwordless console sessions per operator | -| `MAX_DOCKER_CALLS` | 2 docker operations hosted at once, per peer | +| `MAX_DOCKER_CALLS` / `MAX_DOCKER_SLOW` | 4 docker operations hosted at once per peer, of which 2 slow ones (pulls, updates, checks, deploys) | +| `MAX_CHECKS` / `CHECK_CONCURRENCY` | 40 image references asked of registries per check, 6 at a time | +| `SETTLE` | 5 s a recreated container must stay up, or the old one is put back | +| `MAX_ARCHIVE` | 2 MiB, 64 members read out of a container (a Portainer stack's directory) | | `MAX_UPDATE_STACKS` | 32 stacks one node's *Update* may also bring up | | `MAX_GROUPS` / `MAX_GROUP_NODES` | 32 groups / 512 nodes in one | | `MAX_RESPONSE` (docker) | 4 MB read from one Engine API answer | @@ -1334,7 +1512,7 @@ All behind the console session (Bearer or cookie). Details: |---|---|---| | GET | `/api/fleet/state?since=` | snapshot (nodes, notifications, log, capabilities, facts, **jobs**, scans) | | GET | `/api/fleet/shell?sid=&offset=[&wait=1]` | terminal bytes (base64); `wait=1` holds the answer until there are some | -| POST | `/api/fleet/docker` | `{node, op, …}` → one docker operation. The reads are waited on; the ones that pull images hand back a `rid` and are watched in the job list | +| POST | `/api/fleet/docker` | `{node, op, …}` → one docker operation. The reads are waited on; the slow ones (pulls, updates, checks, deploys) hand back a `rid`, and `{node, op: "answer", rid}` returns their answer once it is in, `{pending: true}` until then | | POST | `/api/fleet/stacks` | `{node, stacks[]}` → which stacks *Update* also brings up there | | POST | `/api/fleet/groups` | `{op:set\|remove\|node, …}` → groups, local to this console | | POST | `/api/fleet/update-group` | `{group}` → *Update* on every node in it | @@ -1384,6 +1562,17 @@ All behind the console session (Bearer or cookie). Details: secret leaks nowhere, the bootstrap is not an injection surface, and the publishers a pre-authorisation carries are bounded, de-duplicated, and dropped when they are not keys. +- [`tests/test_fleet_docker.py`](../../tests/test_fleet_docker.py) — the Engine + client against a socket that answers like a daemon (both framings, an error + inside a 200, an oversized answer, an engine that dropped our API version, an + archive carrying a link or a climbing name) and Portainer's client (the + environment handed back on a redeploy, stored git credentials kept, a probe + that sends no token, HTTP only to loopback). +- [`tests/test_fleet_stacks.py`](../../tests/test_fleet_stacks.py) — stacks from + labels (a Portainer stack known by its directory), the update each stack gets, + the replacement of a container (what the old image supplied left out, + anonymous volumes and aliases kept, the old one put back when the new one does + not stay up), the states a check tells apart, and the compose argv. - [`tests/test_console_fleet.py`](../../tests/test_console_fleet.py) — routes and app toggles. - [`tests/integration/test_fleet.py`](../../tests/integration/test_fleet.py) — diff --git a/Docs/Architecture/README.md b/Docs/Architecture/README.md index 6720f8f..74b55de 100644 --- a/Docs/Architecture/README.md +++ b/Docs/Architecture/README.md @@ -72,7 +72,7 @@ is therefore safe to accept from strangers. | `app_web.py` | The routes an app's page calls, **read from the bundle the browser is sent** — what lets an app that declares no operations still be driven (`web` module, MCP tools) without anybody keeping a second list. | | `app_api.py` | **The app API surface**: an app declares its operations (`API` + `api_`), and everything else — another app, the core, a page — calls them the same way. Reject by default: nothing that is not declared, every argument coerced and bounded. See [`../AppAPI/guide`](../AppAPI/guide). | | `app_registry.py` | The registry of **built-in** apps (installed / enabled, persisted) + `AppHost`, which starts and stops them live. | -| `apps/fleet*.py` | The management/deployment app: protocol and roles (`fleet.py`), the capability ledger (`fleet_state.py` — grants, the groups an operator names, and the credentials this node holds for something else), machine facts and the update plan (`fleet_host.py`), LAN scan + SSH over a pty (`fleet_ssh.py`), provisioning bootstrap (`fleet_provision.py`), files under the `shell` right (`fleet_files.py`), **docker** (`fleet_docker.py`: the Engine API over the machine's own socket, and stacks recovered from the labels compose writes — no second source of truth) and the **Portainer** in front of it (`fleet_portainer.py`: a stack it owns is updated through it, or the next thing it does undoes the update), the relay to the local console (`fleet_console.py`), the console bridge (`fleet_web.py`). | +| `apps/fleet*.py` | The management/deployment app: protocol and roles (`fleet.py`), the capability ledger (`fleet_state.py` — grants, the groups an operator names, and the credentials this node holds for something else), machine facts and the update plan (`fleet_host.py`), LAN scan + SSH over a pty (`fleet_ssh.py`), provisioning bootstrap (`fleet_provision.py`), files under the `shell` right (`fleet_files.py`), **docker** (`fleet_docker.py`: the Engine API over the machine's own socket), its **stacks** (`fleet_stacks.py`: recovered from the labels compose writes — no second source of truth — each with an owner, compose or Portainer, told apart by the directory it was deployed from, and updated the way that owner would: through Portainer's API, `compose pull && up -d`, or by recreating each container on its new image through the Engine) and the **Portainer** in front of it (`fleet_portainer.py`: found from its container, connected by a pinned certificate and a write-only token), the relay to the local console (`fleet_console.py`), the console bridge (`fleet_web.py`). | | `session_store.py` | Encrypted persistence: E2E sessions + peers (`SessionStore`), and the **names this node has learned** (`PseudoStore`, its own file and cadence — the session blob is rewritten every couple of seconds and a book of 5 kB claims has no business on that path). | | `join_ticket.py` | **One invitation, both ways round**: the inviter's own endpoint *and* a relay to reach it through, plus the code's seed, an expiry and a checksum, in unpadded base32 (34 characters for a direct IPv4 one, 76 for both routes). That is what removes the second exchange — a ticket used to be the direct case and a block of base64 the other, and an operator had to know which they were in first. Defensive decoding: bounded, everything validated, never anything but a `TicketError`. | | `qr.py` | A QR encoder (ISO/IEC 18004) in pure stdlib: versions 1–10, levels M/L, alphanumeric and byte modes, Reed-Solomon and mask selection. Verified against an independent encoder and a real decoder. | diff --git a/Docs/Architecture/gotchas.md b/Docs/Architecture/gotchas.md index e1b007b..6b0f664 100644 --- a/Docs/Architecture/gotchas.md +++ b/Docs/Architecture/gotchas.md @@ -2729,3 +2729,75 @@ and the same guard file refuses a random port pick anywhere under `tests/`. other and answered yes, so the node was never enrolled. The sweep now removes them all first and judges afterwards. The same shape is why the other two call sites pass `exclude=peer`. + +## Docker: who owns a stack, and what an update must keep + +The fleet app's Docker panel (`src/apps/fleet_stacks.py`, `fleet_portainer.py`; +the whole mechanism is in `Docs/Apps/fleet`, "Docker"). Each trap below was in +the code, and each looked right from reading it. + +- **Portainer labels nothing of its own.** The first version recognised a + Portainer stack by an `io.portainer.stack.name` label; Portainer CE 2.45 + writes no such label (nor any `io.portainer.*` one) on a stack's containers, + so every Portainer stack was listed as a plain compose project and updated + with `compose` on paths under `/data/compose/` — which exist inside + Portainer's container, not on the host. What gives a Portainer stack away is + that directory, and the number in it is Portainer's stack id (BUGSVULNS 117). + Before naming a label as evidence, look at a real container + (`docker inspect … --format '{{json .Config.Labels}}'`). +- **A Portainer redeploy *replaces* the stack's environment.** `PUT + /api/stacks/` with `"Env": []` erases every variable the stack had, and + Portainer recreates the containers without them — reproduced live: + `GREETING=hello` came back `GREETING=`. The environment is read from + `GET /api/stacks/` and handed back unchanged. Same shape for a git stack: + `RepositoryAuthentication: false` tells Portainer to drop the credentials it + stored; asking for them with an empty password makes it use its own. +- **Portainer deploys asynchronously.** A create or redeploy answers at once + with `Status: 3` (deploying) on recent versions, and refuses a second + redeploy while one runs (`Invalid request payload`). An answer is not a + finished deployment: `Portainer._settled` asks `GET /api/stacks/` every + `SETTLE_POLL` until the status leaves 3, and reads a failure from the last + `DeploymentStatus` entry carrying a `Message`. Reported done at the answer, + the page redrew in the middle of the recreation, showing compose's temporary + `_` containers beside the real ones. +- **A copy of Portainer's directory is not Portainer's stack.** The directory + can be read out of the Portainer container through the Engine's archive route, + and `compose` run on the copy. It does not reproduce what runs: a relative + bind (`./data`) was resolved by Portainer against `/data/compose/` — a + path it then handed the daemon as a *host* path — and an `env_file: + stack.env` against the same directory. With `--project-directory + /data/compose/` compose cannot find the env file; without it, the bind + moves and the service's compose hash changes (measured: `web` changed, the + service without a relative path did not). Without a connected Portainer, a + Portainer stack is updated through the Engine, container by container. +- **Recreating a container: what the image gave is not what the container was + given.** A container's `Config` carries its image's environment, command, + labels, ports and volumes merged in. Copied whole, the replacement runs the + new image with the old one's defaults (an `APP_VERSION=1` on version 2). Each + is compared with the old image's own `Config` and dropped when equal. + `Cmd` is dropped only together with `Entrypoint`: `--entrypoint` clears the + image's command, so a command next to a custom entrypoint was given. +- **…and the anonymous volumes are the container's data.** An image `VOLUME` + with no named volume behind it gets an anonymous one, named by a 64-hex id and + listed only under `Mounts`; compose declares its own in `HostConfig.Mounts` + with *no* `Source`. Recreated as written, both give the replacement a fresh, + empty volume. The name is carried over: as a bind for the first, as the + missing `Source` for the second. And the old container is removed without + `v=1`. +- **The old container is put aside, not removed, until the new one stays up.** + Stop, rename aside, create, start, wait `SETTLE`; on any failure the new one + is deleted and the old one renamed back and restarted. Removing first turns a + bad image into an outage. +- **`docker` on the PATH is not compose.** The CLI is very often installed + without the compose plugin; `compose_available()` answered from + `shutil.which("docker")` and offered updates that then failed. Compose is + asked (`docker compose version`, then `docker-compose`). +- **A pinned API version ages.** The client spoke `/v1.41/` only. An engine + raises its minimum over the years (29.x accepts 1.40; later ones will not), so + the version is raised once, from the daemon's own refusal ("Minimum supported + API version is …"), rather than asked before every call. +- **The page's router handler was replaced by its own start.** `enter()` called + `ROUTER.start(() => {})`, and `start` keeps the handler it is given — so the + `ROUTER.onChange` set beside the panels, which loads the Docker panel when it + is opened and refreshes the Logs one, never ran (BUGSVULNS 119). Call + `ROUTER.start()` with no argument where the handler is set elsewhere. diff --git a/Docs/WebConsole/guide b/Docs/WebConsole/guide index 264f7e6..657f406 100644 --- a/Docs/WebConsole/guide +++ b/Docs/WebConsole/guide @@ -810,7 +810,7 @@ now answers `{"error": …}` with the same status. | POST | `/api/pseudo` | `{pseudo}` → renames this node (empty removes the name); signs a fresh claim, announces it, and records it in the configuration file when there is one (the node keeps the name across a restart either way — see [`../Pseudos/guide`](../Pseudos/guide)) | | GET | `/api/fleet/state` | `?since=N` → managed nodes, enrolment notifications, log, capabilities, machine facts. `&proto=2&have=name:rev,…` asks for the **sectioned** shape and gets back only what moved (`src/console_feed.py`); without `proto` it is the flat snapshot it has always been | | GET | `/api/fleet/shell` | `?sid=&offset=` → terminal bytes (base64); `&wait=1` holds the answer until there are some (bounded in time, and in how many are parked at once). `?rid=` instead of `sid` reads the shell a `POST` just opened, or answers `409 {error, failed: true}` with the far node's refusal | -| POST | `/api/fleet/docker` | `{node, op, …}` → one docker operation on a managed node (`docker` right). Reads are waited on; the ones that pull images hand back a `rid` watched in the job list | +| POST | `/api/fleet/docker` | `{node, op, …}` → one docker operation on a managed node (`docker` right). Reads are waited on; the slow ones (pulls, stack updates, update checks, deploys) hand back a `rid`, and `{node, op:"answer", rid}` returns their answer once it is in, `{pending:true}` until then | | GET | `/api/fleet/logs` | `?node=&level=&source=&contains=&since_time=&until_time=` → the lines this node has **already collected** from the machines it manages, newest first, with the policies and what is being followed. Reads our own rings: it never asks the network, so scrolling a log cannot become traffic towards forty machines. Naming a node here is also what says somebody is *looking* at it, which is what the `active` policy follows | | POST | `/api/fleet/logs-fetch` | `{node}` → ask that machine for its log now, whatever its policy says about collecting it | | POST | `/api/fleet/logs-policy` | `{policy?, megabytes?}` for the fleet-wide default, or `{node, policy?, megabytes?, inherit?}` for one machine | @@ -930,8 +930,10 @@ button in the Apps section). The page has six sections, split by the question being asked: **Nodes** (the ones we control, with their status and the update / docker / shell / scan actions), **Who controls this node** (the ones controlling us, with their rights -and the pending requests), **Docker** (containers, stacks, images and the -Portainer in front of them, on one node at a time), **Discover & deploy** (LAN +and the pending requests), **Docker** (stacks — compose's and Portainer's, found +from what runs — with what *Update* will do for each, containers, images, +volumes and networks, and the Portainer in front of them, on one node at a +time; see [`../Apps/fleet`](../Apps/fleet), "Docker"), **Discover & deploy** (LAN discovery and SSH provisioning), **Shell** (an interactive terminal), **Activity** (the log). diff --git a/TEST.md b/TEST.md index 69494ab..b93256f 100644 --- a/TEST.md +++ b/TEST.md @@ -484,13 +484,29 @@ tests/ │ escaped; a container is *built* from a validated shape rather than │ forwarded, so `Privileged` cannot ride in; a bind mount is two absolute │ paths or a volume name; a command is split, never handed to a shell), the -│ stacks recovered from compose's own labels, the multiplexed log header -│ taken off rather than printed, and the wire itself driven against a socket -│ that answers like a daemon — both framings, the daemon's own message -│ surfaced, a failed pull *inside* a 200 still a failure, an oversized answer -│ refused. For Portainer: an address that is a host and nothing else, a token -│ that cannot become a second header, a fingerprint pin, and a git stack -│ redeployed through git rather than handed the file it already had +│ multiplexed log header taken off rather than printed, and the wire itself +│ driven against a socket that answers like a daemon — both framings, the +│ daemon's own message surfaced, a failed pull *inside* a 200 still a +│ failure, an oversized answer refused, an engine that dropped our API +│ version asked again in its own, a reference split where docker splits it, +│ a private registry reached with this account's login and no other's, an +│ archive read without following a link or a climbing name. For Portainer: +│ an address that is a host and nothing else (HTTP to loopback only), a token +│ that cannot become a second header, a fingerprint pin, a probe that sends +│ no token, a git stack redeployed through git with the credentials +│ Portainer stored, the stack's environment handed back rather than erased, +│ and a stack matched to its record by id, never by an ambiguous name +├── test_fleet_stacks.py — stacks recovered from +│ compose's own labels (one-offs left out, bounded, whatever the daemon +│ sends); a Portainer stack known by its directory, a Portainer server or +│ agent by its container; which act each stack's Update is; the replacement +│ of a container (what the old image supplied left to the new one, what the +│ container was given kept, anonymous volumes and aliases carried, the old +│ one put back when the new one does not stay up, a digest left alone, a +│ shared network refused); an update where one service failing costs only +│ itself; the six states a check tells apart; compose asked rather than +│ assumed, its argv, no `--remove-orphans`; a stack's file read without its +│ environment; a new stack's file written privately under the node's state ├── test_fleet_deploy.py — remote deployment and the │ right to update: the authorised script is not inside the node's prefix, the │ rule names one path with no wildcard, the wrapper refuses every argument, diff --git a/pyproject.toml b/pyproject.toml index f5ba731..4ac1811 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.backends.legacy:build" [project] name = "nmesh" -version = "0.4.115" +version = "0.4.116" license = "GPL-3.0-only" [tool.pytest.ini_options] diff --git a/src/apps/fleet.py b/src/apps/fleet.py index 16f4519..164c0e5 100644 --- a/src/apps/fleet.py +++ b/src/apps/fleet.py @@ -67,7 +67,7 @@ from ..app_channel import builtin_id from ..node_id import NodeID from . import (fleet_console, fleet_docker, fleet_files, fleet_host, - fleet_portainer, fleet_provision, fleet_ssh) + fleet_portainer, fleet_provision, fleet_ssh, fleet_stacks) from .fleet_state import (CAPABILITIES, FleetState, _is_node_hex, clean_caps, clean_label, clean_stack_names) @@ -190,11 +190,12 @@ INVITE_TTL_DEFAULT = 300.0 INVITE_TTL_MAX = 6 * 3600.0 MAX_KEY_DATA = 64 * 1024 # an uploaded private key, bounded -# Docker. The engine calls are quick; the ones that pull images are minutes, and -# they are named here rather than guessed at the call site. -DOCKER_TIMEOUT = 60.0 -DOCKER_SLOW_TIMEOUT = 1800.0 # a pull, a `compose up`, a Portainer redeploy -MAX_DOCKER_CALLS = 2 # docker operations we host at once, per peer +# Docker. The reads are one round trip; an update pulls images for minutes and +# holds a slot for all of them, so the two are counted apart — a page must still +# be able to read while a stack it started is being brought up to date. +MAX_DOCKER_CALLS = 4 # docker operations we host at once, per peer +MAX_DOCKER_SLOW = 2 # of which pulls, updates and deploys +_DOCKER_SLOW_OPS = ("pull", "deploy", "deploy_stack", "check", "recreate") # Lines of a stack's own output forwarded while an update runs. A `compose pull` # is a wall of progress bars; what an operator needs is that it is moving, and # every frame past that is a task queued against `MAX_INFLIGHT` for nothing. @@ -541,7 +542,9 @@ def __init__(self, client, auth, *, state: FleetState | None = None, self._reaper: asyncio.Task | None = None self._shells: dict[bytes, _Shell] = {} # agent side self._open_shells: dict[bytes, NodeID] = {} # operator side - self._docker_calls: dict[bytes, int] = {} # agent side, per peer + # agent side, per peer: (operations held, of which slow) + self._docker_calls: dict[bytes, tuple[int, int]] = {} + self._stacks_busy: set[str] = set() # stacks being changed self._inflight: dict[str, tuple[NodeID, str, float]] = {} self._jobs: set[asyncio.Task] = set() # Remote console. Agent side: the loopback client that replays a call @@ -2024,7 +2027,7 @@ async def _run_update(self, src: NodeID, rid: str, "name": f"stack {name}", "elapsed": round(time.monotonic() - started, 1)}}) try: - await self._update_one_stack(name, src, rid, kind="update") + await self._update_one_stack(name, src, rid) done.append(name) except fleet_docker.DockerError as exc: ok, status, failed = False, -1, f"{name}: {exc}"[:200] @@ -2144,15 +2147,16 @@ def _on_docker_request(self, src: NodeID, principal, document: dict) -> None: # Bounded per peer, like the console relay and the file plane: a pull is # minutes of somebody else's disk and network, and "several at once" is # not something an operator ever wanted. - held = self._docker_calls.get(src.raw, 0) - if held >= MAX_DOCKER_CALLS: + slow = _docker_slow(op, document) + held, held_slow = self._docker_calls.get(src.raw, (0, 0)) + if held >= MAX_DOCKER_CALLS or (slow and held_slow >= MAX_DOCKER_SLOW): self._fail(src, rid, "too many docker operations already running") return - self._docker_calls[src.raw] = held + 1 - self._spawn(self._run_docker(src, rid, op, document)) + self._docker_calls[src.raw] = (held + 1, held_slow + int(slow)) + self._spawn(self._run_docker(src, rid, op, document, slow)) async def _run_docker(self, src: NodeID, rid: str, op: str, - document: dict) -> None: + document: dict, slow: bool = False) -> None: try: data = await _DOCKER_OPS[op](self, src, rid, document) except fleet_docker.DockerError as exc: @@ -2164,9 +2168,10 @@ async def _run_docker(self, src: NodeID, rid: str, op: str, self._fail(src, rid, f"that failed ({type(exc).__name__})") return finally: - held = self._docker_calls.get(src.raw, 1) - 1 + held, held_slow = self._docker_calls.get(src.raw, (1, int(slow))) + held, held_slow = held - 1, held_slow - int(slow) if held > 0: - self._docker_calls[src.raw] = held + self._docker_calls[src.raw] = (held, max(held_slow, 0)) else: self._docker_calls.pop(src.raw, None) if data is None: @@ -2189,14 +2194,31 @@ def _on_docker_reply(self, src: NodeID, document: dict) -> None: # -- the operations --------------------------------------------------- async def _docker_overview(self, _src, _rid, _document) -> dict: + """What this machine's docker is, and what manages it — said rather + than raised when there is none, because "no docker here" is an answer + a page draws, not a failure it reports.""" + portainer = self._portainer_state() if not fleet_docker.available(): - raise fleet_docker.DockerError("this machine runs no docker") - info = await fleet_docker.overview() - info["portainer"] = bool(self._portainer_config()) + return {"info": {"engine": False, "portainer": portainer, + "problem": "this machine runs no docker"}} + try: + info = await fleet_docker.overview() + entries = await fleet_docker.raw_containers(True) + except fleet_docker.DockerError as exc: + return {"info": {"engine": False, "portainer": portainer, + "problem": str(exc)[:200]}} + argv, version = await fleet_stacks.compose() + found = fleet_stacks.find_portainer(entries) + portainer.update(server=found["server"], agent=found["agent"]) + info.update(engine=True, compose=version if argv else "", + stacks=len(fleet_stacks.discover(entries, root=self._state_dir)), + portainer=portainer) return {"info": info} async def _docker_containers(self, _src, _rid, _document) -> dict: - return {"items": await fleet_docker.containers(True)} + items = await fleet_docker.containers(True) + await fleet_stacks.name_images(items, "id") + return {"items": items} async def _docker_container(self, _src, _rid, document) -> dict: return {"info": await fleet_docker.inspect(document.get("id"))} @@ -2232,44 +2254,93 @@ async def _docker_deploy(self, _src, _rid, document) -> dict: return {"info": await fleet_docker.create(spec if isinstance(spec, dict) else {})} async def _docker_stacks(self, _src, _rid, _document) -> dict: - items = await fleet_docker.stacks() - # A stack Portainer owns is updated through Portainer, so the list says - # which ones those are — an operator choosing "update" on one of them is - # choosing between two different acts, and has to be able to see that. - portainer = self._portainer() - managed: dict = {} - if portainer is not None: - try: - for entry in await portainer.stacks(): - managed[entry["name"]] = entry - except fleet_docker.DockerError: - managed = {} # unreachable is not fatal here + """Every stack on this machine, each saying who owns it and how + *Update* would bring it up to date.""" + entries = await fleet_docker.raw_containers(True) + items = fleet_stacks.discover(entries, root=self._state_dir) for stack in items: - match = managed.get(stack["name"]) or managed.get(stack.get("portainer") or "") - if match: - stack["portainer_id"] = match["id"] - stack["portainer_endpoint"] = match["endpoint"] - stack["git"] = match["git"] - return {"items": items, "info": {"compose": fleet_docker.compose_available(), - "portainer": portainer is not None}} + await fleet_stacks.name_images(stack["services"]) + argv, version = await fleet_stacks.compose() + listed, problem = await self._portainer_stacks() + for stack in items: + record = fleet_portainer.match(stack, listed) if listed is not None else None + stack["portainer_api"] = record is not None + if record is not None: + stack["portainer_id"] = record["id"] + stack["git"] = record["git"] + stack.update(fleet_stacks.update_method( + stack, portainer_api=record is not None, compose_argv=argv)) + # What Portainer holds and nothing runs: a stopped Portainer stack has + # no container to be found by, and is still somebody's stack — if it + # is this machine's. One Portainer manages other machines through + # their agents, and their stacks are theirs. + seen = {stack.get("portainer_id") for stack in items} + local = await self._portainer_local(listed, seen) + for record in listed or []: + if record["id"] not in seen and not record["active"] \ + and record["endpoint"] == local \ + and len(items) < fleet_stacks.MAX_STACKS: + items.append({"name": record["name"], "owner": "portainer", + "portainer_id": record["id"], "portainer_api": True, + "git": record["git"], "services": [], "running": 0, + "total": 0, "files": [], "workdir": "", + "files_ok": False, "nmesh": False, + "method": "portainer", "why": "stopped in Portainer"}) + return {"items": items, + "info": {"compose": version if argv else "", + "portainer": self._portainer_state(), + "portainer_problem": problem}} + + async def _docker_check(self, _src, _rid, document) -> dict: + entries = await fleet_docker.raw_containers(True) + stack = document.get("stack") + if isinstance(stack, str) and stack: + entries = [entry for entry in entries + if (entry.get("Labels") or {}).get(fleet_stacks.LABEL_PROJECT) == stack] + return {"info": await fleet_stacks.check(entries)} + + async def _docker_recreate(self, _src, _rid, document) -> dict: + return {"info": await fleet_stacks.recreate(document.get("id"))} async def _docker_stack(self, src, rid, document) -> dict: name = clean_stack_names([document.get("stack")]) if not name: raise fleet_docker.DockerError("that is not a stack") action = str(document.get("action"))[:16] - if action == "update": - return {"info": await self._update_one_stack(name[0], src, rid)} - found = await self._stack_record(name[0]) - return {"info": await fleet_docker.stack_act( - name[0], fleet_docker.compose_files(found.get("files", "")), action, - found.get("workdir", ""))} + if action not in ("start", "stop", "restart", "down", "update"): + raise fleet_docker.DockerError("that is not something to do to a stack") + return {"info": await self._stack_do(name[0], action)} + + async def _docker_stack_file(self, _src, _rid, document) -> dict: + name = clean_stack_names([document.get("stack")]) + if not name: + raise fleet_docker.DockerError("that is not a stack") + stack = await self._stack_record(name[0]) + if stack.get("owner") == "portainer": + portainer = self._portainer() + listed, _problem = await self._portainer_stacks() + record = fleet_portainer.match(stack, listed or []) + if portainer is not None and record is not None: + return {"info": {"path": f"Portainer stack {record['id']}", + "text": await portainer.file(record["id"]), + "source": "portainer"}} + entries = await fleet_docker.raw_containers(True) + server = fleet_stacks.find_portainer(entries)["server"] + return {"info": await fleet_stacks.read_file( + stack, server=server["id"] if server else "")} + return {"info": await fleet_stacks.read_file(stack)} async def _docker_deploy_stack(self, _src, _rid, document) -> dict: name = clean_stack_names([document.get("stack")]) if not name: raise fleet_docker.DockerError("that is not a stack name") - return {"info": await fleet_docker.deploy_stack( + if document.get("portainer") is True: + portainer = self._require_portainer() + endpoint = fleet_portainer.local_endpoint(await portainer.endpoints()) + done = await portainer.deploy(name[0], document.get("compose"), endpoint) + done["through"] = "portainer" + return {"info": done} + return {"info": await fleet_stacks.deploy( name[0], document.get("compose"), root=self._state_dir)} async def _docker_portainer_set(self, _src, rid, document) -> dict: @@ -2281,7 +2352,7 @@ async def _docker_portainer_set(self, _src, rid, document) -> dict: than of using it.""" if document.get("clear") is True: self._store_portainer(None) - return {"info": {"configured": False}} + return {"info": self._portainer_state()} config = { "url": fleet_portainer.clean_url(document.get("url")), "token": fleet_portainer.clean_token(document.get("token")), @@ -2290,44 +2361,18 @@ async def _docker_portainer_set(self, _src, rid, document) -> dict: } if not config["url"] or not config["token"]: raise fleet_docker.DockerError( - "a Portainer needs an https address and an access token") + "a Portainer needs an https address (http only on this machine's " + "loopback) and an access token") portainer = fleet_portainer.Portainer( config["url"], config["token"], fingerprint=config["fingerprint"]) # Proved before it is stored: a configuration that has never worked is # a button that fails later, somewhere an operator is not looking. endpoints = await portainer.endpoints() self._store_portainer(config) - return {"info": {"configured": True, "url": config["url"], - "pinned": bool(config["fingerprint"])}, - "items": endpoints} - - async def _docker_portainer_stacks(self, _src, _rid, _document) -> dict: - portainer = self._require_portainer() - return {"items": await portainer.stacks(), - "info": {"url": self._portainer_config().get("url", "")}} + return {"info": self._portainer_state(), "items": endpoints} - async def _docker_portainer_stack(self, _src, _rid, document) -> dict: - portainer = self._require_portainer() - action = str(document.get("action"))[:16] - wanted = document.get("id") - found = None - for entry in await portainer.stacks(): - if entry["id"] == wanted: - found = entry - break - if found is None: - raise fleet_docker.DockerError("Portainer has no such stack") - if action == "redeploy": - return {"info": await portainer.redeploy(found)} - return {"info": await portainer.act(found["id"], found["endpoint"], action)} - - async def _docker_portainer_deploy(self, _src, _rid, document) -> dict: - portainer = self._require_portainer() - name = document.get("stack") - return {"info": await portainer.deploy( - name if isinstance(name, str) else "", - document.get("compose"), - document.get("endpoint"))} + async def _docker_portainer_probe(self, _src, _rid, document) -> dict: + return {"info": await fleet_portainer.probe(document.get("url"))} # -- Portainer configuration (this node's own) ------------------------- @@ -2335,6 +2380,12 @@ def _portainer_config(self) -> dict: raw = self.state.read_secret(_PORTAINER_KEY) return raw if isinstance(raw, dict) and raw.get("url") else {} + def _portainer_state(self) -> dict: + """What may be said about this node's Portainer: where, never how.""" + config = self._portainer_config() + return {"connected": bool(config), "url": config.get("url", ""), + "pinned": bool(config.get("fingerprint"))} + def _store_portainer(self, config: dict | None) -> None: self.state.write_secret(_PORTAINER_KEY, config) @@ -2352,53 +2403,114 @@ def _portainer(self): def _require_portainer(self): portainer = self._portainer() if portainer is None: - raise fleet_docker.DockerError("no Portainer is configured here") + raise fleet_docker.DockerError("no Portainer is connected here") return portainer - # -- bringing one stack up to date ------------------------------------ + async def _portainer_stacks(self) -> tuple[list | None, str]: + """Portainer's stacks, or None and why — an unreachable Portainer + degrades a stack to the Engine's way of updating it, it does not hide + the list.""" + portainer = self._portainer() + if portainer is None: + return None, "" + try: + return await portainer.stacks(), "" + except fleet_docker.DockerError as exc: + return None, str(exc)[:200] + + async def _portainer_local(self, listed, seen: set) -> int: + """Which of Portainer's environments is this machine: the one holding + a stack that runs here, else the one on this machine's socket.""" + for record in listed or []: + if record["id"] in seen: + return record["endpoint"] + portainer = self._portainer() + if not listed or portainer is None: + return 0 + try: + return fleet_portainer.local_endpoint(await portainer.endpoints()) + except fleet_docker.DockerError: + return 0 + + # -- one stack ---------------------------------------------------------- async def _stack_record(self, name: str) -> dict: - for stack in await fleet_docker.stacks(): + entries = await fleet_docker.raw_containers(True) + for stack in fleet_stacks.discover(entries, root=self._state_dir): if stack["name"] == name: return stack raise fleet_docker.DockerError("no stack of that name runs here") - async def _update_one_stack(self, name: str, src: NodeID, rid: str, - *, kind: str = "docker") -> dict: - """Update a stack the way whoever owns it would. - - Portainer first, when Portainer owns it: running `compose up` behind its - back leaves its record stale and the next thing it does undoes the - update. Otherwise compose, with the file the project was deployed from — - pulling images under a project and restarting it is *not* this act and - quietly does less, because a container keeps running the image it was - created from until it is recreated.""" - found = await self._stack_record(name) - portainer = self._portainer() + async def _stack_do(self, name: str, action: str, say=None) -> dict: + """Do one thing to a stack, the way whoever owns it would. + + Portainer first, when Portainer owns it and is connected: anything done + behind its back leaves its record stale. Then compose, with the file + the project came from. Then the Engine, container by container — which + is all that is left when neither is there, and is said to be.""" + if name in self._stacks_busy: + raise fleet_docker.DockerError("that stack is already being changed") + self._stacks_busy.add(name) + try: + return await self._stack_do_now(name, action, say) + finally: + self._stacks_busy.discard(name) + + async def _stack_do_now(self, name: str, action: str, say) -> dict: + listed, _problem = await self._portainer_stacks() + try: + stack = await self._stack_record(name) + except fleet_docker.DockerError: + # Stopped in Portainer: nothing runs, but Portainer can start it. + named = [entry for entry in listed or [] if entry["name"] == name] + if not named or action not in ("start", "update"): + raise + stack = {"name": name, "owner": "portainer", "portainer_id": named[0]["id"]} + record = fleet_portainer.match(stack, listed) if listed is not None else None + portainer = self._portainer() if record is not None else None if portainer is not None: - for entry in await portainer.stacks(): - if entry["name"] == name: - done = await portainer.redeploy(entry) - done["through"] = "portainer" - return done - # Streamed only where there is a stream to carry it. On the docker - # plane a reply *is* the answer — `_on_docker_reply` claims the request - # id and forgets it — so a progress frame there would consume the very - # rid the result has to come back under. + if action == "update": + return await portainer.redeploy(record) + if action == "down": + raise fleet_docker.DockerError( + "Portainer owns that stack; remove it in Portainer") + if action == "restart": + await portainer.act(record["id"], record["endpoint"], "stop") + action = "start" + done = await portainer.act(record["id"], record["endpoint"], action) + done.update(stack=name, through="portainer") + return done + argv, _version = await fleet_stacks.compose() + method = fleet_stacks.update_method(stack, portainer_api=False, compose_argv=argv) + if action == "update": + if method["method"] == "compose": + return await fleet_stacks.compose_update(stack, say=say) + done = await fleet_stacks.engine_update(stack, say=say) + if done["failed"]: + raise fleet_docker.DockerError("; ".join( + f"{step['service']}: {step.get('why', 'failed')}" + for step in done["steps"] if step["result"] == "failed")[:240]) + return done + if method["method"] == "compose": + return await fleet_stacks.compose_act(stack, action, say=say) + if action == "down": + raise fleet_docker.DockerError("taking a stack down needs its compose file") + return await fleet_stacks.engine_act(stack, action) + + async def _update_one_stack(self, name: str, src: NodeID, rid: str) -> dict: + """One stack of a node-wide *Update*, its progress streamed on the + update's own channel — the docker plane has none: a reply there *is* + the answer, and `_on_docker_reply` forgets the rid once it lands.""" sent = [0] def say(line: str) -> None: - if kind != "update" or sent[0] >= _STACK_OUTPUT_LINES: + if sent[0] >= _STACK_OUTPUT_LINES: return sent[0] += 1 self._reply(src, UPDATE_OUTPUT, - {"rid": rid, "kind": kind, "text": line[:UPDATE_CHUNK]}) + {"rid": rid, "kind": "update", "text": line[:UPDATE_CHUNK]}) - done = await fleet_docker.stack_up( - name, fleet_docker.compose_files(found.get("files", "")), - found.get("workdir", ""), on_output=say) - done["through"] = "compose" - return done + return await self._stack_do(name, "update", say) # ====================================================================== # Shell @@ -3596,14 +3708,21 @@ def _shell_env() -> dict: "deploy": FleetApp._docker_deploy, "stacks": FleetApp._docker_stacks, "stack": FleetApp._docker_stack, + "stack_file": FleetApp._docker_stack_file, + "check": FleetApp._docker_check, + "recreate": FleetApp._docker_recreate, "deploy_stack": FleetApp._docker_deploy_stack, "portainer": FleetApp._docker_portainer_set, - "portainer_stacks": FleetApp._docker_portainer_stacks, - "portainer_stack": FleetApp._docker_portainer_stack, - "portainer_deploy": FleetApp._docker_portainer_deploy, + "portainer_probe": FleetApp._docker_portainer_probe, } +def _docker_slow(op: str, document: dict) -> bool: + """Whether an operation pulls images or waits on a redeploy — minutes, + not a round trip.""" + return op in _DOCKER_SLOW_OPS or (op == "stack" and document.get("action") == "update") + + # Dispatch tables, built once. Keeping them beside the handlers makes the set of # accepted message types explicit: anything not listed here is dropped. _SIGNED_INBOUND = { diff --git a/src/apps/fleet_docker.py b/src/apps/fleet_docker.py index 262e7f0..733b384 100644 --- a/src/apps/fleet_docker.py +++ b/src/apps/fleet_docker.py @@ -1,20 +1,12 @@ """ -Docker, and the stacks on top of it — what a managed machine runs in containers. - -Three ways in, and they are deliberately not interchangeable: - -* **The Engine API**, over the machine's own socket. Everything that reads or - moves a container goes through it: listing, inspecting, logs, start/stop, - pulling an image, creating one. No CLI needed, no shell, no quoting surface. -* **`docker compose`**, when the binary is there. A *stack* is a compose - project, and the only honest way to bring one up to date is to re-run the - compose file it was deployed from — pulling images under a project and - restarting it is not the same act and quietly does less. -* **Portainer's API**, when this machine is running one. Portainer deploys - through compose and labels what it deploys, so its stacks already show up in - the list above; what its API adds is the stack *file* (git or string) and the - redeploy that fetches it again. A stack Portainer owns is updated through - Portainer, or the next thing Portainer does will undo it. +Docker's Engine API, spoken over the machine's own socket. + +Everything that reads or moves a container goes through here: listing, +inspecting, logs, start/stop, pulling an image, creating a container, reading a +file out of one. No CLI, no shell, no quoting surface. The stacks built on top +of these containers — compose projects, Portainer's among them — are +`fleet_stacks`'s; this module knows containers and images, and nothing about +who deployed them. **The socket is root.** A process that can talk to `/var/run/docker.sock` can start a privileged container bind-mounting `/`, which is the machine. That is @@ -24,24 +16,27 @@ not been through one. Everything arriving from an operator is hostile input: names, ids, image -references, compose files, Portainer URLs. Each one is validated against a -charset and a bound *before* it reaches a socket or an argv, and a value that -does not validate is refused rather than escaped — there is no escaping here to -get wrong. +references, compose files. Each one is validated against a charset and a bound +*before* it reaches a socket, and a value that does not validate is refused +rather than escaped — there is no escaping here to get wrong. What the daemon +answers is hostile input too, for the same reason a peer's is: every shape +handed on is built field by field and bounded. """ from __future__ import annotations import asyncio +import base64 +import io import json import os import re -import shutil -import time +import tarfile +from urllib.parse import quote # -- bounds ----------------------------------------------------------------- # Every one of these bounds something an operator, or the daemon, can grow. DEFAULT_SOCKET = "/var/run/docker.sock" -API_VERSION = "v1.41" # old enough for every engine still supported +API_VERSION = "1.41" # old enough for every engine still supported CALL_TIMEOUT = 30.0 # one Engine API call PULL_TIMEOUT = 900.0 # an image pull is minutes, not seconds COMPOSE_TIMEOUT = 1800.0 # `compose up` builds and pulls @@ -52,25 +47,22 @@ MAX_VOLUMES = 250 MAX_NETWORKS = 100 MAX_STATS = 40 # containers sampled at once; each sample takes ~1 s -MAX_STACKS = 100 MAX_LOG_TAIL = 2000 # lines # What a reply may carry back. Sized to fit one frame *after* JSON escaping: # `_dump_json` drops whole list entries to make a reply fit, and a string is not # a list — a log too long to trim is a reply nobody receives at all. MAX_LOG_BYTES = 32_000 MAX_COMPOSE = 40_000 # a compose file, so one reply frame still fits +MAX_ARCHIVE = 2 * 1024 * 1024 # a directory read out of a container +MAX_ARCHIVE_MEMBERS = 64 MAX_ENV = 64 # variables on a container we create MAX_PORTS = 32 MAX_MOUNTS = 32 -# Labels compose writes on everything it creates. They are the whole reason a -# "stack" can be recovered from a running machine without a database. +# Labels compose writes on everything it creates. Read here only to say which +# project a container belongs to; what a project *is* is `fleet_stacks`'s. LABEL_PROJECT = "com.docker.compose.project" LABEL_SERVICE = "com.docker.compose.service" -LABEL_CONFIG = "com.docker.compose.project.config_files" -LABEL_WORKDIR = "com.docker.compose.project.working_dir" -# Portainer's own, on the stacks it owns. -LABEL_PORTAINER = "io.portainer.stack.name" _ID_RE = re.compile(r"\A[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z") _IMAGE_RE = re.compile(r"\A[A-Za-z0-9][A-Za-z0-9_.:/@-]{0,255}\Z") @@ -257,10 +249,6 @@ def available() -> bool: return bool(where) -def compose_available() -> bool: - return shutil.which("docker") is not None - - async def _open() -> tuple: kind, where = endpoint() try: @@ -367,16 +355,46 @@ def build_request(method: str, target: str, host: str, payload: bytes, return ("\r\n".join(lines) + "\r\n\r\n").encode("utf-8") + payload +# The API version spoken, raised once if the daemon says ours is too old. An +# engine drops old versions over time; asking `/version` first would be a round +# trip on every call for an answer that changes once a machine's lifetime. +_api = {"version": API_VERSION} +_TOO_OLD = re.compile(r"[Mm]inimum supported API version is (\d{1,2}\.\d{1,3})") + + async def call(method: str, path: str, body=None, *, - timeout: float = CALL_TIMEOUT, raw: bool = False): + timeout: float = CALL_TIMEOUT, raw: bool = False, + headers: tuple = ()): """One Engine API call. Returns parsed JSON, or bytes when ``raw``. Raises :class:`DockerError` and nothing else: a caller of this is answering an operator over the mesh, and an unexpected exception there is a handler that stops handling.""" payload = b"" if body is None else json.dumps(body).encode("utf-8") - head = build_request(method, f"/{API_VERSION}{path}", "docker", payload) + for _attempt in range(2): + status, data = await _exchange(method, path, payload, timeout, headers) + if status == 400: + wanted = _TOO_OLD.search(data[:400].decode("utf-8", "replace")) + if wanted and wanted.group(1) != _api["version"]: + _api["version"] = wanted.group(1) + continue + break + if status >= 400: + raise DockerError(_message(data, status)) + if raw: + return data + if not data: + return None + try: + return json.loads(data) + except ValueError: + raise DockerError("the daemon answered something unreadable") from None + +async def _exchange(method: str, path: str, payload: bytes, timeout: float, + headers: tuple) -> tuple[int, bytes]: + head = build_request(method, f"/v{_api['version']}{path}", "docker", payload, + extra=headers) reader, writer = await _open() try: writer.write(head) @@ -394,17 +412,7 @@ async def call(method: str, path: str, body=None, *, await writer.wait_closed() except (OSError, asyncio.TimeoutError): pass - - if status >= 400: - raise DockerError(_message(data, status)) - if raw: - return data - if not data: - return None - try: - return json.loads(data) - except ValueError: - raise DockerError("the daemon answered something unreadable") from None + return status, data def _message(data: bytes, status: int) -> str: @@ -454,20 +462,23 @@ def _container_row(entry: dict) -> dict: "status": _short(entry.get("Status"), 96), "created": entry.get("Created") if isinstance(entry.get("Created"), int) else 0, "ports": ports[:8], + "image_id": _short(entry.get("ImageID"), 80), "project": _short(labels.get(LABEL_PROJECT), 64), "service": _short(labels.get(LABEL_SERVICE), 64), - "portainer": _short(labels.get(LABEL_PORTAINER), 64), } -async def containers(all_of_them: bool = True) -> list[dict]: +async def raw_containers(all_of_them: bool = True) -> list[dict]: + """The daemon's own list, bounded, for a caller that reads the labels.""" raw = await call("GET", "/containers/json?all=" + ("1" if all_of_them else "0")) - if not isinstance(raw, list): - return [] - return [_container_row(entry) for entry in raw[:MAX_CONTAINERS] + return [entry for entry in (raw if isinstance(raw, list) else [])[:MAX_CONTAINERS] if isinstance(entry, dict)] +async def containers(all_of_them: bool = True) -> list[dict]: + return [_container_row(entry) for entry in await raw_containers(all_of_them)] + + async def images() -> list[dict]: raw = await call("GET", "/images/json") out = [] @@ -579,7 +590,6 @@ async def overview() -> dict: "containers": info.get("Containers") if isinstance(info.get("Containers"), int) else 0, "running": info.get("ContainersRunning") if isinstance(info.get("ContainersRunning"), int) else 0, "images": info.get("Images") if isinstance(info.get("Images"), int) else 0, - "compose": compose_available(), } @@ -672,19 +682,77 @@ async def act(container: str, action: str) -> dict: return {"container": ident, "action": action} +def split_reference(reference: str) -> tuple[str, str, str]: + """``registry/owner/name:tag@digest`` → (repository, tag, digest). + + The tag is the last colon *after* the last slash: a registry's port is a + colon too, and ``127.0.0.1:5000/app`` has no tag at all.""" + reference, _, digest = reference.partition("@") + slash = reference.rfind("/") + colon = reference.rfind(":") + if colon > slash: + return reference[:colon], reference[colon + 1:], digest + return reference, "", digest + + +def registry_of(reference: str) -> str: + """The registry a reference names, as `docker login` keys it.""" + first, sep, _rest = reference.partition("/") + if sep and ("." in first or ":" in first or first == "localhost"): + return first + return "docker.io" + + +def _auth_header(reference: str) -> tuple: + """`X-Registry-Auth` for a private registry, from this account's own + `docker login`. + + Read from the node's account and nowhere else, never sent anywhere but the + local daemon, and never part of an answer. A credential helper keeps its + secrets out of the file, so a registry logged into through one is pulled + anonymously here — and says so in the daemon's refusal.""" + folder = os.environ.get("DOCKER_CONFIG") or os.path.expanduser("~/.docker") + try: + with open(os.path.join(folder, "config.json"), "rb") as handle: + config = json.loads(handle.read(64 * 1024)) + except (OSError, ValueError): + return () + auths = config.get("auths") if isinstance(config, dict) else None + if not isinstance(auths, dict): + return () + registry = registry_of(reference) + keys = ("https://index.docker.io/v1/", "docker.io", "index.docker.io") \ + if registry == "docker.io" else (registry, "https://" + registry) + for key in keys: + entry = auths.get(key) + token = entry.get("auth") if isinstance(entry, dict) else None + if not isinstance(token, str): + continue + try: + user, _, secret = base64.b64decode(token, validate=True).decode().partition(":") + except (ValueError, UnicodeDecodeError): + return () + blob = json.dumps({"username": user, "password": secret, + "serveraddress": key}).encode() + return ("X-Registry-Auth: " + base64.urlsafe_b64encode(blob).decode(),) + return () + + async def pull(image: str) -> dict: """Fetch an image. Minutes long, and the answer is a stream of progress.""" reference = clean_image(image) - if not reference: + if not reference or ".." in reference: raise DockerError("that image name is not usable") - name, _, tag = reference.rpartition(":") - if not name or "/" in tag: - name, tag = reference, "latest" - from urllib.parse import quote - raw = await call( - "POST", - f"/images/create?fromImage={quote(name, safe='')}&tag={quote(tag, safe='')}", - body=None, timeout=PULL_TIMEOUT, raw=True) + repository, tag, digest = split_reference(reference) + if digest: + query = f"fromImage={quote(repository + '@' + digest, safe='')}" + named = f"{repository}@{digest}" + else: + tag = tag or "latest" + query = f"fromImage={quote(repository, safe='')}&tag={quote(tag, safe='')}" + named = f"{repository}:{tag}" + raw = await call("POST", f"/images/create?{query}", body=None, + timeout=PULL_TIMEOUT, raw=True, headers=_auth_header(reference)) text = (raw or b"").decode("utf-8", "replace") # Docker reports a failed pull inside a 200 body, one JSON object per line. for line in reversed(text.strip().splitlines()[-20:]): @@ -694,7 +762,7 @@ async def pull(image: str) -> dict: continue if isinstance(entry, dict) and entry.get("error"): raise DockerError(_short(entry["error"], 200)) - return {"image": f"{name}:{tag}"} + return {"image": named} async def create(spec: dict) -> dict: @@ -719,217 +787,83 @@ async def create(spec: dict) -> dict: # --------------------------------------------------------------------------- -# Stacks +# Reads that stay on this machine # --------------------------------------------------------------------------- # -# A stack is a compose project, and a compose project is recoverable from a -# running machine because compose labels everything it creates. No database, no -# second source of truth — what is running *is* the list. Portainer deploys -# through compose too, so its stacks appear here with no special case; what -# tells them apart is the label it adds, and that decides who is allowed to -# update them. - -def group_stacks(rows: list[dict]) -> list[dict]: - """Containers → the projects they belong to.""" - projects: dict = {} - for row in rows: - name = row.get("project") or "" - if not name: - continue - stack = projects.setdefault(name, { - "name": name, "services": [], "running": 0, "total": 0, - "portainer": row.get("portainer") or "", - }) - if len(stack["services"]) < 64: - stack["services"].append({ - "service": row.get("service") or row.get("name"), - "container": row["id"], "state": row.get("state"), - "image": row.get("image"), - }) - stack["total"] += 1 - if row.get("state") == "running": - stack["running"] += 1 - if row.get("portainer"): - stack["portainer"] = row["portainer"] - ordered = sorted(projects.values(), key=lambda entry: entry["name"]) - return ordered[:MAX_STACKS] - - -async def stacks() -> list[dict]: - rows = await containers(True) - found = group_stacks(rows) - # Where the compose file lives, so an update can re-run it. Read from the - # labels of one container per project rather than guessed. - config: dict = {} - raw = await call("GET", "/containers/json?all=1") - for entry in (raw if isinstance(raw, list) else [])[:MAX_CONTAINERS]: - labels = _labels(entry.get("Labels")) if isinstance(entry, dict) else {} - project = _short(labels.get(LABEL_PROJECT), 64) - if project and project not in config: - config[project] = { - "files": _short(labels.get(LABEL_CONFIG), 1024), - "workdir": _short(labels.get(LABEL_WORKDIR), 512), - } - for stack in found: - stack.update(config.get(stack["name"], {"files": "", "workdir": ""})) - return found - - -def compose_files(files: str) -> list[str]: - """The compose files a project was deployed from, as compose wrote them. - - The label is a comma-separated list of absolute paths. Anything that is not - one is dropped rather than passed to an argv — the label is written by the - daemon, but a daemon is a machine and this runs as root.""" - out = [] - for path in (files or "").split(","): - path = path.strip() - if _PATH_RE.match(path) and len(out) < 8: - out.append(path) - return out +# What `fleet_stacks` needs to recreate a container or find a stack's file. The +# raw documents below are the daemon's own and never leave this node whole — +# a container's configuration routinely carries its secrets. +async def container_raw(container: str) -> dict: + ident = clean_id(container) + if not ident: + raise DockerError("that is not a container") + raw = await call("GET", f"/containers/{ident}/json") + if not isinstance(raw, dict): + raise DockerError("the daemon described that container in no usable way") + return raw -async def _run(argv: list[str], *, cwd: str | None = None, - timeout: float = COMPOSE_TIMEOUT, on_output=None) -> tuple[int, str]: - """Run a command, bounded, never through a shell. - `on_output` is handed each line as it arrives: a compose pull is minutes of - silence otherwise, and silence is indistinguishable from a hang.""" +async def image_raw(reference: str) -> dict | None: + """An image's own document, or None when this machine does not hold it.""" + ident = clean_image(reference) + if not ident or ".." in ident: + raise DockerError("that is not an image") try: - proc = await asyncio.create_subprocess_exec( - *argv, cwd=cwd or None, - stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, - stdin=asyncio.subprocess.DEVNULL) - except (OSError, ValueError) as exc: - raise DockerError(f"could not run docker ({type(exc).__name__})") from None - collected: list[str] = [] - size = 0 + raw = await call("GET", f"/images/{ident}/json") + except DockerError as exc: + if "No such image" in str(exc) or "not found" in str(exc).lower(): + return None + raise + return raw if isinstance(raw, dict) else None - async def drain() -> None: - nonlocal size - while True: - line = await proc.stdout.readline() - if not line: - return - text = line.decode("utf-8", "replace").rstrip("\n") - if size < MAX_LOG_BYTES: - collected.append(text) - size += len(text) + 1 - if on_output is not None: - try: - on_output(text) - except Exception: - pass +async def remote_digest(reference: str) -> str: + """What the registry serves under that reference right now, asked through + the daemon — which holds the network path and the registry's trust, and + reads only the manifest: nothing is downloaded.""" + ident = clean_image(reference) + if not ident or ".." in ident: + raise DockerError("that is not an image") + raw = await call("GET", f"/distribution/{ident}/json", timeout=20.0, + headers=_auth_header(ident)) + descriptor = raw.get("Descriptor") if isinstance(raw, dict) else None + digest = descriptor.get("digest") if isinstance(descriptor, dict) else None + if not isinstance(digest, str) or not re.match(r"\Asha256:[0-9a-f]{64}\Z", digest): + raise DockerError("the registry answered no digest") + return digest + + +async def read_archive(container: str, path: str) -> dict[str, bytes]: + """The regular files under a path inside a container, by name relative to + the directory read. + + Through the daemon's archive route, because the files a container keeps in + its volumes are root's on the host and this node's account is not. Bounded + in bytes and in members; a link, a device or a name that climbs is skipped, + never followed — nothing here is written to disk.""" + ident = clean_id(container) + if not ident or not isinstance(path, str) or not path.startswith("/") \ + or ".." in path or len(path) > 512 or "\0" in path: + raise DockerError("that is not something to read") + raw = await call("GET", f"/containers/{ident}/archive?path={quote(path, safe='/')}", + raw=True) + if len(raw) > MAX_ARCHIVE: + raise DockerError("that directory is too large to read") + out: dict[str, bytes] = {} try: - await asyncio.wait_for(asyncio.gather(drain(), proc.wait()), timeout) - except asyncio.TimeoutError: - try: - proc.kill() - except OSError: - pass - raise DockerError("that took too long and was stopped") from None - return proc.returncode or 0, "\n".join(collected)[-MAX_LOG_BYTES:] - - -def _compose_argv(project: str, files: list[str], *rest: str) -> list[str]: - argv = ["docker", "compose", "--project-name", project] - for path in files: - argv += ["-f", path] - return argv + list(rest) - - -async def stack_up(project: str, files: list[str], workdir: str = "", - *, pull_first: bool = True, on_output=None) -> dict: - """Bring a compose project up to date: pull, then re-run its compose file. - - Pulling under a project and restarting it is *not* this act and quietly does - less — a container keeps running the image it was created from until it is - recreated. So `up -d` is the operation, and it is the compose file that - decides what "up to date" means.""" - name = clean_id(project) - if not name: - raise DockerError("that is not a stack") - if not compose_available(): - raise DockerError("this machine has no docker compose") - if not files: - raise DockerError("that stack has no compose file on this machine") - cwd = workdir if _PATH_RE.match(workdir or "") and os.path.isdir(workdir or "") else None - if pull_first: - code, text = await _run(_compose_argv(name, files, "pull"), - cwd=cwd, on_output=on_output) - if code != 0: - raise DockerError(f"pulling that stack failed: {text[-200:]}") - code, text = await _run(_compose_argv(name, files, "up", "-d", "--remove-orphans"), - cwd=cwd, on_output=on_output) - if code != 0: - raise DockerError(f"bringing that stack up failed: {text[-200:]}") - return {"stack": name, "output": text[-4000:]} - - -async def stack_act(project: str, files: list[str], action: str, - workdir: str = "", *, on_output=None) -> dict: - """start / stop / restart / down, through compose when it is there. - - Without compose the same three are still possible container by container — - it is `down` and `up` that need the file, because they create and destroy. - """ - name = clean_id(project) - if not name: - raise DockerError("that is not a stack") - if action not in ("start", "stop", "restart", "down"): - raise DockerError("that is not something to do to a stack") - if compose_available() and files: - cwd = workdir if _PATH_RE.match(workdir or "") and os.path.isdir(workdir or "") else None - code, text = await _run(_compose_argv(name, files, action), - cwd=cwd, timeout=COMPOSE_TIMEOUT, on_output=on_output) - if code != 0: - raise DockerError(f"that stack refused to {action}: {text[-200:]}") - return {"stack": name, "action": action, "output": text[-4000:]} - if action == "down": - raise DockerError("taking a stack down needs its compose file") - rows = [row for row in await containers(True) if row.get("project") == name] - if not rows: - raise DockerError("no container on this machine belongs to that stack") - for row in rows: - await act(row["id"], action) - return {"stack": name, "action": action, "containers": len(rows)} - - -async def deploy_stack(project: str, content: str, *, root: str = "", - on_output=None) -> dict: - """Deploy a stack from a compose file an operator wrote. - - The file is written under a directory of this node's own, named for the - project, so a redeploy later finds it exactly where compose recorded it — - and so nothing this deploys can be made to overwrite a path chosen by - whoever sent the file: the only thing from the wire that reaches the path is - a name that passed :func:`clean_id`.""" - name = clean_id(project) - if not name: - raise DockerError("that is not a stack name") - body = clean_compose(content) - if not body: - raise DockerError("that compose file is empty or too large") - if not compose_available(): - raise DockerError("this machine has no docker compose") - directory = os.path.join(stack_dir(root), name) - os.makedirs(directory, mode=0o700, exist_ok=True) - path = os.path.join(directory, "docker-compose.yml") - tmp = path + ".new" - with open(tmp, "w", encoding="utf-8") as handle: - os.chmod(tmp, 0o600) - handle.write(body) - os.replace(tmp, path) - return await stack_up(name, [path], directory, on_output=on_output) - - -def stack_dir(root: str = "") -> str: - """Where compose files this node deployed are kept. - - Under the node's own data directory when it has one — the same place its - identity and its session store live, so a compose file that decides what - runs on this machine is backed up and wiped with the rest of its state. - Never `/tmp`, and never a path from the wire.""" - base = root or os.path.expanduser("~/.nmesh") - return os.path.join(base, "stacks") + with tarfile.open(fileobj=io.BytesIO(raw), mode="r:") as archive: + for index, member in enumerate(archive): + if index >= MAX_ARCHIVE_MEMBERS: + break + if not member.isreg() or member.size > MAX_ARCHIVE: + continue + parts = member.name.split("/") + if member.name.startswith("/") or ".." in parts: + continue + handle = archive.extractfile(member) + if handle is not None: + out["/".join(parts[1:])] = handle.read(MAX_ARCHIVE) + except (tarfile.TarError, EOFError, OSError): + raise DockerError("the daemon sent an archive that does not read") from None + return out diff --git a/src/apps/fleet_portainer.py b/src/apps/fleet_portainer.py index 2ce5e93..97b0620 100644 --- a/src/apps/fleet_portainer.py +++ b/src/apps/fleet_portainer.py @@ -1,13 +1,13 @@ """ Portainer — the manager a machine may already have in front of its docker. -Portainer deploys through compose and labels what it deploys, so its stacks are -already in the list this app builds from the running containers. What its API -adds is the two things the labels cannot give: the stack **file** (held by -Portainer, or in a git repository it pulls from) and the **redeploy** that -fetches it again. So a stack Portainer owns is updated *through* Portainer — -running `compose up` behind its back leaves its record stale, and the next thing -it does undoes the update. +Portainer deploys through compose and labels nothing of its own: its stacks are +found from the Engine alone (`fleet_stacks`), by the directory they were +deployed from. What its API adds is what the Engine cannot give: the stack's +**file and environment**, held in Portainer's database, and the **redeploy** +that pulls the images and brings the stack up from them while keeping +Portainer's record true. A stack Portainer owns is updated through here when an +operator connected one. What this holds, and why it is not much --------------------------------------- @@ -15,14 +15,15 @@ drawer, **write-only** from the outside: an operator can set it and can see that it is set, and nothing ever reads it back over the mesh. A token that could be read back would make the `docker` capability a way of stealing the credential -rather than of using it. +rather than of using it. The stack's environment is read from Portainer only to +be handed straight back on a redeploy; it never travels. TLS is verified. A Portainer on a private network very often has a self-signed -certificate, so there is a second way to accept one: **pin its fingerprint**. -That is an anchor an operator chose, which is the only kind of trust this -project has — where "just don't check" is not a setting, because a pinned -certificate is strictly more information than an unchecked one and costs the -same click. +certificate, so there is a second way to accept one: **pin its fingerprint**, +read once by :func:`probe` and confirmed by the operator. That is an anchor an +operator chose, which is the only kind of trust this project has — where "just +don't check" is not a setting. Plain HTTP is accepted only to this machine's +own loopback, where there is no wire to read the token from. Everything from the far side is hostile input, exactly as with the daemon: the answers are large, deep, and written by something this node does not control, so @@ -32,6 +33,8 @@ import asyncio import hashlib +import hmac +import ipaddress import json import re import ssl @@ -45,6 +48,9 @@ MAX_FILE = 40_000 # a stack file, so one reply frame still fits MAX_URL = 256 MAX_TOKEN = 512 +MAX_ENV = 256 # variables a stack's environment may carry +SETTLE_POLL = 2.0 # how often a background deployment is asked about +_ACTIVE, _DEPLOYING = 1, 3 # Portainer's own stack statuses _HOST_RE = re.compile(r"\A[A-Za-z0-9._:\[\]-]{1,200}\Z") _FINGERPRINT_RE = re.compile(r"\A[0-9a-f]{64}\Z") @@ -67,9 +73,34 @@ def clean_url(value) -> str: return "" if "/" in rest or not _HOST_RE.match(rest): return "" + if scheme == "http" and not _loopback(rest): + return "" # a token in the clear, over a wire return f"{scheme}://{rest}" +def _split(url: str) -> tuple[str, str, int, bool]: + """A clean URL → (authority, host, port, is https).""" + scheme, _, rest = url.partition("://") + host, _, port = rest.rpartition(":") + if not host or not port.isdigit() or host.endswith(":"): + host, port = rest, "443" if scheme == "https" else "80" + return rest, host.strip("[]"), int(port), scheme == "https" + + +def _loopback(authority: str) -> bool: + host = authority + if host.startswith("["): + host = host[1:].partition("]")[0] + elif host.count(":") == 1: + host = host.partition(":")[0] + if host == "localhost": + return True + try: + return ipaddress.ip_address(host).is_loopback + except ValueError: + return False + + def clean_token(value) -> str: if not isinstance(value, str): return "" @@ -101,11 +132,7 @@ def __init__(self, url: str, token: str, *, fingerprint: str = "") -> None: # -- the wire --------------------------------------------------------- def _target(self) -> tuple[str, str, int, bool]: - scheme, _, rest = self.url.partition("://") - host, _, port = rest.rpartition(":") - if not host or not port.isdigit(): - host, port = rest, "443" if scheme == "https" else "80" - return rest, host.strip("[]"), int(port), scheme == "https" + return _split(self.url) def _context(self) -> ssl.SSLContext | None: authority, _host, _port, secure = self._target() @@ -173,7 +200,6 @@ def _check_pin(self, writer) -> None: seen = hashlib.sha256(raw).hexdigest() # Constant time: the comparison is against a secret-shaped value an # operator pinned, and there is no reason to leak where it diverges. - import hmac if not hmac.compare_digest(seen, self.fingerprint): raise PortainerError( "that Portainer presented a different certificate " @@ -220,6 +246,47 @@ async def file(self, stack_id: int) -> str: content = raw.get("StackFileContent") if isinstance(raw, dict) else "" return _text(content, MAX_FILE) + async def _detail(self, stack_id: int) -> dict: + """What a redeploy has to hand back unchanged: the stack's environment + and how its repository is reached. Read here, sent straight back, never + part of an answer.""" + raw = await self.call("GET", f"/api/stacks/{_id(stack_id)}") + raw = raw if isinstance(raw, dict) else {} + env = [] + for entry in (raw.get("Env") if isinstance(raw.get("Env"), list) else [])[:MAX_ENV]: + if isinstance(entry, dict) and isinstance(entry.get("name"), str) \ + and isinstance(entry.get("value"), str): + env.append({"name": entry["name"][:256], "value": entry["value"][:8192]}) + git = raw.get("GitConfig") if isinstance(raw.get("GitConfig"), dict) else {} + auth = git.get("Authentication") if isinstance(git.get("Authentication"), dict) else None + return {"env": env, "git": bool(git.get("URL")), + "git_ref": _text(git.get("ReferenceName"), 256), + "git_auth": auth is not None, + "git_user": _text((auth or {}).get("Username"), 256)} + + async def _settled(self, stack_id: int, answer) -> None: + """Wait out a deployment Portainer runs in the background. + + Recent versions answer a redeploy at once, with the stack still + *deploying* (status 3), and recreate its containers afterwards. An + update reported done at that moment is a page drawn in the middle of + the recreation, and a failure nobody hears about.""" + status = _int(answer.get("Status")) if isinstance(answer, dict) else 0 + waited = 0.0 + while status == _DEPLOYING and waited < REDEPLOY_TIMEOUT: + await asyncio.sleep(SETTLE_POLL) + waited += SETTLE_POLL + raw = await self.call("GET", f"/api/stacks/{_id(stack_id)}") + raw = raw if isinstance(raw, dict) else {} + status = _int(raw.get("Status")) + history = raw.get("DeploymentStatus") + last = history[-1] if isinstance(history, list) and history else {} + message = _text(last.get("Message"), 200) if isinstance(last, dict) else "" + if message and _int(last.get("Status")) not in (_ACTIVE, _DEPLOYING): + raise PortainerError(f"Portainer could not deploy it: {message}") + if status == _DEPLOYING: + raise PortainerError("Portainer is still deploying it; look again later") + async def act(self, stack_id: int, endpoint_id: int, action: str) -> dict: if action not in ("start", "stop"): raise PortainerError("that is not something to do to a stack") @@ -230,30 +297,40 @@ async def act(self, stack_id: int, endpoint_id: int, action: str) -> dict: return {"stack": _id(stack_id), "action": action} async def redeploy(self, stack: dict) -> dict: - """Bring one Portainer stack up to date, the way Portainer would. + """Bring one Portainer stack up to date, the way Portainer's own *Pull + and redeploy* would. Two shapes, because Portainer has two: a stack it pulls from git is redeployed by fetching the reference again, and one whose file it holds - is redeployed by handing that file back with `PullImage`. Guessing - wrong is a stack that redeploys the version it already had.""" + is redeployed by handing that file back. Either way with the stack's + own environment: Portainer *replaces* the environment with what a + redeploy carries, so an empty list would wipe it. And a git stack whose + credentials Portainer stored is asked to use them — saying "no + authentication" makes it forget them.""" stack_id, endpoint_id = _id(stack.get("id")), _id(stack.get("endpoint")) - if stack.get("git"): + held = await self._detail(stack_id) + if held["git"]: + body = {"RepositoryReferenceName": held["git_ref"] or _text(stack.get("git_ref"), 96), + "RepositoryAuthentication": held["git_auth"], + "Env": held["env"], "PullImage": True, "Prune": False} + if held["git_auth"]: + body["RepositoryUsername"] = held["git_user"] + body["RepositoryPassword"] = "" answer = await self.call( "PUT", f"/api/stacks/{stack_id}/git/redeploy?endpointId={endpoint_id}", - body={"RepositoryReferenceName": _text(stack.get("git_ref"), 96), - "RepositoryAuthentication": False, "PullImage": True, - "Prune": False}, - timeout=REDEPLOY_TIMEOUT) + body=body, timeout=REDEPLOY_TIMEOUT) else: content = await self.file(stack_id) if not content: raise PortainerError("Portainer holds no file for that stack") answer = await self.call( "PUT", f"/api/stacks/{stack_id}?endpointId={endpoint_id}", - body={"StackFileContent": content, "Env": [], + body={"StackFileContent": content, "Env": held["env"], "Prune": False, "PullImage": True}, timeout=REDEPLOY_TIMEOUT) - return {"stack": stack_id, + await self._settled(stack_id, answer) + return {"stack": _text(stack.get("name"), 96) or str(stack_id), + "portainer_id": stack_id, "through": "portainer", "name": _text((answer or {}).get("Name") if isinstance(answer, dict) else stack.get("name"), 96)} @@ -273,6 +350,78 @@ async def deploy(self, name: str, content: str, endpoint_id: int) -> dict: "name": name} +async def probe(url: str) -> dict: + """What answers at an address, before anything is trusted to it: the + version Portainer reports without a token, and the fingerprint of the + certificate it presents — for the operator to compare and pin. + + No credential is sent, so an impostor here learns nothing; and nothing it + says is acted on until an operator pins what it showed.""" + address = clean_url(url) + if not address: + raise PortainerError("that is not a Portainer address") + authority, host, port, secure = _split(address) + context = None + if secure: + context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.check_hostname = False + context.verify_mode = ssl.CERT_NONE + try: + reader, writer = await asyncio.wait_for( + asyncio.open_connection(host, port, ssl=context), CALL_TIMEOUT) + except (OSError, ssl.SSLError, asyncio.TimeoutError) as exc: + raise PortainerError(f"nothing answered there ({type(exc).__name__})") from None + fingerprint, version = "", "" + try: + if secure: + sock = writer.get_extra_info("ssl_object") + raw = sock.getpeercert(True) if sock is not None else None + fingerprint = hashlib.sha256(raw).hexdigest() if raw else "" + writer.write(build_request("GET", "/api/system/status", authority, b"")) + await writer.drain() + status, _headers, data = await read_response(reader, CALL_TIMEOUT) + if status == 200: + parsed = json.loads(data) + if isinstance(parsed, dict): + version = _text(parsed.get("Version"), 32) + except (OSError, ssl.SSLError, asyncio.TimeoutError, asyncio.IncompleteReadError, + ValueError, DockerError): + pass + finally: + writer.close() + try: + await writer.wait_closed() + except (OSError, ssl.SSLError, asyncio.TimeoutError): + pass + if not version: + raise PortainerError("that address answers, but not as Portainer") + return {"url": address, "version": version, "fingerprint": fingerprint} + + +def local_endpoint(endpoints: list[dict]) -> int: + """The environment that is this machine's own docker socket, else the first.""" + for entry in endpoints: + if str(entry.get("url", "")).startswith("unix://"): + return entry.get("id") or 0 + return endpoints[0].get("id", 0) if endpoints else 0 + + +def match(stack: dict, listed: list[dict]) -> dict | None: + """The Portainer record for a stack found on this machine. + + By id when the directory gave one — names are Portainer's to reuse across + environments — and by name otherwise.""" + number = stack.get("portainer_id") or 0 + if number: + for entry in listed: + if entry.get("id") == number: + return entry + return None + named = [entry for entry in listed if entry.get("name") == stack.get("name")] + return named[0] if len(named) == 1 else None + + def _int(value) -> int: return value if isinstance(value, int) and not isinstance(value, bool) else 0 diff --git a/src/apps/fleet_stacks.py b/src/apps/fleet_stacks.py new file mode 100644 index 0000000..0c62052 --- /dev/null +++ b/src/apps/fleet_stacks.py @@ -0,0 +1,825 @@ +""" +Stacks — what a machine runs as compose projects, who deployed each one, and +how to bring it up to date without undoing whoever did. + +A stack is recovered from the running machine, never from a database of ours: +compose labels every container it creates with its project, its service, the +files it read and the directory it read them from. Portainer deploys through +compose too, and leaves **no label of its own** — what gives one of its stacks +away is the directory, `/data/compose/`, which is a path inside Portainer's +own container. So there are two owners, told apart by that path: + +* **compose** — a project whose files are on this machine. It is brought up to + date the way it was deployed: `compose pull`, then `compose up -d`, with the + same files from the same directory. +* **portainer** — its file and its environment live in Portainer's database and + volume. Through Portainer's API when an operator connected one, which is the + only way Portainer's own record stays true; and when nobody did, through the + Engine, below. Re-running compose from a copy of Portainer's directory is + *not* that act: a relative bind and an `env_file` resolve against a different + directory, and the containers come back configured differently + (`gotchas.md`, "Docker: who owns a stack"). + +The third way works for anything, owner or none: **recreate through the +Engine**. Pull each image the stack's containers were created from, and for +each container whose image moved, create its replacement from its own +configuration — less what the old image supplied, so the new image's defaults +apply — and put the old one back if the new one does not stay up. It needs no +file and no credential, which is why it is the fallback rather than a guess. + +Everything from an operator is a name checked against a charset; everything +from the daemon is read field by field. Nothing here builds a shell command: +every process is an argv. +""" +from __future__ import annotations + +import asyncio +import os +import re +import time + +from . import fleet_docker as docker +from .fleet_docker import DockerError + +# -- labels ------------------------------------------------------------------ +LABEL_PROJECT = "com.docker.compose.project" +LABEL_SERVICE = "com.docker.compose.service" +LABEL_CONFIG = "com.docker.compose.project.config_files" +LABEL_WORKDIR = "com.docker.compose.project.working_dir" +LABEL_ONEOFF = "com.docker.compose.oneoff" +LABEL_IMAGE = "com.docker.compose.image" +LABEL_DEPENDS = "com.docker.compose.depends_on" +LABEL_PORTAINER_SERVER = "io.portainer.server" + +# -- bounds ------------------------------------------------------------------ +MAX_STACKS = 100 +MAX_SERVICES = 64 # containers listed under one stack +MAX_FILES = 8 # compose files one project was deployed from +MAX_CHECKS = 40 # image references asked of registries at once +CHECK_CONCURRENCY = 6 +COMPOSE_TIMEOUT = 1800.0 # `compose pull` and `up` pull and build +PROBE_TIMEOUT = 10.0 +COMPOSE_PROBE_TTL = 600.0 # how long "is compose here" is believed +SETTLE = 5.0 # seconds a recreated container must stay up +MAX_OUTPUT = docker.MAX_LOG_BYTES + +_PATH_RE = re.compile(r"\A/[^\0\n\r]{0,511}\Z") +_PORTAINER_DIR = re.compile(r"\A/data/compose/(\d{1,9})(?:/|\Z)") +_PORTAINER_IMAGE = re.compile(r"(?:\A|/)portainer/portainer(?:-ce|-ee)?(?::|@|\Z)") +_AGENT_IMAGE = re.compile(r"(?:\A|/)portainer/agent(?::|@|\Z)") +_VERSION_RE = re.compile(r"(\d{1,3})\.(\d{1,3})") +_HEX64 = re.compile(r"\A[0-9a-f]{64}\Z") + + +def _short(value, limit: int = 200) -> str: + return value[:limit] if isinstance(value, str) else "" + + +def _labels(entry) -> dict: + labels = entry.get("Labels") if isinstance(entry, dict) else None + return labels if isinstance(labels, dict) else {} + + +def _dict(value) -> dict: + return value if isinstance(value, dict) else {} + + +def _list(value) -> list: + return value if isinstance(value, list) else [] + + +# --------------------------------------------------------------------------- +# docker compose, if this machine has it +# --------------------------------------------------------------------------- +# +# Two spellings exist on machines still in service: the v2 plugin +# (`docker compose`) and the old standalone binary (`docker-compose`). Which one +# answers is asked, not assumed from a `docker` binary being on the PATH — the +# CLI is very often installed without the plugin. + +_compose: dict = {"argv": None, "version": "", "at": None} + + +async def compose() -> tuple[list[str] | None, str]: + """The argv prefix that runs compose here, and its version — or None.""" + now = time.monotonic() + if _compose["at"] is not None and now - _compose["at"] < COMPOSE_PROBE_TTL: + return _compose["argv"], _compose["version"] + found, version = None, "" + for argv in (["docker", "compose"], ["docker-compose"]): + try: + code, text = await run(argv + ["version", "--short"], timeout=PROBE_TIMEOUT) + except DockerError: + continue + if code == 0 and _VERSION_RE.search(text): + found, version = argv, text.strip().splitlines()[-1][:32] + break + _compose.update(argv=found, version=version, at=now) + return found, version + + +def _at_least(version: str, major: int, minor: int) -> bool: + match = _VERSION_RE.search(version or "") + return bool(match) and (int(match.group(1)), int(match.group(2))) >= (major, minor) + + +async def run(argv: list[str], *, cwd: str | None = None, + timeout: float = COMPOSE_TIMEOUT, on_output=None) -> tuple[int, str]: + """Run a command, bounded, never through a shell. + + `on_output` is handed each line as it arrives: a compose pull is minutes of + silence otherwise, and silence is indistinguishable from a hang.""" + try: + proc = await asyncio.create_subprocess_exec( + *argv, cwd=cwd or None, + stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, + stdin=asyncio.subprocess.DEVNULL) + except (OSError, ValueError) as exc: + raise DockerError(f"could not run {argv[0]} ({type(exc).__name__})") from None + collected: list[str] = [] + size = 0 + + async def drain() -> None: + nonlocal size + while True: + line = await proc.stdout.readline() + if not line: + return + text = line.decode("utf-8", "replace").rstrip("\r\n") + if size < MAX_OUTPUT: + collected.append(text) + size += len(text) + 1 + if on_output is not None: + try: + on_output(text) + except Exception: + pass + + try: + await asyncio.wait_for(asyncio.gather(drain(), proc.wait()), timeout) + except asyncio.TimeoutError: + try: + proc.kill() + except OSError: + pass + raise DockerError("that took too long and was stopped") from None + return proc.returncode or 0, "\n".join(collected)[-MAX_OUTPUT:] + + +# --------------------------------------------------------------------------- +# Portainer, as the Engine sees it +# --------------------------------------------------------------------------- + +def _address(port: dict) -> str: + """Where a published port is reached from this machine itself.""" + ip = port.get("IP") if isinstance(port.get("IP"), str) else "" + if ip in ("", "0.0.0.0", "::"): + ip = "127.0.0.1" + if ":" in ip: + ip = f"[{ip}]" + return f"{ip}:{port.get('PublicPort')}" + + +def find_portainer(entries: list[dict]) -> dict: + """A Portainer server or agent among this machine's containers. + + Known by the label Portainer's image carries, and by the image's name for a + build old enough not to carry it. Finding one costs no credential; what it + is told is only what the Engine already says about it.""" + server, agent = None, False + for entry in entries: + image = _short(entry.get("Image"), 256) + labels = _labels(entry) + if labels.get(LABEL_PORTAINER_SERVER) == "true" or _PORTAINER_IMAGE.search(image): + if server is None or entry.get("State") == "running": + server = entry + elif _AGENT_IMAGE.search(image): + agent = True + found: dict = {"server": None, "agent": agent} + if server is None: + return found + names = _list(server.get("Names")) + https = http = "" + for port in _list(server.get("Ports"))[:docker.MAX_PORTS]: + if not isinstance(port, dict) or not isinstance(port.get("PublicPort"), int): + continue + if port.get("PrivatePort") == 9443 and not https: + https = "https://" + _address(port) + elif port.get("PrivatePort") == 9000 and not http: + http = "http://" + _address(port) + image = _short(server.get("Image"), 256) + _repository, tag, _digest = docker.split_reference(image) + found["server"] = { + "id": _short(server.get("Id"), 64), + "name": _short(names[0], 128).lstrip("/") if names and isinstance(names[0], str) else "", + "image": image, + "tag": tag[:64], + "edition": "Business" if "-ee" in image else "Community", + "state": _short(server.get("State"), 32), + # Nothing to suggest when it publishes nothing: the operator knows + # where they reach it, and a guessed address is a wrong one. + "url": https or http, + } + return found + + +def portainer_stack_id(workdir: str, files: list[str]) -> int: + """The id Portainer gave a stack, read off the directory it deployed from; + 0 when the directory is not one of Portainer's.""" + for path in [workdir] + list(files): + match = _PORTAINER_DIR.match(path or "") + if match: + return int(match.group(1)) + return 0 + + +# --------------------------------------------------------------------------- +# Stacks from labels +# --------------------------------------------------------------------------- + +def compose_files(value) -> list[str]: + """The compose files a project was deployed from, as compose wrote them. + + The label is a comma-separated list of absolute paths. Anything that is not + one is dropped rather than passed to an argv — the label is written by + whoever created the container, and this may run as root.""" + out = [] + for path in (value if isinstance(value, str) else "").split(","): + path = path.strip() + if _PATH_RE.match(path) and ".." not in path.split("/") and len(out) < MAX_FILES: + out.append(path) + return out + + +def _readable(path: str) -> bool: + try: + return os.path.isfile(path) and os.access(path, os.R_OK) + except (OSError, ValueError): + return False + + +def discover(entries: list[dict], *, root: str = "") -> list[dict]: + """The daemon's container list → the compose projects it describes. + + One entry per project, with what decides how it can be updated: who owns + it, which files it came from, and whether those files are readable here.""" + projects: dict = {} + for entry in entries: + labels = _labels(entry) + name = docker.clean_id(labels.get(LABEL_PROJECT)) + if not name or labels.get(LABEL_ONEOFF) == "True": + continue # `compose run` leaves one-offs; not services + stack = projects.get(name) + if stack is None: + if len(projects) >= MAX_STACKS: + continue + stack = projects[name] = { + "name": name, "workdir": "", "files": [], "services": [], + "running": 0, "total": 0} + if not stack["workdir"]: + workdir = _short(labels.get(LABEL_WORKDIR), 512) + stack["workdir"] = workdir if _PATH_RE.match(workdir) else "" + if not stack["files"]: + stack["files"] = compose_files(labels.get(LABEL_CONFIG)) + state = _short(entry.get("State"), 32) + stack["total"] += 1 + if state == "running": + stack["running"] += 1 + if len(stack["services"]) < MAX_SERVICES: + names = _list(entry.get("Names")) + stack["services"].append({ + "service": _short(labels.get(LABEL_SERVICE), 64), + "container": _short(entry.get("Id"), 64), + "name": _short(names[0], 128).lstrip("/") + if names and isinstance(names[0], str) else "", + "state": state, + "status": _short(entry.get("Status"), 96), + "image": _short(entry.get("Image"), 256), + "image_id": _short(entry.get("ImageID"), 80), + }) + ours = stack_dir(root) + out = [] + for stack in sorted(projects.values(), key=lambda entry: entry["name"]): + stack["services"].sort(key=lambda row: (row["service"], row["name"])) + number = portainer_stack_id(stack["workdir"], stack["files"]) + stack["owner"] = "portainer" if number else "compose" + stack["portainer_id"] = number + stack["files_ok"] = (not number and bool(stack["files"]) + and all(_readable(path) for path in stack["files"])) + stack["nmesh"] = bool(stack["workdir"]) and ( + stack["workdir"] + "/").startswith(ours + "/") + out.append(stack) + return out + + +async def name_images(rows: list[dict], key: str = "container") -> None: + """Put back the name a container was created from where the daemon lists + an image id instead — which it does once that name points at a newer + image, so it is exactly the container an operator is looking for.""" + for row in rows: + if not str(row.get("image", "")).startswith("sha256:"): + continue + try: + raw = await docker.container_raw(row.get(key, "")) + except DockerError: + continue + named = _short(_dict(raw.get("Config")).get("Image"), 256) + if named: + row["image"] = named + + +def update_method(stack: dict, *, portainer_api: bool, compose_argv) -> dict: + """How *Update* will bring this stack up to date, and why that way. + + Said before anything is done: an operator choosing Update is choosing + between acts that differ in what they keep, and has to be able to see which + one this is.""" + if stack.get("owner") == "portainer": + if portainer_api: + return {"method": "portainer", + "why": "through Portainer's API: it pulls and redeploys " + "with the file and environment it holds"} + return {"method": "engine", + "why": "Portainer is not connected here, so each image is " + "pulled and its container recreated with the settings " + "it runs with"} + if not stack.get("files"): + return {"method": "engine", + "why": "no compose file is recorded for it, so each image is " + "pulled and its container recreated"} + if not stack.get("files_ok"): + return {"method": "engine", + "why": "its compose file is not readable by this node, so each " + "image is pulled and its container recreated"} + if compose_argv is None: + return {"method": "engine", + "why": "docker compose is not installed here, so each image is " + "pulled and its container recreated"} + return {"method": "compose", + "why": "compose pull, then compose up -d, with the file it was " + "deployed from"} + + +# --------------------------------------------------------------------------- +# Is there anything newer? +# --------------------------------------------------------------------------- + +async def check(entries: list[dict]) -> dict: + """For each container, whether its image has something newer. + + ``current``, ``outdated`` (the registry serves another image under that + name), ``pulled`` (a newer one is already here and the container still + runs the old), ``pinned`` (named by digest), ``local`` (never came from a + registry) or ``unknown`` (the registry would not say). Asked through the + daemon's distribution route, which reads a manifest and downloads nothing. + """ + by_ref: dict[str, list[dict]] = {} + for entry in entries: + reference = _short(entry.get("Image"), 256) + if reference.startswith("sha256:"): + # The name it was created from now points at another image. + try: + raw = await docker.container_raw(_short(entry.get("Id"), 64)) + reference = _short(_dict(raw.get("Config")).get("Image"), 256) + except DockerError: + continue + if docker.clean_image(reference): + by_ref.setdefault(reference, []).append(entry) + references = sorted(by_ref)[:MAX_CHECKS] + gate = asyncio.Semaphore(CHECK_CONCURRENCY) + images: dict = {} + + async def one(reference: str) -> None: + async with gate: + images[reference] = await _check_one(reference) + + await asyncio.gather(*(one(reference) for reference in references)) + containers: dict = {} + for reference in references: + verdict = images[reference] + for entry in by_ref[reference]: + state = verdict["state"] + if state in ("current", "outdated") and verdict.get("local_id") and \ + entry.get("ImageID") != verdict["local_id"]: + state = "pulled" + containers[_short(entry.get("Id"), 64)] = state + return {"containers": containers, + "images": {ref: {"state": images[ref]["state"], + "why": images[ref].get("why", "")} + for ref in references}} + + +async def _check_one(reference: str) -> dict: + _repository, _tag, digest = docker.split_reference(reference) + if digest: + return {"state": "pinned"} + try: + local = await docker.image_raw(reference) + except DockerError as exc: + return {"state": "unknown", "why": str(exc)[:160]} + local_id = _short(_dict(local).get("Id"), 80) + held = [value.partition("@")[2] for value in _list(_dict(local).get("RepoDigests")) + if isinstance(value, str)] + if local is not None and not held: + return {"state": "local", "local_id": local_id} + try: + remote = await docker.remote_digest(reference) + except DockerError as exc: + return {"state": "unknown", "why": str(exc)[:160], "local_id": local_id} + return {"state": "current" if remote in held else "outdated", "local_id": local_id} + + +# --------------------------------------------------------------------------- +# Recreate through the Engine +# --------------------------------------------------------------------------- + +def _same(left, right) -> bool: + return (left or None) == (right or None) + + +def recreate_body(raw: dict, image: dict, new_image_id: str) -> tuple[dict, list]: + """A container's own configuration, ready to create its replacement. + + What the old *image* supplied is taken back out — its environment, its + command, its labels, its ports and volumes — so the new image's apply, as + they would to a container created fresh; what the container was *given* is + kept. Its anonymous volumes are carried over by name, or the replacement + would start with empty ones. + + Returns the body and the networks to join after creation: an Engine older + than API 1.44 attaches one network at create time.""" + config = dict(_dict(raw.get("Config"))) + held = _dict(_dict(image).get("Config")) + ident = _short(raw.get("Id"), 64) + if _same(config.get("Entrypoint"), held.get("Entrypoint")): + config.pop("Entrypoint", None) + if _same(config.get("Cmd"), held.get("Cmd")): + config.pop("Cmd", None) + for key in ("WorkingDir", "User", "Healthcheck", "StopSignal"): + if _same(config.get(key), held.get(key)): + config.pop(key, None) + image_env = set(value for value in _list(held.get("Env")) if isinstance(value, str)) + config["Env"] = [value for value in _list(config.get("Env")) + if isinstance(value, str) and value not in image_env] + image_labels = _dict(held.get("Labels")) + labels = {key: value for key, value in _dict(config.get("Labels")).items() + if image_labels.get(key) != value} + if LABEL_IMAGE in labels: + labels[LABEL_IMAGE] = new_image_id + config["Labels"] = labels + for key in ("ExposedPorts", "Volumes"): + own = {name: value for name, value in _dict(config.get(key)).items() + if name not in _dict(held.get(key))} + if own: + config[key] = own + else: + config.pop(key, None) + if config.get("Hostname") and ident.startswith(str(config["Hostname"])): + config.pop("Hostname", None) # the daemon's default: the old id + host = dict(_dict(raw.get("HostConfig"))) + anonymous = {} + for mount in _list(raw.get("Mounts")): + mount = _dict(mount) + name = _short(mount.get("Name"), 64) + target = _short(mount.get("Destination"), 512) + if mount.get("Type") == "volume" and _HEX64.match(name) and target: + anonymous[target] = (name, bool(mount.get("RW", True))) + binds = [value for value in _list(host.get("Binds")) if isinstance(value, str)] + taken = {value.split(":")[1] for value in binds if value.count(":") >= 1} + mounts = [] + for mount in _list(host.get("Mounts")): + mount = dict(_dict(mount)) + target = _short(mount.get("Target"), 512) + taken.add(target) + # compose declares an anonymous volume with no source; left that way, + # the replacement would be given a new, empty one. + if mount.get("Type") == "volume" and not mount.get("Source") \ + and target in anonymous: + mount["Source"] = anonymous[target][0] + mounts.append(mount) + if mounts: + host["Mounts"] = mounts + for target, (name, writable) in anonymous.items(): + if target not in taken: + binds.append(f"{name}:{target}" + ("" if writable else ":ro")) + if binds: + host["Binds"] = binds + mode = _short(host.get("NetworkMode"), 256) + attached = _dict(_dict(raw.get("NetworkSettings")).get("Networks")) + endpoints = {} + for network, settings in list(attached.items())[:16]: + settings = _dict(settings) + endpoint = {"Aliases": [alias for alias in _list(settings.get("Aliases")) + if isinstance(alias, str) and not ident.startswith(alias)]} + for key in ("IPAMConfig", "Links", "DriverOpts"): + if settings.get(key): + endpoint[key] = settings[key] + endpoints[network] = endpoint + body = dict(config) + later = [] + if mode in ("host", "none") or mode.startswith("container:"): + pass # no network of its own to join + elif mode in endpoints: + body["NetworkingConfig"] = {"EndpointsConfig": {mode: endpoints[mode]}} + later = [(network, endpoint) for network, endpoint in endpoints.items() + if network != mode] + elif mode in ("", "default", "bridge") and endpoints: + first = next(iter(endpoints)) + body["NetworkingConfig"] = {"EndpointsConfig": {first: endpoints[first]}} + later = [(network, endpoint) for network, endpoint in endpoints.items() + if network != first] + body["HostConfig"] = host + return body, later + + +async def recreate(container: str, *, pull: bool = True, entries=None, + say=None) -> dict: + """Bring one container onto the newest image of the name it was created + from. ``updated``, ``current`` (nothing newer), or ``pinned``. + + A replacement that does not stay up puts the old container back, so a bad + image costs a few seconds of downtime rather than the service.""" + say = say or (lambda _line: None) + raw = await docker.container_raw(container) + ident = _short(raw.get("Id"), 64) + name = _short(raw.get("Name"), 128).lstrip("/") + reference = _short(_dict(raw.get("Config")).get("Image"), 256) + result = {"container": ident, "name": name, "image": reference} + if not docker.clean_id(name) or not docker.clean_id(ident): + raise DockerError("the daemon named that container in no usable way") + if not docker.clean_image(reference) or reference.startswith("sha256:") \ + or docker.split_reference(reference)[2]: + return dict(result, result="pinned") + if pull: + say(f"pulling {reference}") + await docker.pull(reference) + fresh = await docker.image_raw(reference) + if fresh is None: + raise DockerError(f"{reference} is not on this machine after the pull") + new_id = _short(fresh.get("Id"), 80) + if new_id == raw.get("Image"): + return dict(result, result="current") + if entries is None: + entries = await docker.raw_containers(True) + for entry in entries: + mode = _short(_dict(entry.get("HostConfig")).get("NetworkMode"), 256) + if mode in (f"container:{ident}", f"container:{name}"): + raise DockerError( + f"{name} carries another container's network; update their " + "stack with compose, which recreates them together") + old_image = await docker.image_raw(_short(raw.get("Image"), 80)) or {} + body, later = recreate_body(raw, old_image, new_id) + state = _dict(raw.get("State")) + was_running = bool(state.get("Running") or state.get("Restarting")) + spare = docker.clean_id(f"{name[:100]}-nmesh-old-{ident[:8]}") + say(f"recreating {name}") + await docker.call("POST", f"/containers/{ident}/stop?t=10", timeout=120.0) + await docker.call("POST", f"/containers/{ident}/rename?name={spare}") + fresh_id = "" + try: + answer = await docker.call("POST", f"/containers/create?name={name}", + body=body, timeout=90.0) + fresh_id = _short(_dict(answer).get("Id"), 64) + if not fresh_id: + raise DockerError("docker created it and did not say what it was called") + for network, endpoint in later: + if not docker.clean_id(network): + continue + await docker.call("POST", f"/networks/{docker.clean_id(network)}/connect", + body={"Container": fresh_id, "EndpointConfig": endpoint}) + if was_running: + await docker.call("POST", f"/containers/{fresh_id}/start", timeout=90.0) + await asyncio.sleep(SETTLE) + check_state = _dict((await docker.container_raw(fresh_id)).get("State")) + if not check_state.get("Running"): + raise DockerError( + f"the new {name} stopped with code {check_state.get('ExitCode')}") + except DockerError as exc: + say(f"putting the old {name} back") + if fresh_id: + try: + await docker.call("DELETE", f"/containers/{fresh_id}?force=1") + except DockerError: + pass + await docker.call("POST", f"/containers/{ident}/rename?name={name}") + if was_running: + await docker.call("POST", f"/containers/{ident}/start", timeout=90.0) + raise DockerError(f"{exc}; the old container was put back") from None + await docker.call("DELETE", f"/containers/{ident}?force=1", timeout=60.0) + return dict(result, result="updated", container=fresh_id) + + +def _order(services: list[dict], entries: list[dict]) -> list[dict]: + """Dependencies first, by the `depends_on` compose records on each one.""" + needs: dict[str, set] = {} + for entry in entries: + labels = _labels(entry) + service = _short(labels.get(LABEL_SERVICE), 64) + wanted = _short(labels.get(LABEL_DEPENDS), 1024) + needs[service] = {part.split(":")[0] for part in wanted.split(",") if part} + ordered, seen = [], set() + + def visit(service: str, depth: int) -> None: + if service in seen or depth > MAX_SERVICES: + return + seen.add(service) + for other in sorted(needs.get(service, ())): + visit(other, depth + 1) + ordered.extend(row for row in services if row["service"] == service) + + for row in services: + visit(row["service"], 0) + return ordered + [row for row in services if row not in ordered] + + +async def engine_update(stack: dict, *, say=None) -> dict: + """Pull every image a stack's containers came from, once each, then + recreate the containers whose image moved.""" + say = say or (lambda _line: None) + entries = await docker.raw_containers(True) + ours = [entry for entry in entries + if _labels(entry).get(LABEL_PROJECT) == stack["name"]] + services = _order(list(stack.get("services") or []), ours) + pulled: dict = {} + steps = [] + for row in services: + label = row["service"] or row["name"] + try: + raw = await docker.container_raw(row["container"]) + reference = _short(_dict(raw.get("Config")).get("Image"), 256) + if docker.clean_image(reference) and not docker.split_reference(reference)[2] \ + and not reference.startswith("sha256:"): + if reference not in pulled: + say(f"pulling {reference}") + try: + await docker.pull(reference) + pulled[reference] = "" + except DockerError as exc: + pulled[reference] = str(exc) + if pulled[reference]: + raise DockerError(f"pulling {reference} failed: {pulled[reference]}") + done = await recreate(row["container"], pull=False, entries=entries, say=say) + except DockerError as exc: + # One service that cannot be updated costs only itself: the rest + # of the stack still gets its new images. + steps.append({"service": label, "result": "failed", "why": str(exc)[:200]}) + say(f"{label}: failed — {exc}") + continue + steps.append({"service": label, "result": done["result"]}) + say(f"{label}: {done['result']}") + return {"stack": stack["name"], "through": "engine", "steps": steps, + "updated": sum(1 for step in steps if step["result"] == "updated"), + "failed": sum(1 for step in steps if step["result"] == "failed")} + + +# --------------------------------------------------------------------------- +# Through compose +# --------------------------------------------------------------------------- + +def _compose_argv(prefix: list[str], stack: dict, *rest: str) -> list[str]: + argv = list(prefix) + ["--project-name", stack["name"]] + workdir = stack.get("workdir") or "" + if _PATH_RE.match(workdir) and os.path.isdir(workdir): + argv += ["--project-directory", workdir] + for path in stack.get("files") or []: + argv += ["-f", path] + return argv + list(rest) + + +async def compose_update(stack: dict, *, say=None) -> dict: + """`compose pull`, then `compose up -d`: the project as its file says, + with the images the registry serves now. + + No `--remove-orphans`: a service dropped from the file is somebody's + decision to make about their data, not a side effect of an update.""" + argv, version = await compose() + if argv is None: + raise DockerError("this machine has no docker compose") + if not stack.get("files_ok"): + raise DockerError("that stack's compose file is not readable here") + pull = ["pull"] + (["--ignore-buildable"] if _at_least(version, 2, 15) else []) + code, text = await run(_compose_argv(argv, stack, *pull), on_output=say) + if code != 0: + raise DockerError(f"pulling that stack failed: {text[-300:]}") + code, more = await run(_compose_argv(argv, stack, "up", "-d"), on_output=say) + if code != 0: + raise DockerError(f"bringing that stack up failed: {more[-300:]}") + return {"stack": stack["name"], "through": "compose", + "output": (text + "\n" + more)[-4000:]} + + +async def compose_act(stack: dict, action: str, *, say=None) -> dict: + argv, _version = await compose() + if argv is None or not stack.get("files_ok"): + raise DockerError("that needs the stack's compose file and docker compose") + code, text = await run(_compose_argv(argv, stack, action), on_output=say) + if code != 0: + raise DockerError(f"that stack refused to {action}: {text[-300:]}") + return {"stack": stack["name"], "action": action, "through": "compose", + "output": text[-4000:]} + + +async def engine_act(stack: dict, action: str) -> dict: + """start / stop / restart, container by container — what is possible + without the file. Dependencies start first and stop last.""" + if action not in ("start", "stop", "restart"): + raise DockerError("that needs the stack's compose file") + entries = await docker.raw_containers(True) + ours = [entry for entry in entries + if _labels(entry).get(LABEL_PROJECT) == stack["name"]] + rows = _order(list(stack.get("services") or []), ours) + if action == "stop": + rows.reverse() + if not rows: + raise DockerError("no container on this machine belongs to that stack") + for row in rows: + await docker.act(row["container"], action) + return {"stack": stack["name"], "action": action, "through": "engine", + "containers": len(rows)} + + +# --------------------------------------------------------------------------- +# A stack's file +# --------------------------------------------------------------------------- + +async def read_file(stack: dict, *, server: str = "") -> dict: + """The compose file a stack was deployed from, for reading. + + Only the compose file: Portainer keeps the stack's environment beside it + (`stack.env`), and a container's environment is where its secrets are — + names travel, values do not.""" + files = stack.get("files") or [] + if stack.get("owner") == "portainer": + number = stack.get("portainer_id") or 0 + if not server or not number: + raise DockerError("Portainer keeps that file; connect it, or run it here") + held = await docker.read_archive(server, f"/data/compose/{int(number)}") + wanted = os.path.basename(files[0]) if files else "docker-compose.yml" + for name in (wanted, "docker-compose.yml", "docker-compose.yaml", + "compose.yml", "compose.yaml"): + if name in held and name.endswith((".yml", ".yaml")): + return {"path": f"/data/compose/{int(number)}/{name}", + "text": _text(held[name]), "source": "portainer"} + raise DockerError("Portainer's directory for that stack holds no compose file") + if not files: + raise DockerError("no compose file is recorded for that stack") + path = files[0] + if not path.endswith((".yml", ".yaml")) or not _readable(path): + raise DockerError("that compose file is not readable by this node") + with open(path, "rb") as handle: + return {"path": path, "text": _text(handle.read(docker.MAX_COMPOSE + 1)), + "source": "disk"} + + +def _text(raw: bytes) -> str: + text = raw.decode("utf-8", "replace") + if len(text) > docker.MAX_COMPOSE: + text = text[:docker.MAX_COMPOSE] + "\n# … cut here: the file is longer" + return text + + +# --------------------------------------------------------------------------- +# Deploying a new one +# --------------------------------------------------------------------------- + +def stack_dir(root: str = "") -> str: + """Where compose files this node deployed are kept. + + Under the node's own data directory when it has one — the same place its + identity and its session store live, so a compose file that decides what + runs on this machine is backed up and wiped with the rest of its state. + Never `/tmp`, and never a path from the wire.""" + base = root or os.path.expanduser("~/.nmesh") + return os.path.join(base, "stacks") + + +async def deploy(project: str, content: str, *, root: str = "", say=None) -> dict: + """Deploy a stack from a compose file an operator wrote. + + The file is written under a directory of this node's own, named for the + project, so a later update finds it where compose recorded it — and so + nothing this deploys can be made to overwrite a path chosen by whoever sent + the file: the only thing from the wire that reaches the path is a name that + passed :func:`fleet_docker.clean_id`.""" + name = docker.clean_id(project) + if not name or name != name.lower(): + raise DockerError("a stack name is lowercase letters, digits, - and _") + body = docker.clean_compose(content) + if not body: + raise DockerError("that compose file is empty or too large") + argv, _version = await compose() + if argv is None: + raise DockerError("this machine has no docker compose") + directory = os.path.join(stack_dir(root), name) + os.makedirs(directory, mode=0o700, exist_ok=True) + path = os.path.join(directory, "docker-compose.yml") + tmp = path + ".new" + with open(tmp, "w", encoding="utf-8") as handle: + os.chmod(tmp, 0o600) + handle.write(body) + os.replace(tmp, path) + stack = {"name": name, "files": [path], "workdir": directory, "files_ok": True} + code, text = await run(_compose_argv(argv, stack, "up", "-d"), on_output=say) + if code != 0: + raise DockerError(f"bringing that stack up failed: {text[-300:]}") + return {"stack": name, "through": "compose", "output": text[-4000:]} diff --git a/src/apps/fleet_web.py b/src/apps/fleet_web.py index f353fbd..24f43c7 100644 --- a/src/apps/fleet_web.py +++ b/src/apps/fleet_web.py @@ -49,12 +49,13 @@ MAX_PASSWORDS = 32 # provisioned consoles' passwords, until read MAX_DOCKER_ANSWERS = 32 # docker answers held for the page that asked # What a docker operation costs on the far side. The reads are one round trip; -# the four that pull images are minutes, and a browser is not going to hold a -# request open for those — they are started and watched through the job list. +# the ones that pull images are minutes, and a browser is not going to hold a +# request open for those — they are started, and the page asks for the answer +# (`op: "answer"`) until it is there. _DOCKER_WAIT = 25.0 -_DOCKER_SLOW = ("pull", "deploy", "deploy_stack", "portainer_deploy") -_DOCKER_ACTS = ("act", "pull", "deploy", "stack", "deploy_stack", - "portainer_stack", "portainer_deploy", "portainer") +_DOCKER_SLOW = ("pull", "deploy", "deploy_stack", "check", "recreate") +_DOCKER_ACTS = ("act", "pull", "deploy", "stack", "deploy_stack", "recreate", + "portainer") # How long a caller waits for a managed node to answer with an invitation. A # mesh hop and one signature, not a job: long enough for a slow path, short # enough that a page is never left hanging on a node that went away. @@ -1284,15 +1285,16 @@ def docker(self, node_hex: str, op: str, **arguments) -> dict: """One docker operation on a managed node, and its answer. The reads are waited on — a page with nothing to draw is not a page. - The four that pull images are started and watched through the job list - instead: a browser is not going to hold a request open for minutes, and - an operation that reports nothing until it finishes is one nobody can - tell from a hang.""" + The ones that pull images are started and their ``rid`` handed back + instead: a browser is not going to hold a request open for minutes. The + page then asks ``answer`` with that rid until the node has replied.""" if not self._app.state.may_use(node_hex, "docker"): raise DockerError("that node has not granted docker") + if op == "answer": + return self._docker_answer(node_hex, str(arguments.get("rid") or "")[:64]) rid = self._job(self._call(self._app.request_docker( self._node(node_hex), op, **arguments)), "docker", node_hex) - if op in _DOCKER_SLOW: + if op in _DOCKER_SLOW or (op == "stack" and arguments.get("action") == "update"): return {"rid": rid, "started": True} deadline = time.monotonic() + _DOCKER_WAIT while time.monotonic() < deadline: @@ -1305,6 +1307,23 @@ def docker(self, node_hex: str, op: str, **arguments) -> dict: time.sleep(0.05) raise DockerError("that node did not answer") + def _docker_answer(self, node_hex: str, rid: str) -> dict: + """A slow operation's answer once it is in, or that it is not yet. + + Only for the node it was asked of: a rid is not a credential, and an + answer is handed to whoever asks about the machine it came from.""" + with self._lock: + held = self._docker.get(rid) + if held is not None and held.get("node") == node_hex: + return self._docker.pop(rid) + job = self._jobs.get(rid) + if job is None or job.get("node") != node_hex: + raise DockerError("no such operation") + failure = self._job_failure(rid) + if failure is not None: + raise DockerError(failure) + return {"pending": True, "rid": rid} + def take_docker(self, rid: str) -> dict | None: with self._lock: return self._docker.pop(str(rid), None) @@ -1556,15 +1575,17 @@ def _docker_line(op: str, data: dict) -> str: return f"pulled {info.get('image', 'an image')}" if op == "deploy": return f"deployed {info.get('name') or info.get('image') or 'a container'}" + if op == "recreate": + return f"{info.get('name', 'a container')}: {info.get('result', 'done')}" if op in ("stack", "deploy_stack"): through = info.get("through") - return (f"stack {info.get('stack', '')} " - f"{info.get('action', 'updated')}" + what = info.get("action") or ("deployed" if op == "deploy_stack" else "updated") + if info.get("steps") is not None: + what = f"updated ({info.get('updated', 0)} container(s) recreated)" + return (f"stack {info.get('stack', '')} {what}" + (f" through {through}" if through else "")) - if op in ("portainer_stack", "portainer_deploy"): - return f"Portainer stack {info.get('name') or info.get('stack', '')}" if op == "portainer": - return "Portainer configured" if info.get("configured") else "Portainer forgotten" + return "Portainer connected" if info.get("connected") else "Portainer forgotten" if items is not None: return f"{len(items)} item(s)" return "done" diff --git a/src/version.py b/src/version.py index 8bfc972..f61d8a5 100644 --- a/src/version.py +++ b/src/version.py @@ -10,7 +10,7 @@ import re -__version__ = "0.4.115" +__version__ = "0.4.116" # Releases are tagged ``vX.Y.Z`` (see .github/workflows/release.yml). Anything # after the numbers (``-rc1``, ``+build``) is kept but only used to break ties diff --git a/src/webassets/fleet.py b/src/webassets/fleet.py index 65b25f0..18eaa19 100644 --- a/src/webassets/fleet.py +++ b/src/webassets/fleet.py @@ -140,78 +140,70 @@ @@ -503,6 +495,19 @@ .node-card .stats{grid-template-columns:repeat(auto-fit,minmax(110px,1fr))} .upd{display:flex;flex-direction:column;gap:4px} .cap-pick{display:grid;gap:var(--s-2);grid-template-columns:repeat(auto-fit,minmax(200px,1fr))} + +/* Docker. A stack is a card whose title carries who owns it and where it + stands; the counts on the sub-tabs are plain numbers, hidden while empty. */ +#panel-docker .card-head h2.row{gap:var(--s-2);font-size:var(--fs-md)} +#panel-docker .card-head h2 .badge{font-weight:500} +#dk-tabs .tail{color:var(--text-faint);font-size:var(--fs-xs);margin-left:var(--s-1)} +#dk-tabs .tail:empty{display:none} +#panel-docker td .badge{white-space:nowrap} +#panel-docker td.mono{overflow-wrap:anywhere;min-width:12ch} +#panel-docker .flush{margin:0} +#panel-docker progress.meter{flex:1 1 auto;max-width:240px} +#panel-docker .fp{overflow-wrap:anywhere} +#panel-docker .notice .btn-row,#panel-docker .notice button{flex:none} """ @@ -1448,43 +1453,77 @@ } // ---- docker ---------------------------------------------------------------- -// One node at a time, because everything here is about one machine. The reads -// are waited on by the console; the ones that pull images are started and -// watched through the job list, since a browser will not hold a request open -// for minutes and an operation that says nothing until it ends is one nobody -// can tell from a hang. +// One machine at a time, because everything here is about one machine. Quick +// reads are waited on by the console; the ones that pull images — an update, a +// check, a deploy — hand back a rid and are asked about until they answer, +// because a browser will not hold a request open for minutes and an operation +// that says nothing until it ends is one nobody can tell from a hang. -const DK = {node:"", overview:null, stacks:[], containers:[], images:[], - volumes:[], networks:[], stats:{}, portainer:[], busy:false}; +const DK = {node:"", overview:null, stacks:null, containers:null, images:null, + volumes:null, networks:null, stats:{}, check:null, checkedAt:0, + running:{}, results:{}, scope:"all", probe:null, error:"", loading:false}; function dockerNodes(){ return (ST.managed || []).filter((node) => (node.caps || []).includes("docker")); } -async function dockerCall(op, extra){ - if(!DK.node) throw new Error("pick a node first"); +async function dockerCall(op, extra, node){ + const target = node || DK.node; + if(!target) throw new Error("pick a machine first"); const answer = await apiJson("/api/fleet/docker", "POST", - Object.assign({node:DK.node, op}, extra || {})); + Object.assign({node:target, op}, extra || {})); if(!answer.ok) throw new Error((answer.data && answer.data.error) || "that was refused"); return answer.data; } +// A slow operation: started, then asked about every two seconds until the node +// has answered. Thirty minutes is the far side's own ceiling on a compose run. +async function dockerSlow(op, extra){ + const node = DK.node; + const started = await dockerCall(op, extra, node); + if(!started.rid) return started; + for(let tries = 0; tries < 900; tries++){ + await new Promise((resolve) => setTimeout(resolve, 2000)); + const answer = await dockerCall("answer", {rid:started.rid}, node); + if(!answer.pending) return answer; + } + throw new Error("that machine stopped answering about it"); +} +// Something long running, drawn as running wherever it shows, for as long as +// it runs — and only on the machine it was started on. +async function dockerRun(key, label, op, extra){ + const node = DK.node; + DK.running[key] = {label, since:Date.now(), node}; + paintDocker(); + try{ return await dockerSlow(op, extra); } + finally{ delete DK.running[key]; paintDocker(); } +} +function dockerBusy(key){ + const job = DK.running[key]; + return job && job.node === DK.node ? job : null; +} + async function dockerLoad(){ - if(!DK.node || DK.busy) return; - DK.busy = true; - setMessage("dk-msg", "Reading that machine…"); + if(!DK.node || DK.loading) return; + DK.loading = true; + DK.error = ""; + paintDocker(); try{ - const overview = await dockerCall("overview"); - const stacks = await dockerCall("stacks"); - const containers = await dockerCall("containers"); - DK.overview = overview.info || {}; - DK.stacks = stacks.items || []; - DK.containers = containers.items || []; - setMessage("dk-msg", ""); - dockerSample(); + DK.overview = (await dockerCall("overview")).info || {}; + if(DK.overview.engine){ + const [stacks, containers] = await Promise.all( + [dockerCall("stacks"), dockerCall("containers")]); + DK.stacks = stacks.items || []; + DK.containers = containers.items || []; + if(stacks.info && stacks.info.portainer_problem) + toast("Portainer did not answer", "warn", stacks.info.portainer_problem); + }else{ DK.stacks = []; DK.containers = []; } }catch(error){ - DK.overview = null; DK.stacks = []; DK.containers = []; - setMessage("dk-msg", String(error.message || error), true); - }finally{ DK.busy = false; } + DK.error = String(error.message || error); + }finally{ DK.loading = false; } paintDocker(); + if(DK.overview && DK.overview.engine){ + dockerSample(); + dockerLoadTab(ROUTER.sub); + } } // Usage figures take about a second a container to read, so the list is shown // first and they join it when they arrive. @@ -1495,8 +1534,30 @@ if(node === DK.node){ DK.stats = answer.info || {}; paintDocker(); } }catch(_){ /* the list stands without them */ } } +// The lists only one tab needs are read when that tab is opened, not on every +// refresh of the panel: each is a round trip to somebody else's machine. +async function dockerLoadTab(sub, force){ + if(!DK.node || !DK.overview || !DK.overview.engine) return; + const node = DK.node; + const wanted = {images:["images"], storage:["volumes", "networks"]}[sub] || []; + for(const op of wanted){ + if(DK[op] !== null && !force) continue; + try{ + const items = (await dockerCall(op)).items || []; + if(node === DK.node) DK[op] = items; + }catch(error){ toast(String(error.message || error), "danger"); } + } + paintDocker(); +} +function dockerReset(){ + Object.assign(DK, {overview:null, stacks:null, containers:null, images:null, + volumes:null, networks:null, stats:{}, check:null, checkedAt:0, + probe:null, error:""}); +} + +// ---- what is drawn ----------------------------------------------------------- // Which stacks *Update* also brings up on that node. Read from the ledger, not -// from the checkbox: it is a choice this console remembers about a machine, and +// from a checkbox: it is a choice this console remembers about a machine, and // the node card's Update button reads the same one. function dockerChosen(){ const node = (ST.managed || []).find((entry) => entry.id === DK.node); @@ -1506,149 +1567,556 @@ return '
' + esc(String(value)) + '' + esc(key) + "
"; } -function paintDockerOverview(){ +function stateBadge(state){ + const tone = {running:"ok", restarting:"warn", paused:"warn", created:"", + exited:"danger", dead:"danger"}[state]; + return badge(state || "unknown", tone === undefined ? "" : tone); +} +// What a check said about one container, as a word and a colour. +const UPDATE_WORDS = { + outdated:["update available", "warn"], pulled:["newer image here", "info"], + current:["up to date", "ok"], pinned:["pinned", ""], local:["local image", ""], + unknown:["could not check", ""], +}; +function updateBadge(id){ + const state = DK.check && DK.check.containers && DK.check.containers[id]; + if(!state) return ""; + const [word, tone] = UPDATE_WORDS[state] || [state, ""]; + return badge(word, tone); +} +function stackNeedsUpdate(stack){ + if(!DK.check) return false; + return (stack.services || []).some((row) => + ["outdated", "pulled"].includes(DK.check.containers[row.container])); +} +function portainerWord(info){ + const pt = (info && info.portainer) || {}; + if(pt.connected) return ["connected", "ok"]; + if(pt.server) return ["found", "warn"]; + if(pt.agent) return ["agent", ""]; + return ["none", ""]; +} +function useText(id){ + const use = DK.stats[id]; + return use ? use.cpu.toFixed(1) + "% · " + fmtBytes(use.memory) : ""; +} + +function paintDockerSummary(){ + const nodes = dockerNodes(); + if(!nodes.length){ + setHTML("dk-summary", emptyHTML("No machine has granted docker", + "A machine you manage has to grant the docker right before it can be read here. " + + "It is root on that machine, so it is never granted by default.")); + return; + } + if(DK.error){ + setHTML("dk-summary", errorHTML("That machine did not answer", DK.error)); + return; + } const info = DK.overview; - if(!info){ setHTML("dk-overview", ""); return; } - setHTML("dk-overview", - stat(info.running + "/" + info.containers, "containers running", "accent") + - stat(DK.stacks.length, "stacks") + - stat(info.images || 0, "images") + - stat(info.version || "—", "engine") + - stat(info.compose ? "yes" : "no", "docker compose") + - stat(info.portainer ? "configured" : "none", "portainer")); -} -function stackRow(stack){ + if(!info){ + setHTML("dk-summary", DK.loading ? skeletonHTML(1) : ""); + return; + } + if(!info.engine){ + setHTML("dk-summary", errorHTML("No docker to drive on that machine", + (info.problem || "") + ". Install docker there, or re-run install.sh --docker so " + + "the node's account can reach the socket.")); + return; + } + const [ptWord, ptTone] = portainerWord(info); + let updates = "not checked", tone = ""; + if(DK.check){ + const waiting = Object.values(DK.check.containers || {}) + .filter((state) => state === "outdated" || state === "pulled").length; + updates = waiting ? waiting + " behind" : "up to date"; + tone = waiting ? "warn" : "ok"; + } + setHTML("dk-summary", '
' + + stat((info.running || 0) + " / " + (info.containers || 0), "containers running", "accent") + + stat(DK.stacks ? DK.stacks.length : (info.stacks || 0), "stacks") + + stat(updates, "updates", tone) + + stat(info.version || "—", "docker engine") + + stat(info.compose || "not installed", "docker compose", info.compose ? "" : "warn") + + stat(ptWord, "portainer", ptTone) + "
"); +} + +function stackOwner(stack){ + if(stack.owner === "portainer") + return badge("Portainer" + (stack.portainer_id ? " · stack " + stack.portainer_id : ""), + "accent"); + return badge(stack.nmesh ? "compose · deployed from here" : "compose"); +} +function stackStateBadge(stack){ + if(!stack.total) return badge("stopped"); + if(stack.running === stack.total) return badge(stack.total + " running", "ok"); + if(!stack.running) return badge("stopped", "danger"); + return badge(stack.running + " of " + stack.total + " running", "warn"); +} +// Where the stack comes from, said in the words an operator would look for. +function stackSource(stack){ + if(stack.owner === "portainer"){ + return stack.portainer_api ? "Held by Portainer, which is connected here" + + (stack.git ? " — redeployed from its git repository" : "") + : "Held by Portainer, which is not connected here"; + } + if(!(stack.files || []).length) return "No compose file is recorded for it"; + return (stack.files || []).join(", ") + (stack.files_ok ? "" : " — not readable by the node"); +} +function serviceRows(stack){ + if(!(stack.services || []).length) + return '

Nothing of it runs on this machine.

'; + return '
' + + "" + + stack.services.map((row) => + "" + + '" + + "" + + '" + + "" + + '').join("") + "
ServiceImageStateUseImage status
" + esc(row.service || row.name) + "" + + '
' + esc(row.name) + "
' + esc(row.image) + "" + stateBadge(row.state) + '
' + esc(row.status || "") + + "
' + esc(useText(row.container)) + "" + updateBadge(row.container) + "
"; +} +function stackCard(stack){ + const key = "stack:" + stack.name; + const busy = dockerBusy(key); + const name = esc(stack.name); + const running = stack.running > 0; + const off = busy ? " disabled" : ""; + const result = DK.results[stack.name]; const chosen = dockerChosen().has(stack.name); - const owner = stack.portainer_id ? "portainer" : (stack.files ? "compose" : "unmanaged"); - return '
' + - '' + - '' + esc(stack.name) + "" + - badge(stack.running + "/" + stack.total + " up", - stack.running === stack.total ? "ok" : (stack.running ? "warn" : "")) + - badge(owner, owner === "portainer" ? "accent" : "") + - '' + - '' + - '' + - '' + - '' + - "
"; + return '
' + + '

' + name + " " + stackOwner(stack) + " " + stackStateBadge(stack) + + (stackNeedsUpdate(stack) ? " " + badge("update available", "warn") : "") + "

" + + '
' + esc(stackSource(stack)) + "
" + + '
' + + '" + + '" + + (running ? '" : "") + + '' + + "
" + + '
' + serviceRows(stack) + + '

Update ' + esc(stack.why || "") + ".

" + + (busy ? '
' + esc(busy.label) + "
" : "") + + (!busy && result ? '
' + + '' + esc(result.text) + "" + + (result.output ? '' : "") + "
" : "") + + "
" + + '
" + + "
"; +} +function paintDockerStacks(){ + const all = DK.stacks; + if(all === null){ + setHTML("dk-stacks", DK.loading ? skeletonHTML(3) : ""); + return; + } + const filter = $("dk-stack-filter").value.trim().toLowerCase(); + const shown = all.filter((stack) => !filter || stack.name.toLowerCase().includes(filter) || + (stack.services || []).some((row) => (row.image + " " + row.service).toLowerCase().includes(filter))); + setHTML("dk-stacks", shown.map(stackCard).join("") || + (all.length ? emptyHTML("No stack matches that filter") + : emptyHTML("No stack on this machine", + "A stack is a compose project, deployed by compose or by Portainer. Deploy one, or " + + "look at the containers that run on their own."))); + const behind = all.filter(stackNeedsUpdate); + $("dk-update-all").hidden = !behind.length; + $("dk-update-all").textContent = "Update " + plural(behind.length, "stack"); + $("dk-check").disabled = !!dockerBusy("check"); + $("dk-check").textContent = dockerBusy("check") ? "Checking…" : "Check for updates"; + $("dk-checked").textContent = DK.check + ? "Checked " + fmtAgo((Date.now() - DK.checkedAt) / 1000) : ""; } + function containerRow(entry){ const running = entry.state === "running"; - const use = DK.stats[entry.id]; - return '
' + - '' + esc(entry.name || shortId(entry.id)) + "" + - '' + esc(entry.image) + "" + - (entry.project ? badge(entry.project) : "") + - badge(entry.state || "?", running ? "ok" : "warn") + - (use ? '' + use.cpu.toFixed(1) + "% · " + - fmtBytes(use.memory) + "" : "") + - (entry.ports.length ? '' + - esc(entry.ports.join(" ")) + "" : "") + - '' + - '" + - '' + - '' + - '' + - "
"; -} -function portainerRow(stack){ - return '
' + esc(stack.name) + "" + - badge(stack.active ? "active" : "stopped", stack.active ? "ok" : "") + - (stack.git ? badge("git") : "") + - '' + - '" + - '
'; + const key = "ct:" + entry.id; + const off = dockerBusy(key) ? " disabled" : ""; + const id = esc(entry.id); + return "" + esc(entry.name || shortId(entry.id)) + "" + + (entry.ports.length ? '
' + esc(entry.ports.join(" ")) + + "
" : "") + "" + + '' + esc(entry.image) + "" + + "" + (entry.project ? badge(entry.project) : '—') + "" + + "" + stateBadge(entry.state) + "" + + '' + esc(useText(entry.id)) + "" + + "" + (dockerBusy(key) ? badge("updating…", "info") : updateBadge(entry.id)) + "" + + '' + + '" + + (running ? '" : "") + + // Inside a stack, the stack's Update is the one that knows its owner. + (entry.project ? "" : '") + + '' + + '"; +} +function paintDockerContainers(){ + const all = DK.containers; + if(all === null){ setHTML("dk-containers", DK.loading ? skeletonHTML(4) : ""); return; } + const filter = $("dk-ct-filter").value.trim().toLowerCase(); + const shown = all.filter((entry) => (DK.scope === "all" || !entry.project) && + (!filter || (entry.name + " " + entry.image + " " + entry.project).toLowerCase().includes(filter))); + $$("[data-dk-scope]").forEach((button) => + button.setAttribute("aria-selected", button.dataset.dkScope === DK.scope ? "true" : "false")); + setHTML("dk-containers", shown.length ? '
' + + "" + + "" + shown.map(containerRow).join("") + "
NameImageStackStateUseImage status
" + : emptyHTML(all.length ? "No container matches" : "No container on this machine")); +} + +function paintDockerImages(){ + if(DK.images === null){ setHTML("dk-images", skeletonHTML(3)); return; } + const used = new Set((DK.containers || []).map((entry) => entry.image_id)); + setHTML("dk-images", DK.images.length ? '
' + + '' + + DK.images.map((image) => { + const name = image.tags[0] || image.id; + return '" + + '" + + '" + + '"; + }).join("") + "
ImageSizeCreated
' + esc(name) + + (image.tags.length > 1 ? '
' + esc(image.tags.slice(1).join(", ")) + + "
" : "") + "		' + fmtBytes(image.size) + "' + (image.created + ? esc(new Date(image.created * 1000).toLocaleDateString()) : "—") + "' + (used.has(image.id) ? badge("in use", "ok") + : '') + "
" : emptyHTML("No image on this machine")); +} +function paintDockerStorage(){ + setHTML("dk-volumes", DK.volumes === null ? skeletonHTML(2) : DK.volumes.length + ? '
' + + DK.volumes.map((volume) => '").join("") + "
NameDriver
' + esc(volume.name) + "" + + esc(volume.driver) + "
" + : emptyHTML("No volume")); + setHTML("dk-networks", DK.networks === null ? skeletonHTML(2) : DK.networks.length + ? '
' + + '' + + DK.networks.map((network) => "").join("") + + "
NameDriverSubnetContainers
" + esc(network.name) + "" + + esc(network.driver) + '' + esc(network.subnet || "—") + + '' + network.attached + "
" + : emptyHTML("No network")); +} + +// The Portainer tab is a conversation in three states: none here, found but +// not connected, connected. Rebuilt only when that state changes, so a token +// half typed is never wiped by a poll. +function portainerKey(){ + const pt = (DK.overview && DK.overview.portainer) || {}; + return JSON.stringify([DK.node, !!DK.overview, pt.connected, pt.url, pt.pinned, + pt.server && pt.server.id, pt.agent, DK.probe]); +} +let PT_DRAWN = ""; +function paintDockerPortainer(){ + const key = portainerKey(); + const pt = (DK.overview && DK.overview.portainer) || null; + $("dk-n-portainer").textContent = pt && pt.connected ? "connected" + : pt && pt.server ? "found" : ""; + if(key === PT_DRAWN) return; + PT_DRAWN = key; + if(!pt){ setHTML("dk-portainer", DK.loading ? skeletonHTML(2) : ""); return; } + const server = pt.server; + let html = ""; + if(server){ + html += '

Portainer ' + + "runs on this machine

Found from its container — no credential " + + "is needed to see it, or to see which stacks are its.
" + + stateBadge(server.state) + "
" + + '
' + + "
Container
" + esc(server.name) + "
" + + '
Image
' + esc(server.image) + "
" + + "
Edition
" + esc(server.edition) + "
" + + '
Address here
' + esc(server.url || "publishes no port") + "
" + + "
"; + }else if(pt.agent){ + html += '
This machine runs a Portainer agent: its stacks are deployed ' + + "by a Portainer elsewhere. Connect that Portainer to update them through it.
"; + } + if(pt.connected){ + html += '

Connected

' + + '
Stacks Portainer owns are updated through its API — it pulls and ' + + "redeploys with the file and environment it holds, and its own record stays true." + + "
" + badge("connected", "ok") + "
" + + '
Address
' + esc(pt.url) + + "
Certificate
" + (pt.pinned ? "pinned" : + pt.url.startsWith("http:") ? "none: loopback only" : "verified by its chain") + + "
Token
kept in that node's encrypted drawer, never read back
" + + '
' + + "
"; + }else{ + const suggested = (server && server.url) || ""; + const probe = DK.probe; + html += '

Connect ' + + (server ? "it" : "a Portainer") + "

" + + '
Without it, a Portainer stack is updated by pulling its images and ' + + "recreating its containers with the settings they run with. Connected, it is " + + "redeployed by Portainer itself.
" + + '
' + + '
' + + '
' + + (probe ? '
Portainer ' + esc(probe.version) + + " answers there." + (probe.fingerprint ? ' Its certificate: ' + + esc(probe.fingerprint.match(/../g).join(":")) + "" : "") + "
" + + (probe.fingerprint ? '" : "") : "") + + '" + + '
' + + '

'; + } + if(!server && !pt.agent && !pt.connected){ + html = '
No Portainer runs on this machine. Stacks deployed with compose ' + + "are updated from their own files; connect a Portainer below only if one elsewhere " + + "manages this machine.
" + html; + } + setHTML("dk-portainer", html); } + function paintDocker(){ $("nav-docker").textContent = dockerNodes().length || ""; - paintDockerOverview(); - // Nothing read yet is not the same as nothing there, and a panel that says - // "no stack" before it has looked is a panel that lies once a second. - const waiting = DK.overview === null; - setHTML("dk-stacks", DK.stacks.map(stackRow).join("") || - (waiting ? emptyHTML("Not read yet", "Pick a node, then Refresh.") - : emptyHTML("No stack here", - "A stack is a compose project. Deploy one, or run plain containers."))); - setHTML("dk-containers", DK.containers.map(containerRow).join("") || - (waiting ? "" : emptyHTML("Nothing is running", "No container on that machine."))); - setHTML("dk-images", DK.images.map((image) => - '
' + - esc((image.tags[0] || image.id).slice(0, 80)) + "" + - '' + fmtBytes(image.size) + "" + - '
').join("") || - '

Nothing read yet.

'); - setHTML("dk-volumes", DK.volumes.map((volume) => - '
' + esc(volume.name.slice(0, 64)) + "" + - '' + esc(volume.driver) + "
").join("") || - '

No volume, or not read yet.

'); - setHTML("dk-networks", DK.networks.map((network) => - '
' + esc(network.name) + "" + - '' + esc(network.driver) + "" + - (network.subnet ? '' + esc(network.subnet) + "" : "") + - badge(plural(network.attached, "container")) + "
").join("") || - '

No network, or not read yet.

'); - setHTML("pt-stacks", DK.portainer.map(portainerRow).join("") || - '

Nothing read from Portainer yet.

'); -} - -// A long job — a pull, a deploy, a stack coming up — is started and then -// watched in the job list. Reporting only at the end is indistinguishable from -// a hang, and this is minutes of somebody else's machine. -async function dockerJob(label, op, extra){ - let answer; - try{ answer = await dockerCall(op, extra); } - catch(error){ toast(String(error.message || error), "danger"); return; } - if(!answer.rid){ toast(label + " done", "ok"); await dockerLoad(); return; } - toast(label + " started — it shows in Activity"); - const rid = answer.rid; - for(let tries = 0; tries < 600; tries++){ - await new Promise((resolve) => setTimeout(resolve, 2000)); - const job = (ST.jobs || []).find((entry) => entry.rid === rid); - if(job && job.state !== "running"){ - toast(label + (job.state === "ok" ? " done" : " failed: " + (job.detail || "")), - job.state === "ok" ? "ok" : "danger"); - break; + $("dk-n-stacks").textContent = DK.stacks ? DK.stacks.length : ""; + $("dk-n-containers").textContent = DK.containers ? DK.containers.length : ""; + paintDockerSummary(); + const engine = !!(DK.overview && DK.overview.engine) || DK.loading; + $("dk-tabs").hidden = !engine && !!DK.overview; + paintDockerStacks(); + paintDockerContainers(); + paintDockerImages(); + paintDockerStorage(); + paintDockerPortainer(); + tickDocker(); +} +// The elapsed time on a running job, rewritten in place: a whole repaint a +// second would close whatever the operator had open. +function tickDocker(){ + $$("#panel-docker [data-since]").forEach((element) => { + const seconds = Math.round((Date.now() - Number(element.dataset.since)) / 1000); + const label = element.dataset.label || element.textContent.replace(/ · \d+:\d\d$/, ""); + element.dataset.label = label; + element.textContent = label + " · " + Math.floor(seconds / 60) + ":" + + String(seconds % 60).padStart(2, "0"); + }); +} +setInterval(tickDocker, 1000); + +// ---- what is done ------------------------------------------------------------ + +function stackByName(name){ + return (DK.stacks || []).find((stack) => stack.name === name); +} +// Updating is the one act here that restarts services, so it says what it will +// do before doing it: through whom, and which containers. +async function updateStack(name, quiet){ + const stack = stackByName(name); + if(!stack) return false; + if(!quiet){ + const services = (stack.services || []).map((row) => + "
  • " + esc(row.service || row.name) + ' ' + + esc(row.image) + "
    • ").join(""); + const agreed = await confirmAction({title:"Update " + name + "?", + body:'

    ' + esc(stack.why || "") + ".

    " + + (services ? '
      ' + services + "
    " : "") + + '

    A container whose image changed is recreated, so its service ' + + "is down for a few seconds. Volumes are kept.

    ", + confirmLabel:"Update"}); + if(!agreed) return false; + } + try{ + const answer = await dockerRun("stack:" + name, "Updating", "stack", + {stack:name, action:"update"}); + const info = answer.info || {}; + const steps = info.steps || []; + const text = info.through === "engine" + ? (info.updated ? plural(info.updated, "container") + " recreated on a newer image" + : "Already on the newest images — nothing recreated") + : info.through === "portainer" ? "Redeployed by Portainer, images pulled" + : "Pulled and brought up with compose"; + DK.results[name] = {ok:true, text, output:info.output || + steps.map((step) => step.service + ": " + step.result).join("\n")}; + if(!quiet) toast(name + ": " + text, "ok"); + return true; + }catch(error){ + DK.results[name] = {ok:false, text:String(error.message || error), output:""}; + toast(name + " could not be updated", "danger", String(error.message || error)); + return false; + }finally{ + await dockerLoad(); + if(DK.check) checkUpdates(true); + } +} +async function updateAll(){ + const names = (DK.stacks || []).filter(stackNeedsUpdate).map((stack) => stack.name); + const agreed = await confirmAction({title:"Update " + plural(names.length, "stack") + "?", + body:'

    One after the other, each the way its owner would: ' + + esc(names.join(", ")) + ".

    " + + '

    Each container whose image changed is recreated.

    ', + confirmLabel:"Update all"}); + if(!agreed) return; + let done = 0; + for(const name of names){ if(await updateStack(name, true)) done++; } + toast(plural(done, "stack") + " of " + names.length + " updated", + done === names.length ? "ok" : "warn"); +} +async function checkUpdates(quiet){ + try{ + const answer = await dockerRun("check", "Checking", "check", {}); + DK.check = answer.info || {containers:{}, images:{}}; + DK.checkedAt = Date.now(); + if(!quiet){ + const behind = Object.values(DK.check.containers || {}) + .filter((state) => state === "outdated" || state === "pulled").length; + toast(behind ? plural(behind, "container") + " can be updated" : "Everything is up to date", + behind ? "warn" : "ok"); } + }catch(error){ if(!quiet) toast(String(error.message || error), "danger"); } + paintDocker(); +} +async function stackAct(name, action, button){ + return withBusy(button, async () => { + try{ + await dockerCall("stack", {stack:name, action}); + toast(name + ": " + ({start:"started", stop:"stopped", restart:"restarted"}[action] || action), "ok"); + }catch(error){ toast(String(error.message || error), "danger"); } + await dockerLoad(); + }); +} +async function containerAct(id, action, label, button){ + if(action === "remove"){ + const agreed = await confirmAction({title:"Remove " + (label || "this container") + "?", + body:'

    It is stopped and deleted. Anything it wrote outside a ' + + "volume goes with it.

    ", confirmLabel:"Remove", danger:true}); + if(!agreed) return; } + return withBusy(button, async () => { + try{ await dockerCall("act", {id, action}); } + catch(error){ toast(String(error.message || error), "danger"); } + await dockerLoad(); + }); +} +async function recreateContainer(id, label){ + const agreed = await confirmAction({title:"Update " + (label || "this container") + "?", + body:'

    Its image is pulled again. If the registry has a newer one, the ' + + "container is recreated from it with the settings it runs with, and put back as it " + + "was if the new one does not stay up.

    ", confirmLabel:"Update"}); + if(!agreed) return; + try{ + const info = (await dockerRun("ct:" + id, "Updating", "recreate", {id})).info || {}; + toast((label || "Container") + ": " + (info.result === "updated" ? "recreated on a newer image" + : info.result === "pinned" ? "pinned to a digest, nothing to pull" : "already up to date"), "ok"); + }catch(error){ toast(String(error.message || error), "danger"); } await dockerLoad(); } - -async function dockerLogs(id){ - $("modal-title").textContent = "Logs"; - $("modal-body").innerHTML = '
    Reading…
    '; +function textDialog(title, sub, text){ + $("modal-title").textContent = title; + $("modal-body").innerHTML = (sub ? '

    ' + esc(sub) + "

    " : "") + + '
    ' +
+    '
    ';
+  // Written as text, never as markup: these are bytes a container or a file chose.
+  $("dk-text").textContent = text;
+  $("dk-text-copy").addEventListener("click", () => copyText(text));
   $("modal").showModal();
+}
+async function dockerLogs(id, label){
+  $("modal-title").textContent = "Logs · " + (label || shortId(id));
+  $("modal-body").innerHTML = '
    Reading…
    ' +
+    '
    ';
+  $("modal").showModal();
+  const read = async () => {
+    try{
+      const answer = await dockerCall("logs", {id, tail:400});
+      $("dk-log").textContent = answer.text || "(nothing)";
+      $("dk-log").scrollTop = $("dk-log").scrollHeight;
+    }catch(error){ $("dk-log").textContent = String(error.message || error); }
+  };
+  $("dk-log-more").addEventListener("click", read);
+  await read();
+}
+async function stackFile(name){
   try{
-    const answer = await dockerCall("logs", {id, tail:400});
-    // Written as text, never as markup: these are bytes a container chose.
-    $("dk-log").textContent = answer.text || "(nothing)";
-  }catch(error){ $("dk-log").textContent = String(error.message || error); }
+    const info = (await dockerCall("stack_file", {stack:name})).info || {};
+    textDialog(name + " · compose file", info.path || "", info.text || "");
+  }catch(error){ toast(String(error.message || error), "danger"); }
+}
+async function portainerProbe(button){
+  const url = $("pt-url").value.trim();
+  if(!url){ setMessage("pt-msg", "An address is needed.", true); return; }
+  return withBusy(button, async () => {
+    try{
+      DK.probe = (await dockerCall("portainer_probe", {url})).info || null;
+      paintDocker();
+    }catch(error){ setMessage("pt-msg", String(error.message || error), true); }
+  });
+}
+async function portainerConnect(button){
+  const url = $("pt-url").value.trim();
+  const token = $("pt-token").value;
+  if(!url || !token){ setMessage("pt-msg", "An address and a token are needed.", true); return; }
+  const pin = $("pt-pin") && $("pt-pin").checked && DK.probe && DK.probe.url === url
+    ? DK.probe.fingerprint : "";
+  return withBusy(button, async () => {
+    try{
+      const answer = await dockerCall("portainer", {url, token, fingerprint:pin});
+      // Never echoed back: the token now lives in that node's drawer.
+      $("pt-token").value = "";
+      DK.probe = null;
+      toast("Portainer connected — " + plural((answer.items || []).length, "environment") +
+            " visible", "ok");
+      await dockerLoad();
+    }catch(error){ setMessage("pt-msg", String(error.message || error), true); }
+  });
+}
+async function portainerForget(button){
+  const agreed = await confirmAction({title:"Forget that Portainer?",
+    body:'
    The address and token are deleted from that node\'s drawer. ' +
+      "Its stacks keep running; they are then updated by recreating their containers " +
+      "instead of through Portainer.
    ", confirmLabel:"Forget", danger:true});
+  if(!agreed) return;
+  return withBusy(button, async () => {
+    try{ await dockerCall("portainer", {clear:true}); toast("Portainer forgotten"); }
+    catch(error){ toast(String(error.message || error), "danger"); }
+    await dockerLoad();
+  });
 }
 
 function runDialog(){
   $("modal-title").textContent = "Run a container";
   $("modal-body").innerHTML =
-    '
    Built field by field on the far side — a container body passed ' +
-    "through whole would be every field that node does not know about.
    " +
+    '
    ' +
     '' +
     '' +
-    '' +
-    '' +
-    '' +
+    '' +
+    '' +
+    '' +
     '' +
+    "
    " +
     '
    ' +
     '
    ';
   $("rn-go").addEventListener("click", (event) => withBusy(event.target, async () => {
@@ -1670,9 +2138,13 @@
       if(at > 0) env[line.slice(0, at)] = line.slice(at + 1);
     });
     $("modal").close();
-    await dockerJob("Container", "deploy", {spec:{
-      name:$("rn-name").value.trim(), image, ports, volumes, env,
-      restart:$("rn-restart").value}});
+    try{
+      await dockerRun("run", "Starting " + image, "deploy", {spec:{
+        name:$("rn-name").value.trim(), image, ports, volumes, env,
+        restart:$("rn-restart").value}});
+      toast(image + " is running", "ok");
+    }catch(error){ toast(String(error.message || error), "danger"); }
+    await dockerLoad();
   }));
   $("rn-no").addEventListener("click", () => $("modal").close());
   $("modal").showModal();
@@ -1680,35 +2152,51 @@
 }
 
 function stackDialog(){
-  const viaPortainer = !!(DK.overview && DK.overview.portainer);
+  const connected = !!(DK.overview && DK.overview.portainer && DK.overview.portainer.connected);
+  const compose = !!(DK.overview && DK.overview.compose);
   $("modal-title").textContent = "Deploy a stack";
   $("modal-body").innerHTML =
-    '
    A compose file, kept on that machine under this node\'s own state ' +
-    "so a redeploy later finds it where compose recorded it.
    " +
+    (connected ? '
    ' +
+      '" +
+      '
    ' : "") +
+    '
    ' +
     '' +
-    (viaPortainer ? '" +
-      '' : "") +
+    'placeholder="my-stack" autocomplete="off" spellcheck="false">' +
+    'lowercase letters, digits, - and _' +
     '' +
     '
    ' +
     '
    ';
+  let through = connected && !compose ? "portainer" : "compose";
+  const explain = () => {
+    $("sk-where").textContent = through === "portainer"
+      ? "Portainer keeps the file and owns the stack; it is updated through Portainer."
+      : "The file is kept on that machine under the node's own state, and compose brings it up.";
+    $$("[data-sk-through]").forEach((button) =>
+      button.setAttribute("aria-selected", button.dataset.skThrough === through ? "true" : "false"));
+  };
+  $$("[data-sk-through]").forEach((button) => button.addEventListener("click", () => {
+    through = button.dataset.skThrough; explain();
+  }));
+  explain();
+  if(!compose && !connected)
+    setMessage("sk-msg", "That machine has no docker compose: install it there first.", true);
   $("sk-go").addEventListener("click", (event) => withBusy(event.target, async () => {
     const name = $("sk-name").value.trim();
-    const compose = $("sk-body").value;
-    if(!name || !compose.trim()){
-      setMessage("sk-msg", "A name and a compose file are required.", true); return;
+    const body = $("sk-body").value;
+    if(!/^[a-z0-9][a-z0-9_-]{0,62}$/.test(name) || !body.trim()){
+      setMessage("sk-msg", "A lowercase name and a compose file are needed.", true); return;
     }
-    const through = viaPortainer && $("sk-pt").checked;
     $("modal").close();
-    await dockerJob("Stack " + name,
-                    through ? "portainer_deploy" : "deploy_stack",
-                    through ? {stack:name, compose,
-                               endpoint:parseInt($("sk-endpoint").value, 10) || 1}
-                            : {stack:name, compose});
+    try{
+      await dockerRun("deploy:" + name, "Deploying " + name, "deploy_stack",
+                      {stack:name, compose:body, portainer:through === "portainer"});
+      toast("Stack " + name + " deployed", "ok");
+    }catch(error){ toast("Stack " + name + " failed", "danger", String(error.message || error)); }
+    await dockerLoad();
   }));
   $("sk-no").addEventListener("click", () => $("modal").close());
   $("modal").showModal();
@@ -1804,7 +2292,7 @@
     return openShell();
   }
   if(data.docker){
-    DK.node = data.docker;
+    if(DK.node !== data.docker){ DK.node = data.docker; dockerReset(); }
     $("dk-node").value = data.docker;
     ROUTER.go("docker");
     return dockerLoad();
@@ -1826,60 +2314,6 @@
           started.length ? "ok" : "warn");
     return poll();
   }
-  if(data.stackAct){
-    const name = data.stack;
-    if(data.stackAct === "update"){
-      return dockerJob("Stack " + name, "stack", {stack:name, action:"update"});
-    }
-    return withBusy(button, async () => {
-      try{
-        await dockerCall("stack", {stack:name, action:data.stackAct});
-        toast("Stack " + name + " " + data.stackAct + "ed", "ok");
-      }catch(error){ toast(String(error.message || error), "danger"); }
-      await dockerLoad();
-    });
-  }
-  if(data.imgRm){
-    const agreed = await confirmAction({title:"Remove this image?",
-      body:'
    The daemon refuses while a container uses it.
    ',
-      confirmLabel:"Remove", danger:true});
-    if(!agreed) return;
-    return withBusy(button, async () => {
-      try{ await dockerCall("remove_image", {image:data.imgRm}); }
-      catch(error){ toast(String(error.message || error), "danger"); }
-      try{ DK.images = (await dockerCall("images")).items || []; }catch(_){ /* kept */ }
-      paintDocker();
-    });
-  }
-  if(data.ctLogs) return dockerLogs(data.ctLogs);
-  if(data.ctAct){
-    if(data.ctAct === "remove"){
-      const agreed = await confirmAction({title:"Remove this container?",
-        body:'
    It is stopped and deleted. Anything it wrote outside a ' +
-          "volume goes with it.
    ", confirmLabel:"Remove", danger:true});
-      if(!agreed) return;
-    }
-    return withBusy(button, async () => {
-      try{
-        await dockerCall("act", {id:data.ct, action:data.ctAct});
-      }catch(error){ toast(String(error.message || error), "danger"); }
-      await dockerLoad();
-    });
-  }
-  if(data.ptAct){
-    if(data.ptAct === "redeploy"){
-      return dockerJob("Portainer redeploy", "portainer_stack",
-                       {id:parseInt(data.pt, 10), action:"redeploy"});
-    }
-    return withBusy(button, async () => {
-      try{
-        await dockerCall("portainer_stack",
-                         {id:parseInt(data.pt, 10), action:data.ptAct});
-        DK.portainer = (await dockerCall("portainer_stacks")).items || [];
-        paintDocker();
-      }catch(error){ toast(String(error.message || error), "danger"); }
-    });
-  }
 });
 // ---- an invitation minted by somebody else ---------------------------------
 // The node that will honour the invitation is the one that mints it, so this is
@@ -1969,28 +2403,61 @@
 $("group-new").addEventListener("click", () => groupDialog(""));
 $("dk-node").addEventListener("change", () => {
   DK.node = $("dk-node").value;
-  DK.images = []; DK.volumes = []; DK.networks = []; DK.stats = {}; DK.portainer = [];
+  dockerReset();
   dockerLoad();
 });
 $("dk-refresh").addEventListener("click", (event) => withBusy(event.target, async () => {
+  DK.images = null; DK.volumes = null; DK.networks = null;
   await dockerLoad();
-  // The two lists nothing else needs: read on demand rather than on every
-  // refresh of the panel.
-  try{ DK.images = (await dockerCall("images")).items || []; }catch(_){ DK.images = []; }
-  try{ DK.volumes = (await dockerCall("volumes")).items || []; }catch(_){ DK.volumes = []; }
-  try{ DK.networks = (await dockerCall("networks")).items || []; }catch(_){ DK.networks = []; }
-  if(DK.overview && DK.overview.portainer){
-    try{ DK.portainer = (await dockerCall("portainer_stacks")).items || []; }
-    catch(_){ DK.portainer = []; }
-  }
-  paintDocker();
 }));
 // Read when the panel is opened, not on every poll: this is a round trip to
 // somebody else's machine, and a tab nobody is looking at should cost nothing.
-ROUTER.onChange = (section) => {
-  if(section === "docker" && DK.node && DK.overview === null) dockerLoad();
+ROUTER.onChange = (section, sub) => {
+  if(section === "docker" && DK.node){
+    if(DK.overview === null) dockerLoad(); else dockerLoadTab(sub);
+  }
   if(section === "logs") refreshLogs();
 };
+$("panel-docker").addEventListener("click", (event) => {
+  const scope = event.target.closest("[data-dk-scope]");
+  if(scope){ DK.scope = scope.dataset.dkScope; paintDocker(); return; }
+  const button = event.target.closest("[data-dk]");
+  if(!button) return;
+  const data = button.dataset;
+  const act = {
+    update: () => updateStack(data.stack),
+    stack: () => stackAct(data.stack, data.action, button),
+    file: () => stackFile(data.stack),
+    output: () => {
+      const result = DK.results[data.stack];
+      if(result) textDialog(data.stack + " · last update", "", result.output || result.text);
+    },
+    logs: () => dockerLogs(data.id, data.label),
+    act: () => containerAct(data.id, data.action, data.label, button),
+    remove: () => containerAct(data.id, "remove", data.label, button),
+    recreate: () => recreateContainer(data.id, data.label),
+    rmi: () => removeImage(data.image, button),
+    "pt-probe": () => portainerProbe(button),
+    "pt-connect": () => portainerConnect(button),
+    "pt-forget": () => portainerForget(button),
+  }[data.dk];
+  if(act) act();
+});
+// The switch that decides what Update also brings up. Written the moment it
+// changes, because a preference that needs a second button to save it is one
+// that is routinely lost.
+$("panel-docker").addEventListener("change", async (event) => {
+  const box = event.target.closest("[data-dk-pick]");
+  if(!box) return;
+  const chosen = dockerChosen();
+  if(box.checked) chosen.add(box.dataset.dkPick); else chosen.delete(box.dataset.dkPick);
+  await api("/api/fleet/stacks", "POST", {node:DK.node, stacks:[...chosen]});
+  poll();
+});
+$("dk-stack-filter").addEventListener("input", paintDocker);
+$("dk-ct-filter").addEventListener("input", paintDocker);
+$("dk-check").addEventListener("click", () => checkUpdates(false));
+$("dk-update-all").addEventListener("click", updateAll);
 $("dk-stack-new").addEventListener("click", () => stackDialog());
 $("dk-container-new").addEventListener("click", () => runDialog());
 $("dk-pull-form").addEventListener("submit", async (event) => {
@@ -1998,51 +2465,25 @@
   const image = $("dk-pull").value.trim();
   if(!image) return;
   $("dk-pull").value = "";
-  await dockerJob("Pull of " + image, "pull", {image});
-});
-$("pt-save").addEventListener("click", (event) => withBusy(event.target, async () => {
-  const url = $("pt-url").value.trim();
-  const token = $("pt-token").value;
-  if(!url || !token){
-    setMessage("pt-msg", "An address and a token are required.", true); return;
-  }
-  try{
-    const answer = await dockerCall("portainer",
-      {url, token, fingerprint:$("pt-fp").value.trim()});
-    // Never echoed back: the field is cleared here because the token now lives
-    // in that node's drawer and nothing reads it out again.
-    $("pt-token").value = "";
-    const items = answer.items || [];
-    setMessage("pt-msg", "Saved" + (answer.info && answer.info.pinned ? ", pinned" : "") +
-      " — " + plural(items.length, "environment") + " visible.");
-    DK.portainer = (await dockerCall("portainer_stacks")).items || [];
-    paintDocker();
-  }catch(error){ setMessage("pt-msg", String(error.message || error), true); }
-}));
-$("pt-forget").addEventListener("click", (event) => withBusy(event.target, async () => {
-  const agreed = await confirmAction({title:"Forget that Portainer?",
-    body:'
    The address and token are deleted from that node\'s drawer. ' +
-      "Its stacks keep running; this console just stops being able to redeploy them.
    ",
-    confirmLabel:"Forget", danger:true});
-  if(!agreed) return;
+  toast("Pulling " + image + "…");
   try{
-    await dockerCall("portainer", {clear:true});
-    DK.portainer = [];
-    setMessage("pt-msg", "Forgotten.");
-    await dockerLoad();
-  }catch(error){ setMessage("pt-msg", String(error.message || error), true); }
-}));
-// The tick that decides what Update also brings up. Written the moment it
-// changes, because a preference that needs a second button to save it is one
-// that is routinely lost.
-$("dk-stacks").addEventListener("change", async (event) => {
-  const box = event.target.closest("[data-stack-pick]");
-  if(!box) return;
-  const chosen = $$("#dk-stacks [data-stack-pick]")
-    .filter((entry) => entry.checked).map((entry) => entry.dataset.stackPick);
-  await api("/api/fleet/stacks", "POST", {node:DK.node, stacks:chosen});
-  poll();
+    const info = (await dockerRun("pull", "Pulling " + image, "pull", {image})).info || {};
+    toast("Pulled " + (info.image || image), "ok");
+  }catch(error){ toast(String(error.message || error), "danger"); }
+  await dockerLoadTab("images", true);
 });
+async function removeImage(image, button){
+  const agreed = await confirmAction({title:"Remove this image?",
+    body:'
    ' + esc(image) + "
    " +
+      '
    The daemon refuses while a container uses it.
    ',
+    confirmLabel:"Remove", danger:true});
+  if(!agreed) return;
+  return withBusy(button, async () => {
+    try{ await dockerCall("remove_image", {image}); }
+    catch(error){ toast(String(error.message || error), "danger"); }
+    await dockerLoadTab("images", true);
+  });
+}
 
 $("ssh-sudo").addEventListener("change", syncSudoFields);
 $("deploy-btn").addEventListener("click", deploy);
@@ -2110,7 +2551,9 @@
   $("shell").classList.remove("hidden");
   mountShell();
   CONTEXT.confirm();
-  ROUTER.start(() => {});
+  // No handler passed: the one set beside the panels is the handler, and a
+  // fresh one here would replace it.
+  ROUTER.start();
   await poll();
   await loadKeys();
   // The interval in the top bar drives this page too. It used to be markup with
diff --git a/src/webconsole.py b/src/webconsole.py
index 5352e67..6a1336b 100644
--- a/src/webconsole.py
+++ b/src/webconsole.py
@@ -271,7 +271,7 @@ def close(self) -> None:
 # than forwarded whole: a body passed straight through is every field the far
 # side does not know about, and there is no list of those to reason about.
 _DOCKER_ARGS = ("id", "action", "tail", "image", "spec", "stack", "compose",
-                "url", "token", "fingerprint", "clear", "endpoint")
+                "url", "token", "fingerprint", "clear", "portainer", "rid")
 
 
 class _WebSurface:
diff --git a/tests/test_fleet.py b/tests/test_fleet.py
index 0b68d41..a77a92c 100644
--- a/tests/test_fleet.py
+++ b/tests/test_fleet.py
@@ -24,7 +24,7 @@
     console_path_refusal,
 )
 from src.apps.fleet_state import CAPABILITIES, FleetState
-from src.apps import fleet_files, fleet_provision
+from src.apps import fleet_docker, fleet_files, fleet_provision
 from src.crypto import CryptoIdentity
 from src.node_id import NodeID
 
@@ -1194,13 +1194,190 @@ async def test_only_so_many_docker_operations_at_once(self, operator, agent):
         once" is not something an operator ever wanted, and it is exactly what
         a flood would ask for."""
         await enrol(operator, agent, caps=["docker"])
-        agent.app._docker_calls[operator.id.raw] = fleet.MAX_DOCKER_CALLS
+        agent.app._docker_calls[operator.id.raw] = (fleet.MAX_DOCKER_CALLS, 0)
         await operator.app.request_docker(agent.id, "containers")
         await deliver(operator, agent)
         await deliver(agent, operator)
         failures = [e for e in operator.drain_events() if isinstance(e, Failure)]
         assert failures and "too many docker" in failures[0].error
 
+    async def test_a_long_update_leaves_room_to_read(self, operator, agent):
+        """An update holds its slot for minutes. Counted with the reads, two
+        updates would leave the page unable to draw the very stacks it is
+        updating; counted apart, the slow ones are bounded on their own."""
+        await enrol(operator, agent, caps=["docker"])
+        agent.app._docker_calls[operator.id.raw] = (fleet.MAX_DOCKER_SLOW, fleet.MAX_DOCKER_SLOW)
+        ran = []
+
+        async def overview(self, src, rid, document):
+            ran.append("overview")
+            return {"info": {}}
+        original = fleet._DOCKER_OPS["overview"]
+        fleet._DOCKER_OPS["overview"] = overview
+        try:
+            await operator.app.request_docker(agent.id, "stack", stack="site", action="update")
+            await operator.app.request_docker(agent.id, "overview")
+            await deliver(operator, agent)
+            await asyncio.sleep(0)
+            await deliver(agent, operator)
+        finally:
+            fleet._DOCKER_OPS["overview"] = original
+        failures = [e for e in operator.drain_events() if isinstance(e, Failure)]
+        assert len(failures) == 1 and "too many docker" in failures[0].error
+        assert ran == ["overview"]
+
+    def test_which_operations_are_slow(self):
+        assert fleet._docker_slow("stack", {"action": "update"})
+        assert not fleet._docker_slow("stack", {"action": "stop"})
+        for op in ("pull", "check", "recreate", "deploy_stack", "deploy"):
+            assert fleet._docker_slow(op, {})
+        for op in ("stacks", "containers", "overview", "logs", "stack_file"):
+            assert not fleet._docker_slow(op, {})
+
+
+class _FakePortainer:
+    def __init__(self, listed):
+        self.listed = listed
+        self.done = []
+
+    async def stacks(self):
+        return self.listed
+
+    async def redeploy(self, record):
+        self.done.append(("redeploy", record["id"]))
+        return {"stack": record["name"], "through": "portainer"}
+
+    async def act(self, stack_id, endpoint_id, action):
+        self.done.append((action, stack_id))
+        return {"stack": stack_id, "action": action}
+
+
+class TestOneStackTheWayItsOwnerWould:
+    """Which act a stack gets. Portainer when it owns the stack and is
+    connected; compose with a readable file; the Engine otherwise — and never
+    two of them at once on one stack."""
+
+    def _stack(self, owner="portainer", **extra):
+        stack = {"name": "blog", "owner": owner, "portainer_id": 7 if owner == "portainer" else 0,
+                 "files": ["/data/compose/7/docker-compose.yml"], "workdir": "/data/compose/7",
+                 "files_ok": False, "services": [{"service": "web", "container": "c1"}]}
+        stack.update(extra)
+        return stack
+
+    def _wire(self, agent, monkeypatch, stack, portainer=None, compose=None):
+        from src.apps import fleet_stacks
+        calls = []
+
+        async def record(name):
+            return stack
+
+        async def engine_update(found, *, say=None):
+            calls.append(("engine", found["name"]))
+            return {"stack": found["name"], "through": "engine", "steps": [],
+                    "updated": 0, "failed": 0}
+
+        async def compose_update(found, *, say=None):
+            calls.append(("compose", found["name"]))
+            return {"stack": found["name"], "through": "compose"}
+
+        async def compose_probe():
+            return compose, "2.29.1" if compose else ""
+
+        monkeypatch.setattr(agent.app, "_stack_record", record)
+        monkeypatch.setattr(agent.app, "_portainer", lambda: portainer)
+        monkeypatch.setattr(fleet_stacks, "engine_update", engine_update)
+        monkeypatch.setattr(fleet_stacks, "compose_update", compose_update)
+        monkeypatch.setattr(fleet_stacks, "compose", compose_probe)
+        return calls
+
+    async def test_a_portainer_stack_goes_through_a_connected_portainer(self, agent,
+                                                                       monkeypatch):
+        portainer = _FakePortainer([{"id": 7, "name": "blog", "endpoint": 1,
+                                     "active": True, "git": False}])
+        calls = self._wire(agent, monkeypatch, self._stack(), portainer)
+        done = await agent.app._stack_do("blog", "update")
+        assert done["through"] == "portainer" and portainer.done == [("redeploy", 7)]
+        assert calls == []
+
+    async def test_and_through_the_engine_when_none_is_connected(self, agent, monkeypatch):
+        calls = self._wire(agent, monkeypatch, self._stack(), None, ["docker", "compose"])
+        done = await agent.app._stack_do("blog", "update")
+        assert done["through"] == "engine" and calls == [("engine", "blog")]
+
+    async def test_a_portainer_that_does_not_know_the_stack_is_not_asked(self, agent,
+                                                                        monkeypatch):
+        portainer = _FakePortainer([{"id": 8, "name": "blog", "endpoint": 1,
+                                     "active": True, "git": False}])
+        calls = self._wire(agent, monkeypatch, self._stack(), portainer)
+        await agent.app._stack_do("blog", "update")
+        assert portainer.done == [] and calls == [("engine", "blog")]
+
+    async def test_a_compose_stack_goes_through_compose(self, agent, monkeypatch):
+        stack = self._stack("compose", files=["/srv/blog/docker-compose.yml"],
+                            workdir="/srv/blog", files_ok=True)
+        calls = self._wire(agent, monkeypatch, stack, None, ["docker", "compose"])
+        await agent.app._stack_do("blog", "update")
+        assert calls == [("compose", "blog")]
+
+    async def test_a_failed_service_fails_the_update(self, agent, monkeypatch):
+        from src.apps import fleet_stacks
+        self._wire(agent, monkeypatch, self._stack(), None, None)
+
+        async def engine_update(found, *, say=None):
+            return {"stack": "blog", "through": "engine", "updated": 0, "failed": 1,
+                    "steps": [{"service": "web", "result": "failed", "why": "no space left"}]}
+        monkeypatch.setattr(fleet_stacks, "engine_update", engine_update)
+        with pytest.raises(fleet_docker.DockerError) as failure:
+            await agent.app._stack_do("blog", "update")
+        assert "web: no space left" in str(failure.value)
+
+    async def test_portainer_has_no_restart_so_it_is_a_stop_and_a_start(self, agent,
+                                                                        monkeypatch):
+        portainer = _FakePortainer([{"id": 7, "name": "blog", "endpoint": 1,
+                                     "active": True, "git": False}])
+        self._wire(agent, monkeypatch, self._stack(), portainer)
+        await agent.app._stack_do("blog", "restart")
+        assert portainer.done == [("stop", 7), ("start", 7)]
+
+    async def test_a_portainer_stack_is_not_taken_down_behind_its_back(self, agent,
+                                                                       monkeypatch):
+        portainer = _FakePortainer([{"id": 7, "name": "blog", "endpoint": 1,
+                                     "active": True, "git": False}])
+        self._wire(agent, monkeypatch, self._stack(), portainer)
+        with pytest.raises(fleet_docker.DockerError):
+            await agent.app._stack_do("blog", "down")
+        assert portainer.done == []
+
+    async def test_one_stack_is_changed_by_one_act_at_a_time(self, agent, monkeypatch):
+        from src.apps import fleet_stacks
+        self._wire(agent, monkeypatch, self._stack(), None, None)
+        gate = asyncio.Event()
+
+        async def slow(found, *, say=None):
+            await gate.wait()
+            return {"stack": "blog", "through": "engine", "steps": [], "updated": 0, "failed": 0}
+        monkeypatch.setattr(fleet_stacks, "engine_update", slow)
+        first = asyncio.ensure_future(agent.app._stack_do("blog", "update"))
+        await asyncio.sleep(0)
+        with pytest.raises(fleet_docker.DockerError) as failure:
+            await agent.app._stack_do("blog", "update")
+        assert "already being changed" in str(failure.value)
+        gate.set()
+        await first
+        assert agent.app._stacks_busy == set()
+
+    async def test_no_docker_is_an_answer_not_a_failure(self, agent, monkeypatch):
+        monkeypatch.setattr(fleet_docker, "available", lambda: False)
+        answer = await agent.app._docker_overview(None, "", {})
+        assert answer["info"]["engine"] is False and answer["info"]["problem"]
+
+    async def test_the_token_never_comes_back(self, agent):
+        agent.app._store_portainer({"url": "https://127.0.0.1:9443", "token": "ptr_secret",
+                                    "fingerprint": "ab" * 32})
+        state = agent.app._portainer_state()
+        assert state == {"connected": True, "url": "https://127.0.0.1:9443", "pinned": True}
+        assert "ptr_secret" not in repr(state)
+
 
 class TestUpdateWithStacks:
     """`Update` brings a machine up to date — its packages, and the stacks this
diff --git a/tests/test_fleet_docker.py b/tests/test_fleet_docker.py
index b278198..6c703f8 100644
--- a/tests/test_fleet_docker.py
+++ b/tests/test_fleet_docker.py
@@ -1,5 +1,7 @@
 """
-Docker and Portainer: the validation in front of them, and the wire under them.
+Docker's Engine API and Portainer's: the validation in front of them, and the
+wire under them. What is built on them — stacks, updates — is
+`test_fleet_stacks.py`'s.
 
 Two things are worth proving here and they are not the same thing.
 
@@ -104,35 +106,6 @@ def test_environment_names_are_names(self):
         assert body["Env"] == ["GOOD=1"]
 
 
-class TestStacksFromLabels:
-    def test_containers_group_into_the_projects_that_created_them(self):
-        rows = [
-            {"id": "a", "name": "web", "project": "site", "service": "web",
-             "state": "running", "image": "nginx", "portainer": ""},
-            {"id": "b", "name": "db", "project": "site", "service": "db",
-             "state": "exited", "image": "pg", "portainer": ""},
-            {"id": "c", "name": "loose", "project": "", "service": "",
-             "state": "running", "image": "busybox", "portainer": ""},
-            {"id": "d", "name": "pt", "project": "other", "service": "app",
-             "state": "running", "image": "img", "portainer": "other"},
-        ]
-        stacks = docker.group_stacks(rows)
-        assert [stack["name"] for stack in stacks] == ["other", "site"]
-        site = stacks[1]
-        assert site["total"] == 2 and site["running"] == 1
-        assert {entry["service"] for entry in site["services"]} == {"web", "db"}
-        # A container in no project is in no stack — not in one called "".
-        assert all(stack["name"] for stack in stacks)
-        assert stacks[0]["portainer"] == "other"
-
-    def test_a_compose_file_label_that_is_not_a_path_is_dropped(self):
-        assert docker.compose_files("/srv/a.yml,/srv/b.yml") == ["/srv/a.yml", "/srv/b.yml"]
-        assert docker.compose_files("a.yml") == []
-        assert docker.compose_files("/srv/a.yml,relative.yml") == ["/srv/a.yml"]
-        assert docker.compose_files("") == []
-        assert docker.compose_files("/a\nb.yml") == []
-
-
 class TestLogStream:
     def test_the_multiplexed_header_is_taken_off_rather_than_printed(self):
         """A container with no tty has its output framed: eight bytes per frame,
@@ -293,15 +266,166 @@ async def test_an_oversized_answer_is_refused_rather_than_read(self):
 # Portainer
 # ---------------------------------------------------------------------------
 
+class TestEngineClientBeyondTheBasics:
+    async def test_an_engine_that_dropped_our_version_is_asked_again_in_its_own(self):
+        """An engine raises its minimum API version over the years. Asking
+        `/version` before every call would cost a round trip each time for an
+        answer that changes once in a machine's life; the refusal names it."""
+        refusal = json.dumps({"message": "client version 1.41 is too old. "
+                              "Minimum supported API version is 1.44, please "
+                              "upgrade your client to a newer version"}).encode()
+        daemon = _Daemon([_answer(refusal, status=400), _answer(b'{"Version":"30.0"}'),
+                          _answer(b'{"Version":"30.0"}')])
+        await daemon.start()
+        saved = dict(docker._api)
+        try:
+            assert (await docker.call("GET", "/version"))["Version"] == "30.0"
+            await docker.call("GET", "/version")
+            assert [line.split()[1] for line, _ in daemon.requests] == [
+                "/v1.41/version", "/v1.44/version", "/v1.44/version"]
+        finally:
+            docker._api.update(saved)
+            await daemon.stop()
+
+    async def test_a_400_that_is_not_about_the_version_is_not_retried(self):
+        daemon = _Daemon([_answer(b'{"message":"bad parameter"}', status=400)])
+        await daemon.start()
+        try:
+            with pytest.raises(docker.DockerError) as failure:
+                await docker.call("GET", "/version")
+            assert "bad parameter" in str(failure.value)
+            assert len(daemon.requests) == 1
+        finally:
+            await daemon.stop()
+
+    def test_a_reference_splits_where_docker_splits_it(self):
+        """A registry's port is a colon too: `127.0.0.1:5000/app` has no tag."""
+        assert docker.split_reference("127.0.0.1:5000/app") == ("127.0.0.1:5000/app", "", "")
+        assert docker.split_reference("127.0.0.1:5000/app:2") == ("127.0.0.1:5000/app", "2", "")
+        assert docker.split_reference("nginx") == ("nginx", "", "")
+        assert docker.split_reference("img:1@sha256:ab") == ("img", "1", "sha256:ab")
+        assert docker.registry_of("nginx") == "docker.io"
+        assert docker.registry_of("owner/app") == "docker.io"
+        assert docker.registry_of("ghcr.io/owner/app") == "ghcr.io"
+        assert docker.registry_of("localhost/app") == "localhost"
+
+    async def test_a_pull_by_digest_asks_for_the_digest(self):
+        digest = "sha256:" + "a" * 64
+        daemon = _Daemon([_answer(b'{"status":"done"}'), _answer(b'{"status":"done"}')])
+        await daemon.start()
+        try:
+            assert (await docker.pull("img@" + digest))["image"] == "img@" + digest
+            assert (await docker.pull("127.0.0.1:5000/app"))["image"] == "127.0.0.1:5000/app:latest"
+            first, second = (line for line, _ in daemon.requests)
+            assert "fromImage=img%40sha256%3A" in first and "tag=" not in first
+            assert "fromImage=127.0.0.1%3A5000%2Fapp&tag=latest" in second
+        finally:
+            await daemon.stop()
+
+    def test_a_private_registry_is_reached_with_this_accounts_own_login(self, tmp_path,
+                                                                         monkeypatch):
+        import base64
+        token = base64.b64encode(b"deploy:hunter2").decode()
+        (tmp_path / "config.json").write_text(json.dumps(
+            {"auths": {"ghcr.io": {"auth": token}}}))
+        monkeypatch.setenv("DOCKER_CONFIG", str(tmp_path))
+        header = docker._auth_header("ghcr.io/owner/app:1")
+        assert header and header[0].startswith("X-Registry-Auth: ")
+        blob = json.loads(base64.urlsafe_b64decode(header[0].split(": ", 1)[1]))
+        assert blob == {"username": "deploy", "password": "hunter2", "serveraddress": "ghcr.io"}
+        # Another registry is not handed this one's password.
+        assert docker._auth_header("docker.io/library/nginx") == ()
+        assert docker._auth_header("quay.io/x/y") == ()
+
+    def test_a_login_file_that_is_not_one_costs_nothing(self, tmp_path, monkeypatch):
+        monkeypatch.setenv("DOCKER_CONFIG", str(tmp_path))
+        for content in ("", "not json", "[]", '{"auths": 3}',
+                        '{"auths": {"ghcr.io": {"auth": "%%%"}}}'):
+            (tmp_path / "config.json").write_text(content)
+            assert docker._auth_header("ghcr.io/a/b") == ()
+
+    async def test_an_archive_gives_its_files_and_nothing_else(self):
+        """Read through the daemon because a volume's files are root's. A link
+        is never followed, a name that climbs is never a key, and nothing is
+        written anywhere."""
+        import io
+        import tarfile
+        buffer = io.BytesIO()
+        with tarfile.open(fileobj=buffer, mode="w") as archive:
+            def add(name, data=b"", kind=tarfile.REGTYPE, link=""):
+                info = tarfile.TarInfo(name)
+                info.type, info.linkname, info.size = kind, link, len(data)
+                archive.addfile(info, io.BytesIO(data) if data else None)
+            add("1", kind=tarfile.DIRTYPE)
+            add("1/docker-compose.yml", b"services: {}\n")
+            add("1/stack.env", b"A=1\n")
+            add("1/link", kind=tarfile.SYMTYPE, link="/etc/shadow")
+            add("1/../../escape", b"x")
+        daemon = _Daemon([_answer(buffer.getvalue())])
+        await daemon.start()
+        try:
+            held = await docker.read_archive("portainer", "/data/compose/1")
+            assert held == {"docker-compose.yml": b"services: {}\n", "stack.env": b"A=1\n"}
+            assert "path=/data/compose/1" in daemon.requests[0][0]
+        finally:
+            await daemon.stop()
+
+    async def test_an_archive_path_that_is_not_one_never_reaches_the_socket(self):
+        daemon = _Daemon([])
+        await daemon.start()
+        try:
+            for container, path in (("portainer", "relative"), ("portainer", "/a/../etc"),
+                                    ("bad name", "/data"), ("portainer", "/a\0b")):
+                with pytest.raises(docker.DockerError):
+                    await docker.read_archive(container, path)
+            assert daemon.requests == []
+        finally:
+            await daemon.stop()
+
+    async def test_an_archive_that_is_not_one_is_a_dockererror(self):
+        daemon = _Daemon([_answer(b"definitely not a tar file" * 40)])
+        await daemon.start()
+        try:
+            with pytest.raises(docker.DockerError):
+                await docker.read_archive("portainer", "/data")
+        finally:
+            await daemon.stop()
+
+    async def test_the_registry_digest_is_read_and_checked(self):
+        good = "sha256:" + "b" * 64
+        daemon = _Daemon([_answer(json.dumps({"Descriptor": {"digest": good}}).encode()),
+                          _answer(b'{"Descriptor": {"digest": "md5:xyz"}}'),
+                          _answer(b'[]')])
+        await daemon.start()
+        try:
+            assert await docker.remote_digest("nginx:1") == good
+            for _ in range(2):
+                with pytest.raises(docker.DockerError):
+                    await docker.remote_digest("nginx:1")
+            assert daemon.requests[0][0].split()[1].endswith("/distribution/nginx:1/json")
+        finally:
+            await daemon.stop()
+
+
 class TestPortainerConfiguration:
     def test_an_address_is_a_scheme_and_a_host_and_nothing_more(self):
         assert portainer.clean_url("https://portainer.lan:9443") == "https://portainer.lan:9443"
-        assert portainer.clean_url("http://10.0.0.5:9000/") == "http://10.0.0.5:9000"
+        assert portainer.clean_url("https://10.0.0.5:9443/") == "https://10.0.0.5:9443"
         for bad in ("", "portainer.lan", "ftp://x", "https://x/api/stacks",
                     "https://x?a=b", "https://" + "x" * 300, None,
                     "https://a b"):
             assert portainer.clean_url(bad) == ""
 
+    def test_a_token_travels_in_the_clear_only_to_this_machine(self):
+        """Plain HTTP carries the token as written. On loopback there is no
+        wire to read it from; anywhere else there is."""
+        for good in ("http://127.0.0.1:9000", "http://localhost:9000",
+                     "http://[::1]:9000", "http://127.0.0.2:9000"):
+            assert portainer.clean_url(good) == good
+        for bad in ("http://192.0.2.4:9000", "http://portainer.lan:9000",
+                    "http://[2001:db8::1]:9000", "http://localhost.example:9000"):
+            assert portainer.clean_url(bad) == ""
+
     def test_a_token_that_could_be_a_second_header_is_refused(self):
         assert portainer.clean_token("ptr_abc") == "ptr_abc"
         for bad in ("", "a\r\nX-Admin: 1", "a\nb", "tab\there", "é",
@@ -383,27 +507,94 @@ async def test_a_git_stack_is_redeployed_through_git(self):
         redeploys the version it already had."""
         stack = {"id": 3, "endpoint": 1, "name": "site", "git": True,
                  "git_ref": "main"}
-        server, client = await self._client([_answer(b'{"Name":"site"}')])
+        detail = {"Id": 3, "Env": [], "GitConfig": {"URL": "https://git/x",
+                                                    "ReferenceName": "main"}}
+        server, client = await self._client([_answer(json.dumps(detail).encode()),
+                                             _answer(b'{"Name":"site"}')])
         try:
             await client.redeploy(stack)
-            line, _headers, body = server.requests[0]
+            line, _headers, body = server.requests[1]
             assert line.startswith("PUT /api/stacks/3/git/redeploy?endpointId=1")
-            assert json.loads(body)["PullImage"] is True
+            sent = json.loads(body)
+            assert sent["PullImage"] is True and sent["RepositoryReferenceName"] == "main"
+            assert sent["RepositoryAuthentication"] is False
         finally:
             await server.stop()
 
     async def test_a_file_stack_is_redeployed_with_its_file(self):
         server, client = await self._client([
+            _answer(b'{"Id":3,"Env":[]}'),
             _answer(json.dumps({"StackFileContent": "services: {}"}).encode()),
             _answer(b'{"Name":"site"}'),
         ])
         try:
             await client.redeploy({"id": 3, "endpoint": 1, "name": "site",
                                    "git": False})
-            assert server.requests[0][0].startswith("GET /api/stacks/3/file")
-            line, _headers, body = server.requests[1]
+            assert server.requests[0][0].startswith("GET /api/stacks/3 ")
+            assert server.requests[1][0].startswith("GET /api/stacks/3/file")
+            line, _headers, body = server.requests[2]
             assert line.startswith("PUT /api/stacks/3?endpointId=1")
             assert json.loads(body)["StackFileContent"] == "services: {}"
+            assert json.loads(body)["PullImage"] is True
+        finally:
+            await server.stop()
+
+    async def test_a_redeploy_hands_the_stacks_environment_back(self):
+        """Portainer replaces a stack's environment with what a redeploy
+        carries. An empty list there is every variable of the stack erased, and
+        the containers recreated without them."""
+        env = [{"name": "DB_PASSWORD", "value": "s3cret"}, {"name": "TZ", "value": "UTC"}]
+        server, client = await self._client([
+            _answer(json.dumps({"Id": 3, "Env": env}).encode()),
+            _answer(json.dumps({"StackFileContent": "services: {}"}).encode()),
+            _answer(b'{"Name":"site"}'),
+        ])
+        try:
+            done = await client.redeploy({"id": 3, "endpoint": 1, "name": "site"})
+            assert json.loads(server.requests[2][2])["Env"] == env
+            # Handed back to Portainer, and to nobody else.
+            assert "s3cret" not in json.dumps(done)
+        finally:
+            await server.stop()
+
+    async def test_a_git_stack_keeps_the_credentials_portainer_stored(self):
+        """Saying "no authentication" on a redeploy makes Portainer forget the
+        repository's credentials, and the next pull of a private repository
+        fails. Asking for them with an empty password makes it use its own."""
+        detail = {"Id": 3, "Env": [], "GitConfig": {
+            "URL": "https://git/x", "ReferenceName": "refs/heads/main",
+            "Authentication": {"Username": "deploy", "Password": ""}}}
+        server, client = await self._client([_answer(json.dumps(detail).encode()),
+                                             _answer(b'{"Name":"site"}')])
+        try:
+            await client.redeploy({"id": 3, "endpoint": 1, "name": "site", "git": True})
+            sent = json.loads(server.requests[1][2])
+            assert sent["RepositoryAuthentication"] is True
+            assert sent["RepositoryUsername"] == "deploy"
+            assert sent["RepositoryPassword"] == ""
+            assert sent["RepositoryReferenceName"] == "refs/heads/main"
+        finally:
+            await server.stop()
+
+    async def test_a_probe_reports_the_version_and_sends_no_token(self):
+        server = _PortainerServer([_answer(b'{"Version":"2.45.2","InstanceID":"x"}')])
+        await server.start()
+        try:
+            seen = await portainer.probe(f"http://127.0.0.1:{server.port}")
+            assert seen["version"] == "2.45.2" and seen["fingerprint"] == ""
+            line, headers, _body = server.requests[0]
+            assert line.startswith("GET /api/system/status")
+            assert "x-api-key" not in headers
+        finally:
+            await server.stop()
+
+    async def test_a_probe_of_something_else_says_so(self):
+        server = _PortainerServer([_answer(b'hello')])
+        await server.start()
+        try:
+            with pytest.raises(portainer.PortainerError) as failure:
+                await portainer.probe(f"http://127.0.0.1:{server.port}")
+            assert "not as Portainer" in str(failure.value)
         finally:
             await server.stop()
 
@@ -445,30 +636,6 @@ async def test_an_unreachable_portainer_is_a_message_not_a_traceback(self):
         assert "Portainer" in str(failure.value)
 
 
-class TestWhereAComposeFileLands:
-    """A file that decides what runs on a machine belongs with that node's own
-    state — backed up and wiped with it, never in `/tmp` and never at a path
-    somebody sent."""
-
-    def test_it_is_under_the_nodes_data_directory(self):
-        assert docker.stack_dir("/var/lib/nmesh") == "/var/lib/nmesh/stacks"
-
-    def test_and_falls_back_to_the_accounts_own_directory(self):
-        assert docker.stack_dir().endswith("/.nmesh/stacks")
-
-    async def test_a_name_that_could_climb_out_never_reaches_a_path(self, tmp_path):
-        for bad in ("../../etc/cron.d", "/etc/nmesh", "a/b", "", "."):
-            with pytest.raises(docker.DockerError):
-                await docker.deploy_stack(bad, "services: {}", root=str(tmp_path))
-        assert list(tmp_path.iterdir()) == []
-
-    async def test_an_empty_or_oversized_compose_file_is_refused(self, tmp_path):
-        for bad in ("", "   ", "x" * (docker.MAX_COMPOSE + 1), "a\0b"):
-            with pytest.raises(docker.DockerError):
-                await docker.deploy_stack("ok", bad, root=str(tmp_path))
-        assert list(tmp_path.iterdir()) == []
-
-
 def test_a_log_reply_is_cut_to_fit_the_frame_that_carries_it():
     """A reply too long to trim is not a truncated answer — it is a reply nobody
     receives at all, because the far side cannot parse half a JSON document.
@@ -487,6 +654,28 @@ def test_a_log_reply_is_cut_to_fit_the_frame_that_carries_it():
     assert answer["text"] and log.endswith(answer["text"])
 
 
+class TestMatchingAStackToPortainersRecord:
+    def test_by_the_id_its_directory_gave(self):
+        listed = [{"id": 3, "name": "site"}, {"id": 9, "name": "site"}]
+        assert portainer.match({"name": "site", "portainer_id": 9}, listed)["id"] == 9
+        assert portainer.match({"name": "site", "portainer_id": 4}, listed) is None
+
+    def test_by_name_only_when_the_name_is_unambiguous(self):
+        """Two environments may each hold a stack called `site`; updating the
+        other machine's would be updating the wrong machine."""
+        listed = [{"id": 3, "name": "site"}, {"id": 9, "name": "site"},
+                  {"id": 5, "name": "blog"}]
+        assert portainer.match({"name": "blog", "portainer_id": 0}, listed)["id"] == 5
+        assert portainer.match({"name": "site", "portainer_id": 0}, listed) is None
+
+    def test_the_local_environment_is_the_one_on_this_socket(self):
+        endpoints = [{"id": 2, "url": "tcp://192.0.2.9:9001"},
+                     {"id": 7, "url": "unix:///var/run/docker.sock"}]
+        assert portainer.local_endpoint(endpoints) == 7
+        assert portainer.local_endpoint(endpoints[:1]) == 2
+        assert portainer.local_endpoint([]) == 0
+
+
 class TestReachingTheSocket:
     async def test_a_socket_this_account_may_not_open_says_which(self, tmp_path):
         """"No docker here" and "docker is here and this account cannot reach
@@ -586,3 +775,44 @@ async def test_an_image_is_removed_by_the_name_it_was_given(self):
             assert daemon.requests[0][0].startswith("DELETE /v1.41/images/ghcr.io/o/i:1 ")
         finally:
             await daemon.stop()
+
+
+class TestAPortainerDeploymentInTheBackground:
+    async def _client(self, answers):
+        server = _PortainerServer(answers)
+        await server.start()
+        return server, portainer.Portainer(f"http://127.0.0.1:{server.port}", "ptr_tok")
+
+    async def test_an_update_is_done_when_portainer_says_so(self, monkeypatch):
+        """Recent Portainer answers a redeploy with the stack still deploying
+        and recreates its containers afterwards."""
+        monkeypatch.setattr(portainer, "SETTLE_POLL", 0.01)
+        server, client = await self._client([
+            _answer(b'{"Id":3,"Env":[]}'),
+            _answer(json.dumps({"StackFileContent": "services: {}"}).encode()),
+            _answer(b'{"Name":"site","Status":3}'),
+            _answer(b'{"Status":3}'),
+            _answer(b'{"Status":1,"DeploymentStatus":[{"Status":3},{"Status":1}]}'),
+        ])
+        try:
+            await client.redeploy({"id": 3, "endpoint": 1, "name": "site"})
+            assert len(server.requests) == 5
+        finally:
+            await server.stop()
+
+    async def test_a_deployment_that_failed_is_a_failure(self, monkeypatch):
+        monkeypatch.setattr(portainer, "SETTLE_POLL", 0.01)
+        failed = {"Status": 2, "DeploymentStatus": [
+            {"Status": 3}, {"Status": 4, "Message": "pull access denied"}]}
+        server, client = await self._client([
+            _answer(b'{"Id":3,"Env":[]}'),
+            _answer(json.dumps({"StackFileContent": "services: {}"}).encode()),
+            _answer(b'{"Name":"site","Status":3}'),
+            _answer(json.dumps(failed).encode()),
+        ])
+        try:
+            with pytest.raises(portainer.PortainerError) as failure:
+                await client.redeploy({"id": 3, "endpoint": 1, "name": "site"})
+            assert "pull access denied" in str(failure.value)
+        finally:
+            await server.stop()
diff --git a/tests/test_fleet_stacks.py b/tests/test_fleet_stacks.py
new file mode 100644
index 0000000..cc79f31
--- /dev/null
+++ b/tests/test_fleet_stacks.py
@@ -0,0 +1,808 @@
+"""
+Stacks: finding them on a machine, telling who owns each one, and bringing it
+up to date without undoing whoever did.
+
+Three things are worth proving and they are different things.
+
+**Discovery reads the machine, not a database.** Compose labels what it
+creates; Portainer labels nothing of its own and is known by the directory it
+deploys from. A stack is found, or missed, from those labels alone.
+
+**An update keeps what the container was given.** Recreating through the
+Engine is the fallback for anything without a readable file or a connected
+Portainer, and the one thing it must not do is lose a setting, a volume or a
+network alias on the way — or leave a service down when the new image will not
+start.
+
+**What reaches an argv or a socket is checked.** Names from the operator,
+paths from labels: each is refused, never escaped.
+
+No real daemon here: the Engine calls are replaced by a fake that records them,
+the way `test_fleet_docker.py` drives the client itself against a fake socket.
+"""
+import os
+
+import pytest
+
+from src.apps import fleet_docker as docker
+from src.apps import fleet_stacks as stacks
+
+
+def _entry(ident, project="", service="", *, state="running", image="img:1",
+           workdir="", files="", oneoff=False, extra=None, image_id=""):
+    labels = {}
+    if project:
+        labels[stacks.LABEL_PROJECT] = project
+        labels[stacks.LABEL_SERVICE] = service
+        labels[stacks.LABEL_WORKDIR] = workdir
+        labels[stacks.LABEL_CONFIG] = files
+        if oneoff:
+            labels[stacks.LABEL_ONEOFF] = "True"
+    labels.update(extra or {})
+    return {"Id": ident, "Names": ["/" + (service or ident)], "Image": image,
+            "ImageID": image_id or "sha256:" + ident, "State": state,
+            "Status": "Up", "Labels": labels}
+
+
+# ---------------------------------------------------------------------------
+# Discovery
+# ---------------------------------------------------------------------------
+
+class TestFindingStacks:
+    def test_containers_group_into_the_projects_that_created_them(self):
+        found = stacks.discover([
+            _entry("a", "site", "web", workdir="/srv/site", files="/srv/site/compose.yml"),
+            _entry("b", "site", "db", state="exited"),
+            _entry("c"),                                     # in no project
+            _entry("d", "other", "app"),
+        ])
+        assert [stack["name"] for stack in found] == ["other", "site"]
+        site = found[1]
+        assert site["total"] == 2 and site["running"] == 1
+        assert [row["service"] for row in site["services"]] == ["db", "web"]
+        assert site["workdir"] == "/srv/site"
+        assert site["files"] == ["/srv/site/compose.yml"]
+
+    def test_a_one_off_container_is_not_a_service(self):
+        """`compose run` leaves containers labelled with the project; they are
+        not part of what the stack runs, and recreating one would run its
+        command again."""
+        found = stacks.discover([_entry("a", "site", "web"),
+                                 _entry("b", "site", "web", oneoff=True)])
+        assert found[0]["total"] == 1
+
+    def test_a_project_name_that_is_not_a_name_is_no_stack(self):
+        assert stacks.discover([_entry("a", "../etc", "x"), _entry("b", "a b", "y")]) == []
+
+    def test_portainer_is_known_by_the_directory_it_deploys_from(self):
+        """Portainer writes no label of its own on a stack's containers —
+        measured on Portainer 2.45 — so the name of a label is not how it is
+        recognised. Its directory is: a path inside its own container."""
+        found = stacks.discover([
+            _entry("a", "blog", "web", workdir="/data/compose/12",
+                   files="/data/compose/12/docker-compose.yml"),
+            _entry("b", "plain", "web", workdir="/srv/plain",
+                   files="/srv/plain/docker-compose.yml"),
+            _entry("c", "nested", "web", workdir="/opt/data/compose/3",
+                   files="/opt/data/compose/3/docker-compose.yml"),
+        ])
+        by_name = {stack["name"]: stack for stack in found}
+        assert by_name["blog"]["owner"] == "portainer"
+        assert by_name["blog"]["portainer_id"] == 12
+        assert by_name["blog"]["files_ok"] is False      # Portainer's, not ours to read
+        assert by_name["plain"]["owner"] == "compose"
+        assert by_name["nested"]["owner"] == "compose"
+
+    def test_a_file_counts_as_usable_only_when_this_node_can_read_it(self, tmp_path):
+        readable = tmp_path / "compose.yml"
+        readable.write_text("services: {}\n")
+        found = stacks.discover([
+            _entry("a", "here", "web", workdir=str(tmp_path), files=str(readable)),
+            _entry("b", "gone", "web", workdir="/nonexistent",
+                   files="/nonexistent/compose.yml"),
+            _entry("c", "none", "web"),
+        ])
+        by_name = {stack["name"]: stack for stack in found}
+        assert by_name["here"]["files_ok"] is True
+        assert by_name["gone"]["files_ok"] is False
+        assert by_name["none"]["files_ok"] is False
+
+    def test_a_stack_this_node_deployed_says_so(self, tmp_path):
+        ours = os.path.join(stacks.stack_dir(str(tmp_path)), "app")
+        found = stacks.discover([_entry("a", "app", "web", workdir=ours)], root=str(tmp_path))
+        assert found[0]["nmesh"] is True
+        beside = stacks.stack_dir(str(tmp_path)) + "-other/app"
+        found = stacks.discover([_entry("a", "app", "web", workdir=beside)],
+                                root=str(tmp_path))
+        assert found[0]["nmesh"] is False
+
+    def test_the_count_of_stacks_and_services_is_bounded(self):
+        entries = [_entry(f"c{i}", f"p{i}", "s") for i in range(stacks.MAX_STACKS + 20)]
+        entries += [_entry(f"d{i}", "p0", f"s{i}") for i in range(stacks.MAX_SERVICES + 5)]
+        found = stacks.discover(entries)
+        assert len(found) == stacks.MAX_STACKS
+        p0 = next(stack for stack in found if stack["name"] == "p0")
+        assert len(p0["services"]) == stacks.MAX_SERVICES
+        assert p0["total"] == stacks.MAX_SERVICES + 6
+
+    def test_whatever_the_daemon_sends_there_is_a_list(self):
+        hostile = [{}, {"Labels": None}, {"Labels": {stacks.LABEL_PROJECT: 7}},
+                   {"Labels": {stacks.LABEL_PROJECT: "ok"}, "Names": "nope",
+                    "State": ["x"]}, {"Labels": []}]
+        found = stacks.discover(hostile)
+        assert [stack["name"] for stack in found] == ["ok"]
+
+    def test_a_compose_file_label_that_is_not_a_path_is_dropped(self):
+        assert stacks.compose_files("/srv/a.yml,/srv/b.yml") == ["/srv/a.yml", "/srv/b.yml"]
+        assert stacks.compose_files("a.yml") == []
+        assert stacks.compose_files("/srv/a.yml,relative.yml") == ["/srv/a.yml"]
+        assert stacks.compose_files("/srv/../etc/a.yml") == []
+        assert stacks.compose_files("") == []
+        assert stacks.compose_files(None) == []
+        assert stacks.compose_files("/a\nb.yml") == []
+        many = ",".join(f"/f{i}.yml" for i in range(20))
+        assert len(stacks.compose_files(many)) == stacks.MAX_FILES
+
+
+class TestFindingPortainer:
+    def _server(self, **override):
+        entry = {"Id": "p" * 64, "Names": ["/portainer"], "State": "running",
+                 "Image": "portainer/portainer-ce:2.45.2",
+                 "Labels": {"io.portainer.server": "true"},
+                 "Ports": [{"IP": "0.0.0.0", "PrivatePort": 9000, "PublicPort": 9000},
+                           {"IP": "0.0.0.0", "PrivatePort": 9443, "PublicPort": 9443}]}
+        entry.update(override)
+        return entry
+
+    def test_by_its_label_and_where_it_answers(self):
+        found = stacks.find_portainer([_entry("a"), self._server()])
+        assert found["server"]["name"] == "portainer"
+        assert found["server"]["edition"] == "Community"
+        assert found["server"]["tag"] == "2.45.2"
+        # HTTPS before HTTP, and a wildcard bind reached on loopback.
+        assert found["server"]["url"] == "https://127.0.0.1:9443"
+
+    def test_by_its_image_when_the_label_is_missing(self):
+        server = self._server(Labels={}, Image="registry.example/portainer/portainer-ee:2.19")
+        found = stacks.find_portainer([server])
+        assert found["server"]["edition"] == "Business"
+
+    def test_an_address_it_is_bound_to_is_the_one_suggested(self):
+        server = self._server(Ports=[{"IP": "127.0.0.1", "PrivatePort": 9000,
+                                      "PublicPort": 19000}])
+        assert stacks.find_portainer([server])["server"]["url"] == "http://127.0.0.1:19000"
+        server = self._server(Ports=[{"IP": "::1", "PrivatePort": 9443, "PublicPort": 9443}])
+        assert stacks.find_portainer([server])["server"]["url"] == "https://[::1]:9443"
+
+    def test_one_that_publishes_nothing_is_given_no_guessed_address(self):
+        found = stacks.find_portainer([self._server(Ports=[])])
+        assert found["server"]["url"] == ""
+
+    def test_an_agent_is_not_a_server(self):
+        found = stacks.find_portainer([_entry("a", image="portainer/agent:2.45.2")])
+        assert found == {"server": None, "agent": True}
+
+    def test_nothing_is_found_where_there_is_nothing(self):
+        assert stacks.find_portainer([_entry("a"), _entry("b", image="myportainer/x")]) == \
+            {"server": None, "agent": False}
+
+
+class TestHowAStackIsUpdated:
+    def test_through_portainer_when_it_owns_it_and_is_connected(self):
+        stack = {"owner": "portainer", "files": ["/data/compose/1/docker-compose.yml"]}
+        assert stacks.update_method(stack, portainer_api=True,
+                                    compose_argv=["docker", "compose"])["method"] == "portainer"
+
+    def test_never_through_compose_when_portainer_owns_it(self):
+        """A copy of Portainer's directory resolves a relative bind and an
+        `env_file` against the wrong place: the containers would come back
+        configured differently. Without the API, the Engine."""
+        stack = {"owner": "portainer", "files": ["/data/compose/1/docker-compose.yml"],
+                 "files_ok": True}
+        assert stacks.update_method(stack, portainer_api=False,
+                                    compose_argv=["docker", "compose"])["method"] == "engine"
+
+    def test_through_compose_with_a_readable_file(self):
+        stack = {"owner": "compose", "files": ["/srv/a.yml"], "files_ok": True}
+        assert stacks.update_method(stack, portainer_api=False,
+                                    compose_argv=["docker", "compose"])["method"] == "compose"
+
+    @pytest.mark.parametrize("stack,argv", [
+        ({"owner": "compose", "files": [], "files_ok": False}, ["docker", "compose"]),
+        ({"owner": "compose", "files": ["/srv/a.yml"], "files_ok": False}, ["docker", "compose"]),
+        ({"owner": "compose", "files": ["/srv/a.yml"], "files_ok": True}, None),
+    ])
+    def test_through_the_engine_when_compose_cannot(self, stack, argv):
+        chosen = stacks.update_method(stack, portainer_api=False, compose_argv=argv)
+        assert chosen["method"] == "engine" and chosen["why"]
+
+
+# ---------------------------------------------------------------------------
+# Recreating through the Engine
+# ---------------------------------------------------------------------------
+
+OLD_ID = "1" * 64
+
+
+def _raw(**override):
+    raw = {
+        "Id": OLD_ID, "Name": "/demo-web-1", "Image": "sha256:old",
+        "State": {"Running": True},
+        "Config": {
+            "Image": "registry.example/app:latest", "Hostname": OLD_ID[:12],
+            "Env": ["GREETING=hello", "PATH=/usr/bin", "APP_VERSION=1"],
+            "Cmd": ["serve"], "Entrypoint": ["/entry"], "WorkingDir": "/app",
+            "Labels": {"com.docker.compose.project": "demo",
+                       "com.docker.compose.image": "sha256:old",
+                       "org.opencontainers.image.version": "1",
+                       "mine": "kept"},
+            "ExposedPorts": {"80/tcp": {}, "9000/tcp": {}},
+            "Volumes": {"/var/lib/app": {}},
+        },
+        "HostConfig": {"Binds": ["/data/compose/1/data:/data:rw"],
+                       "NetworkMode": "demo_default",
+                       "RestartPolicy": {"Name": "always"}},
+        "Mounts": [
+            {"Type": "bind", "Source": "/data/compose/1/data", "Destination": "/data", "RW": True},
+            {"Type": "volume", "Name": "a" * 64, "Destination": "/var/lib/app", "RW": True},
+        ],
+        "NetworkSettings": {"Networks": {
+            "demo_default": {"Aliases": ["demo-web-1", "web", OLD_ID[:12]],
+                             "IPAddress": "172.18.0.2", "NetworkID": "n1",
+                             "IPAMConfig": {"IPv4Address": "172.18.0.50"}},
+            "backend": {"Aliases": ["web"], "IPAddress": "172.19.0.4"},
+        }},
+    }
+    raw.update(override)
+    return raw
+
+
+IMAGE = {"Config": {"Env": ["PATH=/usr/bin", "APP_VERSION=1"], "Cmd": ["serve"],
+                    "Entrypoint": ["/entry"], "WorkingDir": "/app",
+                    "Labels": {"org.opencontainers.image.version": "1"},
+                    "ExposedPorts": {"80/tcp": {}}, "Volumes": {"/var/lib/app": {}}}}
+
+
+class TestTheReplacementsConfiguration:
+    def test_what_the_old_image_supplied_is_left_to_the_new_one(self):
+        """An image's own environment, command and labels are baked into the
+        container's configuration. Kept, the replacement would run the new
+        image with the old one's defaults — a version variable that lies, a
+        command that no longer exists."""
+        body, _later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert body["Env"] == ["GREETING=hello"]
+        assert "Cmd" not in body and "Entrypoint" not in body and "WorkingDir" not in body
+        assert "org.opencontainers.image.version" not in body["Labels"]
+        assert body["ExposedPorts"] == {"9000/tcp": {}}
+        assert "Volumes" not in body
+
+    def test_what_the_container_was_given_is_kept(self):
+        body, _later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert body["Image"] == "registry.example/app:latest"
+        assert body["Labels"]["mine"] == "kept"
+        assert body["Labels"]["com.docker.compose.project"] == "demo"
+        assert body["HostConfig"]["RestartPolicy"] == {"Name": "always"}
+        assert "/data/compose/1/data:/data:rw" in body["HostConfig"]["Binds"]
+
+    def test_compose_is_told_which_image_it_now_runs(self):
+        """Left at the old id, compose reads the container as diverged from
+        its image and recreates it again on its next `up`."""
+        body, _later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert body["Labels"]["com.docker.compose.image"] == "sha256:new"
+
+    def test_a_given_command_survives_with_a_given_entrypoint(self):
+        """`--entrypoint` clears the image's command: a command equal to the
+        image's next to a custom entrypoint was given, not inherited."""
+        raw = _raw()
+        raw["Config"]["Entrypoint"] = ["/custom"]
+        body, _later = stacks.recreate_body(raw, IMAGE, "sha256:new")
+        assert body["Entrypoint"] == ["/custom"] and body["Cmd"] == ["serve"]
+
+    def test_an_anonymous_volume_is_carried_over_by_name(self):
+        """Without it the replacement starts with a fresh, empty volume where
+        the old one kept its data."""
+        body, _later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert f"{'a' * 64}:/var/lib/app" in body["HostConfig"]["Binds"]
+
+    def test_an_anonymous_volume_compose_declared_gets_its_source_back(self):
+        raw = _raw()
+        raw["HostConfig"]["Mounts"] = [{"Type": "volume", "Target": "/var/lib/app"}]
+        body, _later = stacks.recreate_body(raw, IMAGE, "sha256:new")
+        assert body["HostConfig"]["Mounts"] == [
+            {"Type": "volume", "Target": "/var/lib/app", "Source": "a" * 64}]
+        assert not any(bind.endswith(":/var/lib/app") for bind in body["HostConfig"]["Binds"])
+
+    def test_the_old_id_is_not_carried_as_a_name(self):
+        body, _later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert "Hostname" not in body
+        endpoint = body["NetworkingConfig"]["EndpointsConfig"]["demo_default"]
+        assert endpoint["Aliases"] == ["demo-web-1", "web"]
+        assert endpoint["IPAMConfig"] == {"IPv4Address": "172.18.0.50"}
+        assert "IPAddress" not in endpoint and "NetworkID" not in endpoint
+
+    def test_a_hostname_somebody_chose_is_kept(self):
+        raw = _raw()
+        raw["Config"]["Hostname"] = "web.internal"
+        body, _later = stacks.recreate_body(raw, IMAGE, "sha256:new")
+        assert body["Hostname"] == "web.internal"
+
+    def test_the_other_networks_are_joined_after_creation(self):
+        """Before API 1.44 the Engine attaches one network at create time."""
+        body, later = stacks.recreate_body(_raw(), IMAGE, "sha256:new")
+        assert list(body["NetworkingConfig"]["EndpointsConfig"]) == ["demo_default"]
+        assert later == [("backend", {"Aliases": ["web"]})]
+
+    def test_host_networking_takes_no_endpoint(self):
+        raw = _raw()
+        raw["HostConfig"]["NetworkMode"] = "host"
+        raw["NetworkSettings"] = {"Networks": {"host": {}}}
+        body, later = stacks.recreate_body(raw, IMAGE, "sha256:new")
+        assert "NetworkingConfig" not in body and later == []
+
+    def test_whatever_the_daemon_sent_there_is_a_body(self):
+        for hostile in ({}, {"Config": None, "HostConfig": [], "Mounts": "x",
+                             "NetworkSettings": {"Networks": None}},
+                        {"Config": {"Env": [1, None, "A=1"], "Labels": []}}):
+            body, later = stacks.recreate_body(hostile, {"Config": "nope"}, "sha256:new")
+            assert isinstance(body, dict) and isinstance(later, list)
+
+
+class _Engine:
+    """The Engine calls `fleet_stacks` makes, answered from a script and
+    recorded. Patched over the client's functions, not over a socket: what is
+    under test here is the order and the content of the acts."""
+
+    def __init__(self, raw, *, fresh_id="sha256:new", starts=True, local=None):
+        self.raw = raw
+        self.fresh_id = fresh_id
+        self.starts = starts
+        self.calls = []
+        self.pulled = []
+        self.local = local if local is not None else {}
+
+    def install(self, monkeypatch):
+        monkeypatch.setattr(docker, "container_raw", self.container_raw)
+        monkeypatch.setattr(docker, "image_raw", self.image_raw)
+        monkeypatch.setattr(docker, "pull", self.pull)
+        monkeypatch.setattr(docker, "call", self.call)
+        monkeypatch.setattr(docker, "raw_containers", self.raw_containers)
+        monkeypatch.setattr(stacks, "SETTLE", 0.0)
+        return self
+
+    async def container_raw(self, ident):
+        if ident == "new-id":
+            return {"State": {"Running": self.starts, "ExitCode": 0 if self.starts else 1}}
+        if "Id" in self.raw:
+            return self.raw
+        return self.raw[ident]
+
+    async def image_raw(self, reference):
+        if reference in self.local:
+            return self.local[reference]
+        if reference.startswith("sha256:"):
+            return IMAGE
+        return {"Id": self.fresh_id}
+
+    async def pull(self, reference):
+        self.pulled.append(reference)
+        return {"image": reference}
+
+    async def raw_containers(self, _all=True):
+        return []
+
+    async def call(self, method, path, body=None, **_options):
+        self.calls.append((method, path.split("?")[0], path, body))
+        if path.startswith("/containers/create"):
+            return {"Id": "new-id"}
+        return None
+
+
+class TestRecreating:
+    async def test_a_newer_image_replaces_the_container(self, monkeypatch):
+        engine = _Engine(_raw()).install(monkeypatch)
+        done = await stacks.recreate(OLD_ID)
+        assert done["result"] == "updated" and done["container"] == "new-id"
+        assert engine.pulled == ["registry.example/app:latest"]
+        acts = [(method, short) for method, short, _path, _body in engine.calls]
+        assert acts == [
+            ("POST", f"/containers/{OLD_ID}/stop"),
+            ("POST", f"/containers/{OLD_ID}/rename"),
+            ("POST", "/containers/create"),
+            ("POST", "/networks/backend/connect"),
+            ("POST", "/containers/new-id/start"),
+            ("DELETE", f"/containers/{OLD_ID}"),
+        ]
+        create = engine.calls[2]
+        assert "name=demo-web-1" in create[2]
+        # Volumes outlive the old container: it is removed without them.
+        assert "v=1" not in engine.calls[-1][2]
+
+    async def test_nothing_is_touched_when_the_image_did_not_move(self, monkeypatch):
+        engine = _Engine(_raw(), fresh_id="sha256:old").install(monkeypatch)
+        done = await stacks.recreate(OLD_ID)
+        assert done["result"] == "current" and engine.calls == []
+
+    @pytest.mark.parametrize("reference", ["app@sha256:" + "c" * 64, "sha256:" + "d" * 64])
+    async def test_an_image_named_by_digest_is_left_alone(self, monkeypatch, reference):
+        raw = _raw()
+        raw["Config"]["Image"] = reference
+        engine = _Engine(raw).install(monkeypatch)
+        assert (await stacks.recreate(OLD_ID))["result"] == "pinned"
+        assert engine.calls == [] and engine.pulled == []
+
+    async def test_a_replacement_that_does_not_stay_up_puts_the_old_one_back(self,
+                                                                              monkeypatch):
+        """A bad image costs a few seconds of downtime, not the service."""
+        engine = _Engine(_raw(), starts=False).install(monkeypatch)
+        with pytest.raises(docker.DockerError) as failure:
+            await stacks.recreate(OLD_ID)
+        assert "put back" in str(failure.value)
+        tail = [(method, short) for method, short, _path, _body in engine.calls][-3:]
+        assert tail == [("DELETE", "/containers/new-id"),
+                        ("POST", f"/containers/{OLD_ID}/rename"),
+                        ("POST", f"/containers/{OLD_ID}/start")]
+        assert "name=demo-web-1" in engine.calls[-2][2]
+        assert not any(method == "DELETE" and short == f"/containers/{OLD_ID}"
+                       for method, short, _path, _body in engine.calls)
+
+    async def test_a_container_that_was_stopped_is_not_started(self, monkeypatch):
+        engine = _Engine(_raw(State={"Running": False})).install(monkeypatch)
+        assert (await stacks.recreate(OLD_ID))["result"] == "updated"
+        assert not any(short.endswith("/start") for _m, short, _p, _b in engine.calls)
+
+    async def test_a_container_another_one_shares_a_network_with_is_refused(self,
+                                                                             monkeypatch):
+        """`network_mode: container:web` names the old container; recreating
+        it alone would leave its partner on a network that no longer exists."""
+        engine = _Engine(_raw()).install(monkeypatch)
+        partner = {"Id": "2" * 64, "HostConfig": {"NetworkMode": f"container:{OLD_ID}"}}
+        with pytest.raises(docker.DockerError) as failure:
+            await stacks.recreate(OLD_ID, entries=[partner])
+        assert "network" in str(failure.value)
+        assert not any(short.endswith("/stop") for _m, short, _p, _b in engine.calls)
+
+    async def test_a_name_the_daemon_should_not_have_given_never_reaches_a_path(self,
+                                                                               monkeypatch):
+        engine = _Engine(_raw(Name="/bad name&x=1")).install(monkeypatch)
+        with pytest.raises(docker.DockerError):
+            await stacks.recreate(OLD_ID)
+        assert engine.calls == []
+
+
+class TestUpdatingAStackThroughTheEngine:
+    def _stack(self):
+        return {"name": "demo", "services": [
+            {"service": "web", "container": "w", "name": "demo-web-1"},
+            {"service": "db", "container": "d", "name": "demo-db-1"},
+            {"service": "worker", "container": "k", "name": "demo-worker-1"}]}
+
+    async def test_each_image_is_pulled_once_and_a_failure_costs_only_its_service(
+            self, monkeypatch):
+        web, db, worker = _raw(), _raw(), _raw()
+        web["Name"], db["Name"], worker["Name"] = "/demo-web-1", "/demo-db-1", "/demo-worker-1"
+        db["Config"] = dict(db["Config"], Image="postgres:16")
+        worker["Config"] = dict(worker["Config"], Image="registry.example/broken:1")
+        engine = _Engine({"w": web, "d": db, "k": worker}).install(monkeypatch)
+
+        async def pull(reference):
+            if "broken" in reference:
+                raise docker.DockerError("manifest unknown")
+            engine.pulled.append(reference)
+            return {"image": reference}
+        monkeypatch.setattr(docker, "pull", pull)
+        done = await stacks.engine_update(self._stack())
+        results = {step["service"]: step["result"] for step in done["steps"]}
+        assert results == {"web": "updated", "db": "updated", "worker": "failed"}
+        assert sorted(engine.pulled) == ["postgres:16", "registry.example/app:latest"]
+        assert done["updated"] == 2 and done["failed"] == 1
+
+    def test_dependencies_come_first(self):
+        services = [{"service": "web"}, {"service": "db"}, {"service": "cache"}]
+        entries = [
+            _entry("a", "demo", "web", extra={stacks.LABEL_DEPENDS: "db:service_started:false,cache:service_healthy:true"}),
+            _entry("b", "demo", "db"),
+            _entry("c", "demo", "cache", extra={stacks.LABEL_DEPENDS: "db:service_started:false"}),
+        ]
+        ordered = [row["service"] for row in stacks._order(services, entries)]
+        assert ordered.index("db") < ordered.index("cache") < ordered.index("web")
+
+    def test_a_dependency_cycle_still_ends(self):
+        services = [{"service": "a"}, {"service": "b"}]
+        entries = [_entry("1", "p", "a", extra={stacks.LABEL_DEPENDS: "b:x:y"}),
+                   _entry("2", "p", "b", extra={stacks.LABEL_DEPENDS: "a:x:y"})]
+        assert sorted(row["service"] for row in stacks._order(services, entries)) == ["a", "b"]
+
+
+# ---------------------------------------------------------------------------
+# Is there anything newer?
+# ---------------------------------------------------------------------------
+
+class TestCheckingForUpdates:
+    async def test_each_state_is_told_apart(self, monkeypatch):
+        digest_a, digest_b = "sha256:" + "a" * 64, "sha256:" + "b" * 64
+        local = {
+            "fresh:1": {"Id": "sha256:f", "RepoDigests": [f"fresh@{digest_a}"]},
+            "stale:1": {"Id": "sha256:s", "RepoDigests": [f"stale@{digest_a}"]},
+            "built:1": {"Id": "sha256:l", "RepoDigests": []},
+            "moved:1": {"Id": "sha256:m2", "RepoDigests": [f"moved@{digest_a}"]},
+        }
+
+        async def image_raw(reference):
+            return local.get(reference)
+
+        async def remote_digest(reference):
+            if reference == "private:1":
+                raise docker.DockerError("unauthorized")
+            return digest_b if reference == "stale:1" else digest_a
+
+        monkeypatch.setattr(docker, "image_raw", image_raw)
+        monkeypatch.setattr(docker, "remote_digest", remote_digest)
+        entries = [
+            {"Id": "c1", "Image": "fresh:1", "ImageID": "sha256:f"},
+            {"Id": "c2", "Image": "stale:1", "ImageID": "sha256:s"},
+            {"Id": "c3", "Image": "built:1", "ImageID": "sha256:l"},
+            {"Id": "c4", "Image": "moved:1", "ImageID": "sha256:m1"},
+            {"Id": "c5", "Image": "app@" + digest_a, "ImageID": "sha256:p"},
+            {"Id": "c6", "Image": "private:1", "ImageID": "sha256:x"},
+        ]
+        done = await stacks.check(entries)
+        assert done["containers"] == {"c1": "current", "c2": "outdated", "c3": "local",
+                                      "c4": "pulled", "c5": "pinned", "c6": "unknown"}
+        assert "unauthorized" in done["images"]["private:1"]["why"]
+
+    async def test_a_container_whose_name_moved_is_checked_under_its_name(self, monkeypatch):
+        """The daemon lists a container by image id once its tag points at
+        another image — which is exactly the container that is behind."""
+        async def container_raw(ident):
+            return {"Config": {"Image": "moved:1"}}
+
+        async def image_raw(reference):
+            return {"Id": "sha256:new", "RepoDigests": ["moved@sha256:" + "a" * 64]}
+
+        async def remote_digest(reference):
+            return "sha256:" + "a" * 64
+
+        monkeypatch.setattr(docker, "container_raw", container_raw)
+        monkeypatch.setattr(docker, "image_raw", image_raw)
+        monkeypatch.setattr(docker, "remote_digest", remote_digest)
+        done = await stacks.check([{"Id": "c1", "Image": "sha256:old", "ImageID": "sha256:old"}])
+        assert done["containers"] == {"c1": "pulled"}
+
+    async def test_registries_are_asked_a_bounded_number_of_times(self, monkeypatch):
+        asked = []
+
+        async def image_raw(reference):
+            return {"Id": "sha256:x", "RepoDigests": [reference + "@sha256:" + "a" * 64]}
+
+        async def remote_digest(reference):
+            asked.append(reference)
+            return "sha256:" + "a" * 64
+
+        monkeypatch.setattr(docker, "image_raw", image_raw)
+        monkeypatch.setattr(docker, "remote_digest", remote_digest)
+        entries = [{"Id": f"c{i}", "Image": f"img{i}:1", "ImageID": "sha256:x"}
+                   for i in range(stacks.MAX_CHECKS + 30)]
+        entries += [{"Id": "dup", "Image": "img0:1", "ImageID": "sha256:x"}]
+        await stacks.check(entries)
+        assert len(asked) == stacks.MAX_CHECKS == len(set(asked))
+
+
+# ---------------------------------------------------------------------------
+# Through compose
+# ---------------------------------------------------------------------------
+
+class _Runner:
+    def __init__(self, answers=None):
+        self.argvs = []
+        self.answers = list(answers or [])
+
+    async def __call__(self, argv, *, cwd=None, timeout=None, on_output=None):
+        self.argvs.append(list(argv))
+        return self.answers.pop(0) if self.answers else (0, "ok")
+
+
+class TestCompose:
+    async def test_the_plugin_is_asked_before_the_old_binary(self, monkeypatch):
+        runner = _Runner([(1, "unknown command"), (0, "1.29.2")])
+        monkeypatch.setattr(stacks, "run", runner)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        assert await stacks.compose() == (["docker-compose"], "1.29.2")
+        assert runner.argvs == [["docker", "compose", "version", "--short"],
+                                ["docker-compose", "version", "--short"]]
+        # Believed for a while: asked once, not on every page refresh.
+        assert await stacks.compose() == (["docker-compose"], "1.29.2")
+        assert len(runner.argvs) == 2
+
+    async def test_a_docker_cli_without_the_plugin_is_no_compose(self, monkeypatch):
+        """A `docker` binary on the PATH said nothing about compose: the CLI is
+        very often installed without the plugin."""
+        async def missing(argv, **_options):
+            if argv[0] == "docker-compose":
+                raise docker.DockerError("could not run docker-compose")
+            return 1, "docker: 'compose' is not a docker command."
+        monkeypatch.setattr(stacks, "run", missing)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        assert await stacks.compose() == (None, "")
+
+    async def test_an_update_pulls_then_brings_it_up_with_its_own_files(self, monkeypatch,
+                                                                       tmp_path):
+        runner = _Runner()
+        monkeypatch.setattr(stacks, "run", runner)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        runner.answers = [(0, "2.29.1")]
+        stack = {"name": "site", "files": ["/srv/site/a.yml", "/srv/site/b.yml"],
+                 "workdir": str(tmp_path), "files_ok": True}
+        done = await stacks.compose_update(stack)
+        assert done["through"] == "compose"
+        prefix = ["docker", "compose", "--project-name", "site",
+                  "--project-directory", str(tmp_path),
+                  "-f", "/srv/site/a.yml", "-f", "/srv/site/b.yml"]
+        assert runner.argvs[1:] == [prefix + ["pull", "--ignore-buildable"],
+                                    prefix + ["up", "-d"]]
+        # A service dropped from the file is somebody's data, not a side effect.
+        assert not any("--remove-orphans" in argv for argv in runner.argvs)
+
+    async def test_a_failed_pull_stops_before_anything_is_recreated(self, monkeypatch):
+        runner = _Runner([(0, "2.29.1"), (1, "pull access denied")])
+        monkeypatch.setattr(stacks, "run", runner)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        stack = {"name": "site", "files": ["/srv/a.yml"], "workdir": "", "files_ok": True}
+        with pytest.raises(docker.DockerError) as failure:
+            await stacks.compose_update(stack)
+        assert "pull access denied" in str(failure.value)
+        assert not any("up" in argv for argv in runner.argvs[1:])
+
+    async def test_an_unreadable_file_is_never_handed_to_compose(self, monkeypatch):
+        runner = _Runner([(0, "2.29.1")])
+        monkeypatch.setattr(stacks, "run", runner)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        with pytest.raises(docker.DockerError):
+            await stacks.compose_update({"name": "x", "files": ["/a.yml"], "files_ok": False})
+        assert len(runner.argvs) == 1
+
+    def test_versions_compare_as_numbers(self):
+        assert stacks._at_least("2.15.0", 2, 15) and stacks._at_least("v5.6.0", 2, 15)
+        assert not stacks._at_least("2.9.1", 2, 15) and not stacks._at_least("", 2, 15)
+
+
+class TestRunningAProcess:
+    async def test_output_is_bounded_and_the_exit_code_kept(self):
+        code, text = await stacks.run(["sh", "-c", "yes x | head -c 200000; exit 3"])
+        assert code == 3 and len(text) <= stacks.MAX_OUTPUT
+
+    async def test_a_command_that_outlives_its_bound_is_stopped(self):
+        with pytest.raises(docker.DockerError):
+            await stacks.run(["sleep", "5"], timeout=0.2)
+
+    async def test_a_missing_binary_is_a_dockererror(self):
+        with pytest.raises(docker.DockerError):
+            await stacks.run(["/nonexistent/compose"])
+
+
+# ---------------------------------------------------------------------------
+# Start, stop, the file, a new stack
+# ---------------------------------------------------------------------------
+
+class TestActingWithoutTheFile:
+    async def test_dependencies_start_first_and_stop_last(self, monkeypatch):
+        acted = []
+
+        async def raw_containers(_all=True):
+            return [_entry("w", "demo", "web", extra={stacks.LABEL_DEPENDS: "db:x:y"}),
+                    _entry("d", "demo", "db")]
+
+        async def act(container, action):
+            acted.append((container, action))
+            return {}
+
+        monkeypatch.setattr(docker, "raw_containers", raw_containers)
+        monkeypatch.setattr(docker, "act", act)
+        stack = {"name": "demo", "services": [{"service": "web", "container": "w"},
+                                              {"service": "db", "container": "d"}]}
+        await stacks.engine_act(stack, "start")
+        await stacks.engine_act(stack, "stop")
+        assert acted == [("d", "start"), ("w", "start"), ("w", "stop"), ("d", "stop")]
+
+    async def test_down_needs_the_file(self):
+        with pytest.raises(docker.DockerError):
+            await stacks.engine_act({"name": "demo", "services": []}, "down")
+
+
+class TestReadingAStacksFile:
+    async def test_portainers_compose_file_and_never_its_environment(self, monkeypatch):
+        """Portainer keeps the stack's variables beside its file, in
+        `stack.env`. Those are the stack's secrets; names may travel, values
+        never do."""
+        async def read_archive(container, path):
+            assert (container, path) == ("p" * 64, "/data/compose/12")
+            return {"docker-compose.yml": b"services: {}\n", "stack.env": b"PASSWORD=x\n"}
+        monkeypatch.setattr(docker, "read_archive", read_archive)
+        stack = {"owner": "portainer", "portainer_id": 12,
+                 "files": ["/data/compose/12/docker-compose.yml"]}
+        done = await stacks.read_file(stack, server="p" * 64)
+        assert done == {"path": "/data/compose/12/docker-compose.yml",
+                        "text": "services: {}\n", "source": "portainer"}
+
+    async def test_a_portainer_stack_is_never_read_from_the_hosts_path(self):
+        with pytest.raises(docker.DockerError):
+            await stacks.read_file({"owner": "portainer", "portainer_id": 12,
+                                    "files": ["/data/compose/12/docker-compose.yml"]})
+
+    async def test_a_file_on_disk_is_read_bounded(self, tmp_path):
+        path = tmp_path / "docker-compose.yml"
+        path.write_text("x" * (docker.MAX_COMPOSE + 500))
+        done = await stacks.read_file({"owner": "compose", "files": [str(path)]})
+        assert done["source"] == "disk"
+        assert len(done["text"]) < docker.MAX_COMPOSE + 100
+        assert done["text"].endswith("the file is longer")
+
+    async def test_a_label_naming_something_else_is_not_read(self, tmp_path):
+        """The label is written by whoever created the container. A label
+        pointing at a file that is not compose's is not an invitation."""
+        secret = tmp_path / "shadow"
+        secret.write_text("root:x")
+        for files in ([str(secret)], [str(tmp_path / "gone.yml")], []):
+            with pytest.raises(docker.DockerError):
+                await stacks.read_file({"owner": "compose", "files": files})
+
+
+class TestDeployingAStack:
+    """A file that decides what runs on a machine belongs with that node's own
+    state — backed up and wiped with it, never in `/tmp` and never at a path
+    somebody sent."""
+
+    def test_it_is_under_the_nodes_data_directory(self):
+        assert stacks.stack_dir("/var/lib/nmesh") == "/var/lib/nmesh/stacks"
+
+    def test_and_falls_back_to_the_accounts_own_directory(self):
+        assert stacks.stack_dir().endswith("/.nmesh/stacks")
+
+    async def test_a_name_that_could_climb_out_never_reaches_a_path(self, tmp_path):
+        for bad in ("../../etc/cron.d", "/etc/nmesh", "a/b", "", ".", "UPPER"):
+            with pytest.raises(docker.DockerError):
+                await stacks.deploy(bad, "services: {}", root=str(tmp_path))
+        assert list(tmp_path.iterdir()) == []
+
+    async def test_an_empty_or_oversized_compose_file_is_refused(self, tmp_path):
+        for bad in ("", "   ", "x" * (docker.MAX_COMPOSE + 1), "a\0b", None):
+            with pytest.raises(docker.DockerError):
+                await stacks.deploy("ok", bad, root=str(tmp_path))
+        assert list(tmp_path.iterdir()) == []
+
+    async def test_the_file_is_written_privately_and_brought_up(self, monkeypatch, tmp_path):
+        runner = _Runner([(0, "2.29.1"), (0, "started")])
+        monkeypatch.setattr(stacks, "run", runner)
+        monkeypatch.setitem(stacks._compose, "at", None)
+        done = await stacks.deploy("site", "services: {}\n", root=str(tmp_path))
+        path = os.path.join(stacks.stack_dir(str(tmp_path)), "site", "docker-compose.yml")
+        assert open(path).read() == "services: {}\n"
+        assert os.stat(path).st_mode & 0o077 == 0
+        assert done["stack"] == "site"
+        assert runner.argvs[1][-2:] == ["up", "-d"]
+        assert ["-f", path] == runner.argvs[1][runner.argvs[1].index("-f"):][:2]
+
+
+def test_nothing_here_runs_through_a_shell():
+    """Every process is an argv: a stack name in a command string would be a
+    command of its own."""
+    import inspect
+    source = inspect.getsource(stacks)
+    assert "create_subprocess_shell" not in source
+    assert "shell=True" not in source
+    assert "os.system" not in source
+
+
+
+class TestNamingImages:
+    async def test_an_image_id_is_replaced_by_the_name_it_was_created_from(self, monkeypatch):
+        async def container_raw(ident):
+            if ident == "gone":
+                raise docker.DockerError("no such container")
+            return {"Config": {"Image": "registry.example/app:latest"}}
+        monkeypatch.setattr(docker, "container_raw", container_raw)
+        rows = [{"container": "a", "image": "sha256:" + "1" * 64},
+                {"container": "b", "image": "alpine:3.20"},
+                {"container": "gone", "image": "sha256:" + "2" * 64}]
+        await stacks.name_images(rows)
+        assert [row["image"] for row in rows] == [
+            "registry.example/app:latest", "alpine:3.20", "sha256:" + "2" * 64]