fix: stop proxied copies, the owner migration and the in-app update from failing owners

CLAUDE.md

@@ -41,7 +41,7 @@ python -m zensical build --clean --strict
 python -I .github/scripts/site-check.py site
 ```
 
-Two containers: `cashpilot-ui` (port 8080, web dashboard + earnings collection) and `cashpilot-worker` (port 8081, Docker agent requiring socket access).
+Three long-running containers in `docker-compose.yml`: `cashpilot-ui` (port 8080, web dashboard + earnings collection), `cashpilot-worker` (port 8081, Docker agent) and `cashpilot-socket-proxy` (holds the host's Docker socket). Since v1.71.0 the worker has no Docker socket: it reaches Docker through the proxy's filtered socket (`DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock`), and since v1.73.0 the proxy's owner isolation limits it to containers, volumes and networks that carry the worker's owner label (`cashpilot.deployed-by=worker`); images are not scoped. See `docs/security-defaults.md`. The Unraid templates and the TrueNAS app still mount the raw socket in the worker (`docs/app-stores.md`).
 
 ### Testing traps
 
@@ -114,7 +114,7 @@ CashPilot is split into two **always-separate** components. The UI never touches
 | Component | Description |
 |-----------|-------------|
 | **CashPilot UI** | The single web dashboard. Collects all earnings centrally, shows global fleet view, manages workers. **Has NO Docker socket.** Can be hosted anywhere. Only ONE UI instance exists. |
-| **CashPilot Worker** | Agent running on each server that has Docker. **Must have Docker socket access**. Manages local containers, reports status to UI via heartbeats. Has a minimal config page. |
+| **CashPilot Worker** | Agent running on each server that has Docker. **Reaches Docker through the socket proxy** (`cashpilot-socket-proxy`) in the shipped compose files; the Unraid and TrueNAS listings still mount the raw socket. Manages local containers, reports status to UI via heartbeats. Has a minimal config page. |
 
 Two separate Docker images:
 - **`drumsergio/cashpilot`** — UI image: FastAPI, Jinja2, templates, static assets, collectors, APScheduler. **No Docker SDK.**
@@ -125,7 +125,7 @@ Two separate Docker images:
 ### Core Principles
 
 1. **Separation of concerns.** UI handles: dashboard, earnings collection, scheduling, user auth. Workers handle: Docker container lifecycle, health reporting.
-2. **Workers must be privileged.** A worker without Docker socket is useless.
+2. **Workers need Docker, through the socket proxy.** A worker without Docker access is useless, but in the shipped compose files it never gets the raw socket: the proxy refuses privileged containers and binds of `/` or the socket (unless the operator lists one of them in `CASHPILOT_PROXY_BIND_MOUNTS` or `CASHPILOT_COMPOSE_DIR`, which it takes as given), and limits the worker to containers, volumes and networks with its owner label. The Unraid and TrueNAS listings still give it the raw socket, which is root on the host (`docs/app-stores.md`).
 3. **Single source of truth.** The UI instance is the only one that collects earnings and stores historical data.
 4. **Earnings are never duplicated.** Since only the UI collects, there is no risk of double-counting.
 5. **Workers are stateless satellites.** A worker knows which containers to keep running and the UI URL to report to.
@@ -204,7 +204,7 @@ filtered on `service` silently matched nothing in production. Use
 ## Key Rules
 - Never hardcode credentials; all secrets via environment variables
 - Docker images: `drumsergio/cashpilot` with semver tags, never `:latest`
-- Worker requires Docker socket access (`/var/run/docker.sock`)
+- In the shipped compose files the worker reaches Docker only through the socket proxy (`DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock`) and never mounts `/var/run/docker.sock` (the proxy does, and so does the one-shot `cashpilot-migrate`)
 - Always use bind mounts (not named volumes) on Unraid
 - License: GPL-3.0
 

UPGRADING.md

@@ -93,7 +93,9 @@ ones:
 - every volume of an earner the worker started is copied to a backup named
   `<volume>-pre-isolation`, checked, recreated with the label, filled back from
   the backup and checked again, and the earner is recreated as it was. A
-  Mysterium identity keeps its address. Each backup carries the label
+  Mysterium identity keeps its address, and a backup you confirmed on the
+  Payouts page stays confirmed (if you migrated with 1.73.0 or 1.73.1, the
+  page shows those boxes as changed: confirm them again). Each backup carries the label
   `cashpilot.backup-of=<volume>`, and the log line
   *Backup … holds the data of …* says which earner it belonged to;
 - the worker's own volumes are copied into new ones with `_owned` names

app/constants.py

@@ -40,6 +40,13 @@
 UNDEPLOYABLE_STATUSES = frozenset({"broken", "dead", "dropped"})
 
 
+#: On a volume the owner migration made again: the creation time of the volume
+#: it replaced. The backup fingerprint (orchestrator.state_id) reads it instead of
+#: the new volume's own, so moving the same data onto a labelled volume does not
+#: void a confirmed backup of it.
+LABEL_STATE_CREATED = "cashpilot.state-created"
+
+
 #: deployed-by values only CashPilot itself writes: "worker" by deploy_raw, and
 #: "direct" by the deploy path before workers (removed in #85), whose containers
 #: still run on old installs. "compose" is an exported file the user runs by hand.

app/exchange_rates.py

@@ -35,6 +35,8 @@
     # renamed from ATOR and CoinGecko kept the original slug. Taken from the
     # collector, which has been querying that id for the price all along.
     "ANYONE": "airtor-protocol",
+    # UpRock's token; its payout minimum is set in UPT (services/depin/uprock.yml).
+    "UPT": "uprock",
 }
 
 CACHE_TTL = 900  # 15 minutes

app/isolation_check.py

@@ -413,6 +413,12 @@ def placement_findings(earners: list[Any], network: str, mode: str | None = None
                 "and can reach your LAN and this host. Turn it on with the button on this worker's row",
                 ISOLATION_PAGE,
             )
+        elif mode == lan_fence.MODE_UNSUPPORTED:
+            why, docs = (
+                "This worker runs on Podman, which CashPilot cannot fence yet, so these earners sit on Podman's "
+                "default network and can reach your LAN and this host",
+                ISOLATION_PAGE,
+            )
         elif mode == lan_fence.MODE_OFF:
             why, docs = (
                 "Isolation is turned off on this worker with CASHPILOT_ISOLATION=off, so these earners sit on "

app/lan_fence.py

@@ -42,6 +42,13 @@
   ``off`` deletes the file.
 * ``legacy``: an upgraded install whose earners are on the default bridge.
   Nothing changes until the owner turns isolation on from the fleet page.
+* ``unsupported``: a Podman engine, with the setting not ``off``. Podman's
+  Docker-compatible API keeps the interface name out of the network's options
+  and drops the masquerade and inter-container options, and netavark adds its
+  own masquerade, so none of what the fence relies on can be set or checked
+  there. Earners deploy on the default network and the fleet page says they
+  are not fenced; with CASHPILOT_ISOLATION=on, deploys are refused, since on
+  was asked for and cannot be given.
 
 The top of this module is stdlib only: the helper container imports it to
 render the rules, and the UI image imports its constants.
@@ -57,6 +64,7 @@
 import re
 import threading
 import time
+from datetime import datetime
 from pathlib import Path
 from typing import Any
 
@@ -76,6 +84,12 @@
 MODE_OFF = "off"
 MODE_ON = "on"
 MODE_LEGACY = "legacy"
+MODE_UNSUPPORTED = "unsupported"
+
+PODMAN_UNSUPPORTED = (
+    "CashPilot cannot fence earners off the LAN on Podman yet: its Docker-compatible API does not keep "
+    "the network settings the fence relies on"
+)
 
 #: Seconds between checks that the table is still in place.
 RECHECK_SECONDS = 300
@@ -627,13 +641,35 @@ def _remember(on: bool) -> None:
             )
 
 
+def is_podman(client: Any) -> bool:
+    """A Podman engine behind the Docker-compatible API.
+
+    Podman names itself in /version ("Podman Engine"). An engine that does not
+    answer cannot be told apart, and a Podman taken for Docker gets a network
+    whose bridge is not the one the fence's rules name, so this raises
+    RuntimeError instead of guessing: isolation fails closed.
+    """
+    try:
+        version = client.version() or {}
+    except Exception as exc:  # noqa: BLE001 - re-raised as the documented RuntimeError
+        raise RuntimeError(
+            f"Could not read the container engine's /version ({exc}), so it cannot tell Docker from Podman "
+            "and the LAN fence was not applied."
+        ) from exc
+    names = [str((version.get("Platform") or {}).get("Name") or "")]
+    names += [str((c or {}).get("Name") or "") for c in version.get("Components") or []]
+    return any("podman" in n.lower() for n in names)
+
+
 def resolve_mode(client: Any) -> str:
     if _orchestrator()._CONTAINER_NETWORK:
         return MODE_OPERATOR
     chosen = setting()
     if chosen == "off":
         _remember(False)
         return MODE_OFF
+    if is_podman(client):
+        return MODE_UNSUPPORTED
     if (
         chosen == "on"
         or _remembered_on()
@@ -770,58 +806,108 @@ def _helper_image(client: Any) -> str:
     return f"drumsergio/cashpilot-worker:{version.current()}"
 
 
-def _remove_helper(client: Any) -> None:
-    """Remove a leftover helper by name, only when it carries the helper's label.
+#: How long a helper that was just created or has just exited still belongs to
+#: the process that started it: that process starts it, then reads its logs and
+#: removes it, all within moments. HELPER_POLL is how often a busy helper is
+#: looked at again.
+HELPER_GRACE = 10.0
+HELPER_POLL = 0.25
+
+
+def _stamp_age(stamp: Any, now: float) -> float | None:
+    """Seconds since a Docker timestamp, or None when it is unset or unreadable."""
+    try:
+        when = datetime.fromisoformat(str(stamp))
+    except ValueError:
+        return None
+    if when.tzinfo is None or when.year < 2000:
+        return None
+    return now - when.timestamp()
+
+
+def _helper_busy(attrs: dict[str, Any], now: float) -> bool:
+    """A helper another process is still using: running, or created or exited within HELPER_GRACE."""
+    state = attrs.get("State") or {}
+    if state.get("Running") or state.get("Restarting"):
+        return True
+    status = state.get("Status")
+    if status == "created":
+        stamp = attrs.get("Created")
+    elif status in ("exited", "dead"):
+        stamp = state.get("FinishedAt")
+    else:
+        return False
+    age = _stamp_age(stamp, now)
+    return age is not None and age < HELPER_GRACE
+
+
+def _clear_helper(client: Any) -> None:
+    """Make room for this process's helper.
+
+    Only a container that carries the helper's label is touched: a container
+    someone else named cashpilot-fence is left alone, and the run that follows
+    fails on the name instead of deleting it.
 
-    A container someone else named cashpilot-fence is left alone; the run that
-    follows then fails on the name instead of deleting it.
+    The Desktop app runs the same helper, by the same name and label, so on a
+    host that runs both, a helper of ours can belong to another process that is
+    still waiting for its result. That one is waited for, never killed: killing
+    it would make the other side record a failed fence and refuse its deploy.
+    Only a leftover, one idle past HELPER_GRACE or still there after the helper
+    timeout, is removed.
     """
     from docker.errors import NotFound
 
+    # Per helper: one another process created after the last poll gets its own wait.
+    waiting_on, deadline = None, 0.0
+    while True:
+        try:
+            found = client.containers.get(HELPER_NAME)
+            if (getattr(found, "labels", None) or {}).get(PROBE_LABEL) != "fence":
+                return
+            if getattr(found, "id", None) != waiting_on:
+                waiting_on, deadline = getattr(found, "id", None), time.monotonic() + HELPER_TIMEOUT + HELPER_GRACE
+            if not _helper_busy(found.attrs or {}, time.time()) or time.monotonic() > deadline:
+                found.remove(force=True, v=True)
+                return
+        except NotFound:
+            return
+        except Exception as exc:  # noqa: BLE001 - logged; the run that follows fails on the name
+            logger.warning("Could not remove %s: %s", HELPER_NAME, exc)
+            return
+        time.sleep(HELPER_POLL)
+
+
+def _remove_own(container: Any) -> None:
+    """Remove the helper this process started, by its own ID, never by name."""
+    from docker.errors import NotFound
+
     try:
-        found = client.containers.get(HELPER_NAME)
-        if (getattr(found, "labels", None) or {}).get(PROBE_LABEL) == "fence":
-            found.remove(force=True, v=True)
+        container.remove(force=True, v=True)
     except NotFound:
         pass
-    except Exception as exc:  # noqa: BLE001 - logged; the next run removes it by name first
+    except Exception as exc:  # noqa: BLE001 - logged; the next run clears it as a leftover
         logger.warning("Could not remove %s: %s", HELPER_NAME, exc)
 
 
 def run_helper(client: Any, action: str, subnet: str, filtered: bool = True) -> dict[str, Any]:
     """Run the helper to completion and return what it printed. Removed on every path."""
+    from docker.errors import APIError
+
     with _lock:
-        _remove_helper(client)
+        container = None
         try:
-            container = client.containers.run(
-                image=_helper_image(client),
-                name=HELPER_NAME,
-                entrypoint=["python", "-c", HELPER_SCRIPT],
-                command=[action],
-                environment={
-                    "FENCE_IFACE": BRIDGE_INTERFACE,
-                    "FENCE_SUBNET": subnet,
-                    "FENCE_FILTER": "1" if filtered else "0",
-                    "FENCE_BACKEND": _BACKEND,
-                    "FENCE_EGRESS": _egress["ip"] or "",
-                },
-                network_mode="host",
-                user="0:0",
-                read_only=True,
-                # iptables-legacy takes its lock in /run.
-                tmpfs={"/run": "size=1m"},
-                cap_drop=["ALL"],
-                # NET_RAW only matters to iptables-legacy, which talks to the
-                # kernel over a raw socket. Both run our own script, never an earner.
-                cap_add=["NET_ADMIN", "NET_RAW"],
-                security_opt=["no-new-privileges:true"],
-                privileged=False,
-                pids_limit=64,
-                mem_limit="64m",
-                healthcheck={"test": ["NONE"]},
-                labels={PROBE_LABEL: "fence"},
-                detach=True,
-            )
+            # Twice: another process can create its helper between our clear and our run.
+            for attempt in range(2):
+                _clear_helper(client)
+                try:
+                    container = _create_helper(client, action, subnet, filtered)
+                    break
+                except APIError as exc:
+                    if attempt or getattr(exc, "status_code", None) != 409:
+                        raise
+            # Created and started apart, so a start that fails still leaves an
+            # ID to remove: nothing is ever removed by name after the create.
+            container.start()
             status = container.wait(timeout=HELPER_TIMEOUT)
             text = container.logs(stdout=True, stderr=False).decode(errors="replace").strip()
             if (status or {}).get("StatusCode") != 0:
@@ -831,7 +917,51 @@ def run_helper(client: Any, action: str, subnet: str, filtered: bool = True) ->
                 )
             return json.loads(text.splitlines()[-1])
         finally:
-            _remove_helper(client)
+            if container is not None:
+                _remove_own(container)
+
+
+def _create_helper(client: Any, action: str, subnet: str, filtered: bool) -> Any:
+    """Create the helper container, not started, pulling its image once if it is missing."""
+    from docker.errors import ImageNotFound
+
+    image = _helper_image(client)
+    try:
+        return client.containers.create(image=image, **_helper_spec(action, subnet, filtered))
+    except ImageNotFound:
+        client.images.pull(image)
+        return client.containers.create(image=image, **_helper_spec(action, subnet, filtered))
+
+
+def _helper_spec(action: str, subnet: str, filtered: bool) -> dict[str, Any]:
+    """The helper's container settings: host networking, NET_ADMIN and NET_RAW, every other capability dropped."""
+    return dict(
+        name=HELPER_NAME,
+        entrypoint=["python", "-c", HELPER_SCRIPT],
+        command=[action],
+        environment={
+            "FENCE_IFACE": BRIDGE_INTERFACE,
+            "FENCE_SUBNET": subnet,
+            "FENCE_FILTER": "1" if filtered else "0",
+            "FENCE_BACKEND": _BACKEND,
+            "FENCE_EGRESS": _egress["ip"] or "",
+        },
+        network_mode="host",
+        user="0:0",
+        read_only=True,
+        # iptables-legacy takes its lock in /run.
+        tmpfs={"/run": "size=1m"},
+        cap_drop=["ALL"],
+        # NET_RAW only matters to iptables-legacy, which talks to the
+        # kernel over a raw socket. Both run our own script, never an earner.
+        cap_add=["NET_ADMIN", "NET_RAW"],
+        security_opt=["no-new-privileges:true"],
+        privileged=False,
+        pids_limit=64,
+        mem_limit="64m",
+        healthcheck={"test": ["NONE"]},
+        labels={PROBE_LABEL: "fence"},
+    )
 
 
 def _set_fence(state: str, result: dict[str, Any] | None = None, error: str = "") -> None:
@@ -904,8 +1034,17 @@ def network_for_deploy() -> str | None:
     """
     orch = _orchestrator()
     client = orch._get_client()
-    mode = resolve_mode(client)
+    try:
+        mode = resolve_mode(client)
+    except RuntimeError as exc:
+        raise orch.ContainerNetworkError(f"{exc} The running container was not touched. {_off_hint()}") from None
     _state["mode"] = mode
+    if mode == MODE_UNSUPPORTED and setting() == "on":
+        raise orch.ContainerNetworkError(
+            f"CASHPILOT_ISOLATION=on, but {PODMAN_UNSUPPORTED}. The running container was not touched. "
+            "Set CASHPILOT_ISOLATION=auto or off on this worker to deploy on Podman's default network, "
+            "without the fence (docs/isolation.md)."
+        )
     if mode != MODE_ON:
         return None
     try:
@@ -971,6 +1110,8 @@ def reconcile() -> dict[str, Any]:
         if mode == MODE_ON:
             with contextlib.suppress(RuntimeError, IsolationConflict):
                 apply_fence(client)
+        elif mode == MODE_UNSUPPORTED:
+            _set_fence("unsupported", error=PODMAN_UNSUPPORTED + ".")
         elif _needs_reduced(client, mode, orch._CONTAINER_NETWORK):
             try:
                 reduce_fence(client)
@@ -1045,6 +1186,8 @@ def enable() -> dict[str, Any]:
             "This worker has CASHPILOT_ISOLATION=off. Remove that setting and restart the worker to turn isolation on."
         )
     client = orch._get_client()
+    if is_podman(client):
+        raise IsolationConflict(PODMAN_UNSUPPORTED + ".")
     apply_fence(client)
     _remember(True)
     _state["mode"] = MODE_ON