fix(mysterium): keep the running node when Docker refuses its replacement

UPGRADING.md

@@ -46,19 +46,28 @@ drift, and **Check isolation** fails on it.
    worker refuses to redeploy Mysterium, since on Docker's default bridge the
    other earners could reach its WebUI; the running node is left as it is. On
    your own network (`CASHPILOT_CONTAINER_NETWORK`) it runs under your rules.
-3. Redeploy Mysterium once from the dashboard. It keeps its data volume or
+3. Move the UDP range below 32768 first: set `UDP_PORTS` to a
+   free range below 32768 other than the default, such as `23100:23129`,
+   and change the router forward to match. A value equal to the default
+   (`23000:23029`) reads as a field left as it was, so the range on record
+   would stay; to take the default itself, remove the service from that
+   machine (its data volume stays) and deploy again.
+   The worker publishes the range 1:1 through Docker, and a node's range on
+   record (`56000:56100` through 2.0.6, `56000:56029` in 2.0.7 and 2.1.0) sits in
+   Linux's ephemeral range, 32768-60999, where any program on the host can
+   hold a port for a moment. One taken port fails the whole start. The
+   worker then puts the old node back and names the port, so nothing is
+   lost, but the node does not move. A node that keeps its old range still
+   works while no host program holds one of its ports. 30 ports also save
+   about 175 MB of memory in Docker's port proxy against 101. UPnP no longer
+   works for the node, so forward the range by hand if you relied on it.
+4. Redeploy Mysterium once from the dashboard. It keeps its data volume or
    data folder, so it keeps its identity and its earnings: the worker reads
    the identity from the running node before it removes it, compares it with
    the new one's, and stops the new node if they differ. That needs both
    reads, so back up the keystore first anyway
    ([Backing up node identities](docs/backup.md)) and check the identity after
    (`docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities`).
-4. Nothing changes on the router: the worker publishes the UDP range 1:1. The
-   default range is now `56000:56029`, inside the old `56000:56100`, so the
-   old forward keeps working. A node with `56000:56100` on record keeps it;
-   setting `UDP_PORTS` to `56000:56029` before the redeploy saves about 175 MB
-   of memory in Docker's port proxy. UPnP no longer works for the node, so
-   forward the range by hand if you relied on it.
 
 **If you reach the WebUI through a reverse proxy.** `UI_ADDRESS` is now the
 host addresses port 4449 is published on. A Docker network gateway in it is

app/lan_fence.py

@@ -776,7 +776,8 @@ def reattach(client: Any, net: Any) -> list[str]:
         try:
             net.disconnect(c, force=True)
             net.connect(c)
-            if "network" in error and "not found" in error:
+            # A fence-gated one is started by the guard alone, after all its checks.
+            if "network" in error and "not found" in error and not _is_gated(c):
                 c.start()
             moved.append(c.name)
         except Exception as exc:  # noqa: BLE001 - logged; the next check tries again
@@ -786,6 +787,13 @@ def reattach(client: Any, net: Any) -> list[str]:
     return moved
 
 
+def _is_gated(container: Any) -> bool:
+    from app.constants import LABEL_SERVICE
+
+    slug = str((getattr(container, "labels", None) or {}).get(LABEL_SERVICE) or "")
+    return _orchestrator()._gated_container(slug, container)
+
+
 def _subnet(net: Any) -> str:
     for conf in ((net.attrs or {}).get("IPAM") or {}).get("Config") or []:
         subnet = conf.get("Subnet") or ""
@@ -1158,12 +1166,14 @@ def guard() -> dict[str, str]:
     """
     orch = _orchestrator()
     held: dict[str, str] = {}
+    scanned = False
     try:
         if not orch.docker_available():
             _state["unfenced"] = []
             return {}
         client = orch._get_client()
         _state["unfenced"] = _unfenced(client)
+        scanned = True
         gated = _gated(client)
         if not gated:
             _state["guard"] = {}
@@ -1190,27 +1200,81 @@ def guard() -> dict[str, str]:
                     with contextlib.suppress(RuntimeError, IsolationConflict):
                         apply_fence(client)
             stopped = orch.user_stopped()
+            failed = orch.deploy_failed()
             for c in gated:
                 running = c.status == "running" and c.name not in stopped_now
-                problem = fence or orch.gated_spec_problem(c, mode) or (None if deploying else orch.identity_problem(c))
+                problem = (
+                    fence
+                    or (FAILED_DEPLOY if c.name in failed else None)
+                    or orch.gated_spec_problem(c, mode)
+                    or (None if deploying else orch.identity_problem(c))
+                )
                 if problem:
                     held[c.name] = problem
                     if running:
                         _guard_stop(c, problem)
-                elif not deploying and c.status in ("exited", "created") and c.name not in stopped:
+                elif (
+                    not deploying
+                    and c.status in ("exited", "created")
+                    and c.name not in stopped
+                    # The old node a deploy moved aside: put back by its roll-back, never by the guard.
+                    and not c.name.endswith(orch.REPLACED_SUFFIX)
+                ):
                     c.start()
-                    logger.info("Started %s: the LAN fence is confirmed", c.name)
+                    if problem := _started_unattached(c, mode):
+                        # Up with no address on its network: never started again
+                        # by the guard until a deploy succeeds.
+                        held[c.name] = problem
+                        _guard_stop(c, problem)
+                        if not problem.startswith(UNREAD):
+                            orch._mark_failed(c.name, True)
+                    else:
+                        logger.info("Started %s: the LAN fence is confirmed and it is on its network", c.name)
         finally:
             if not deploying:
                 orch.GATED_DEPLOY_LOCK.release()
     except Exception as exc:  # noqa: BLE001 - the next pass tries again
         logger.warning("Fence guard pass failed: %s", exc)
-        # Not a fresh scan: no Fleet button on a report that may be stale.
-        _state["unfenced"] = []
+        # No Fleet button on a report the scan could not refresh. A later step
+        # failing (another node's start) leaves a fresh report as it is.
+        if not scanned:
+            _state["unfenced"] = []
     _state["guard"] = held
     return held
 
 
+#: Why the guard holds a container a failed deploy left behind.
+FAILED_DEPLOY = "its last deploy failed and left this container behind. Redeploy it from the dashboard"
+
+
+#: A start the guard could not read back: stopped, but tried again next pass.
+UNREAD = "it could not be read back after its start"
+
+
+def _started_unattached(container: Any, mode: str | None) -> str | None:
+    """Why a container just started must stop: not running, or no address on its network.
+
+    Judged in ``on`` and ``operator`` mode, the only ones a fence-gated
+    container runs in (mode_problem).
+    """
+    if mode not in (MODE_ON, MODE_OPERATOR):
+        return None
+    expected = NETWORK_NAME if mode == MODE_ON else _orchestrator()._CONTAINER_NETWORK
+    try:
+        container.reload()
+    except Exception as exc:  # noqa: BLE001 - not confirmed is not attached
+        return f"{UNREAD} ({exc})"
+    attrs = container.attrs if isinstance(getattr(container, "attrs", None), dict) else {}
+    state = attrs.get("State") if isinstance(attrs.get("State"), dict) else {}
+    nets = ((attrs.get("NetworkSettings") or {}).get("Networks") or {}) if isinstance(attrs, dict) else {}
+    address = str(((nets.get(expected) or {}) if isinstance(nets, dict) else {}).get("IPAddress") or "")
+    if state and not state.get("Running"):
+        return "it exited right after its start"
+    if not address:
+        return f"it started with no address on {expected}, so it has no network"
+    return None
+
+
 def _guard_stop(container: Any, reason: str) -> None:
     from app.constants import LABEL_SERVICE