fix(isolation): fence Mysterium like every other earner, so a compromised node cannot rewrite the host firewall

README.md

@@ -17,7 +17,7 @@ CashPilot is a self-hosted platform that runs passive income services on your ow
 ## New in 2.0
 
 - **The worker is boxed in.** It reaches Docker only through a pinned socket proxy, and only the containers, volumes and networks it created, so a compromised worker cannot read the dashboard's data or its encryption key ([security defaults](https://geiserx.github.io/CashPilot/security-defaults/#docker-socket-proxy)).
-- **Earners are fenced off your LAN**, host, tailnet and cloud metadata on new installs; host-networked Mysterium is the exception ([network isolation](https://geiserx.github.io/CashPilot/isolation/)).
+- **Earners are fenced off your LAN**, host, tailnet and cloud metadata on new installs; Mysterium too, in its own network namespace ([network isolation](https://geiserx.github.io/CashPilot/isolation/)).
 - **One box, several IPs.** Proxied copies run an earner through SOCKS5, HTTP CONNECT or a Tailscale exit node, and stop instead of leaking when the proxy dies ([proxied copies](https://geiserx.github.io/CashPilot/proxies/)).
 - Coming from 1.x? Read [What's new in 2.0](https://geiserx.github.io/CashPilot/whats-new-2.0/#upgrading-from-1x) before you change the image tags.
 

UPGRADING.md

@@ -19,6 +19,64 @@ commands, and [Backing up node identities](docs/backup.md) covers the rest.
 
 ---
 
+## After 2.0.6: redeploy Mysterium once to move it off the host network
+
+**Affects you if** you run Mysterium through CashPilot. Through 2.0.6 the node
+ran on the host's network with `NET_ADMIN`, so a compromised node or a
+malicious image update could rewrite the host's firewall, the LAN fence
+included, and reach your LAN, your tailnet and every other container. Later
+releases run it in its own network namespace on `cashpilot-isolated`, behind
+the fence, and the worker starts it only once the fence is in place
+([Mysterium on the fenced bridge](docs/isolation.md#mysterium-on-the-fenced-bridge)).
+
+A running node keeps working exactly as it is until you redeploy it, in any
+isolation mode: the upgraded worker does not stop, start or change it. The
+dashboard lists it under drift ("shares the host's network, where its
+NET_ADMIN reaches the host's firewall. Switch isolation on in Fleet, then
+redeploy Mysterium once"), and **Check isolation** fails on it.
+
+**What to do, in this order.**
+
+1. Upgrade the UI **and the worker that runs Mysterium**. The dashboard refuses
+   to redeploy Mysterium onto an older worker.
+2. Switch isolation on in Fleet: if the Fleet page offers **Turn on
+   isolation**, press it. A worker with `CASHPILOT_ISOLATION=off` needs it set
+   to `auto` and a restart. In `off` or `legacy` mode, and on Podman, the
+   worker refuses to redeploy Mysterium, since on Docker's default bridge the
+   other earners could reach its WebUI; the running node is left as it is. On
+   your own network (`CASHPILOT_CONTAINER_NETWORK`) it runs under your rules.
+3. Redeploy Mysterium once from the dashboard. It keeps its data volume or
+   data folder, so it keeps its identity and its earnings: the worker reads
+   the identity from the running node before it removes it, compares it with
+   the new one's, and stops the new node if they differ. That needs both
+   reads, so back up the keystore first anyway
+   ([Backing up node identities](docs/backup.md)) and check the identity after
+   (`docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities`).
+4. Nothing changes on the router: the worker publishes the UDP range 1:1. The
+   default range is now `56000:56029`, inside the old `56000:56100`, so the
+   old forward keeps working. A node with `56000:56100` on record keeps it;
+   setting `UDP_PORTS` to `56000:56029` before the redeploy saves about 175 MB
+   of memory in Docker's port proxy. UPnP no longer works for the node, so
+   forward the range by hand if you relied on it.
+
+**If you reach the WebUI through a reverse proxy.** `UI_ADDRESS` is now the
+host addresses port 4449 is published on. A Docker network gateway in it is
+published as before. A LAN address is published too, but a host `INPUT` rule
+no longer restricts it, since published ports go through Docker's forwarding:
+use a `DOCKER-USER` rule or keep the proxy on the same machine.
+
+**If the worker runs gVisor by default** (`CASHPILOT_CONTAINER_RUNTIME=runsc`):
+nothing to do. Mysterium's catalog entry requires `runc` and wins over it.
+
+**To go back** to 2.0.6, a redeploy is not enough: 2.0.6 keeps the network on
+record, so the node would stay on the bridge with its WebUI listening only
+inside its own namespace. Remove Mysterium from 2.0.6's dashboard, keeping its
+data volume (the default), then deploy it again: a fresh deploy uses 2.0.6's
+catalog, host network and UDP range included, on the same data volume or
+folder. Back it up first, as for the redeploy.
+
+---
+
 ## 2.0: change the image tags
 
 **Affects everyone on 1.x.** The compose files pin a release series, so
@@ -276,7 +334,7 @@ ranges, and **Check isolation** on the Fleet page flags it.
 
 **What to do.** Upgrade the UI and the worker that runs Mysterium, then redeploy
 Mysterium from the dashboard. The identity stays in its data volume. Details:
-[Mysterium, which uses host networking](docs/isolation-check.md#mysterium-which-uses-host-networking).
+[Mysterium](docs/isolation-check.md#mysterium).
 
 ---
 

app/catalog.py

@@ -74,6 +74,131 @@ def parse_port(mapping: object) -> tuple[str, int | list[str | int]]:
     return key, [address, host]
 
 
+#: Capabilities that reach a network namespace's firewall, routes and
+#: interfaces (SYS_ADMIN reaches far more). Held in the host's namespace they
+#: rewrite the host firewall, the LAN fence included, so a service asking for
+#: one must run in a namespace of its own and on an image named exactly.
+NETNS_CAPABILITIES = frozenset({"NET_ADMIN", "SYS_ADMIN"})
+
+#: The most UDP ports one udp_port_range may publish. Each published port is a
+#: rule and, with Docker's userland proxy, a process on the host.
+MAX_UDP_PORTS = 1000
+
+
+def is_port_range(value: str) -> bool:
+    """A blank value, or "start:end" with 1 <= start <= end <= 65535.
+
+    A pattern can check the shape but not the order, and Mysterium refuses to
+    start on a reversed or out-of-range --udp.ports, so the node stops earning.
+    """
+    if not value:
+        return True
+    start, sep, end = value.partition(":")
+    if not (sep and start.isascii() and start.isdigit() and end.isascii() and end.isdigit()):
+        return False
+    return 1 <= int(start) <= int(end) <= 65535
+
+
+def expand_udp_range(value: str, cap: int = MAX_UDP_PORTS) -> dict[str, int]:
+    """``"56000:56100"`` as Docker port bindings published 1:1: ``{"56000/udp": 56000, ...}``.
+
+    A NAT traversal range only works when each port reaches the same port in
+    the container. Raises ValueError for a value that is not a range, a
+    reversed one, or one wider than ``cap``. A blank value publishes nothing.
+    """
+    value = str(value or "").strip()
+    if not value:
+        return {}
+    if not is_port_range(value):
+        raise ValueError(f"{value!r} is not a port range start:end with start <= end <= 65535")
+    start, _, end = value.partition(":")
+    first, last = int(start), int(end)
+    if last - first + 1 > cap:
+        raise ValueError(f"{value!r} spans {last - first + 1} ports; at most {cap} can be published")
+    return {f"{port}/udp": port for port in range(first, last + 1)}
+
+
+def ui_bindings(value: str, port: int) -> list[list[str | int]]:
+    """``[[address, port], ...]`` for each address in a comma-separated list.
+
+    ``localhost`` becomes 127.0.0.1, since Docker binds addresses, not names.
+    Raises ValueError for anything that is not an IPv4 address.
+    """
+    found: list[list[str | int]] = []
+    for raw in str(value or "").split(","):
+        address = raw.strip()
+        if not address:
+            continue
+        if address == "localhost":
+            address = "127.0.0.1"
+        try:
+            ipaddress.IPv4Address(address)
+        except ValueError:
+            raise ValueError(f"{address!r} is not an IPv4 address") from None
+        if [address, port] not in found:
+            found.append([address, port])
+    return found
+
+
+def fence_gated(svc: dict[str, Any] | None) -> bool:
+    """Whether the worker starts this service only behind a confirmed LAN fence."""
+    return bool(((svc or {}).get("docker") or {}).get("fence_gated") is True)
+
+
+def _validate_network_shape(docker: dict[str, Any], name: str) -> list[str]:
+    """The keys that decide where a service's network namespace lives."""
+    errors: list[str] = []
+    env_vars = {v.get("key"): v for v in docker.get("env") or [] if isinstance(v, dict)}
+    rng = docker.get("udp_port_range")
+    if rng is not None:
+        if not isinstance(rng, dict) or not isinstance(rng.get("env"), str):
+            errors.append(f"{name}: docker.udp_port_range must be a mapping with an env key")
+        else:
+            var = env_vars.get(rng["env"])
+            if not var or var.get("port_range") is not True:
+                errors.append(f"{name}: docker.udp_port_range.env must name a docker.env key with port_range: true")
+            cap = rng.get("max_ports", MAX_UDP_PORTS)
+            if isinstance(cap, bool) or not isinstance(cap, int) or not 1 <= cap <= MAX_UDP_PORTS:
+                errors.append(f"{name}: docker.udp_port_range.max_ports must be 1 to {MAX_UDP_PORTS}")
+            elif var and str(var.get("default") or ""):
+                try:
+                    expand_udp_range(str(var["default"]), cap)
+                except ValueError as exc:
+                    errors.append(f"{name}: docker.udp_port_range default: {exc}")
+    ui = docker.get("ui_port")
+    if ui is not None:
+        port = ui.get("port") if isinstance(ui, dict) else None
+        if (
+            not isinstance(ui, dict)
+            or ui.get("env") not in env_vars
+            or isinstance(port, bool)
+            or not isinstance(port, int)
+            or not 1 <= port <= 65535
+        ):
+            errors.append(f"{name}: docker.ui_port must name a docker.env key and a port 1-65535")
+    if "fence_gated" in docker and not isinstance(docker["fence_gated"], bool):
+        errors.append(f"{name}: docker.fence_gated must be a boolean")
+    runtime = docker.get("runtime")
+    if runtime is not None and (not isinstance(runtime, str) or not runtime.strip()):
+        errors.append(f"{name}: docker.runtime must be a runtime name")
+    caps = {str(c).upper().removeprefix("CAP_") for c in docker.get("cap_add") or []}
+    if held := sorted(caps & NETNS_CAPABILITIES):
+        # The threat class this rule closes: a container with these in the host's
+        # namespace can rewrite the host firewall, the LAN fence included, and
+        # no host rule holds against it. Mysterium ran that way through 2.0.6.
+        if str(docker.get("network_mode") or "").strip().lower() == "host":
+            errors.append(
+                f"{name}: a service with {', '.join(held)} cannot use network_mode: host; it would hold them "
+                "in the host's network namespace"
+            )
+        images = [docker.get("image")] + list((docker.get("image_by_arch") or {}).values())
+        if not all(isinstance(i, str) and re.search(r"@sha256:[0-9a-f]{64}$", i) for i in images if i):
+            errors.append(f"{name}: a service with {', '.join(held)} must pin its image by @sha256: digest")
+        if docker.get("fence_gated") is not True:
+            errors.append(f"{name}: a service with {', '.join(held)} must be fence_gated: true")
+    return errors
+
+
 def _validate(data: dict[str, Any], path: Path) -> list[str]:
     """Return a list of validation errors (empty = OK).
 
@@ -206,6 +331,7 @@ def _validate(data: dict[str, Any], path: Path) -> list[str]:
                         f"(already docker.ports[{seen[key]}])"
                     )
                 seen.setdefault(key, i)
+        errors.extend(_validate_network_shape(docker, path.name))
 
     reqs = data.get("requirements")
     if isinstance(reqs, dict):

app/compose_generator.py

@@ -18,7 +18,7 @@
 import yaml
 
 from app import arch as arch_mod
-from app.catalog import get_service, get_services
+from app.catalog import fence_gated, get_service, get_services
 from app.constants import (
     CONTAINER_PREFIX,
     LABEL_CATEGORY,
@@ -171,9 +171,21 @@ def _service_to_compose(
         compose_svc["environment"] = {key: _escape_value(value) for key, value in env.items()}
 
     # Ports
-    ports = docker_conf.get("ports", [])
+    ports = [str(p) for p in docker_conf.get("ports", [])]
+    # Mysterium's WebUI on each UI_ADDRESS address and its UDP range 1:1, as the
+    # worker publishes them; Compose takes a range in one line.
+    ui = docker_conf.get("ui_port")
+    if isinstance(ui, dict):
+        for address in str(env.get(ui.get("env")) or "127.0.0.1").split(","):
+            address = "127.0.0.1" if address.strip() == "localhost" else address.strip()
+            if address:
+                ports.append(f"{address}:{ui.get('port')}:{ui.get('port')}")
+    rng = docker_conf.get("udp_port_range")
+    if isinstance(rng, dict) and env.get(rng.get("env")):
+        span = str(env[rng["env"]]).replace(":", "-")
+        ports.append(f"{span}:{span}/udp")
     if ports:
-        compose_svc["ports"] = [str(p) for p in ports]
+        compose_svc["ports"] = ports
 
     # Volumes — fill ${VAR} host paths from known values, escape whatever remains.
     #
@@ -251,6 +263,13 @@ def generate_compose_single(
         # something that cannot earn — the export must refuse like the deploy
         # route does, not hand the user a working-looking YAML.
         raise ValueError(f"Service {slug} is no longer available ({svc.get('status')})")
+    if fence_gated(svc):
+        # Compose would start it on Docker's say-so, before any LAN fence is in
+        # place, on a network the other services share.
+        raise ValueError(
+            f"{svc.get('name') or slug} runs only behind a CashPilot worker's LAN fence, so it has no Compose "
+            "export. Deploy it from the dashboard."
+        )
 
     compose_svc = _service_to_compose(svc, env_vars, hostname, arch)
     if not compose_svc:
@@ -279,7 +298,7 @@ def generate_compose_multi(
         svc = get_service(slug)
         if not svc:
             continue
-        if svc.get("status") in UNDEPLOYABLE_STATUSES:
+        if svc.get("status") in UNDEPLOYABLE_STATUSES or fence_gated(svc):
             continue
         compose_svc = _service_to_compose(svc, env_map.get(slug), hostname, arch)
         if compose_svc:

app/drift.py

@@ -17,6 +17,7 @@
 
 from __future__ import annotations
 
+import contextlib
 import hashlib
 import json
 import re
@@ -44,6 +45,10 @@
     "volumes",
     "env",
     "command",
+    "fence_gated",
+    "ui_port",
+    "udp_port_range",
+    "runtime",
 )
 
 
@@ -167,17 +172,54 @@ def container_drift(
         found.append("starts with a different entrypoint than the catalog declares")
 
     # Network mode and ports: the record wins on a redeploy, the catalog otherwise.
+    # A fence-gated service is the exception: its namespace is the catalog's
+    # alone (app/main.py _fence_gated_spec), so a record from when it shared
+    # the host's network does not count.
     rec = recorded or {}
-    network_mode = rec.get("network_mode") if "network_mode" in rec else docker_conf.get("network_mode")
+    gated = catalog.fence_gated(svc)
+    if gated:
+        network_mode = docker_conf.get("network_mode")
+    else:
+        network_mode = rec.get("network_mode") if "network_mode" in rec else docker_conf.get("network_mode")
     want_host = network_mode == "host"
     have_host = shape.get("network_mode") == "host"
     if want_host != have_host:
-        found.append("uses its own network, but should share the host's" if want_host else "shares the host's network")
+        reach = (
+            "its NET_ADMIN reaches the host's firewall"
+            if "NET_ADMIN" in have_caps
+            else "the LAN fence does not cover it"
+        )
+        found.append(
+            "uses its own network, but should share the host's"
+            if want_host
+            else "shares the host's network"
+            + (
+                f", where {reach}. Switch isolation on in Fleet, then redeploy {svc.get('name') or 'it'} once to "
+                "move it onto the fenced bridge"
+                if gated
+                else ""
+            )
+        )
     elif not want_host:
-        recorded_ports = rec.get("ports")
+        recorded_ports = None if gated else rec.get("ports")
         want_ports = set(recorded_ports) if recorded_ports else _catalog_port_keys(docker_conf)
+        if gated and isinstance(docker_conf.get("ui_port"), dict):
+            want_ports.add(f"{docker_conf['ui_port'].get('port')}/tcp")
+        rng = docker_conf.get("udp_port_range")
+        if gated and isinstance(rng, dict):
+            # The range a redeploy publishes: the recorded value, else the catalog default.
+            value = (rec.get("env") or {}).get(rng.get("env")) or next(
+                (v.get("default") for v in docker_conf.get("env") or [] if v.get("key") == rng.get("env")), ""
+            )
+            with contextlib.suppress(ValueError):
+                want_ports |= set(catalog.expand_udp_range(str(value or ""), int(rng.get("max_ports") or 1000)))
         if missing := want_ports - set(shape.get("ports") or []):
             found.append(f"does not publish port {_names(missing)}")
+    if gated and shape.get("restart_policy") not in (None, "no"):
+        found.append(
+            f"Docker starts it on its own (restart policy {shape['restart_policy']}), before the worker has "
+            "confirmed the LAN fence"
+        )
 
     # Mounts: a redeploy keeps the recorded ones and adds the ones the catalog gained.
     want_mounts = _volume_targets(docker_conf.get("volumes")) | _volume_targets(rec.get("volumes"))
@@ -216,6 +258,10 @@ def container_drift(
     # "Outdated" by the image check, so only the tag is judged here.
     want_image = custom_image or arch.image_for(docker_conf, worker_arch) or ""
     have_image = str(shape.get("image") or "")
+    if gated and want_image and have_image and have_image != want_image:
+        # It holds NET_ADMIN, so it runs only the image the catalog names exactly.
+        found.append(f"runs {have_image}; the catalog pins {want_image}")
+        return found
     if want_image.startswith(OWN_IMAGE_PREFIX) and have_image:
         w_repo, w_tag = _split_ref(want_image)
         h_repo, h_tag = _split_ref(have_image)