diff --git a/README.md b/README.md
index 96c2aefc..b8ff4940 100644
--- a/README.md
+++ b/README.md
@@ -17,7 +17,7 @@ CashPilot is a self-hosted platform that runs passive income services on your ow
## New in 2.0
- **The worker is boxed in.** It reaches Docker only through a pinned socket proxy, and only the containers, volumes and networks it created, so a compromised worker cannot read the dashboard's data or its encryption key ([security defaults](https://geiserx.github.io/CashPilot/security-defaults/#docker-socket-proxy)).
-- **Earners are fenced off your LAN**, host, tailnet and cloud metadata on new installs; host-networked Mysterium is the exception ([network isolation](https://geiserx.github.io/CashPilot/isolation/)).
+- **Earners are fenced off your LAN**, host, tailnet and cloud metadata on new installs; Mysterium too, in its own network namespace ([network isolation](https://geiserx.github.io/CashPilot/isolation/)).
- **One box, several IPs.** Proxied copies run an earner through SOCKS5, HTTP CONNECT or a Tailscale exit node, and stop instead of leaking when the proxy dies ([proxied copies](https://geiserx.github.io/CashPilot/proxies/)).
- Coming from 1.x? Read [What's new in 2.0](https://geiserx.github.io/CashPilot/whats-new-2.0/#upgrading-from-1x) before you change the image tags.
diff --git a/UPGRADING.md b/UPGRADING.md
index 3dd47d16..0769805b 100644
--- a/UPGRADING.md
+++ b/UPGRADING.md
@@ -19,6 +19,64 @@ commands, and [Backing up node identities](docs/backup.md) covers the rest.
---
+## After 2.0.6: redeploy Mysterium once to move it off the host network
+
+**Affects you if** you run Mysterium through CashPilot. Through 2.0.6 the node
+ran on the host's network with `NET_ADMIN`, so a compromised node or a
+malicious image update could rewrite the host's firewall, the LAN fence
+included, and reach your LAN, your tailnet and every other container. Later
+releases run it in its own network namespace on `cashpilot-isolated`, behind
+the fence, and the worker starts it only once the fence is in place
+([Mysterium on the fenced bridge](docs/isolation.md#mysterium-on-the-fenced-bridge)).
+
+A running node keeps working exactly as it is until you redeploy it, in any
+isolation mode: the upgraded worker does not stop, start or change it. The
+dashboard lists it under drift ("shares the host's network, where its
+NET_ADMIN reaches the host's firewall. Switch isolation on in Fleet, then
+redeploy Mysterium once"), and **Check isolation** fails on it.
+
+**What to do, in this order.**
+
+1. Upgrade the UI **and the worker that runs Mysterium**. The dashboard refuses
+ to redeploy Mysterium onto an older worker.
+2. Switch isolation on in Fleet: if the Fleet page offers **Turn on
+ isolation**, press it. A worker with `CASHPILOT_ISOLATION=off` needs it set
+ to `auto` and a restart. In `off` or `legacy` mode, and on Podman, the
+ worker refuses to redeploy Mysterium, since on Docker's default bridge the
+ other earners could reach its WebUI; the running node is left as it is. On
+ your own network (`CASHPILOT_CONTAINER_NETWORK`) it runs under your rules.
+3. Redeploy Mysterium once from the dashboard. It keeps its data volume or
+ data folder, so it keeps its identity and its earnings: the worker reads
+ the identity from the running node before it removes it, compares it with
+ the new one's, and stops the new node if they differ. That needs both
+ reads, so back up the keystore first anyway
+ ([Backing up node identities](docs/backup.md)) and check the identity after
+ (`docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities`).
+4. Nothing changes on the router: the worker publishes the UDP range 1:1. The
+ default range is now `56000:56029`, inside the old `56000:56100`, so the
+ old forward keeps working. A node with `56000:56100` on record keeps it;
+ setting `UDP_PORTS` to `56000:56029` before the redeploy saves about 175 MB
+ of memory in Docker's port proxy. UPnP no longer works for the node, so
+ forward the range by hand if you relied on it.
+
+**If you reach the WebUI through a reverse proxy.** `UI_ADDRESS` is now the
+host addresses port 4449 is published on. A Docker network gateway in it is
+published as before. A LAN address is published too, but a host `INPUT` rule
+no longer restricts it, since published ports go through Docker's forwarding:
+use a `DOCKER-USER` rule or keep the proxy on the same machine.
+
+**If the worker runs gVisor by default** (`CASHPILOT_CONTAINER_RUNTIME=runsc`):
+nothing to do. Mysterium's catalog entry requires `runc` and wins over it.
+
+**To go back** to 2.0.6, a redeploy is not enough: 2.0.6 keeps the network on
+record, so the node would stay on the bridge with its WebUI listening only
+inside its own namespace. Remove Mysterium from 2.0.6's dashboard, keeping its
+data volume (the default), then deploy it again: a fresh deploy uses 2.0.6's
+catalog, host network and UDP range included, on the same data volume or
+folder. Back it up first, as for the redeploy.
+
+---
+
## 2.0: change the image tags
**Affects everyone on 1.x.** The compose files pin a release series, so
@@ -276,7 +334,7 @@ ranges, and **Check isolation** on the Fleet page flags it.
**What to do.** Upgrade the UI and the worker that runs Mysterium, then redeploy
Mysterium from the dashboard. The identity stays in its data volume. Details:
-[Mysterium, which uses host networking](docs/isolation-check.md#mysterium-which-uses-host-networking).
+[Mysterium](docs/isolation-check.md#mysterium).
---
diff --git a/app/catalog.py b/app/catalog.py
index 16075441..af540897 100644
--- a/app/catalog.py
+++ b/app/catalog.py
@@ -74,6 +74,131 @@ def parse_port(mapping: object) -> tuple[str, int | list[str | int]]:
return key, [address, host]
+#: Capabilities that reach a network namespace's firewall, routes and
+#: interfaces (SYS_ADMIN reaches far more). Held in the host's namespace they
+#: rewrite the host firewall, the LAN fence included, so a service asking for
+#: one must run in a namespace of its own and on an image named exactly.
+NETNS_CAPABILITIES = frozenset({"NET_ADMIN", "SYS_ADMIN"})
+
+#: The most UDP ports one udp_port_range may publish. Each published port is a
+#: rule and, with Docker's userland proxy, a process on the host.
+MAX_UDP_PORTS = 1000
+
+
+def is_port_range(value: str) -> bool:
+ """A blank value, or "start:end" with 1 <= start <= end <= 65535.
+
+ A pattern can check the shape but not the order, and Mysterium refuses to
+ start on a reversed or out-of-range --udp.ports, so the node stops earning.
+ """
+ if not value:
+ return True
+ start, sep, end = value.partition(":")
+ if not (sep and start.isascii() and start.isdigit() and end.isascii() and end.isdigit()):
+ return False
+ return 1 <= int(start) <= int(end) <= 65535
+
+
+def expand_udp_range(value: str, cap: int = MAX_UDP_PORTS) -> dict[str, int]:
+ """``"56000:56100"`` as Docker port bindings published 1:1: ``{"56000/udp": 56000, ...}``.
+
+ A NAT traversal range only works when each port reaches the same port in
+ the container. Raises ValueError for a value that is not a range, a
+ reversed one, or one wider than ``cap``. A blank value publishes nothing.
+ """
+ value = str(value or "").strip()
+ if not value:
+ return {}
+ if not is_port_range(value):
+ raise ValueError(f"{value!r} is not a port range start:end with start <= end <= 65535")
+ start, _, end = value.partition(":")
+ first, last = int(start), int(end)
+ if last - first + 1 > cap:
+ raise ValueError(f"{value!r} spans {last - first + 1} ports; at most {cap} can be published")
+ return {f"{port}/udp": port for port in range(first, last + 1)}
+
+
+def ui_bindings(value: str, port: int) -> list[list[str | int]]:
+ """``[[address, port], ...]`` for each address in a comma-separated list.
+
+ ``localhost`` becomes 127.0.0.1, since Docker binds addresses, not names.
+ Raises ValueError for anything that is not an IPv4 address.
+ """
+ found: list[list[str | int]] = []
+ for raw in str(value or "").split(","):
+ address = raw.strip()
+ if not address:
+ continue
+ if address == "localhost":
+ address = "127.0.0.1"
+ try:
+ ipaddress.IPv4Address(address)
+ except ValueError:
+ raise ValueError(f"{address!r} is not an IPv4 address") from None
+ if [address, port] not in found:
+ found.append([address, port])
+ return found
+
+
+def fence_gated(svc: dict[str, Any] | None) -> bool:
+ """Whether the worker starts this service only behind a confirmed LAN fence."""
+ return bool(((svc or {}).get("docker") or {}).get("fence_gated") is True)
+
+
+def _validate_network_shape(docker: dict[str, Any], name: str) -> list[str]:
+ """The keys that decide where a service's network namespace lives."""
+ errors: list[str] = []
+ env_vars = {v.get("key"): v for v in docker.get("env") or [] if isinstance(v, dict)}
+ rng = docker.get("udp_port_range")
+ if rng is not None:
+ if not isinstance(rng, dict) or not isinstance(rng.get("env"), str):
+ errors.append(f"{name}: docker.udp_port_range must be a mapping with an env key")
+ else:
+ var = env_vars.get(rng["env"])
+ if not var or var.get("port_range") is not True:
+ errors.append(f"{name}: docker.udp_port_range.env must name a docker.env key with port_range: true")
+ cap = rng.get("max_ports", MAX_UDP_PORTS)
+ if isinstance(cap, bool) or not isinstance(cap, int) or not 1 <= cap <= MAX_UDP_PORTS:
+ errors.append(f"{name}: docker.udp_port_range.max_ports must be 1 to {MAX_UDP_PORTS}")
+ elif var and str(var.get("default") or ""):
+ try:
+ expand_udp_range(str(var["default"]), cap)
+ except ValueError as exc:
+ errors.append(f"{name}: docker.udp_port_range default: {exc}")
+ ui = docker.get("ui_port")
+ if ui is not None:
+ port = ui.get("port") if isinstance(ui, dict) else None
+ if (
+ not isinstance(ui, dict)
+ or ui.get("env") not in env_vars
+ or isinstance(port, bool)
+ or not isinstance(port, int)
+ or not 1 <= port <= 65535
+ ):
+ errors.append(f"{name}: docker.ui_port must name a docker.env key and a port 1-65535")
+ if "fence_gated" in docker and not isinstance(docker["fence_gated"], bool):
+ errors.append(f"{name}: docker.fence_gated must be a boolean")
+ runtime = docker.get("runtime")
+ if runtime is not None and (not isinstance(runtime, str) or not runtime.strip()):
+ errors.append(f"{name}: docker.runtime must be a runtime name")
+ caps = {str(c).upper().removeprefix("CAP_") for c in docker.get("cap_add") or []}
+ if held := sorted(caps & NETNS_CAPABILITIES):
+ # The threat class this rule closes: a container with these in the host's
+ # namespace can rewrite the host firewall, the LAN fence included, and
+ # no host rule holds against it. Mysterium ran that way through 2.0.6.
+ if str(docker.get("network_mode") or "").strip().lower() == "host":
+ errors.append(
+ f"{name}: a service with {', '.join(held)} cannot use network_mode: host; it would hold them "
+ "in the host's network namespace"
+ )
+ images = [docker.get("image")] + list((docker.get("image_by_arch") or {}).values())
+ if not all(isinstance(i, str) and re.search(r"@sha256:[0-9a-f]{64}$", i) for i in images if i):
+ errors.append(f"{name}: a service with {', '.join(held)} must pin its image by @sha256: digest")
+ if docker.get("fence_gated") is not True:
+ errors.append(f"{name}: a service with {', '.join(held)} must be fence_gated: true")
+ return errors
+
+
def _validate(data: dict[str, Any], path: Path) -> list[str]:
"""Return a list of validation errors (empty = OK).
@@ -206,6 +331,7 @@ def _validate(data: dict[str, Any], path: Path) -> list[str]:
f"(already docker.ports[{seen[key]}])"
)
seen.setdefault(key, i)
+ errors.extend(_validate_network_shape(docker, path.name))
reqs = data.get("requirements")
if isinstance(reqs, dict):
diff --git a/app/compose_generator.py b/app/compose_generator.py
index ee12e312..89c720d0 100644
--- a/app/compose_generator.py
+++ b/app/compose_generator.py
@@ -18,7 +18,7 @@
import yaml
from app import arch as arch_mod
-from app.catalog import get_service, get_services
+from app.catalog import fence_gated, get_service, get_services
from app.constants import (
CONTAINER_PREFIX,
LABEL_CATEGORY,
@@ -171,9 +171,21 @@ def _service_to_compose(
compose_svc["environment"] = {key: _escape_value(value) for key, value in env.items()}
# Ports
- ports = docker_conf.get("ports", [])
+ ports = [str(p) for p in docker_conf.get("ports", [])]
+ # Mysterium's WebUI on each UI_ADDRESS address and its UDP range 1:1, as the
+ # worker publishes them; Compose takes a range in one line.
+ ui = docker_conf.get("ui_port")
+ if isinstance(ui, dict):
+ for address in str(env.get(ui.get("env")) or "127.0.0.1").split(","):
+ address = "127.0.0.1" if address.strip() == "localhost" else address.strip()
+ if address:
+ ports.append(f"{address}:{ui.get('port')}:{ui.get('port')}")
+ rng = docker_conf.get("udp_port_range")
+ if isinstance(rng, dict) and env.get(rng.get("env")):
+ span = str(env[rng["env"]]).replace(":", "-")
+ ports.append(f"{span}:{span}/udp")
if ports:
- compose_svc["ports"] = [str(p) for p in ports]
+ compose_svc["ports"] = ports
# Volumes — fill ${VAR} host paths from known values, escape whatever remains.
#
@@ -251,6 +263,13 @@ def generate_compose_single(
# something that cannot earn — the export must refuse like the deploy
# route does, not hand the user a working-looking YAML.
raise ValueError(f"Service {slug} is no longer available ({svc.get('status')})")
+ if fence_gated(svc):
+ # Compose would start it on Docker's say-so, before any LAN fence is in
+ # place, on a network the other services share.
+ raise ValueError(
+ f"{svc.get('name') or slug} runs only behind a CashPilot worker's LAN fence, so it has no Compose "
+ "export. Deploy it from the dashboard."
+ )
compose_svc = _service_to_compose(svc, env_vars, hostname, arch)
if not compose_svc:
@@ -279,7 +298,7 @@ def generate_compose_multi(
svc = get_service(slug)
if not svc:
continue
- if svc.get("status") in UNDEPLOYABLE_STATUSES:
+ if svc.get("status") in UNDEPLOYABLE_STATUSES or fence_gated(svc):
continue
compose_svc = _service_to_compose(svc, env_map.get(slug), hostname, arch)
if compose_svc:
diff --git a/app/drift.py b/app/drift.py
index 8aee78d1..b866233b 100644
--- a/app/drift.py
+++ b/app/drift.py
@@ -17,6 +17,7 @@
from __future__ import annotations
+import contextlib
import hashlib
import json
import re
@@ -44,6 +45,10 @@
"volumes",
"env",
"command",
+ "fence_gated",
+ "ui_port",
+ "udp_port_range",
+ "runtime",
)
@@ -167,17 +172,54 @@ def container_drift(
found.append("starts with a different entrypoint than the catalog declares")
# Network mode and ports: the record wins on a redeploy, the catalog otherwise.
+ # A fence-gated service is the exception: its namespace is the catalog's
+ # alone (app/main.py _fence_gated_spec), so a record from when it shared
+ # the host's network does not count.
rec = recorded or {}
- network_mode = rec.get("network_mode") if "network_mode" in rec else docker_conf.get("network_mode")
+ gated = catalog.fence_gated(svc)
+ if gated:
+ network_mode = docker_conf.get("network_mode")
+ else:
+ network_mode = rec.get("network_mode") if "network_mode" in rec else docker_conf.get("network_mode")
want_host = network_mode == "host"
have_host = shape.get("network_mode") == "host"
if want_host != have_host:
- found.append("uses its own network, but should share the host's" if want_host else "shares the host's network")
+ reach = (
+ "its NET_ADMIN reaches the host's firewall"
+ if "NET_ADMIN" in have_caps
+ else "the LAN fence does not cover it"
+ )
+ found.append(
+ "uses its own network, but should share the host's"
+ if want_host
+ else "shares the host's network"
+ + (
+ f", where {reach}. Switch isolation on in Fleet, then redeploy {svc.get('name') or 'it'} once to "
+ "move it onto the fenced bridge"
+ if gated
+ else ""
+ )
+ )
elif not want_host:
- recorded_ports = rec.get("ports")
+ recorded_ports = None if gated else rec.get("ports")
want_ports = set(recorded_ports) if recorded_ports else _catalog_port_keys(docker_conf)
+ if gated and isinstance(docker_conf.get("ui_port"), dict):
+ want_ports.add(f"{docker_conf['ui_port'].get('port')}/tcp")
+ rng = docker_conf.get("udp_port_range")
+ if gated and isinstance(rng, dict):
+ # The range a redeploy publishes: the recorded value, else the catalog default.
+ value = (rec.get("env") or {}).get(rng.get("env")) or next(
+ (v.get("default") for v in docker_conf.get("env") or [] if v.get("key") == rng.get("env")), ""
+ )
+ with contextlib.suppress(ValueError):
+ want_ports |= set(catalog.expand_udp_range(str(value or ""), int(rng.get("max_ports") or 1000)))
if missing := want_ports - set(shape.get("ports") or []):
found.append(f"does not publish port {_names(missing)}")
+ if gated and shape.get("restart_policy") not in (None, "no"):
+ found.append(
+ f"Docker starts it on its own (restart policy {shape['restart_policy']}), before the worker has "
+ "confirmed the LAN fence"
+ )
# Mounts: a redeploy keeps the recorded ones and adds the ones the catalog gained.
want_mounts = _volume_targets(docker_conf.get("volumes")) | _volume_targets(rec.get("volumes"))
@@ -216,6 +258,10 @@ def container_drift(
# "Outdated" by the image check, so only the tag is judged here.
want_image = custom_image or arch.image_for(docker_conf, worker_arch) or ""
have_image = str(shape.get("image") or "")
+ if gated and want_image and have_image and have_image != want_image:
+ # It holds NET_ADMIN, so it runs only the image the catalog names exactly.
+ found.append(f"runs {have_image}; the catalog pins {want_image}")
+ return found
if want_image.startswith(OWN_IMAGE_PREFIX) and have_image:
w_repo, w_tag = _split_ref(want_image)
h_repo, h_tag = _split_ref(have_image)
diff --git a/app/isolation_check.py b/app/isolation_check.py
index dcf59f65..6cba2cfd 100644
--- a/app/isolation_check.py
+++ b/app/isolation_check.py
@@ -51,7 +51,7 @@
DOCS = "https://geiserx.github.io/CashPilot/"
FIX_PAGE = DOCS + "isolation-check/#2-fix-it-by-platform"
OVERLAY_PAGE = DOCS + "isolation-check/#with-tailscale-or-another-overlay-network-on-the-host"
-NODE_FIREWALL_PAGE = DOCS + "isolation-check/#mysterium-which-uses-host-networking"
+NODE_FIREWALL_PAGE = DOCS + "isolation-check/#mysterium"
BRIDGE_PAGE = DOCS + "home-network-security/#put-the-containers-on-their-own-bridge"
RULES_PAGE = DOCS + "home-network-security/#the-rules"
DNS_PAGE = DOCS + "home-network-security/#name-lookups-need-a-resolver-the-rules-allow"
@@ -69,9 +69,10 @@
#: Reads a node's own nat chain from every iptables backend without creating
#: anything. The chain name and the blackhole address come from the service's
-#: ``docker.node_firewall`` in the catalog. The node runs on host networking as
-#: root, so a legacy lookup of a nat table that does not exist would make the
-#: kernel load ``iptable_nat`` and register an empty legacy table on the host.
+#: ``docker.node_firewall`` in the catalog. The node runs as root with
+#: NET_ADMIN (in its own namespace after 2.0.6, in the host's before), so a
+#: legacy lookup of a nat table that does not exist would make the kernel load
+#: ``iptable_nat`` and register an empty legacy table there.
#: The legacy backend is asked only when the host already has its nat table;
#: the image's plain ``iptables`` only when it is the nf_tables one or that
#: table exists. Each call gets ``timeout 5`` when the image has it, so a hung
@@ -400,6 +401,44 @@ def _name(container: Any) -> str:
return str(getattr(container, "name", "") or "")
+#: Capabilities that reach a network namespace's firewall, routes and interfaces.
+NETNS_CAPABILITIES = frozenset({"NET_ADMIN", "SYS_ADMIN"})
+
+
+def host_netns_findings(containers: list[Any]) -> list[dict[str, Any]]:
+ """Managed containers holding NET_ADMIN or SYS_ADMIN in the host's network namespace.
+
+ Such a container can rewrite the host's firewall, the LAN fence included,
+ add routes and addresses, and bind any port on every host address. No host
+ rule holds against it, so it is an exposure on its own, whatever the probes
+ find. Mysterium ran this way through 2.0.6; a redeploy moves it onto the
+ fenced bridge.
+ """
+ held = []
+ for c in containers:
+ host = ((c.attrs or {}).get("HostConfig") or {}) if isinstance(getattr(c, "attrs", None), dict) else {}
+ caps = {str(cap).upper().removeprefix("CAP_") for cap in host.get("CapAdd") or []}
+ if host.get("Privileged"):
+ caps |= NETNS_CAPABILITIES
+ if _network_mode(c) == "host" and caps & NETNS_CAPABILITIES:
+ held.append(f"{_name(c)} ({', '.join(sorted(caps & NETNS_CAPABILITIES))})")
+ if not held:
+ return []
+ return [
+ {
+ "kind": "net_admin_in_host_netns",
+ "exposure": True,
+ "detail": (
+ "These containers hold firewall capabilities in this host's own network namespace, so they can "
+ "rewrite the host's firewall, the LAN fence included, and reach everything the host reaches: "
+ f"{', '.join(sorted(held))}. Switch isolation on in Fleet, then redeploy them once from the dashboard to "
+ "move them onto the fenced bridge."
+ ),
+ "docs": ISOLATION_PAGE,
+ }
+ ]
+
+
def placement_findings(earners: list[Any], network: str, mode: str | None = None) -> list[dict[str, Any]]:
"""Bridge earners that no rule on the isolated bridge can cover."""
bridge = [c for c in earners if _is_bridge(c)]
@@ -730,7 +769,11 @@ def _run(client: Any) -> dict[str, Any]:
"internet": None,
"not_answering_from_host": [],
"addresses_left_out": [],
- "findings": placement_findings(every, network, mode) + ipv6_findings(client, every, network),
+ "findings": (
+ host_netns_findings(every)
+ + placement_findings(every, network, mode)
+ + ipv6_findings(client, every, network)
+ ),
"mode": mode,
"fence": None,
"node_firewall": None,
diff --git a/app/lan_fence.py b/app/lan_fence.py
index 4b053e6c..b43ecfca 100644
--- a/app/lan_fence.py
+++ b/app/lan_fence.py
@@ -93,6 +93,9 @@
#: Seconds between checks that the table is still in place.
RECHECK_SECONDS = 300
+#: Seconds between the guard's checks while a fence-gated service (one that
+#: holds NET_ADMIN in its own namespace, Mysterium) is deployed on this worker.
+GUARD_SECONDS = 30
#: (rule name, destination, sentinel address the isolation check connects to).
#: A sentinel is an address nobody uses, so the only evidence a connect to it
@@ -524,6 +527,8 @@ class IsolationConflict(RuntimeError):
"fence": {"state": "n/a", "backend": None, "error": "", "applied_at": None},
"earners_outside": [],
"earners_outside_stopped": [],
+ # Fence-gated containers the guard holds stopped, by name, with the reason.
+ "guard": {},
}
#: The worker's public egress IP, set from the heartbeat's detection
#: (app/worker_api.py). The next apply adds its drop.
@@ -1008,10 +1013,16 @@ def apply_fence(client: Any) -> dict[str, Any]:
def read_fence(client: Any) -> dict[str, Any] | None:
- """The table's per-rule counters, or None when our network does not exist."""
+ """The table's per-rule counters, or None when our network does not exist.
+
+ Raises IsolationConflict when the network no longer has the settings the
+ rules depend on: they match its interface by name, so a copy recreated
+ under another interface would read complete and cover nothing.
+ """
net = managed_network(client)
if net is None:
return None
+ _check_managed(net)
return run_helper(client, "read", _subnet(net), filtered=True)
@@ -1055,6 +1066,138 @@ def network_for_deploy() -> str | None:
return NETWORK_NAME
+def mode_problem(mode: str, service: str = "this service") -> str | None:
+ """Why a fence-gated service cannot run in this isolation mode at all, or None.
+
+ Only ``on`` (behind our fence) and ``operator`` (on the operator's own
+ network, under their rules) qualify. In ``off``, ``legacy`` and
+ ``unsupported`` the node would sit on Docker's default bridge, where every
+ other earner reaches its WebUI and nothing keeps it off the LAN.
+ ``service`` is the catalog name the message gives.
+ """
+ if mode in (MODE_ON, MODE_OPERATOR):
+ return None
+ what = {
+ MODE_LEGACY: f"Switch isolation on in Fleet, then redeploy {service}.",
+ MODE_OFF: f"Set CASHPILOT_ISOLATION=auto on this worker and restart it, then redeploy {service}.",
+ MODE_UNSUPPORTED: "Run it on Docker with isolation on, or on your own network (CASHPILOT_CONTAINER_NETWORK).",
+ }.get(mode, f"Switch isolation on in Fleet, then redeploy {service}.")
+ return f"isolation is {mode} on this worker, and {service} runs only behind the LAN fence. {what}"
+
+
+def fence_problem(client: Any, service: str = "this service") -> str | None:
+ """Why a fence-gated container must not start now, or None.
+
+ In ``on`` mode the table is read back by the helper and every rule it
+ should hold must be there, IPv6 drops included. On the operator's own
+ network there is no fence of ours to wait for: it carries their rules. Any
+ other mode does not run the node at all (mode_problem).
+ """
+ try:
+ mode = resolve_mode(client)
+ except RuntimeError as exc:
+ return f"the isolation mode could not be read ({exc})"
+ if mode != MODE_ON:
+ return mode_problem(mode, service)
+ try:
+ result = read_fence(client)
+ except Exception as exc: # noqa: BLE001 - the reason is the answer
+ return f"the LAN fence could not be read ({exc})"
+ if result is None:
+ return f"the {NETWORK_NAME} network does not exist, so the LAN fence is not in place"
+ if not result.get("present") or not result.get("matches"):
+ return "the LAN fence is not complete on this host, so the node stays stopped until the worker restores it"
+ return None
+
+
+def _gated(client: Any) -> list[Any]:
+ orch = _orchestrator()
+ from app.constants import LABEL_SERVICE
+
+ return [
+ c
+ for c in _managed(client)
+ if orch.fence_gated(str((c.labels or {}).get(LABEL_SERVICE) or ""))
+ and not orch.is_proxied(c)
+ # A node from 2.0.6 or earlier, on the host's network, is left exactly
+ # as it is until its owner redeploys it (orchestrator.predates_fence).
+ and not orch.predates_fence(c)
+ ]
+
+
+def guard() -> dict[str, str]:
+ """Every GUARD_SECONDS: hold each fence-gated container to the fence. Never raises.
+
+ A fence-gated service has no Docker restart policy, so this is what starts
+ it: after a deploy that left it created, after a reboot or a daemon
+ restart, after a crash. It starts one only when the fence is confirmed and
+ the container's shape is the catalog's, and never one the owner stopped.
+
+ It also stops one within GUARD_SECONDS when either goes: a rule deleted
+ from the table, the container recreated on the host network, privileged,
+ with a capability or an image the catalog does not give it. The fence is
+ then applied again; the next pass starts the container once it reads
+ complete. Returns the containers held stopped, with the reason.
+ """
+ orch = _orchestrator()
+ held: dict[str, str] = {}
+ try:
+ if not orch.docker_available():
+ return {}
+ client = orch._get_client()
+ gated = _gated(client)
+ if not gated:
+ _state["guard"] = {}
+ return {}
+ # A deploy replacing a gated container holds this lock: the pass then
+ # only stops (orchestrator.GATED_DEPLOY_LOCK says why).
+ deploying = not orch.GATED_DEPLOY_LOCK.acquire(blocking=False)
+ try:
+ try:
+ mode: str | None = resolve_mode(client)
+ fence = fence_problem(client)
+ except RuntimeError as exc:
+ # Not knowing the mode is not knowing the fence is there.
+ mode, fence = None, f"the isolation mode could not be read ({exc})"
+ # Container.status is read-only in docker-py; what this pass stopped is kept apart.
+ stopped_now: set[str] = set()
+ if fence and mode in (MODE_ON, None):
+ # Stop first, so nothing runs while the table is put back.
+ for c in gated:
+ if c.status == "running":
+ _guard_stop(c, fence)
+ stopped_now.add(c.name)
+ if mode == MODE_ON:
+ with contextlib.suppress(RuntimeError, IsolationConflict):
+ apply_fence(client)
+ stopped = orch.user_stopped()
+ for c in gated:
+ running = c.status == "running" and c.name not in stopped_now
+ problem = fence or orch.gated_spec_problem(c, mode) or (None if deploying else orch.identity_problem(c))
+ if problem:
+ held[c.name] = problem
+ if running:
+ _guard_stop(c, problem)
+ elif not deploying and c.status in ("exited", "created") and c.name not in stopped:
+ c.start()
+ logger.info("Started %s: the LAN fence is confirmed", c.name)
+ finally:
+ if not deploying:
+ orch.GATED_DEPLOY_LOCK.release()
+ except Exception as exc: # noqa: BLE001 - the next pass tries again
+ logger.warning("Fence guard pass failed: %s", exc)
+ _state["guard"] = held
+ return held
+
+
+def _guard_stop(container: Any, reason: str) -> None:
+ from app.constants import LABEL_SERVICE
+
+ slug = str((container.labels or {}).get(LABEL_SERVICE) or "")
+ logger.warning("Stopping %s: %s", container.name, reason)
+ container.stop(timeout=_orchestrator()._get_stop_timeout(slug))
+
+
def effective_network(client: Any) -> tuple[str, str]:
"""(mode, the network earners should be on, or "" for the default bridge)."""
mode = resolve_mode(client)
@@ -1204,6 +1347,7 @@ def status() -> dict[str, Any]:
"fence": dict(_state["fence"]),
"earners_outside": list(_state["earners_outside"]),
"earners_outside_stopped": list(_state.get("earners_outside_stopped") or []),
+ "guard": dict(_state.get("guard") or {}),
}
diff --git a/app/lan_isolation.py b/app/lan_isolation.py
index 99f6018f..105879f3 100644
--- a/app/lan_isolation.py
+++ b/app/lan_isolation.py
@@ -21,9 +21,10 @@
**It names every exception.** The worker fences bridge earners off the LAN by
default (app/lan_fence.py): its own bridge plus one firewall table, with replies
to published ports still allowed, so Storj keeps its inbound port and its
-dashboard. What a bridge cannot cover is said here per service: Mysterium runs
-on the host network, where no bridge applies at all. Operators who run their own
-rules (CASHPILOT_CONTAINER_NETWORK) get the same list to write them from.
+dashboard. What a bridge cannot cover is said here per service. No catalog
+service runs on the host network any more: Mysterium moved onto the fenced
+bridge in its own namespace, with its UDP range published. Operators who run
+their own rules (CASHPILOT_CONTAINER_NETWORK) get the same list to write them from.
"""
from __future__ import annotations
@@ -77,6 +78,22 @@ def published_ports(service: dict[str, Any] | None) -> list[str]:
if isinstance(binding, list) and ipaddress.ip_address(binding[0]).is_loopback:
continue
found.append(str(mapping))
+ ui = _docker(service).get("ui_port")
+ if isinstance(ui, dict):
+ default = next(
+ (str(v.get("default") or "") for v in _docker(service).get("env") or [] if v.get("key") == ui.get("env")),
+ "",
+ )
+ if not all(_loopback(a) for a in default.split(",")):
+ found.append(f"{default}:{ui.get('port')}")
+ rng = _docker(service).get("udp_port_range")
+ if isinstance(rng, dict):
+ default = next(
+ (str(v.get("default") or "") for v in _docker(service).get("env") or [] if v.get("key") == rng.get("env")),
+ "",
+ )
+ if default:
+ found.append(f"{default}/udp")
return found
@@ -258,3 +275,14 @@ def compose_snippet(network_name: str = DEFAULT_NETWORK_NAME, bridge_interface:
f"# ...while still allowing the exceptions listed for each service, or the\n"
f"# services that need inbound ports will silently stop earning.\n"
)
+
+
+def _loopback(address: str) -> bool:
+ """Blank, ``localhost`` or any loopback address: published to this machine only."""
+ address = address.strip()
+ if address in ("", "localhost"):
+ return True
+ try:
+ return ipaddress.ip_address(address).is_loopback
+ except ValueError:
+ return False
diff --git a/app/main.py b/app/main.py
index 3c693e28..5d8a8cab 100644
--- a/app/main.py
+++ b/app/main.py
@@ -549,6 +549,12 @@ def _container_down_message(slug: str, copies: list[dict[str, Any]]) -> str:
codes = sorted({c["exit_code"] for c in copies if isinstance(c.get("exit_code"), int)})
how = f" (exit code {', '.join(str(code) for code in codes)})" if codes else ""
message = f"{name} has not been running since {since} UTC on {nodes}{how}, so it is not earning."
+ if catalog.fence_gated(catalog.get_service(slug)):
+ # No restart policy on purpose: the worker starts it once the LAN fence is confirmed.
+ return message + (
+ " The worker starts it as soon as the LAN fence is confirmed; while it stays down, the Fleet page "
+ "says what holds it. If you stopped it on purpose, start it from the dashboard when you want it back."
+ )
policies = sorted({str(c["restart_policy"]) for c in copies if c.get("restart_policy")} - set(_RESTARTING_POLICIES))
if policies:
message += (
@@ -3494,6 +3500,8 @@ async def _deploy(slug: str, body: DeployRequest, worker_id: int | None) -> dict
]
if invalid:
raise HTTPException(status_code=400, detail=f"Invalid value for: {', '.join(invalid)}")
+ if catalog.fence_gated(svc):
+ spec = await _fence_gated_spec(spec, docker_conf, merged_env, worker_id, slug, divergence, custom_image)
if raw_command:
spec["command"] = re.sub(r"\$\{(\w+)\}", lambda m: merged_env.get(m.group(1), m.group(0)), raw_command)
@@ -3574,18 +3582,75 @@ async def _deploy(slug: str, body: DeployRequest, worker_id: int | None) -> dict
return response
-def _is_port_range(value: str) -> bool:
- """A blank value, or "start:end" with 1 <= start <= end <= 65535.
+_is_port_range = catalog.is_port_range
+
+
+def fence_gated_ports(docker_conf: dict[str, Any], env: dict[str, Any]) -> dict[str, Any]:
+ """The ports of a fence-gated service, always from the catalog and the env.
- A pattern can check the shape but not the order, and Mysterium refuses to
- start on a reversed or out-of-range --udp.ports, so the node stops earning.
+ The catalog's static ports, the WebUI port on each address ``ui_port.env``
+ lists (one binding, or several), and the UDP range ``udp_port_range.env``
+ names, published 1:1. Raises ValueError naming what is wrong.
"""
- if not value:
- return True
- start, sep, end = value.partition(":")
- if not (sep and start.isascii() and start.isdigit() and end.isascii() and end.isdigit()):
- return False
- return 1 <= int(start) <= int(end) <= 65535
+ ports: dict[str, Any] = {}
+ for mapping in docker_conf.get("ports") or []:
+ key, binding = catalog.parse_port(mapping)
+ ports[key] = binding
+ ui = docker_conf.get("ui_port")
+ if isinstance(ui, dict):
+ bindings = catalog.ui_bindings(str(env.get(ui["env"]) or "127.0.0.1"), int(ui["port"]))
+ # One binding travels as [address, port], which every worker reads; a
+ # list of them only to a worker that can publish one port several times.
+ ports[f"{int(ui['port'])}/tcp"] = bindings[0] if len(bindings) == 1 else bindings
+ rng = docker_conf.get("udp_port_range")
+ if isinstance(rng, dict):
+ ports.update(catalog.expand_udp_range(str(env.get(rng["env"]) or ""), int(rng.get("max_ports") or 1000)))
+ return ports
+
+
+async def _fence_gated_spec(
+ spec: dict[str, Any],
+ docker_conf: dict[str, Any],
+ env: dict[str, Any],
+ worker_id: int,
+ slug: str,
+ divergence: list[str],
+ custom_image: str,
+) -> dict[str, Any]:
+ """Where a fence-gated service's network namespace lives is the catalog's alone.
+
+ The record wins for an ordinary service's network mode and ports, so a
+ redeploy reproduces them. Here they are the fence: a node recorded on the
+ host network (every Mysterium through 2.0.6) must move onto the fenced
+ bridge on its next redeploy, with its ports rebuilt for it, or the redeploy
+ would carry NET_ADMIN back into the host's namespace. The identity volume is
+ a mount, so it is kept as before.
+ """
+ if custom_image:
+ raise HTTPException(
+ status_code=400,
+ detail=f"{slug} runs only the image the catalog pins, since it holds NET_ADMIN in its own namespace",
+ )
+ if "fence_gated" not in await _worker_features(worker_id):
+ raise HTTPException(
+ status_code=409,
+ detail=(
+ f"This worker is too old to run {slug} behind the LAN fence. Update the worker first; the node "
+ "keeps running as it is until then."
+ ),
+ )
+ try:
+ ports = fence_gated_ports(docker_conf, env)
+ except ValueError as exc:
+ raise HTTPException(status_code=400, detail=f"Invalid value for {slug}: {exc}") from None
+ merged = dict(spec)
+ # The record's network mode and ports do not win here, so saying they were kept would be false.
+ divergence[:] = [d for d in divergence if not d.startswith(("network_mode: keeping", "ports: keeping"))]
+ if (merged.get("network_mode") or None) != (docker_conf.get("network_mode") or None):
+ divergence.append("network_mode: moving to the fenced bridge the catalog declares now")
+ merged["network_mode"] = docker_conf.get("network_mode") or None
+ merged["ports"] = ports
+ return merged
def _merge_recorded_spec(
diff --git a/app/orchestrator.py b/app/orchestrator.py
index 223d8dbc..e5e786ba 100644
--- a/app/orchestrator.py
+++ b/app/orchestrator.py
@@ -9,11 +9,13 @@
import collections
import contextlib
+import functools
import hashlib
import ipaddress
import logging
import os
import re
+import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import UTC, datetime
@@ -400,6 +402,7 @@ def _shape(container: Any) -> dict[str, Any]:
"mounts": sorted({str(m.get("Destination") or "").rstrip("/") for m in mounts if isinstance(m, dict)} - {""}),
"env_keys": sorted({str(item).split("=", 1)[0] for item in config.get("Env") or []}),
"entrypoint": [str(part) for part in entrypoint] if isinstance(entrypoint, list) else None,
+ "restart_policy": str((host.get("RestartPolicy") or {}).get("Name") or "no"),
}
@@ -707,15 +710,22 @@ def container_runtime(slug: str, network_mode: str | None, requested: str | None
"""The runtime a new container runs under, or None for Docker's default.
A runtime named in the deploy request wins; the worker API has already
- checked it against the daemon. Otherwise a container on bridge networking
- takes the service's override or the worker's default. Host and none
- networking keep Docker's default: a sandbox gains little on the host's own
- network stack, and Mysterium's TUN device does not work under gVisor.
+ checked it against the daemon. A runtime the service's catalog entry
+ requires wins over both. Otherwise a container on bridge networking takes
+ the service's override or the worker's default. Host and none networking
+ keep Docker's default: a sandbox gains little on the host's own network
+ stack. Mysterium's catalog entry requires runc: a TUN device works under
+ gVisor, but iptables-nft and kernel WireGuard do not.
Like container_network, a runtime the daemon does not provide is refused
here, BEFORE the running container is stopped, so a typo in the compose
file cannot take a service down.
"""
+ if required := catalog_runtime(slug):
+ # The service's own catalog entry names the one runtime it works under
+ # (Mysterium: runc, since gVisor has no iptables-nft or kernel WireGuard).
+ # Neither the request nor the operator's default can change it.
+ return _proxy_allows_runtime(slug, required)
if requested:
return _proxy_allows_runtime(slug, requested)
if (network_mode or "bridge") not in ("bridge", "default"):
@@ -738,6 +748,28 @@ def container_runtime(slug: str, network_mode: str | None, requested: str | None
return runtime
+#: Held while deploy_raw replaces a fence-gated container (Mysterium). The
+#: guard (lan_fence.guard) takes it for each pass, and while a deploy holds it
+#: the guard only stops: it starts nothing and reads no identity. Otherwise a
+#: pass could start the old node between its stop and its removal, or read the
+#: old node's identity, match the record deploy_raw just wrote, and delete it,
+#: so the new node would never be compared.
+GATED_DEPLOY_LOCK = threading.Lock()
+
+
+def _holds_gated_lock(fn: Any) -> Any:
+ @functools.wraps(fn)
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ slug = kwargs.get("slug", args[0] if args else "")
+ if not fence_gated(str(slug)):
+ return fn(*args, **kwargs)
+ with GATED_DEPLOY_LOCK:
+ return fn(*args, **kwargs)
+
+ return wrapper
+
+
+@_holds_gated_lock
def deploy_raw(
slug: str,
image: str,
@@ -780,6 +812,7 @@ def deploy_raw(
client = _get_client()
name = name or _container_name(slug, instance)
joined = (network_mode or "").startswith("container:")
+ gated = fence_gated(slug)
# Decided before the old container is touched: a network or runtime that
# cannot be used must fail the deploy while the running service is still running.
@@ -787,6 +820,13 @@ def deploy_raw(
network = container_network(network_mode)
runtime = container_runtime(slug, network_mode, runtime)
dns = container_dns(network)
+ if gated:
+ from app import lan_fence
+
+ # A mode that cannot run it at all (off, legacy) is refused here too,
+ # while the running node is still running.
+ if problem := lan_fence.mode_problem(lan_fence.resolve_mode(client), service_name(slug)):
+ raise ContainerNetworkError(f"{slug} was not deployed: {problem} The running container was not touched.")
# The pull is on this side of the line for the same reason: a registry outage
# or a typo'd image must not cost the running container. And once the image
@@ -853,6 +893,14 @@ def deploy_raw(
# chance to flush (storj declares 300 s for that reason) and pays for it
# with an unclean-shutdown recovery. A stop that fails is not a reason to
# abandon the deploy - the forced removal below is what ran before.
+ if gated and not _pending_identity(name):
+ # Read while it still runs; the guard compares the new one with it.
+ # A redeploy over a stopped node clears the record instead. A record
+ # still pending is kept: the container being replaced is itself a
+ # replacement whose identity was never confirmed, so this retry is
+ # compared with the node as it was before the first redeploy.
+ identity = read_node_identity(old, slug) if getattr(old, "status", "") == "running" else None
+ _remember_identity(name, identity)
try:
old.stop(timeout=_get_stop_timeout(slug))
except APIError as exc:
@@ -871,6 +919,10 @@ def deploy_raw(
}
if labels:
all_labels.update(labels)
+ # What the guard, the owner checks and every lookup find a container by:
+ # never the caller's to change.
+ all_labels[LABEL_SERVICE] = slug
+ all_labels[LABEL_MANAGED] = "true"
if instance > 1:
all_labels[LABEL_INSTANCE] = str(instance)
@@ -890,7 +942,9 @@ def deploy_raw(
# This line makes `docker logs cashpilot-worker` state what THIS worker
# applied, so an operator comparing against the catalog can see the gap.
logger.info("Creating container %s from %s (resources: %s)", name, image, resource_kwargs or "docker defaults")
- container = client.containers.run(
+ # A fence-gated container is created, then started only once the fence is
+ # confirmed again: containers.run would start it in the same call.
+ run_kwargs: dict[str, Any] = dict(
image=image,
name=name,
environment=env or {},
@@ -927,18 +981,27 @@ def deploy_raw(
stop_timeout=_get_stop_timeout(slug),
labels=all_labels,
hostname=None if joined else (hostname or name),
- detach=True,
# A container inside another's namespace is never started by Docker on
# its own: after a reboot or a daemon restart it would join the tunnel's
# namespace before the tunnel's firewall is loaded. The worker starts it
# once the tunnel says it is ready (proxy_tunnel.status).
- restart_policy={"Name": "no"} if joined else {"Name": "unless-stopped"},
+ # A fence-gated one (Mysterium) neither: Docker would start it after a
+ # reboot before this worker has put the fence back. The worker's guard
+ # starts it once the fence is confirmed (lan_fence.guard).
+ restart_policy={"Name": "no"} if joined or gated else {"Name": "unless-stopped"},
# None means Docker's default runtime. See container_runtime for when
# the operator's CASHPILOT_CONTAINER_RUNTIME applies instead.
runtime=runtime or None,
**resource_kwargs,
)
-
+ container = client.containers.create(**run_kwargs) if gated else client.containers.run(detach=True, **run_kwargs)
+
+ if gated:
+ _mark_stopped(name, False)
+ # Left created, not started, when the fence is not confirmed: the guard
+ # starts it as soon as it is, and nothing runs before that.
+ _require_fence(slug)
+ container.start()
logger.info("Container %s started: %s", name, container.short_id)
return container.id
@@ -1065,20 +1128,274 @@ def _get_stop_timeout(slug: str) -> int:
return 30
+def fence_gated(slug: str) -> bool:
+ """Whether this worker's own catalog starts ``slug`` only behind a confirmed fence."""
+ if not get_service:
+ return False
+ return ((get_service(slug) or {}).get("docker") or {}).get("fence_gated") is True
+
+
+def catalog_runtime(slug: str) -> str | None:
+ """The runtime this worker's own catalog requires for ``slug``, if any."""
+ if not get_service:
+ return None
+ runtime = ((get_service(slug) or {}).get("docker") or {}).get("runtime")
+ return runtime.strip() if isinstance(runtime, str) and runtime.strip() else None
+
+
+#: Names of the fence-gated containers the owner stopped. Docker restarts none
+#: of them (restart policy "no"), so the worker's guard starts each one that is
+#: not running, except these: a Stop from the dashboard must stay a stop.
+STOPPED_MARKER = os.path.join(os.getenv("CASHPILOT_DATA_DIR", "/data"), ".fence_gated_stopped")
+
+
+def user_stopped() -> set[str]:
+ try:
+ with open(STOPPED_MARKER, encoding="utf-8") as fh:
+ return {line.strip() for line in fh if line.strip()}
+ except OSError:
+ return set()
+
+
+def _mark_stopped(name: str, stopped: bool, *, strict: bool = False) -> None:
+ """Record or clear an owner's Stop. ``strict`` raises when it cannot be written.
+
+ A Stop that is not recorded is undone by the guard's next pass, so Stop is
+ strict. A Start or a deploy that cannot clear the record only leaves the
+ node stopped, which is safe, so they log it.
+ """
+ names = user_stopped()
+ if (name in names) == stopped:
+ return
+ names = names | {name} if stopped else names - {name}
+ try:
+ with open(STOPPED_MARKER, "w", encoding="utf-8") as fh:
+ fh.write("".join(f"{n}\n" for n in sorted(names)))
+ except OSError as exc:
+ if strict:
+ raise ContainerRuntimeError(
+ f"{name} was not stopped: the stop could not be recorded ({exc}), and without that record the "
+ "worker would start it again within a minute."
+ ) from None
+ logger.warning("Could not record that %s was %s: %s", name, "stopped" if stopped else "started", exc)
+
+
+def predates_fence(container: Any) -> bool:
+ """A fence-gated service's container from 2.0.6 or earlier: on the host's network.
+
+ Every Mysterium through 2.0.6 runs this way, and keeps running exactly as
+ it is until its owner redeploys it: the guard neither stops nor starts it,
+ and Start, Stop and Restart treat it as before. Drift and the isolation
+ check flag it, and the redeploy moves it onto the fenced bridge.
+ """
+ attrs = container.attrs if isinstance(getattr(container, "attrs", None), dict) else {}
+ host = attrs.get("HostConfig") if isinstance(attrs.get("HostConfig"), dict) else {}
+ return str(host.get("NetworkMode") or "") == "host"
+
+
+def _gated_container(slug: str, container: Any) -> bool:
+ return fence_gated(slug) and not predates_fence(container)
+
+
+def service_name(slug: str) -> str:
+ """The catalog's display name for ``slug``, or the slug itself."""
+ return str(((get_service(slug) if get_service else None) or {}).get("name") or slug)
+
+
def stop_service(slug: str, instance: int = 1) -> None:
"""Stop the container for a service."""
container = _find_container(slug, instance)
- container.stop(timeout=_get_stop_timeout(slug))
+ if _gated_container(slug, container):
+ # Recorded before the stop, under the guard's lock: a guard pass that
+ # read the records before this Stop would otherwise start it after.
+ with GATED_DEPLOY_LOCK:
+ _mark_stopped(container.name, True, strict=True)
+ container.stop(timeout=_get_stop_timeout(slug))
+ else:
+ container.stop(timeout=_get_stop_timeout(slug))
logger.info("Stopped container %s", container.name)
def restart_service(slug: str, instance: int = 1) -> None:
"""Restart the container for a service."""
container = _find_container(slug, instance)
+ if _gated_container(slug, container):
+ _require_startable(container, slug)
+ _mark_stopped(container.name, False)
container.restart(timeout=_get_stop_timeout(slug))
logger.info("Restarted container %s", container.name)
+#: The node identity a fence-gated container had before a redeploy replaced
+#: it, one file per container name. The guard reads the new container's
+#: identity until it matches, then deletes the file; a different identity
+#: holds the node stopped (lan_fence.guard), since its data volume is not the
+#: one it ran on.
+IDENTITY_DIR = os.path.join(os.getenv("CASHPILOT_DATA_DIR", "/data"), ".fence_gated_identity")
+
+
+def read_node_identity(container: Any, slug: str) -> str | None:
+ """The node's own identity from inside it (collector.node_read), or None.
+
+ Mysterium answers its TequilAPI identity on the node's loopback; the
+ command and the pattern are the catalog's, the same the earnings read uses.
+ """
+ svc = (get_service(slug) if get_service else None) or {}
+ spec = (svc.get("collector") or {}).get("node_read") or {}
+ command, pattern = spec.get("command"), spec.get("node")
+ if not (isinstance(command, list) and command and isinstance(pattern, str)):
+ return None
+ try:
+ code, output = container.exec_run(command)
+ match = re.search(pattern, (output or b"").decode(errors="replace")) if code == 0 else None
+ except Exception as exc: # noqa: BLE001 - unknown, not different
+ logger.debug("Could not read the identity of %s: %s", getattr(container, "name", "?"), exc)
+ return None
+ return match.group(1) if match else None
+
+
+def _identity_file(name: str) -> str:
+ return os.path.join(IDENTITY_DIR, re.sub(r"[^A-Za-z0-9_.-]", "_", name))
+
+
+def _pending_identity(name: str) -> str | None:
+ """The identity a replacement still has to match, or None (no record, or a held mismatch)."""
+ try:
+ with open(_identity_file(name), encoding="utf-8") as fh:
+ recorded = fh.read().strip()
+ except OSError:
+ return None
+ return None if not recorded or recorded.startswith("MISMATCH ") else recorded
+
+
+def _remember_identity(name: str, identity: str | None) -> None:
+ path = _identity_file(name)
+ try:
+ if identity:
+ os.makedirs(IDENTITY_DIR, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(identity)
+ elif os.path.exists(path):
+ os.remove(path)
+ except OSError as exc:
+ logger.warning("Could not record the identity of %s: %s", name, exc)
+
+
+def identity_problem(container: Any) -> str | None:
+ """Why a fence-gated node must stop: a different identity than before its redeploy.
+
+ None while there is nothing to compare (no record, or the node not
+ answering yet) and once it matches, when the record is deleted.
+ """
+ path = _identity_file(container.name)
+ try:
+ with open(path, encoding="utf-8") as fh:
+ expected = fh.read().strip()
+ except OSError:
+ return None
+ held, _, rest = expected.partition(" ")
+ if held == "MISMATCH":
+ # Sticky until the next deploy rewrites the record: a stopped node
+ # reads no identity, and must not be started again because of that.
+ return rest
+ if getattr(container, "status", "") != "running":
+ return None
+ slug = str((getattr(container, "labels", None) or {}).get(LABEL_SERVICE) or "")
+ current = read_node_identity(container, slug)
+ if current is None:
+ return None
+ if current.lower() == expected.lower():
+ _remember_identity(container.name, None)
+ logger.info("%s kept its identity %s across the redeploy", container.name, current)
+ return None
+ reason = (
+ f"its identity is {current}, not {expected} as before the redeploy, so it is not running on the data "
+ "volume that holds its keystore. It stays stopped: restore that volume, then redeploy"
+ )
+ _remember_identity(container.name, f"MISMATCH {reason}")
+ return reason
+
+
+def _require_fence(slug: str) -> None:
+ """Refuse to start a fence-gated service unless the fence is confirmed now.
+
+ Raises ContainerNetworkError with the reason. Outside ``on`` mode there is
+ no fence of ours to wait for: the operator's network carries their own
+ rules, and ``off`` or ``legacy`` was the owner's choice.
+ """
+ from app import lan_fence
+
+ problem = lan_fence.fence_problem(_get_client(), service_name(slug))
+ if problem:
+ raise ContainerNetworkError(f"{slug} was not started: {problem}")
+
+
+def _require_startable(container: Any, slug: str) -> None:
+ """Refuse Start or Restart of an existing fence-gated container the guard would stop.
+
+ The fence first, then what the guard also holds against the container
+ itself: its shape (host network, privileged, a capability, an image or a
+ restart policy the catalog does not give it) and a different identity
+ than before its last redeploy. Without this the dashboard reported it
+ started, and it ran until the guard's next pass stopped it again.
+ """
+ from app import lan_fence
+
+ _require_fence(slug)
+ try:
+ mode = lan_fence.resolve_mode(_get_client())
+ except RuntimeError as exc:
+ raise ContainerNetworkError(f"{slug} was not started: the isolation mode could not be read ({exc})") from None
+ problem = gated_spec_problem(container, mode) or identity_problem(container)
+ if problem:
+ raise ContainerNetworkError(f"{slug} was not started: {problem}")
+
+
+def gated_spec_problem(container: Any, mode: str | None = None) -> str | None:
+ """Why a fence-gated container must not run as it is, or None.
+
+ Read from the live container, so a container someone recreated by hand, or
+ a worker of an older release made, is judged on what it is: on the host
+ network, privileged, with a capability its catalog entry does not grant,
+ started by Docker on its own, or on an image other than the pinned one.
+ """
+ labels = getattr(container, "labels", None) or {}
+ slug = str(labels.get(LABEL_SERVICE) or "")
+ svc = (get_service(slug) if get_service else None) or {}
+ docker_conf = svc.get("docker") or {}
+ attrs = container.attrs if isinstance(getattr(container, "attrs", None), dict) else {}
+ host = attrs.get("HostConfig") if isinstance(attrs.get("HostConfig"), dict) else {}
+ config = attrs.get("Config") if isinstance(attrs.get("Config"), dict) else {}
+ network_mode = str(host.get("NetworkMode") or "")
+ if network_mode == "host":
+ return "it shares the host's network, where its NET_ADMIN would reach the host's firewall"
+ if host.get("Privileged"):
+ return "it runs privileged"
+ want_caps = {str(c).upper().removeprefix("CAP_") for c in docker_conf.get("cap_add") or []}
+ have_caps = {str(c).upper().removeprefix("CAP_") for c in host.get("CapAdd") or []}
+ if extra := sorted(have_caps - want_caps):
+ return f"it holds {', '.join(extra)}, which its catalog entry does not grant"
+ policy = str((host.get("RestartPolicy") or {}).get("Name") or "no")
+ if policy != "no":
+ return f"Docker would start it on its own (restart policy {policy}), before the fence is confirmed"
+ pinned = {docker_conf.get("image")} | set((docker_conf.get("image_by_arch") or {}).values())
+ pinned -= {None, ""}
+ if pinned and str(config.get("Image") or "") not in pinned:
+ return f"it runs {config.get('Image') or 'an unknown image'}, not the image the catalog pins"
+ from app import lan_fence
+
+ if mode == lan_fence.MODE_ON and network_mode != lan_fence.NETWORK_NAME:
+ return f"it is not on {lan_fence.NETWORK_NAME}, so the fence does not cover it"
+ if mode == lan_fence.MODE_OPERATOR and network_mode != _CONTAINER_NETWORK:
+ return f"it is not on {_CONTAINER_NETWORK}, the network whose rules this worker runs earners under"
+ if mode in (lan_fence.MODE_ON, lan_fence.MODE_OPERATOR):
+ settings = attrs.get("NetworkSettings") if isinstance(attrs.get("NetworkSettings"), dict) else {}
+ attached = settings.get("Networks") if isinstance(settings.get("Networks"), dict) else {}
+ if extra := sorted(set(attached) - {network_mode}):
+ return f"it is also attached to {', '.join(extra)}, where the fence does not cover it"
+ return None
+
+
def _keep_live_mounts(
old: Any,
volumes: dict[str, dict[str, str]] | None,
@@ -1489,6 +1806,9 @@ def remove_service(slug: str, delete_volumes: bool = False, allow_delete_critica
def start_service(slug: str, instance: int = 1) -> None:
"""Start a stopped container for a service."""
container = _find_container(slug, instance)
+ if _gated_container(slug, container):
+ _require_startable(container, slug)
+ _mark_stopped(container.name, False)
container.start()
logger.info("Started container %s", container.name)
diff --git a/app/socket_policy.py b/app/socket_policy.py
index 1924c0ff..d3c6dbdb 100644
--- a/app/socket_policy.py
+++ b/app/socket_policy.py
@@ -120,11 +120,14 @@
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [{capabilities}]
allowed_devices: [{devices}]
allowed_runtimes: [{runtimes}]
diff --git a/app/templates/fleet.html b/app/templates/fleet.html
index 6e0fb447..5fe8d7cc 100644
--- a/app/templates/fleet.html
+++ b/app/templates/fleet.html
@@ -493,6 +493,11 @@
Proxies
} else if (iso.mode === 'off' && outside.length === 0 && fence.state === 'reduced') {
line = 'Isolation is off (CASHPILOT_ISOLATION=off). Earners still on the isolated network keep their internet; redeploy them to move them back to Docker\'s default bridge.';
}
+ // Fence-gated earners (Mysterium) the worker holds stopped, with why.
+ const held = Object.entries(iso.guard || {});
+ if (held.length) {
+ line += `${line ? '' : ''}Held stopped: ${held.map(([n, why]) => `${esc(n)} (${esc(why)})`).join('; ')}. A missing LAN fence clears by itself; any other reason needs a redeploy from the dashboard.${line ? '
' : ''}`;
+ }
if (sw) {
const tone = sw.error ? 'text-fail' : sw.busy ? 'muted' : 'text-ok';
line += `${esc(sw.error || sw.busy || sw.done)}
`;
diff --git a/app/worker_api.py b/app/worker_api.py
index 2b3ad9d8..c1514a24 100644
--- a/app/worker_api.py
+++ b/app/worker_api.py
@@ -722,6 +722,9 @@ async def _send_heartbeat() -> None:
# This worker fences its earners off the LAN by itself and takes
# POST /api/isolation to switch an upgraded install over (app/lan_fence.py).
payload["system_info"]["features"].append("isolation_default")
+ # Starts a fence_gated service (Mysterium) only behind a confirmed fence,
+ # on the bridge, with its port range and several WebUI addresses.
+ payload["system_info"]["features"].append("fence_gated")
# Only what the owner asked for, and only a finished round.
sent_seq = _activity_seq
if _activity_enabled and _pending_activity is not None:
@@ -897,6 +900,24 @@ async def _fence_loop() -> None:
await _check_fence()
+_guard_task: asyncio.Task | None = None
+
+
+async def _guard_once() -> None:
+ """One pass of the fence guard (lan_fence.guard). Never raises."""
+ try:
+ await asyncio.wait_for(asyncio.to_thread(lan_fence.guard), FENCE_STARTUP_TIMEOUT)
+ except Exception as exc: # noqa: BLE001 - the next pass tries again
+ logger.warning("Fence guard pass did not finish: %s", exc)
+
+
+async def _guard_loop() -> None:
+ """Every GUARD_SECONDS: start fence-gated services behind a confirmed fence, stop them without one."""
+ while True:
+ await asyncio.sleep(lan_fence.GUARD_SECONDS)
+ await _guard_once()
+
+
def _apply_activity_reply(resp: Any, delivered_seq: int | None) -> None:
"""Follow the UI's activity switch, and drop a round it has stored.
@@ -1019,7 +1040,7 @@ def _get_local_ip() -> str:
@asynccontextmanager
async def lifespan(app: FastAPI):
- global _heartbeat_task, _fence_task, _activity_task
+ global _heartbeat_task, _fence_task, _activity_task, _guard_task
logger.info("CashPilot Worker '%s' starting", WORKER_NAME)
docker_mode = "direct" if await asyncio.to_thread(orchestrator.docker_available) else "monitor-only"
@@ -1039,6 +1060,11 @@ async def lifespan(app: FastAPI):
lan_fence.set_egress_ip(await _detect_egress_ip())
await _check_fence(startup=True)
_fence_task = asyncio.create_task(_fence_loop())
+ if docker_mode == "direct":
+ # Right after the fence: a fence-gated node (Mysterium) has no restart
+ # policy, so after a reboot this is what starts it, and only now.
+ await _guard_once()
+ _guard_task = asyncio.create_task(_guard_loop())
if docker_mode == "direct":
# A proxy Test tunnel this worker left behind when it stopped mid-test.
try:
@@ -1058,7 +1084,7 @@ async def lifespan(app: FastAPI):
yield
- for task in (_heartbeat_task, _fence_task, _activity_task):
+ for task in (_heartbeat_task, _fence_task, _activity_task, _guard_task):
if task and not task.done():
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
@@ -1213,9 +1239,11 @@ class ResourceSpec(BaseModel):
class DeploySpec(BaseModel):
image: str
env: dict[str, str] = {}
- # A host port, or (host address, host port) for a bind to one interface.
- # Docker reads a list as several bindings, so the pair must be a tuple.
- ports: dict[str, int | tuple[str, int]] = {}
+ # A host port, or (host address, host port) for a bind to one interface, or
+ # several of those pairs for one container port published on several
+ # addresses (Mysterium's WebUI on each UI_ADDRESS). Docker reads a list as
+ # several bindings, so each pair must be a tuple.
+ ports: dict[str, int | tuple[str, int] | list[tuple[str, int]]] = {}
volumes: dict[str, dict[str, str]] = {}
# Mount targets the operator relocated on purpose this deploy. Every other
# mount of a container being replaced is kept where it is.
@@ -1500,9 +1528,39 @@ def _validate_instance(instance: int) -> None:
raise HTTPException(status_code=400, detail=f"instance must be between 1 and {MAX_INSTANCES}")
+#: The most container ports one deploy may publish: a 1000-port UDP range
+#: (catalog.MAX_UDP_PORTS) plus a handful of TCP ones.
+MAX_PUBLISHED_PORTS = 1024
+
+
+def _catalog_service(slug: str | None) -> dict[str, Any] | None:
+ if not _catalog_get_services or not slug:
+ return None
+ return next((svc for svc in _catalog_get_services() if svc.get("slug") == slug), None)
+
+
+def _validate_fence_gated(spec: DeploySpec, slug: str | None) -> None:
+ """A fence-gated service runs exactly as this worker's own catalog says.
+
+ Decided here from the worker's catalog, never from the request: the image
+ must be the pinned one and the network a bridge, so a caller cannot put a
+ NET_ADMIN service back on the host network or swap its image.
+ """
+ docker_conf = (_catalog_service(slug) or {}).get("docker") or {}
+ if docker_conf.get("fence_gated") is not True:
+ return
+ pinned = {docker_conf.get("image")} | set((docker_conf.get("image_by_arch") or {}).values())
+ if spec.image not in pinned - {None, ""}:
+ raise HTTPException(status_code=403, detail=f"'{slug}' runs only the image this worker's catalog pins")
+ if (spec.network_mode or "bridge") not in ("bridge", "default"):
+ raise HTTPException(status_code=403, detail=f"'{slug}' runs only on the fenced bridge")
+
+
def _validate_deploy_spec(spec: DeploySpec, slug: str | None = None) -> None:
_validate_runtime(spec.runtime)
_validate_instance(spec.instance)
+ if len(spec.ports) > MAX_PUBLISHED_PORTS:
+ raise HTTPException(status_code=400, detail=f"At most {MAX_PUBLISHED_PORTS} ports can be published")
if spec.instance > 1 and (spec.volumes or spec.ports):
# Two copies would share the same host directories and fight over the
# same host ports. The UI refuses this first; this is the backstop.
@@ -1528,6 +1586,7 @@ def _validate_deploy_spec(spec: DeploySpec, slug: str | None = None) -> None:
raise HTTPException(status_code=403, detail=f"Network mode '{spec.network_mode}' is not allowed")
if spec.network_mode == "host" and slug not in _catalog_host_network_slugs():
raise HTTPException(status_code=403, detail=f"Network mode 'host' is not allowed for '{slug}'")
+ _validate_fence_gated(spec, slug)
proxy_roots = orchestrator._proxy_bind_roots() if orchestrator.proxied() else None
for source in spec.volumes:
if source in orchestrator.PROXY_VOLUMES:
diff --git a/contrib/app-stores/casaos/CashPilot/docker-compose.yml b/contrib/app-stores/casaos/CashPilot/docker-compose.yml
index 14bc894a..58bf53eb 100644
--- a/contrib/app-stores/casaos/CashPilot/docker-compose.yml
+++ b/contrib/app-stores/casaos/CashPilot/docker-compose.yml
@@ -166,7 +166,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The folders the proxy allows, so the worker can say what to set before
# the proxy refuses a host-folder bind. Empty means none.
- CASHPILOT_COMPOSE_DIR=
@@ -259,11 +259,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/contrib/app-stores/runtipi/cashpilot/docker-compose.yml b/contrib/app-stores/runtipi/cashpilot/docker-compose.yml
index afbad50f..f94a266b 100644
--- a/contrib/app-stores/runtipi/cashpilot/docker-compose.yml
+++ b/contrib/app-stores/runtipi/cashpilot/docker-compose.yml
@@ -117,7 +117,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The values the proxy gets, so the worker can say what to set before the
# proxy refuses a host-folder bind.
- CASHPILOT_COMPOSE_DIR=${CASHPILOT_COMPOSE_DIR:-}
@@ -183,11 +183,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/docker-compose.build.yml b/docker-compose.build.yml
index 0f2ee66c..adc53e92 100644
--- a/docker-compose.build.yml
+++ b/docker-compose.build.yml
@@ -164,7 +164,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The same two values the proxy gets, so the worker can say what to set
# before the proxy refuses (the in-app update, a host-folder bind).
- CASHPILOT_COMPOSE_DIR=${CASHPILOT_COMPOSE_DIR:-}
@@ -244,11 +244,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/docker-compose.fleet.yml b/docker-compose.fleet.yml
index 2ed0ceb7..e478bfd0 100644
--- a/docker-compose.fleet.yml
+++ b/docker-compose.fleet.yml
@@ -221,7 +221,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The same two values the proxy gets, so the worker can say what to set
# before the proxy refuses (the in-app update, a host-folder bind).
- CASHPILOT_COMPOSE_DIR=${CASHPILOT_COMPOSE_DIR:-}
@@ -309,11 +309,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/docker-compose.worker.yml b/docker-compose.worker.yml
index f8cf7b83..12ca1284 100644
--- a/docker-compose.worker.yml
+++ b/docker-compose.worker.yml
@@ -125,7 +125,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The same two values the proxy gets, so the worker can say what to set
# before the proxy refuses (the in-app update, a host-folder bind).
- CASHPILOT_COMPOSE_DIR=${CASHPILOT_COMPOSE_DIR:-}
@@ -202,11 +202,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/docker-compose.yml b/docker-compose.yml
index 2b2f96c6..8c629416 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -229,7 +229,7 @@ services:
- DOCKER_HOST=unix:///run/cashpilot-docker/docker.sock
# The proxy policy's fingerprint (scripts/socket_proxy_policy.py), so the
# worker can say when its image needs a newer policy than this file has.
- - CASHPILOT_PROXY_POLICY=e9a30411b564dc75
+ - CASHPILOT_PROXY_POLICY=4c547983d8a0aaa1
# The same two values the proxy gets, so the worker can say what to set
# before the proxy refuses (the in-app update, a host-folder bind).
- CASHPILOT_COMPOSE_DIR=${CASHPILOT_COMPOSE_DIR:-}
@@ -313,11 +313,14 @@ configs:
request_body:
container_create:
allow_privileged: false
- allow_host_network: true # Mysterium runs on host networking
+ # The LAN fence helper and the isolation check's control probe run on the
+ # host network; no earner does (the worker refuses it for every catalog
+ # service, and the catalog refuses it to any service with NET_ADMIN).
+ allow_host_network: true
allow_host_pid: false
allow_host_ipc: false
allow_host_userns: false
- allow_host_cgroupns: true # the Activity probe of a host-network earner (app/activity.py)
+ allow_host_cgroupns: false
allowed_capabilities: [NET_ADMIN, NET_RAW, SETUID, SETGID]
allowed_devices: [/dev/net/tun]
allowed_runtimes: [runc, runsc, runsc-hostnet, runsc-hostnet-raw]
diff --git a/docs/activity.md b/docs/activity.md
index 15b3afe7..45bbd355 100644
--- a/docs/activity.md
+++ b/docs/activity.md
@@ -64,7 +64,8 @@ for the dashboard's usage graphs; Activity stores no second byte counter.
## Limits
-- **Host network** (Mysterium, for example): the earner shares the machine's
+- **Host network** (no catalog service; Mysterium shared it through 2.0.6, and
+ the socket proxy now refuses the helper described here): the earner shares the machine's
socket table, so reading `/proc/net` would show every socket on the machine.
Its helper joins the host network instead and asks the kernel for the socket
list over `sock_diag`, which reports the cgroup each socket was created in.
diff --git a/docs/comparison.md b/docs/comparison.md
index d66f993e..9fb803fe 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -30,7 +30,7 @@ Left out: [earning-machine](https://github.com/jf-m/earning-machine) and [passiv
## Side by side
-CashPilot's column describes v1.64.0, except the proxy, Docker socket and LAN isolation rows, which describe v1.71.1 (`0170dee`, 2026-11-06). The other columns describe the commits in the table above.
+CashPilot's column describes v1.64.0, except the proxy, Docker socket and LAN isolation rows, which describe v1.71.1 (`0170dee`, 2026-11-06), and Mysterium's network in the container settings row, which describes the release after 2.0.6 (`01e7310`). The other columns describe the commits in the table above.
| | CashPilot | money4band 5.0.0 | IGM | InternetIncome | CashFactory | HA add-ons |
|---|---|---|---|---|---|---|
@@ -42,7 +42,7 @@ CashPilot's column describes v1.64.0, except the proxy, Docker socket and LAN is
| Payouts | Proposes a payout when a balance drops, and records it once you confirm ([schema][cp-payouts]) | None | None | None | None | None |
| Daily reward claiming | No | Yes, sidecars ([README][m4b-features]) | Yes ([README][igm-features], [compose][igm-sidecars]) | No | No | No |
| Credentials at rest | Encrypted (Fernet) in the UI's database; the key file sits in the same data volume ([code][cp-fernet]). Earner credentials reach each worker as container environment variables, so they are plain text in that worker's Docker config (`docker inspect`) | Encrypted (Fernet); the key file sits in the data directory ([code][m4b-crypto]) | Encrypted (AES-256 with openssl); the key file sits beside the vault ([code][igm-encryptor]) | Plain text in `properties.conf`; some passwords are passed as `docker run` arguments ([script][ii-password-arg]) | Plain text in `.env` ([file][cf-env]) | Add-on options, stored by the Supervisor in `/data/options.json` ([HA docs][ha-options]) |
-| Earner container settings | Every capability dropped, then the catalog adds back what a service needs; `no-new-privileges`; a pids limit; privileged refused ([code][cp-hardening], [code][cp-privileged]). Mysterium runs with host networking and `NET_ADMIN`, `SETUID` and `SETGID` ([catalog][cp-myst]) | `no-new-privileges` on every non-privileged service ([code][m4b-nnp]); the proxy container is privileged with `NET_ADMIN` ([template][m4b-tun]) | Nothing set on app services; Mysterium gets `NET_ADMIN` ([compose][igm-hosting-myst]); the proxy container gets `NET_ADMIN`, with `no-new-privileges` and a pids limit ([compose][igm-proxy]) | Ebesucher runs with `seccomp=unconfined` ([script][ii-seccomp]); proxy containers and Mysterium get `NET_ADMIN` | Nothing set | No add-on asks for privileged, host network or Docker API access ([example config][ha-config]) |
+| Earner container settings | Every capability dropped, then the catalog adds back what a service needs; `no-new-privileges`; a pids limit; privileged refused ([code][cp-hardening], [code][cp-privileged]). Mysterium runs in its own network namespace on the fenced bridge with `NET_ADMIN`, `SETUID` and `SETGID`, on an image pinned by digest ([catalog][cp-myst]) | `no-new-privileges` on every non-privileged service ([code][m4b-nnp]); the proxy container is privileged with `NET_ADMIN` ([template][m4b-tun]) | Nothing set on app services; Mysterium gets `NET_ADMIN` ([compose][igm-hosting-myst]); the proxy container gets `NET_ADMIN`, with `no-new-privileges` and a pids limit ([compose][igm-proxy]) | Ebesucher runs with `seccomp=unconfined` ([script][ii-seccomp]); proxy containers and Mysterium get `NET_ADMIN` | Nothing set | No add-on asks for privileged, host network or Docker API access ([example config][ha-config]) |
| Docker socket mounted into | Since v1.71.0, a socket proxy (sockguard 2.4.5, pinned by digest), read-only. The worker gets a filtered copy that refuses privileged containers, binds of `/` or of the socket, and capabilities or devices outside the catalog's ([compose][cp-sock], [details](security-defaults.md)); never an earner ([code][cp-mounts]) | Watchtower ([template][m4b-watchtower]) | Watchtower, which is read-only with `no-new-privileges` ([compose][igm-watchtower]) | Five restart helper containers ([script][ii-sock-helper]) and Watchtower ([script][ii-watchtower]) | Portainer, published on ports 8000, 9000 and 9443 on all interfaces ([compose][cf-portainer]), and two Watchtowers ([compose][cf-watchtower-ps], [compose][cf-watchtower-efm]) | None |
| Image tags | 4 of the 17 earner images are pinned to a version or digest; the rest float | All 15 app images use `:latest` ([example][m4b-image]); proxy container pinned to v2.6.0, Watchtower to `:1` | No tag, so Docker uses `latest` ([compose][igm-hosting]); a tag is set only for some ARM builds | Mostly `:latest` ([example][ii-seccomp]); EarnApp `:lite`, Honeygain pinned to 0.6.6 when proxied on amd64 ([script][ii-honeygain]), Traffmonetizer per-arch tags on ARM; Watchtower `:nightly` | `:latest` or no tag, except EarnApp `:lite` ([compose][cf-compose]) | One version per add-on, bumped by a daily CI job ([workflow][ha-bump], [example][ha-dockerfile]) |
| App image updates | Pulled again on each deploy or redeploy; no automatic updater | Watchtower ([template][m4b-watchtower]) | Watchtower every 4 hours ([compose][igm-watchtower]) | Watchtower once a day, if you turn it on ([script][ii-watchtower]) | Two Watchtowers, each limited to one app, PacketStream ([compose][cf-watchtower-ps]) and EarnFM ([compose][cf-watchtower-efm]); the other 9 apps are not updated | Home Assistant's add-on updates |
@@ -155,7 +155,7 @@ We did not check whether each app on each list still pays out.
[cp-hardening]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/app/orchestrator.py#L745-L752
[cp-privileged]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/app/worker_api.py#L1244
[cp-mounts]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/app/worker_api.py#L992-L1040
-[cp-myst]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/services/bandwidth/mysterium.yml#L83-L91
+[cp-myst]: https://github.com/GeiserX/CashPilot/blob/01e7310cf1d923ba35523d8944ebaf2478f39079/services/bandwidth/mysterium.yml#L19-L113
[cp-sock]: https://github.com/GeiserX/CashPilot/blob/0170deec45c1a5fe6e72682b94adc07e5878d2bf/docker-compose.yml#L69-L106
[cp-update]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/app/update_check.py#L80
[cp-rates]: https://github.com/GeiserX/CashPilot/blob/c7a12bd4a1e5ce06f5be0ad303afc14c28ec17b8/app/exchange_rates.py#L101-L119
diff --git a/docs/guides/mysterium.md b/docs/guides/mysterium.md
index 9f4b4193..a3a6f724 100644
--- a/docs/guides/mysterium.md
+++ b/docs/guides/mysterium.md
@@ -6,7 +6,7 @@
## Description
-MystNodes (Mysterium Network) is a decentralized VPN and proxy network built on blockchain technology. You earn MYST tokens by running a node that provides VPN, proxy, and data scraping services to users. Requires NET_ADMIN, SETUID and SETGID capabilities and host networking for full functionality. Includes a built-in web UI for node management. Works on both residential and VPS connections.
+MystNodes (Mysterium Network) is a decentralized VPN and proxy network built on blockchain technology. You earn MYST tokens by running a node that provides VPN, proxy, and data scraping services to users. Requires NET_ADMIN, SETUID and SETGID capabilities and a TUN device, which it uses inside its own network namespace on the fenced earner bridge. Includes a built-in web UI for node management. Works on both residential and VPS connections.
## Earning Estimates
@@ -18,7 +18,7 @@ MystNodes (Mysterium Network) is a decentralized VPN and proxy network built on
| Payout frequency | Automatic on-chain settlement each time a node holds 5 MYST unsettled |
| Payment methods | Crypto |
-> Earnings in MYST tokens. Residential IPs earn significantly more. Node WebUI at port 4449 for management, bound to the machine it runs on; from another machine open an SSH tunnel first (`ssh -L 4449:127.0.0.1:4449 `, then browse `http://127.0.0.1:4449`), or put a reverse proxy in front (see **WebUI address** below). VPS accepted. Important: after first run, set your beneficiary (settlement) wallet via the node WebUI or CLI to match your mystnodes.com account -- this links on-chain earnings to your cloud dashboard.
+> Earnings in MYST tokens. Residential IPs earn significantly more. Node WebUI at port 4449 for management, published on the machine it runs on only (127.0.0.1); from another machine open an SSH tunnel first (`ssh -L 4449:127.0.0.1:4449 `, then browse `http://127.0.0.1:4449`), or put a reverse proxy in front (see **WebUI address** below). VPS accepted. Important: after first run, set your beneficiary (settlement) wallet via the node WebUI or CLI to match your mystnodes.com account -- this links on-chain earnings to your cloud dashboard.
> **One node per public IP.** Mysterium strictly enforces one active node per public IP address. Additional nodes on the same IP show as offline and earn nothing. Do not run on a phone if a Docker node is already running on the same network. Use separate public IPs (e.g. dual WAN, different locations) for additional nodes.
@@ -63,29 +63,41 @@ In the CashPilot web UI, open **Catalog** and click **Add** on the **MystNodes**
| Variable | Default | What it does |
|---|---|---|
-| `UI_ADDRESS` | `127.0.0.1` | Where the node's WebUI (port 4449) listens. Comma separated, no spaces. |
-| `UDP_PORTS` | `56000:56100` | The UDP ports the node uses for sessions, as `start:end`. Forward the same range. To go back to the default after changing it, remove the service from that machine (its data volume stays) and deploy again. |
+| `UI_ADDRESS` | `127.0.0.1` | The host addresses the node's WebUI (port 4449) is published on. Comma separated, no spaces. |
+| `UDP_PORTS` | `56000:56029` | The UDP ports the node uses for sessions, as `start:end`. Forward the same range. To go back to the default after changing it, remove the service from that machine (its data volume stays) and deploy again. |
### Port forwarding
Each session first tries to reach the node directly on its UDP range, and falls
back to hole punching when it cannot, which many buyers fail at. The node uses
-only the range in `UDP_PORTS` (default `56000:56100`), so forward exactly that
+only the range in `UDP_PORTS` (default `56000:56029`), so forward exactly that
range. Create one rule on your router:
| Field | Value |
|---|---|
| Protocol | UDP |
-| External ports | 56000-56100 (your `UDP_PORTS`) |
-| Internal ports | the same, 56000-56100 |
+| External ports | 56000-56029 (your `UDP_PORTS`) |
+| Internal ports | the same, 56000-56029 |
| Internal IP | the LAN address of the machine running the node |
Give that machine a fixed LAN address (a DHCP reservation), or the rule points
-at the wrong host after a lease change. UPnP on the router works too; the node
-uses it by itself. DMZ is the last resort.
-
-Each session takes about two ports, so 101 ports carry about 50 sessions at
-once, far more than a home node gets.
+at the wrong host after a lease change. UPnP does not work: the node sits behind
+the LAN fence, which drops the router, so it cannot ask the router for a
+forward. Forward the range by hand. DMZ is the last resort.
+
+The node runs on CashPilot's fenced bridge, and the worker publishes the range
+1:1 on the machine (`56000/udp` to `56000/udp`, and so on), so the forward on
+the router is the same as when the node shared the host's network.
+
+Each session takes about two ports, so the default 30 ports carry about 15
+sessions at once, more than a home node gets. Every published port costs about
+2.5 MB of memory on the machine (Docker starts two small proxy processes per
+port; 101 ports measured about 250 MB), so a wider range only costs more. A
+range of 20 to 30 ports is enough. Through 2.0.6 the default was
+`56000:56100`: a router forward for that range keeps working, since the new
+default is inside it. A node that has `56000:56100` on record keeps it on a
+redeploy; to save the memory, set `UDP_PORTS` to `56000:56029` and redeploy,
+with no change on the router.
**Check it.** After a few sessions, on the machine running the node:
@@ -107,9 +119,11 @@ you open it.
### WebUI address
-The node runs with host networking, so the WebUI listens on the host itself.
-By default that is `127.0.0.1` only: nothing outside the machine can reach it,
-and CashPilot does not need it. The worker reads the node's balance and lifetime from the node's own Tequila API, inside the container.
+The node runs in its own network namespace on CashPilot's fenced bridge. Its
+WebUI listens on every address inside that namespace, and the worker publishes
+port 4449 on the host addresses in `UI_ADDRESS`. By default that is
+`127.0.0.1` only: nothing outside the machine can reach it, and CashPilot does
+not need it. The worker reads the node's balance and lifetime from the node's own Tequila API, inside the container.
To put a reverse proxy in front, add the one address the proxy connects to and
keep `127.0.0.1` first, for example `127.0.0.1,172.18.0.1`:
@@ -117,15 +131,16 @@ keep `127.0.0.1` first, for example `127.0.0.1,172.18.0.1`:
- **Proxy in Docker on the same machine:** the gateway of the proxy's Docker
network (`docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}'`).
Only containers on that network can reach it. Create that network before the
- node starts: an address that does not exist yet is skipped, and the node has to
- be restarted to pick it up.
+ node is deployed: Docker refuses to publish on an address the host does not
+ have, and the deploy fails.
- **Proxy on another machine:** this machine's LAN address. Anything on the LAN
- can then reach the WebUI, so let only the proxy's address in with the host
- firewall.
+ can then reach the WebUI. A published port goes through Docker's forwarding,
+ so a host `INPUT` rule does not restrict it; limit it with a rule in the
+ `DOCKER-USER` chain, or keep the proxy on the same machine.
-Never `0.0.0.0`: with host networking that is every interface of the machine.
-The Tequila API (4050) always stays on `127.0.0.1`; the WebUI reaches it from
-inside the node.
+Never `0.0.0.0`: that publishes the WebUI on every address of the machine.
+The Tequila API (4050) is never published; the WebUI reaches it from inside
+the node.
Set it in the deploy form and redeploy that machine's node; each worker keeps
its own value. To go back to loopback only, enter `localhost`: the form treats
@@ -134,12 +149,10 @@ machine.
### Activity page
-Because the node shares the machine's network, the [Activity](../activity.md)
-page matches its connections by the node's cgroup rather than by a network of
-its own. The row shows the node's destinations and alerts, but not its
-connection counters, which are the machine's. Traffic the node forwards for VPN
-customers is routed by the kernel and does not appear there. On a cgroup v1
-machine the row reads "host network: cannot be told apart".
+The node has a network namespace of its own, so the [Activity](../activity.md)
+page reads it like any other earner. Traffic the node forwards for VPN
+customers is routed by the kernel inside that namespace and does not appear
+there.
## Troubleshooting
@@ -224,21 +237,39 @@ uses now (`docker inspect cashpilot-mysterium --format '{{json .Mounts}}'`); a
different one starts a new node identity.
```bash
-docker run -d --name cashpilot-mysterium \
- --network host --restart unless-stopped \
+docker create --name cashpilot-mysterium \
+ --network cashpilot-isolated --restart no --runtime runc \
+ -p 127.0.0.1:4449:4449 -p 56000-56029:56000-56029/udp \
--cap-drop ALL --cap-add NET_ADMIN --cap-add SETUID --cap-add SETGID \
--device /dev/net/tun \
--security-opt no-new-privileges:true \
-v /path/to/your/myst/data:/var/lib/mysterium-node \
- mysteriumnetwork/myst:1.39.7-alpine \
- --ui.address=127.0.0.1 --tequilapi.address=127.0.0.1 \
+ mysteriumnetwork/myst:1.39.9-alpine \
+ --ui.address=0.0.0.0 --tequilapi.address=127.0.0.1 \
--firewall.protected.networks=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8,100.64.0.0/10,169.254.0.0/16 \
- --udp.ports=56000:56100 \
+ --udp.ports=56000:56029 \
service --agreed-terms-and-conditions
```
-If a reverse proxy reaches this node's WebUI, use the same `--ui.address` list
-the container has now (`docker inspect cashpilot-mysterium --format '{{json .Config.Cmd}}'`).
+If a reverse proxy reaches this node's WebUI, add a `-p :4449:4449` for
+each address it connects to. The worker's guard covers only nodes CashPilot
+deployed, so this hand-made container is created, not started, and has no
+restart policy: Docker must not start it before the worker has put the fence
+in place. Start it, now and after every reboot, only when the fence's host
+rule is there (nftables backend; on the `iptables-legacy` one, test
+`sudo iptables-legacy -t mangle -S CP-IN | grep -q host` instead):
+
+```bash
+sudo nft list table ip cashpilot 2>/dev/null | grep -q 'comment "host"' \
+ && docker start cashpilot-mysterium || echo "fence not in place: node left stopped"
+```
+
+This checks that the table is there, not every rule in it, as the worker
+does. A redeploy from the dashboard does not take this container over: behind
+the Docker socket proxy the worker cannot stop or remove a container it did
+not start, so the redeploy fails until you remove this one yourself. A node
+the worker deploys keeps its identity on the `mysterium-data` volume, so it is
+a new node unless that volume already holds this node's data.
Verify it took:
@@ -252,7 +283,8 @@ docker logs cashpilot-mysterium 2>&1 | grep -i wireguard | tail -3 # expect "W
**Your identity is safe.** It lives in the mounted data directory
(`/var/lib/mysterium-node/keystore/`), not in the container, so recreating the
container keeps the same node identity and its accumulated reputation. Confirm
-with `curl -s http://127.0.0.1:4050/identities`; the address must be unchanged.
+with `docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities`
+(the Tequila API listens only inside the node); the address must be unchanged.
**On the host**, `/dev/net/tun` must exist (`ls -l /dev/net/tun`). If it does
not, load the module with `modprobe tun`.
diff --git a/docs/home-network-security.md b/docs/home-network-security.md
index df689ae1..76f6b9fa 100644
--- a/docs/home-network-security.md
+++ b/docs/home-network-security.md
@@ -48,9 +48,9 @@ with nothing else you care about.
- **A VLAN** does the same thing on a wired network, with a firewall rule on the
router that lets the VLAN reach the internet but not your other VLANs.
-A segment protects against everything on that machine, including services that
-use the host's network directly, such as Mysterium. The container firewall
-below cannot confine those.
+A segment protects against everything on that machine, including a service
+that uses the host's network directly or breaks out of its container. The
+container firewall below cannot confine those.
Storj, Sia (hostd) and Anyone Protocol need an inbound port forwarded, and
Mysterium earns more with its UDP port range forwarded. Point those forwards
@@ -240,10 +240,12 @@ add INPUT -i "$BR" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
We checked that these rules load, but tested the blocking only for IPv4.
**What it cannot cover.** A service on host networking uses the host's own
-interfaces, so a rule on a bridge does not apply to it. That is Mysterium today.
-Use a separate segment for the machine, run such services somewhere else, or
-fence its session interfaces as in
-[Mysterium, which uses host networking](isolation-check.md#mysterium-which-uses-host-networking).
+interfaces, so a rule on a bridge does not apply to it. No catalog service runs
+that way: after 2.0.6 Mysterium sits on the bridge in its own network
+namespace, with its UDP range published
+([Mysterium on the fenced bridge](isolation.md#mysterium-on-the-fenced-bridge)).
+For a node you run yourself on host networking, use a separate segment, or
+fence its session interfaces as in [Mysterium](isolation-check.md#mysterium).
!!! warning "Tailscale on the host can bypass these rules"
@@ -289,8 +291,14 @@ shipped compose files, and with the raw Docker socket, which is root, in the
`CASHPILOT_WORKER_BIND_ADDR`: the one the dashboard reaches it on, never
`0.0.0.0` on a machine with a public address.
- **Don't publish a node's own management UI to your LAN** unless you need it.
- Mysterium's WebUI and API bind to the machine itself by default. Reach them
- over an SSH tunnel, or add only your reverse proxy's address.
+ Mysterium's WebUI is published on `127.0.0.1` only by default, and its API
+ never leaves the node. Reach the WebUI over an SSH tunnel, or add to
+ `UI_ADDRESS` only the one address of this machine your reverse proxy
+ connects to: a Docker network gateway such as `172.18.0.1` for a proxy on
+ the same machine, or this machine's LAN address for a proxy elsewhere.
+ `UI_ADDRESS` names addresses on this machine, never the proxy's own. A published port goes through
+ Docker's forwarding, so a host `INPUT` rule does not restrict it: limit it
+ with the address you publish on, or a rule in `DOCKER-USER`.
## 4. Don't run it next to what you cannot lose
@@ -376,9 +384,11 @@ was the limit that day, and CPU stayed at about six times runc's: 19 to 20 s for
that capability back.
- **A service on host networking gains little.** gVisor's own documentation says
network passthrough "decreases the isolation to the host", and a service that
- already shares the host's network, like Mysterium, has nothing left to
- confine it there. A service that needs host networking
- plus a TUN device, like Mysterium, is a poor fit.
+ already shares the host's network has nothing left to confine it there.
+- **Mysterium does not run under it.** gVisor gives it a TUN device, but not
+ `iptables-nft` or kernel WireGuard, which the node uses for its customers.
+ Its catalog entry requires `runc`, so `CASHPILOT_CONTAINER_RUNTIME=runsc`
+ does not reach it.
- **Stock Unraid cannot run it at all.** Unraid runs from its initial RAM
filesystem. That root has no parent mount, so gVisor cannot `pivot_root` out
of it, and working around that with a chroot hits a second kernel rule: no new
@@ -429,8 +439,8 @@ For CashPilot's networks, add the passthrough runtimes to
On an Ubuntu 24.04 host, each of these ran under `runsc-hostnet` on
`cashpilot-isolated` without restarting and without name-lookup errors: EarnApp, Earn.fm, Honeygain, IPRoyal Pawns, PacketStream, ProxyBase,
ProxyLite, ProxyRack, Repocket and Traffmonetizer, plus Bitping under
-`runsc-hostnet-raw`. We did not try Mysterium, which needs host networking and a
-TUN device, or Storj, whose disk traffic would pay gVisor's overhead on every
+`runsc-hostnet-raw`. Mysterium cannot run under it (see above). We did not try
+Storj, whose disk traffic would pay gVisor's overhead on every
read and write.
To make that stick across redeploys, set it on the worker, next to
@@ -443,7 +453,8 @@ CASHPILOT_CONTAINER_RUNTIME_OVERRIDES: bitping=runsc-hostnet-raw
```
The runtime applies to containers on bridge networking when they are next
-deployed; host-networked services such as Mysterium keep Docker's default.
+deployed; a service whose catalog entry names a runtime, such as Mysterium
+(`runc`), keeps that one.
`slug=runc` in the overrides runs one service under `runc`, the runtime Docker ships with; Storj is a good candidate. If the
daemon does not provide a runtime you named, the deploy is refused and the running
container is left alone. Nothing selects gVisor unless you set it
diff --git a/docs/isolation-check.md b/docs/isolation-check.md
index 90caec58..928629bb 100644
--- a/docs/isolation-check.md
+++ b/docs/isolation-check.md
@@ -295,12 +295,43 @@ What works here:
The test in section 1 still works on Docker Desktop, so you can see what an
earner reaches before and after.
-### Mysterium, which uses host networking
+### Mysterium
+
+After 2.0.6 a Mysterium node CashPilot deploys runs in its own network
+namespace on `cashpilot-isolated`, like every other earner, with its UDP range
+published ([Mysterium on the fenced bridge](isolation.md#mysterium-on-the-fenced-bridge)).
+Its customers' traffic arrives through WireGuard interfaces inside that
+namespace, and leaves it through the bridge, where the fence drops anything
+headed for your LAN, the host, a tailnet or cloud metadata. Two layers apply,
+and **Check isolation** reads both.
+
+**The host's fence, and a red line for the old setup.** The check fails with
+"These containers hold firewall capabilities in this host's own network
+namespace" for any CashPilot container on host networking that holds
+`NET_ADMIN` or `SYS_ADMIN`, or runs privileged. Such a container can rewrite
+the host's firewall, so no probe result can make it safe. A node deployed
+through 2.0.6 shows it until you redeploy it from the dashboard (UPGRADING.md).
+
+To see the fence hold from the node's own namespace, join it with a probe that
+has no capabilities. A target nothing listens on prints `blocked` with or
+without a fence, so pick targets that answer on your network (your router's
+web page, this machine's SSH on its LAN address, a tailnet DNS if you run one)
+and run the same loop from Docker's default bridge first, as the control:
-A Mysterium node runs with `network_mode: host`. Its customers' traffic does not
-leave through a Docker bridge. It arrives through WireGuard interfaces named
-`myst0`, `myst1` and so on, which exist only while a session is open, and the
-host forwards it. Two layers apply.
+```sh
+T='192.168.1.1:80 192.168.1.20:22 100.100.100.100:53 1.1.1.1:443' # yours
+P='for t in '"$T"'; do nc -z -w 3 ${t%:*} ${t#*:} && echo "REACHED $t" || echo "blocked $t"; done'
+docker run --rm --network bridge --cap-drop ALL --read-only busybox sh -c "$P" # control
+docker run --rm --network container:cashpilot-mysterium --cap-drop ALL --read-only busybox sh -c "$P"
+```
+
+Only a target the control reached counts. From the node, each of those but
+`1.1.1.1:443` should be blocked, and the fence's drop counters on the host
+should go up between a run and the next: `sudo nft list table ip cashpilot` on
+the nftables backend, or `sudo iptables-legacy -t mangle -L CP-FWD -v -n` and
+`sudo iptables-legacy -t mangle -L CP-IN -v -n` on the `iptables-legacy` one
+(the worker's log says which: "LAN fence applied (nft backend" or "(legacy
+backend").
**The node's own list.** Mysterium blocks some destinations for its customers
itself. Its default list is `10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
@@ -308,7 +339,8 @@ itself. Its default list is `10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
that Tailscale uses. Since 1.46.0 CashPilot's catalog starts the node with the
full list, `--firewall.protected.networks=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8,100.64.0.0/10,169.254.0.0/16`.
A node deployed before that keeps the default list until you redeploy it from
-the dashboard; the identity lives in its data volume and is kept. For a node
+the dashboard; the identity lives in its data volume and is kept. Inside its
+own namespace the list stays as a second layer behind the host's fence. For a node
you run outside CashPilot, add the ranges in its config file
(`/var/lib/mysterium-node/config-mainnet.toml` inside the container) and
restart it:
@@ -323,11 +355,19 @@ when a range is missing. When the next customer connects, the node's NAT rules
should list the new ranges:
```sh
-sudo iptables -t nat -S MYST | grep 240.0.0.1 # Unraid: iptables-nft -t nat -S MYST
+docker exec cashpilot-mysterium sudo -n iptables -t nat -S MYST | grep 240.0.0.1
```
-**A host rule as a backstop.** The node's list only applies while the node is
-running and configured. Add these lines to the end of the `mangle` script. The
+Those rules live in the node's own namespace, so the host's `iptables` does
+not show them. Only a node still on the host's network (2.0.6 or earlier, or
+one you run yourself that way) has them on the host:
+`sudo iptables -t nat -S MYST` (Unraid: `iptables-nft`).
+
+**A node you run yourself on host networking.** Everything from here on is
+for a node outside CashPilot that still shares the host's network. Prefer the
+setup above: on the host's network the node holds `NET_ADMIN` in the host's own
+namespace and can rewrite any rule below. The node's list only applies while
+the node is running and configured. Add these lines to the end of the `mangle` script. The
script rebuilds `EARNER-IN` on every run, so the two DNS lines are added fresh
each time, just ahead of its final drop. They are the one exception: the node
answers its customers' DNS on port 11253 of the host (its `MYST` NAT chain shows
diff --git a/docs/isolation.md b/docs/isolation.md
index aaa3945e..fec0a8ec 100644
--- a/docs/isolation.md
+++ b/docs/isolation.md
@@ -72,8 +72,9 @@ How it works:
it an earner could reach any host service listening on `[::]`.
- Every service the worker deploys onto a bridge joins that network, with DNS
servers `1.1.1.1` and `9.9.9.9` (`CASHPILOT_CONTAINER_DNS` changes them).
- Services on host networking, such as Mysterium, are not covered: see
- [Mysterium, which uses host networking](isolation-check.md#mysterium-which-uses-host-networking).
+ No catalog service runs on host networking. Mysterium joins the network
+ too, in its own network namespace: see
+ [Mysterium on the fenced bridge](#mysterium-on-the-fenced-bridge).
**It fails closed, with three conditions.** The NAT and the drops live in the
same table. When something removes the table (a reboot, `nft flush ruleset`, a
@@ -146,9 +147,101 @@ Set `CASHPILOT_ISOLATION=off` on the worker and recreate the worker, then
redeploy the earners to move them back to Docker's default bridge. There is no
off button on purpose: one click should not take the fence down.
+### Mysterium on the fenced bridge
+
+Mysterium is a VPN exit node. It needs `NET_ADMIN` to build its WireGuard
+interfaces, routes and NAT for its customers. Through 2.0.6 it ran on the host's
+network, so it held `NET_ADMIN` in the host's own namespace. From there a
+compromised node, or a malicious image update, could rewrite the host's
+firewall (this fence included), add routes and addresses, bind any port on
+every host address, and reach your LAN, your tailnet, cloud metadata and every
+other container. No host rule holds against a process that can rewrite the
+host's rules: a seccomp filter leaves netlink open, and a table owned by a
+process stops filtering when that process exits.
+
+So the node now runs in its own network namespace on `cashpilot-isolated`, like
+every other earner:
+
+- Its `NET_ADMIN` reaches only its own interfaces, routes and tables. The
+ fence matches on the bridge's interface, so whatever the node does inside
+ its namespace (flush its tables, take a LAN address, add a LAN route), what
+ it sends to your LAN, the host, a tailnet or cloud metadata is dropped on
+ the host.
+- Its UDP range (`UDP_PORTS`, 56000:56029 by default, inside the 56000:56100
+ default through 2.0.6) is published 1:1, so the forward on your router stays
+ valid. Each port goes to the same port in
+ the node, with the customer's own source address. At most 1000 ports.
+- Its WebUI listens on `0.0.0.0` inside its namespace and is published only
+ on the host addresses in `UI_ADDRESS`, `127.0.0.1` by default. The Tequila
+ API is never published.
+- Its resolvers are the bridge's (`CASHPILOT_CONTAINER_DNS`), so customers no
+ longer resolve names through a tailnet's or a LAN's DNS.
+- Its own `--firewall.protected.networks` list stays in place, as a second
+ layer inside its namespace.
+- The image is pinned by digest, and the node always runs under `runc`, since
+ gVisor has no `iptables-nft` or kernel WireGuard.
+
+The catalog refuses any service that asks for `NET_ADMIN` or `SYS_ADMIN` with
+host networking, or without an image pinned by digest, or without
+`fence_gated: true`. The worker refuses host networking and any other image for
+it, from its own catalog, whatever the request says.
+
+**Started only behind the fence.** Docker gives the node no restart policy, so
+it never starts it on its own after a reboot or a daemon restart, before the
+worker has put the table back. The worker starts it once it has read the table
+back with every rule in place: at deploy, when the worker starts, and after a
+crash. Every 30 seconds it checks again, and it stops the node when a rule is
+missing, or when the container was recreated privileged, with a capability or
+an image its catalog entry does not give it, with a restart policy, or off the
+network the mode puts it on. It then writes the table again, and starts the node on the next
+check once the table reads complete. A node you stopped from the dashboard
+stays stopped. The fleet page names a node held stopped and says why. In the
+lab, a deleted rule stopped the node within 30 seconds, and after a Docker
+restart with the table deleted the node stayed stopped until the worker had
+written the table back.
+
+**The identity is checked when it can be read.** On a redeploy of a running
+node the worker reads the node's identity before it removes the old container,
+and reads it again from the new one. If they differ, the new node is not
+running on the data volume that holds its keystore: the worker stops it and
+keeps it stopped until the next redeploy. The check needs both reads: a node
+that was already stopped, or that did not answer, leaves nothing to compare,
+and the new node runs. It is a safety net, not a backup: keep a copy of the
+keystore ([Backing up node identities](backup.md)) and check the data mount
+before a redeploy.
+
+On your own network (`CASHPILOT_CONTAINER_NETWORK`) there is no fence of
+CashPilot's to wait for: the node starts as soon as it is deployed, under your
+rules, and keeping other containers off its WebUI is up to those rules too. In
+`off` and `legacy` mode, and on Podman, the worker does not run the node at
+all: on Docker's default bridge every other earner could reach its WebUI and
+change its payout wallet, and nothing would keep it off your LAN. A deploy is
+refused before the running node is touched, with the order to follow: switch
+isolation on in Fleet, then redeploy Mysterium. A fenced node already running
+in such a mode is stopped.
+
+**A node from 2.0.6 or earlier keeps running as it is.** A Mysterium container
+on the host's network, which is what every node deployed through 2.0.6 is, is
+left exactly as it is in every mode until you redeploy it: the guard neither
+stops nor starts it, and Start, Stop and Restart work on it as before. Drift
+and **Check isolation** keep flagging it. It moves onto the fenced bridge on
+its redeploy; see UPGRADING.md.
+
### Limits
-- Host-networked services (Mysterium) are not covered.
+- The node still holds `NET_ADMIN` inside its own namespace, so the kernel's
+ netlink, nf_tables, WireGuard and tun code is reachable from it. A kernel bug
+ there would reach the host. Running the node in userspace mode without
+ `NET_ADMIN` is the next step.
+- Docker's userland proxy starts two processes per published UDP port.
+ Measured with 101 ports: 205 processes, about 250 MB of memory in total
+ (proportional set size), so about 2.5 MB per port. The default range is 30
+ ports for that reason (about 60 processes, 75 MB), which carries about 15
+ sessions at once; 20 to 30 ports is enough for a home node. A node with a
+ wider range on record keeps it until you narrow `UDP_PORTS` and redeploy;
+ a router forward for the wider range keeps working.
+- UPnP on the router no longer works for the node, since the fence drops the
+ router. Forward the range by hand.
- IPv6: the fenced network has IPv6 off. On Docker 26 and later the earners
get no IPv6 address at all; on older engines they get a link-local one,
which the IPv6 table drops. A host booted with IPv6 disabled gets no IPv6
@@ -190,9 +283,10 @@ allowed through: `GET /api/fleet/isolation-guide`.
| **needs exceptions** | It can be confined, but specific things must stay reachable. |
| **not isolatable** | It runs on the host network, so a container bridge does not apply. Use a VLAN or a separate machine. |
-Today `mysterium` is the third case (it declares `network_mode: "host"`), and
-`storj` is the second: it needs its inbound port **and** its local dashboard
-reachable, because that dashboard is where CashPilot reads its earnings.
+No catalog service is the third case today. `storj` is the second: it needs
+its inbound port **and** its local dashboard reachable, because that dashboard
+is where CashPilot reads its earnings. So is `mysterium`, whose UDP range must
+accept inbound packets.
## Running your own rules
diff --git a/docs/security-defaults.md b/docs/security-defaults.md
index 019d7b83..cd1cfcea 100644
--- a/docs/security-defaults.md
+++ b/docs/security-defaults.md
@@ -117,8 +117,8 @@ dangerous containers, not only dangerous URLs:
container update, prune, image push or load. The allow list is the set of
calls the live test recorded from the worker, its update helper and Compose.
- **Containers.** No privileged mode, no host PID, IPC or user namespace
- (host networking and the host cgroup namespace are allowed, see *What a
- compromised worker can still do* below), `no-new-privileges` required, no unconfined seccomp, AppArmor or
+ (host networking is allowed, see *What a compromised worker can still do*
+ below; the host cgroup namespace is not), `no-new-privileges` required, no unconfined seccomp, AppArmor or
system paths. Added capabilities only from `NET_ADMIN`, `NET_RAW`, `SETUID`,
`SETGID` (what the catalog declares), devices only `/dev/net/tun`, runtimes
only `runc` and the gVisor ones (the worker refuses any other runtime itself,
@@ -206,10 +206,13 @@ stands between the worker and root.
network aliases, which sockguard always lets through. It adds little reach, since the worker
can already start a container on host networking, but a compromised worker
can take an earner off the fence.
-- Start a container in the host's cgroup namespace. The Activity reader of a
- host-network earner such as Mysterium needs it to tell that earner's
- sockets from the host's. The cgroup tree it shows is read-only and names
- the host's containers and services, which the worker can list anyway.
+- Start a container on host networking. The LAN fence's helper and the
+ isolation check's control probe need it. The worker itself refuses host
+ networking to every earner, and the catalog refuses it to any service with
+ `NET_ADMIN`, so only those two use it. The proxy no longer allows the host's
+ cgroup namespace: only the Activity reader of a host-network earner needed
+ it, and no earner shares the host's network since Mysterium moved onto the
+ fenced bridge.
- Pull images from any registry (custom services need it) and set namespaced
sysctls on its own containers (the tunnel sidecar needs it).
diff --git a/docs/whats-new-2.0.md b/docs/whats-new-2.0.md
index ddf0bfcc..1aedc4f8 100644
--- a/docs/whats-new-2.0.md
+++ b/docs/whats-new-2.0.md
@@ -82,7 +82,11 @@ network and writes one firewall table that drops traffic from them to:
An upgraded install turns it on with **Turn on isolation** on the Fleet page,
and **Check isolation** starts two short-lived containers to test that it holds.
-Limits: Mysterium runs on host networking and is not covered by the fence. A
+Limits: through 2.0.6 Mysterium ran on host networking and was not covered by
+the fence; later releases move it onto the fenced bridge in its own network
+namespace on its next redeploy, and run it only with isolation on or on your
+own network
+([Mysterium on the fenced bridge](isolation.md#mysterium-on-the-fenced-bridge)). A
second WAN or extra public IPs are not dropped. Docker Desktop, rootless Docker,
and Unraid are not tested yet. On Podman the fence is not available yet: a
worker there deploys on Podman's default network and the Fleet page says so.
diff --git a/pyproject.toml b/pyproject.toml
index 55806f0d..43e611a8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -93,5 +93,6 @@ addopts = "-m 'not live'"
markers = [
"real_custom_images: use the real database.get_custom_images instead of the empty default",
"real_state_backups: use the real database.get_state_backups instead of the empty default",
+ "real_gated_mode: keep the rule that a fence-gated service runs only in on or operator isolation mode",
"live: hits a real provider API; needs credentials; excluded from normal runs",
]
diff --git a/services/bandwidth/mysterium.yml b/services/bandwidth/mysterium.yml
index 2ead2916..39a9a526 100644
--- a/services/bandwidth/mysterium.yml
+++ b/services/bandwidth/mysterium.yml
@@ -7,8 +7,9 @@ description: >
MystNodes (Mysterium Network) is a decentralized VPN and proxy network built on
blockchain technology. You earn MYST tokens by running a node that provides VPN,
proxy, and data scraping services to users. Requires NET_ADMIN, SETUID and SETGID capabilities and
- host networking for full functionality. Includes a built-in web UI for node
- management. Works on both residential and VPS connections.
+ a TUN device, which it uses inside its own network namespace on the fenced
+ earner bridge. Includes a built-in web UI for node management. Works on both
+ residential and VPS connections.
short_description: Decentralized VPN node - blockchain-based, works on VPS
referral:
@@ -16,7 +17,12 @@ referral:
code: "do7v7YOoBBpbOstKQovX2pUvZYKia4ZhH3QIdNtE"
docker:
- image: mysteriumnetwork/myst
+ # Pinned by digest: the node holds NET_ADMIN, so the catalog refuses it any
+ # image it has not named exactly (app/catalog.py). Bump tag and digest together.
+ image: "mysteriumnetwork/myst:1.39.9-alpine@sha256:f89347d244efeeaf29555e876404cf8aebcd41561c5cbed06c365ba64beb8564"
+ # gVisor cannot run this node (no iptables-nft, no kernel WireGuard), so an
+ # operator default of CASHPILOT_CONTAINER_RUNTIME=runsc must not reach it.
+ runtime: runc
platforms: [linux/amd64, linux/arm64, linux/arm/v7]
resources:
mem_limit: "768m"
@@ -27,13 +33,13 @@ docker:
required: false
secret: false
description: >-
- Where the node's WebUI (port 4449) listens. 127.0.0.1 keeps it on this
- machine. To reach it through a reverse proxy, add the address the proxy
- connects to, comma separated with no spaces: 127.0.0.1,172.18.0.1.
+ The host addresses the node's WebUI (port 4449) is published on.
+ 127.0.0.1 keeps it on this machine. To reach it through a reverse proxy,
+ add the address the proxy connects to, comma separated with no spaces:
+ 127.0.0.1,172.18.0.1.
default: "127.0.0.1"
- # The value lands on the node's command line, so anything but addresses
- # would become an argument: "127.0.0.1, 10.0.0.1" puts a stray word before
- # `service`, the node refuses to start and stops earning.
+ # Each address becomes one host binding of port 4449 (ui_port below), so
+ # anything but addresses would fail the deploy.
# Each octet 0 to 255, ASCII digits only ([0-9], not \d, which in Python also
# matches other scripts' digits): 256.0.0.1 is not an address, and the WebUI
# listener would fail to bind while the node kept running.
@@ -44,15 +50,32 @@ docker:
secret: false
description: >-
The UDP ports the node uses for VPN sessions, as start:end. Forward this
- same range on your router.
- default: "56000:56100"
- # Without it the node picks from 10000:60000, so a forwarded 56000-56100
- # is never used and every session falls back to hole punching.
+ same range on your router. Each session takes about two ports, so the
+ default 30 carry about 15 sessions at once, more than a home node gets.
+ Every published port costs about 2.5 MB of memory in Docker's port
+ proxy, so a wider range only costs more. A forward you already have for
+ a wider range that contains this one (56000-56100) keeps working.
+ default: "56000:56029"
+ # Without it the node picks from 10000:60000, so a forwarded range is
+ # never used and every session falls back to hole punching. 30 ports, a
+ # sub-range of the 56000:56100 default through 2.0.6, so the router
+ # forwards made for that one stay valid.
# ASCII digits only, as above; port_range checks start <= end <= 65535.
pattern: '[0-9]{4,5}:[0-9]{4,5}'
port_range: true
- ports:
- - "4449:4449"
+ # No static ports. The WebUI is published on each UI_ADDRESS address, and the
+ # VPN range 1:1 on every address, from the env (app/catalog.py).
+ ui_port:
+ env: UI_ADDRESS
+ port: 4449
+ udp_port_range:
+ env: UDP_PORTS
+ max_ports: 1000
+ # Docker never starts this node on its own (restart policy "no"). The worker
+ # starts it only after it has confirmed the LAN fence is in place, at deploy,
+ # after a reboot and after a crash, and stops it within 30 s if the fence or
+ # the container's shape changes (app/lan_fence.py guard).
+ fence_gated: true
volumes:
- "mysterium-data:/var/lib/mysterium-node"
critical_volumes:
@@ -60,19 +83,20 @@ docker:
holds: >-
Node identity keystore. Destroying it creates a brand-new identity: the
node loses its reputation and any earnings still held against the old one.
- # Loopback by default, on purpose. With host networking, 0.0.0.0 publishes
- # the node's WebUI (4449) and Tequila API (4050) on every interface of the
- # machine, and nothing off the machine needs them: the worker reads the
- # node's earnings from inside the container (collector.node_read below), not
- # over the network. Reach the WebUI from
- # another machine over an SSH tunnel (ssh -L 4449:127.0.0.1:4449 ), or
- # put a reverse proxy in front by adding the one address it connects to in
- # UI_ADDRESS (the node binds every address in the comma-separated list). The
- # Tequila API stays on loopback: the WebUI proxies to it from inside the node.
+ # The node runs in its own network namespace on the fenced earner bridge, so
+ # 0.0.0.0 inside it reaches only that namespace: the WebUI (4449) is
+ # reachable from the host only where ui_port publishes it, loopback by
+ # default. Nothing off the machine needs it: the worker reads the node's
+ # earnings from inside the container (collector.node_read below). Reach the
+ # WebUI from another machine over an SSH tunnel (ssh -L 4449:127.0.0.1:4449
+ # ), or add the one address a reverse proxy connects to in UI_ADDRESS.
+ # The Tequila API stays on the node's own loopback: the WebUI proxies to it
+ # from inside the node.
# --firewall.protected.networks: the node blackholes these for its customers.
# Its default leaves out 100.64.0.0/10 (CGNAT and Tailscale) and
- # 169.254.0.0/16 (cloud metadata), so a VPN customer could reach the tailnet.
- command: "--ui.address=${UI_ADDRESS} --tequilapi.address=127.0.0.1 --firewall.protected.networks=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8,100.64.0.0/10,169.254.0.0/16 --udp.ports=${UDP_PORTS} service --agreed-terms-and-conditions"
+ # 169.254.0.0/16 (cloud metadata). It stays as a second layer inside the
+ # node's namespace; the host's fence is the first.
+ command: "--ui.address=0.0.0.0 --tequilapi.address=127.0.0.1 --firewall.protected.networks=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8,100.64.0.0/10,169.254.0.0/16 --udp.ports=${UDP_PORTS} service --agreed-terms-and-conditions"
# The node's own firewall for its customers, read by the worker's isolation
# check: the ranges in this flag of `command` go to the nat chain MYST, which
# sends each one to the blackhole address.
@@ -80,7 +104,6 @@ docker:
flag: "--firewall.protected.networks="
chain: MYST
blackhole: "240.0.0.1"
- network_mode: "host"
# SETUID and SETGID are for sudo. The node runs as root and still configures
# its interface and firewall through `sudo ip ...` / `sudo iptables ...`, and
# sudo switches uid and gid on the way (setresuid, setgroups). Every container
@@ -88,6 +111,11 @@ docker:
# "sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted", every
# session dies at setup, and the node keeps registering and looking healthy.
# no-new-privileges stays on: the node is already root, so it blocks nothing here.
+ # NET_ADMIN reaches only the node's own network namespace: its tun and wg
+ # interfaces, its routes and its own NAT. It cannot touch the host's
+ # firewall, routes or interfaces, and the host's fence on the bridge drops
+ # what the node sends to the LAN, the host, the tailnet and cloud metadata
+ # whatever the node does to its own tables.
cap_add: [NET_ADMIN, SETUID, SETGID]
# Wireguard/dvpn need a TUN device. Without it the node starts, registers and
# advertises itself to the network but carries no traffic - it looks healthy
@@ -147,7 +175,7 @@ earnings:
monthly_high: 10
currency: USD
per: "device"
- notes: "Earnings in MYST tokens. Residential IPs earn significantly more. Node WebUI at port 4449 for management. VPS accepted. Port forwarding UDP 56000-56100 (the UDP_PORTS default) strongly recommended -- without it, nodes get Strict NAT status and many VPN sessions fail, severely reducing income. Important: after first run, set your beneficiary (settlement) wallet via the node WebUI or CLI to match your mystnodes.com account -- this links on-chain earnings to your cloud dashboard."
+ notes: "Earnings in MYST tokens. Residential IPs earn significantly more. Node WebUI at port 4449 (loopback by default) for management. VPS accepted. Port forwarding UDP 56000-56029 (the UDP_PORTS default) strongly recommended -- without it, nodes get Strict NAT status and many VPN sessions fail, severely reducing income. Important: after first run, set your beneficiary (settlement) wallet via the node WebUI or CLI to match your mystnodes.com account -- this links on-chain earnings to your cloud dashboard."
cashout:
method: auto
diff --git a/tests/conftest.py b/tests/conftest.py
index 5f1553ce..2df51c94 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -50,6 +50,24 @@ def _no_custom_images(request):
yield
+@pytest.fixture(autouse=True)
+def _gated_services_run_in_any_mode(request):
+ """A fence-gated service (Mysterium) deploys in the suite's ``off`` mode.
+
+ The worker refuses one outside ``on`` and ``operator`` (lan_fence.mode_problem),
+ and the suite runs with isolation off (above), where tests about volumes,
+ mounts or capabilities deploy Mysterium as their example. Tests marked
+ ``real_gated_mode`` get the real rule; tests/test_mysterium_fenced.py is.
+ """
+ if request.node.get_closest_marker("real_gated_mode"):
+ yield
+ return
+ from unittest.mock import patch
+
+ with patch("app.lan_fence.mode_problem", return_value=None):
+ yield
+
+
@pytest.fixture(autouse=True)
def _no_state_backups(request):
"""No backup is confirmed, unless a test asks for the real table.
diff --git a/tests/test_beads_batch_56.py b/tests/test_beads_batch_56.py
index 1e848f72..b32c7f34 100644
--- a/tests/test_beads_batch_56.py
+++ b/tests/test_beads_batch_56.py
@@ -404,6 +404,8 @@ async def fake_deploy(worker_id, slug, spec):
patch("app.main.database.get_deployment_spec", new_callable=AsyncMock, return_value=None),
patch("app.main.database.save_deployment", new_callable=AsyncMock),
patch("app.main.database.record_health_event", new_callable=AsyncMock),
+ # Mysterium deploys only to a worker that runs it behind the fence.
+ patch("app.main._worker_features", new_callable=AsyncMock, return_value={"fence_gated"}),
):
asyncio.run(main_module.api_deploy(_FakeRequest(), "mysterium", body, worker_id=1))
diff --git a/tests/test_catalog_devices.py b/tests/test_catalog_devices.py
index 4fea003f..d280f73d 100644
--- a/tests/test_catalog_devices.py
+++ b/tests/test_catalog_devices.py
@@ -55,7 +55,10 @@ def test_an_unknown_slug_is_denied_not_defaulted(self):
class TestValidationRefuses:
def _spec(self, **kw):
- return DeploySpec(image="x", **kw)
+ # Mysterium's pinned image: the worker refuses it any other.
+ from app import catalog
+
+ return DeploySpec(image=catalog.get_service("mysterium")["docker"]["image"], **kw)
def test_the_declared_device_is_accepted(self):
_validate_deploy_spec(self._spec(devices=["/dev/net/tun"]), slug="mysterium")
diff --git a/tests/test_compose_unfilled_paths.py b/tests/test_compose_unfilled_paths.py
index 651051b6..eab10dfb 100644
--- a/tests/test_compose_unfilled_paths.py
+++ b/tests/test_compose_unfilled_paths.py
@@ -39,8 +39,8 @@ def test_supplied_paths_are_written_as_they_are():
def test_a_real_named_volume_is_still_declared():
- compose = yaml.safe_load(compose_generator.generate_compose_single("mysterium"))
- assert "mysterium-data" in compose["volumes"]
+ compose = yaml.safe_load(compose_generator.generate_compose_single("bitping"))
+ assert "bitping-data" in compose["volumes"]
def test_the_message_cannot_break_out_of_the_variable():
diff --git a/tests/test_container_runtime.py b/tests/test_container_runtime.py
index 7f241607..31c1cdd1 100644
--- a/tests/test_container_runtime.py
+++ b/tests/test_container_runtime.py
@@ -57,7 +57,7 @@ def test_set_applies_to_a_bridge_networked_service(self):
@pytest.mark.parametrize("mode", ["host", "none"])
def test_host_and_none_networking_keep_dockers_default(self, mode):
- kwargs = _deploy(_client(), default="runsc-hostnet", slug="mysterium", network_mode=mode)
+ kwargs = _deploy(_client(), default="runsc-hostnet", slug="honeygain", network_mode=mode)
assert kwargs["runtime"] is None
def test_a_runtime_named_in_the_request_wins(self):
diff --git a/tests/test_lan_fence.py b/tests/test_lan_fence.py
index 363c65d7..11c706d3 100644
--- a/tests/test_lan_fence.py
+++ b/tests/test_lan_fence.py
@@ -693,7 +693,7 @@ def test_host_networking_is_never_moved(self):
patch.object(orchestrator, "_get_client", return_value=client),
patch.object(lan_fence, "run_helper") as helper,
):
- orchestrator.deploy_raw(slug="mysterium", image="mysteriumnetwork/myst", network_mode="host")
+ orchestrator.deploy_raw(slug="hostnode", image="example/hostnode", network_mode="host")
assert client.containers.run.call_args.kwargs["network_mode"] == "host"
helper.assert_not_called()
diff --git a/tests/test_lan_isolation.py b/tests/test_lan_isolation.py
index 2c19d4b4..63e91c19 100644
--- a/tests/test_lan_isolation.py
+++ b/tests/test_lan_isolation.py
@@ -71,10 +71,20 @@ def test_most_of_the_catalog_is_undocumented_and_says_so(self):
class TestWhatCanActuallyBeIsolated:
def test_a_host_networked_service_cannot_be_confined_by_a_bridge(self):
- out = li.assess(catalog.get_service("mysterium"))
+ svc = {"slug": "x", "name": "X", "docker": {"network_mode": "host"}}
+ out = li.assess(svc)
assert out["verdict"] == li.NOT_ISOLATABLE
assert "VLAN or a separate machine" in out["summary"]
+ def test_mysterium_is_fenced_with_its_udp_range_as_the_exception(self):
+ """After 2.0.6 it runs on the fenced bridge in its own namespace."""
+ out = li.assess(catalog.get_service("mysterium"))
+ assert out["verdict"] == li.NEEDS_EXCEPTIONS
+ details = " ".join(e["detail"] for e in out["exceptions"])
+ assert "56000:56029/udp" in details
+ assert "host network" not in details
+ assert "4449" not in details, "the WebUI is published on loopback by default"
+
def test_a_service_needing_inbound_ports_needs_exceptions(self):
"""Missing these costs a storage node its held payout balance."""
out = li.assess(catalog.get_service("storj"))
@@ -158,7 +168,7 @@ def test_deploy_risk_returns_both_halves(self):
out = self._call(main.api_deploy_risk, "mysterium")
assert out["attribution"]["documented"] is True
- assert out["isolation"]["verdict"] == li.NOT_ISOLATABLE
+ assert out["isolation"]["verdict"] == li.NEEDS_EXCEPTIONS
def test_deploy_risk_for_a_storage_node_has_no_attribution_notice(self):
from app import main
@@ -179,7 +189,8 @@ def test_the_guide_sorts_services_into_the_three_buckets(self):
out = self._call(main.api_isolation_guide)
assert "honeygain" in out["isolatable"]
- assert any(s["slug"] == "mysterium" for s in out["not_isolatable"])
+ assert any(s["slug"] == "mysterium" for s in out["needs_exceptions"])
+ assert not out["not_isolatable"], "no catalog service shares the host's network"
assert any(s["slug"] == "storj" for s in out["needs_exceptions"])
def test_the_guide_hands_back_a_pasteable_snippet(self):
diff --git a/tests/test_mysterium_fenced.py b/tests/test_mysterium_fenced.py
new file mode 100644
index 00000000..faf06ec7
--- /dev/null
+++ b/tests/test_mysterium_fenced.py
@@ -0,0 +1,983 @@
+"""Mysterium is fenced like every other earner, and Docker never starts it unfenced.
+
+Through 2.0.6 the node ran with ``network_mode: host`` and NET_ADMIN. That put
+NET_ADMIN in the host's own network namespace: the node, or a malicious image
+update, could rewrite the host's firewall (the LAN fence included), add routes
+and addresses, bind any port on every host address and reach the LAN, the
+tailnet, cloud metadata and every other container. No host rule holds against
+a process that can rewrite the host's rules.
+
+Now the node runs in its own namespace on the fenced bridge, with its UDP range
+published 1:1, so its NET_ADMIN reaches only its own tables and the host's
+fence drops what it sends inward whatever it does to them. Docker gives it no
+restart policy: the worker starts it only once the fence reads complete, and
+stops it within 30 s if the fence or the container's shape goes.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from contextlib import asynccontextmanager
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
+
+os.environ.setdefault("CASHPILOT_API_KEY", "test-fleet-key")
+
+import pytest
+import yaml
+from fastapi import HTTPException
+from fastapi.testclient import TestClient
+
+from app import catalog, drift, isolation_check, lan_fence, orchestrator, socket_policy, worker_api
+from app.worker_api import DeploySpec, _validate_deploy_spec
+
+#: The mode rule (lan_fence.mode_problem) is part of what these tests check.
+pytestmark = pytest.mark.real_gated_mode
+
+ROOT = Path(__file__).resolve().parent.parent
+MYST = catalog.get_service("mysterium")
+PINNED = MYST["docker"]["image"]
+LABELS = {"cashpilot.service": "mysterium", "cashpilot.managed": "true"}
+
+
+def _entry(**docker):
+ entry = {"name": "X", "slug": "x", "category": "bandwidth", "status": "active", "description": "x"}
+ entry["docker"] = {"image": "x/x@sha256:" + "a" * 64, "fence_gated": True, **docker}
+ return entry
+
+
+# ---------------------------------------------------------------------------
+# (a) The catalog: no host network, and the rule that keeps the threat class out
+# ---------------------------------------------------------------------------
+
+
+class TestTheCatalog:
+ def test_mysterium_no_longer_shares_the_hosts_network(self):
+ raw = yaml.safe_load((ROOT / "services/bandwidth/mysterium.yml").read_text())["docker"]
+ assert "network_mode" not in raw
+ assert raw["fence_gated"] is True
+ assert raw["runtime"] == "runc"
+ assert "4449:4449" not in [str(p) for p in raw.get("ports") or []]
+
+ def test_its_image_is_pinned_by_digest(self):
+ assert "@sha256:" in PINNED and PINNED.startswith("mysteriumnetwork/myst:")
+
+ def test_its_own_firewall_stays_as_the_second_layer(self):
+ cmd = MYST["docker"]["command"]
+ assert "--ui.address=0.0.0.0 " in cmd
+ assert "--tequilapi.address=127.0.0.1 " in cmd
+ assert "100.64.0.0/10" in cmd and "169.254.0.0/16" in cmd
+
+ def test_net_admin_on_the_host_network_is_refused(self, tmp_path):
+ errors = catalog._validate(_entry(cap_add=["NET_ADMIN"], network_mode="host"), tmp_path / "x.yml")
+ assert any("cannot use network_mode: host" in e for e in errors)
+
+ def test_sys_admin_is_held_to_the_same_rule(self, tmp_path):
+ errors = catalog._validate(_entry(cap_add=["CAP_SYS_ADMIN"], network_mode="host"), tmp_path / "x.yml")
+ assert any("cannot use network_mode: host" in e for e in errors)
+
+ def test_net_admin_needs_a_digest(self, tmp_path):
+ errors = catalog._validate(_entry(cap_add=["NET_ADMIN"], image="x/x:1.0"), tmp_path / "x.yml")
+ assert any("@sha256:" in e for e in errors)
+
+ def test_net_admin_needs_the_fence_gate(self, tmp_path):
+ errors = catalog._validate(_entry(cap_add=["NET_ADMIN"], fence_gated=False), tmp_path / "x.yml")
+ assert any("fence_gated" in e for e in errors)
+
+ def test_control_a_pinned_fenced_net_admin_service_loads(self, tmp_path):
+ assert catalog._validate(_entry(cap_add=["NET_ADMIN"]), tmp_path / "x.yml") == []
+
+ def test_a_udp_range_must_name_a_port_range_env(self, tmp_path):
+ bad = _entry(env=[{"key": "P", "default": "1:2"}], udp_port_range={"env": "P"})
+ assert any("port_range" in e for e in catalog._validate(bad, tmp_path / "x.yml"))
+ good = _entry(env=[{"key": "P", "default": "1:2", "port_range": True}], udp_port_range={"env": "P"})
+ assert catalog._validate(good, tmp_path / "x.yml") == []
+
+ def test_a_udp_range_cap_above_the_ceiling_is_refused(self, tmp_path):
+ bad = _entry(
+ env=[{"key": "P", "default": "1:2", "port_range": True}],
+ udp_port_range={"env": "P", "max_ports": 5000},
+ )
+ assert any("max_ports" in e for e in catalog._validate(bad, tmp_path / "x.yml"))
+
+
+# ---------------------------------------------------------------------------
+# (b) The UDP range, published 1:1
+# ---------------------------------------------------------------------------
+
+
+class TestTheUdpRange:
+ def test_the_default_range_is_30_ports_each_on_itself(self):
+ default = next(e["default"] for e in MYST["docker"]["env"] if e["key"] == "UDP_PORTS")
+ ports = catalog.expand_udp_range(default)
+ # About two ports per session: 15 at once, and about 2.5 MB of
+ # docker-proxy memory per port, so a wider default only costs memory.
+ assert len(ports) == 30
+ assert ports["56000/udp"] == 56000 and ports["56029/udp"] == 56029
+
+ def test_the_default_stays_inside_the_range_routers_forward_since_2_0_6(self):
+ default = next(e["default"] for e in MYST["docker"]["env"] if e["key"] == "UDP_PORTS")
+ assert set(catalog.expand_udp_range(default)) <= set(catalog.expand_udp_range("56000:56100"))
+
+ @pytest.mark.parametrize("value", ["56100:56000", "1:70000", "56000", "a:b", "56000-56100"])
+ def test_anything_but_a_range_is_refused(self, value):
+ with pytest.raises(ValueError):
+ catalog.expand_udp_range(value)
+
+ def test_a_range_over_the_cap_is_refused(self):
+ with pytest.raises(ValueError, match="at most 1000"):
+ catalog.expand_udp_range("10000:11000")
+ assert len(catalog.expand_udp_range("10000:10999")) == 1000
+
+ def test_the_webui_addresses_become_bindings(self):
+ assert catalog.ui_bindings("localhost,172.18.0.1", 4449) == [["127.0.0.1", 4449], ["172.18.0.1", 4449]]
+ with pytest.raises(ValueError):
+ catalog.ui_bindings("::1", 4449)
+
+
+# ---------------------------------------------------------------------------
+# (c) The deploy: a node recorded on the host network moves onto the bridge
+# ---------------------------------------------------------------------------
+
+
+@asynccontextmanager
+async def _noop_lifespan(a):
+ yield
+
+
+@pytest.fixture
+def ui():
+ from app.main import app
+
+ app.router.lifespan_context = _noop_lifespan
+ with TestClient(app, raise_server_exceptions=False) as c:
+ yield c
+
+
+def _ui_deploy(client, body, recorded=None, features=("fence_gated",)):
+ worker = {
+ "id": 1,
+ "name": "w1",
+ "status": "online",
+ "url": "http://192.168.1.10:8081",
+ "system_info": json.dumps({"features": list(features)}),
+ }
+ sent: dict = {}
+
+ async def _fake_deploy(worker_id, slug, spec):
+ sent.update(spec)
+ return {"container_id": "abc123"}
+
+ with (
+ patch("app.main.auth.get_current_user", return_value={"uid": 1, "u": "admin", "r": "owner"}),
+ patch("app.main.database.list_workers", new_callable=AsyncMock, return_value=[worker]),
+ patch("app.main.database.get_worker", new_callable=AsyncMock, return_value=worker),
+ patch("app.main.database.get_deployment_spec", new_callable=AsyncMock, return_value=recorded),
+ patch("app.main._proxy_worker_deploy", side_effect=_fake_deploy),
+ patch("app.main.database.save_deployment", new_callable=AsyncMock),
+ patch("app.main.database.record_health_event", new_callable=AsyncMock),
+ patch("app.main._run_collection", new_callable=AsyncMock),
+ ):
+ resp = client.post("/api/deploy/mysterium", json=body)
+ return resp, sent
+
+
+#: What every Mysterium deployed through 2.0.6 has on record.
+HOST_RECORD = {
+ "image": "mysteriumnetwork/myst",
+ "env": {"UI_ADDRESS": "127.0.0.1", "UDP_PORTS": "56000:56100"},
+ "network_mode": "host",
+ "ports": {"4449/tcp": 4449},
+ "volumes": {"/mnt/appdata/myst/data": {"bind": "/var/lib/mysterium-node", "mode": "rw"}},
+}
+
+
+class TestTheDeploy:
+ def test_a_first_deploy_is_on_the_bridge_with_its_range_and_loopback_webui(self, ui):
+ resp, sent = _ui_deploy(ui, {"env": {}})
+ assert resp.status_code == 200, resp.text
+ assert sent["network_mode"] is None
+ assert sum(k.endswith("/udp") for k in sent["ports"]) == 30
+ assert sent["ports"]["4449/tcp"] == ["127.0.0.1", 4449]
+
+ def test_a_node_recorded_on_the_host_network_moves_onto_the_bridge(self, ui):
+ resp, sent = _ui_deploy(ui, {"env": {}}, recorded=HOST_RECORD)
+ assert resp.status_code == 200, resp.text
+ assert sent["network_mode"] is None, "the record must not carry NET_ADMIN back into the host's namespace"
+ # 4449 on every interface was the host-network record; now loopback only.
+ assert sent["ports"]["4449/tcp"] == ["127.0.0.1", 4449]
+ # The range on record is kept, so the router's forward still matches.
+ assert sum(k.endswith("/udp") for k in sent["ports"]) == 101
+ # The identity lives in the data directory, which the redeploy keeps.
+ assert sent["volumes"] == HOST_RECORD["volumes"]
+ kept = " ".join(resp.json().get("kept_from_previous_deployment") or [])
+ assert "fenced bridge" in kept and "keeping the deployed value" not in kept
+
+ def test_a_worker_too_old_to_gate_it_is_refused_and_the_node_left_alone(self, ui):
+ resp, sent = _ui_deploy(ui, {"env": {}}, recorded=HOST_RECORD, features=("instances",))
+ assert resp.status_code == 409
+ assert "Update the worker" in resp.json()["detail"]
+ assert not sent
+
+ def test_another_image_is_refused(self, ui):
+ resp, sent = _ui_deploy(ui, {"env": {}, "image": "mysteriumnetwork/myst:latest"})
+ assert resp.status_code == 400
+ assert not sent
+
+
+# ---------------------------------------------------------------------------
+# The worker checks its own catalog, never the request
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def real_catalog(monkeypatch):
+ monkeypatch.setattr(worker_api, "_catalog_get_services", catalog.get_services)
+
+
+class TestTheWorkerRefuses:
+ def test_mysterium_on_the_host_network(self, real_catalog):
+ with pytest.raises(HTTPException) as refused:
+ _validate_deploy_spec(DeploySpec(image=PINNED, network_mode="host"), "mysterium")
+ assert refused.value.status_code == 403
+
+ def test_mysterium_on_another_image(self, real_catalog):
+ with pytest.raises(HTTPException) as refused:
+ _validate_deploy_spec(DeploySpec(image="mysteriumnetwork/myst:latest"), "mysterium")
+ assert refused.value.status_code == 403
+
+ def test_more_ports_than_a_range_can_need(self, real_catalog):
+ ports = {f"{p}/udp": p for p in range(20000, 21100)}
+ with pytest.raises(HTTPException) as refused:
+ _validate_deploy_spec(DeploySpec(image="x", ports=ports), "honeygain")
+ assert refused.value.status_code == 400
+
+ def test_control_the_catalog_spec_passes(self, real_catalog):
+ docker = MYST["docker"]
+ ports = {**catalog.expand_udp_range("56000:56100"), "4449/tcp": [("127.0.0.1", 4449), ("10.0.0.2", 4449)]}
+ spec = DeploySpec(image=PINNED, cap_add=docker["cap_add"], devices=docker["devices"], ports=ports)
+ _validate_deploy_spec(spec, "mysterium")
+ assert spec.ports["4449/tcp"] == [("127.0.0.1", 4449), ("10.0.0.2", 4449)]
+
+
+# ---------------------------------------------------------------------------
+# (d) deploy_raw: no restart policy, started only behind a confirmed fence
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def markers(tmp_path, monkeypatch):
+ monkeypatch.setattr(orchestrator, "STOPPED_MARKER", str(tmp_path / "stopped"))
+ monkeypatch.setattr(orchestrator, "IDENTITY_DIR", str(tmp_path / "identity"))
+ return tmp_path
+
+
+def _raw_deploy(problem=None, slug="mysterium", old=None, mode=lan_fence.MODE_ON, **kwargs):
+ real = lan_fence.mode_problem
+ created = MagicMock(id="new", short_id="new")
+ client = MagicMock()
+ if old is None:
+ client.containers.get.side_effect = orchestrator.NotFound("nope")
+ else:
+ client.containers.get.return_value = old
+ client.containers.create.return_value = created
+ client.containers.run.return_value = created
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch.object(orchestrator, "_CONTAINER_RUNTIME", "runsc"),
+ patch.object(orchestrator, "available_runtimes", return_value={"runc", "runsc"}),
+ patch("app.lan_fence.fence_problem", return_value=problem),
+ patch("app.lan_fence.mode_problem", side_effect=lambda _mode, service="this service": real(mode, service)),
+ ):
+ orchestrator.deploy_raw(slug=slug, image=PINNED, **kwargs)
+ return client, created
+
+
+class TestDeployRaw:
+ def test_docker_never_starts_it_on_its_own(self, markers):
+ client, created = _raw_deploy()
+ kwargs = client.containers.create.call_args.kwargs
+ assert kwargs["restart_policy"] == {"Name": "no"}
+ client.containers.run.assert_not_called()
+ created.start.assert_called_once()
+
+ def test_runc_wins_over_an_operator_default_of_gvisor(self, markers):
+ client, _ = _raw_deploy()
+ assert client.containers.create.call_args.kwargs["runtime"] == "runc"
+
+ def test_without_the_fence_nothing_starts(self, markers):
+ created = MagicMock(id="new", short_id="new")
+ client = MagicMock()
+ client.containers.get.side_effect = orchestrator.NotFound("nope")
+ client.containers.create.return_value = created
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value="missing"),
+ patch("app.lan_fence.mode_problem", return_value=None),
+ pytest.raises(orchestrator.ContainerNetworkError),
+ ):
+ orchestrator.deploy_raw(slug="mysterium", image=PINNED)
+ created.start.assert_not_called()
+ client.containers.run.assert_not_called()
+
+ @pytest.mark.parametrize("mode", [lan_fence.MODE_LEGACY, lan_fence.MODE_OFF])
+ def test_on_the_default_bridge_it_is_refused_before_the_old_node_is_touched(self, markers, mode):
+ old = MagicMock()
+ old.name = "cashpilot-mysterium"
+ old.labels = dict(LABELS)
+ with pytest.raises(orchestrator.ContainerNetworkError, match="not touched"):
+ _raw_deploy(old=old, mode=mode)
+ old.stop.assert_not_called()
+ old.remove.assert_not_called()
+
+ def test_control_on_the_operators_own_network_it_deploys(self, markers):
+ client, created = _raw_deploy(mode=lan_fence.MODE_OPERATOR)
+ created.start.assert_called_once()
+
+ def test_control_an_ordinary_earner_keeps_unless_stopped(self, markers):
+ client, _ = _raw_deploy(slug="honeygain")
+ assert client.containers.run.call_args.kwargs["restart_policy"] == {"Name": "unless-stopped"}
+
+ def test_the_identity_is_read_before_the_old_node_goes(self, markers):
+ old = MagicMock()
+ old.name = "cashpilot-mysterium"
+ old.status = "running"
+ old.labels = dict(LABELS)
+ old.attrs = {"Mounts": []}
+ old.exec_run.return_value = (0, b'{"identities":[{"id":"0x' + b"a" * 40 + b'"}]} {"id":"0x' + b"a" * 40 + b'"}')
+ _raw_deploy(old=old)
+ assert (markers / "identity" / "cashpilot-mysterium").read_text() == "0x" + "a" * 40
+
+
+# ---------------------------------------------------------------------------
+# (e) The socket proxy no longer lets anything join the host's cgroup namespace
+# ---------------------------------------------------------------------------
+
+
+def test_the_socket_policy_keeps_host_cgroupns_off():
+ create = yaml.safe_load(socket_policy.policy())["request_body"]["container_create"]
+ assert create["allow_host_cgroupns"] is False
+ assert create["allow_privileged"] is False
+
+
+# ---------------------------------------------------------------------------
+# (f) Drift: a node still on the host network is flagged, whatever its record says
+# ---------------------------------------------------------------------------
+
+
+#: The 30-port default range, published 1:1 as the worker does.
+UDP_BINDINGS = {f"{p}/udp": [{"HostIp": "", "HostPort": str(p)}] for p in range(56000, 56030)}
+
+
+def _shape(network_mode="cashpilot-isolated", restart="no", image=PINNED, udp=True):
+ attrs = {
+ "Config": {"Image": image, "Env": ["UI_ADDRESS=127.0.0.1", "UDP_PORTS=56000:56029"]},
+ "HostConfig": {
+ "CapAdd": ["NET_ADMIN", "SETUID", "SETGID"],
+ "NetworkMode": network_mode,
+ "Devices": [{"PathInContainer": "/dev/net/tun"}],
+ "PortBindings": {
+ "4449/tcp": [{"HostIp": "127.0.0.1", "HostPort": "4449"}],
+ **(UDP_BINDINGS if udp else {}),
+ },
+ "RestartPolicy": {"Name": restart},
+ },
+ "Mounts": [{"Destination": "/var/lib/mysterium-node"}],
+ }
+ return orchestrator._shape(SimpleNamespace(attrs=attrs))
+
+
+class TestDrift:
+ def test_a_node_on_the_host_network_is_flagged_though_its_record_says_host(self):
+ found = drift.container_drift(MYST, _shape(network_mode="host"), HOST_RECORD)
+ assert any("shares the host's network" in f and "NET_ADMIN" in f for f in found)
+
+ def test_a_restart_policy_is_flagged(self):
+ found = drift.container_drift(MYST, _shape(restart="unless-stopped"), None)
+ assert any("restart policy unless-stopped" in f for f in found)
+
+ def test_another_image_is_flagged(self):
+ found = drift.container_drift(MYST, _shape(image="mysteriumnetwork/myst:latest"), None)
+ assert any("the catalog pins" in f for f in found)
+
+ def test_control_a_migrated_node_has_no_drift(self):
+ assert drift.container_drift(MYST, _shape(), None) == []
+
+
+# ---------------------------------------------------------------------------
+# (g) The isolation check goes red on NET_ADMIN in the host's namespace
+# ---------------------------------------------------------------------------
+
+
+def _c(name, mode, caps=(), privileged=False):
+ return SimpleNamespace(
+ name=name, attrs={"HostConfig": {"NetworkMode": mode, "CapAdd": list(caps), "Privileged": privileged}}
+ )
+
+
+class TestIsolationCheck:
+ def test_net_admin_in_the_host_netns_is_an_exposure(self):
+ found = isolation_check.host_netns_findings([_c("cashpilot-mysterium", "host", ["NET_ADMIN", "SETUID"])])
+ assert found and found[0]["exposure"] is True
+ assert "cashpilot-mysterium (NET_ADMIN)" in found[0]["detail"]
+ assert "Switch isolation on in Fleet, then redeploy them once" in found[0]["detail"]
+
+ def test_control_net_admin_in_its_own_namespace_is_not(self):
+ assert (
+ isolation_check.host_netns_findings([_c("cashpilot-mysterium", "cashpilot-isolated", ["NET_ADMIN"])]) == []
+ )
+
+ def test_control_the_host_network_without_those_capabilities_is_not(self):
+ assert isolation_check.host_netns_findings([_c("cashpilot-x", "host", ["NET_RAW"])]) == []
+
+
+# ---------------------------------------------------------------------------
+# (h) The guard: started behind the fence, stopped without it
+# ---------------------------------------------------------------------------
+
+
+class _Container(MagicMock):
+ """Like docker-py's Container: ``status`` is a read-only property."""
+
+ @property
+ def status(self):
+ return self._status
+
+
+def _node(status="running", network_mode="cashpilot-isolated", caps=("NET_ADMIN", "SETUID", "SETGID"), image=PINNED):
+ c = _Container()
+ c._status = status
+ c.name = "cashpilot-mysterium"
+ c.labels = dict(LABELS)
+ c.attrs = {
+ "Config": {"Image": image},
+ "HostConfig": {"NetworkMode": network_mode, "CapAdd": list(caps), "RestartPolicy": {"Name": "no"}},
+ }
+ c.exec_run.return_value = (1, b"")
+ return c
+
+
+def _guard(containers, problem=None, mode=lan_fence.MODE_ON):
+ client = MagicMock()
+ client.containers.list.return_value = containers
+ with (
+ patch.object(orchestrator, "docker_available", return_value=True),
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch.object(lan_fence, "resolve_mode", return_value=mode),
+ patch.object(lan_fence, "fence_problem", return_value=problem),
+ patch.object(lan_fence, "apply_fence") as apply,
+ ):
+ held = lan_fence.guard()
+ return held, apply
+
+
+class TestTheGuard:
+ def test_a_missing_fence_stops_the_node_and_puts_the_fence_back(self, markers):
+ node = _node()
+ held, apply = _guard([node], problem="the LAN fence is not complete on this host")
+ node.stop.assert_called_once()
+ apply.assert_called_once()
+ assert "not complete" in held["cashpilot-mysterium"]
+ assert lan_fence.status()["guard"] == held
+
+ def test_a_stopped_node_starts_once_the_fence_reads_complete(self, markers):
+ node = _node(status="exited")
+ held, _ = _guard([node])
+ node.start.assert_called_once()
+ assert held == {}
+
+ def test_a_node_the_owner_stopped_stays_stopped(self, markers):
+ orchestrator._mark_stopped("cashpilot-mysterium", True)
+ node = _node(status="exited")
+ _guard([node])
+ node.start.assert_not_called()
+
+ def test_without_the_fence_a_stopped_node_is_not_started(self, markers):
+ node = _node(status="exited")
+ _guard([node], problem="missing")
+ node.start.assert_not_called()
+
+ @pytest.mark.parametrize(
+ "change",
+ [
+ {"caps": ("NET_ADMIN", "SETUID", "SETGID", "SYS_ADMIN")},
+ {"image": "mysteriumnetwork/myst:latest"},
+ {"network_mode": "bridge"},
+ ],
+ )
+ def test_a_node_recreated_another_way_is_stopped(self, markers, change):
+ node = _node(**change)
+ held, _ = _guard([node])
+ node.stop.assert_called_once()
+ assert held["cashpilot-mysterium"]
+
+ def test_a_node_with_another_identity_is_stopped_and_stays_stopped(self, markers):
+ orchestrator._remember_identity("cashpilot-mysterium", "0x" + "a" * 40)
+ node = _node()
+ node.exec_run.return_value = (0, b'{"id":"0x' + b"b" * 40 + b'"}')
+ held, _ = _guard([node])
+ node.stop.assert_called_once()
+ assert "0x" + "a" * 40 in held["cashpilot-mysterium"]
+ # Stopped, it reads no identity, and must not be started again because of that.
+ stopped = _node(status="exited")
+ held, _ = _guard([stopped])
+ stopped.start.assert_not_called()
+ assert held["cashpilot-mysterium"]
+
+ def test_the_same_identity_clears_the_record(self, markers):
+ orchestrator._remember_identity("cashpilot-mysterium", "0x" + "a" * 40)
+ node = _node()
+ node.exec_run.return_value = (0, b'{"id":"0x' + b"a" * 40 + b'"}')
+ held, _ = _guard([node])
+ node.stop.assert_not_called()
+ assert held == {}
+ assert not (markers / "identity" / "cashpilot-mysterium").exists()
+
+ def test_control_a_running_node_behind_the_fence_is_left_alone(self, markers):
+ node = _node()
+ held, apply = _guard([node])
+ node.stop.assert_not_called()
+ node.start.assert_not_called()
+ apply.assert_not_called()
+ assert held == {}
+
+ def test_on_the_operators_own_network_there_is_no_fence_to_wait_for(self, markers, monkeypatch):
+ monkeypatch.setattr(orchestrator, "_CONTAINER_NETWORK", "operator-net")
+ node = _node(status="exited", network_mode="operator-net")
+ _guard([node], mode=lan_fence.MODE_OPERATOR)
+ node.start.assert_called_once()
+
+ @pytest.mark.parametrize("network_mode", ["bridge", "cashpilot-isolated"])
+ def test_in_operator_mode_a_node_off_the_operators_network_is_held(self, markers, monkeypatch, network_mode):
+ # Docker's default bridge has inter-container traffic on, and nothing of the operator's applies there.
+ monkeypatch.setattr(orchestrator, "_CONTAINER_NETWORK", "operator-net")
+ running, stopped = _node(network_mode=network_mode), _node(status="exited", network_mode=network_mode)
+ held, _ = _guard([running, stopped], mode=lan_fence.MODE_OPERATOR)
+ running.stop.assert_called_once()
+ stopped.start.assert_not_called()
+ assert "operator-net" in held["cashpilot-mysterium"]
+
+ @pytest.mark.parametrize("mode", [lan_fence.MODE_LEGACY, lan_fence.MODE_OFF])
+ def test_on_the_default_bridge_it_does_not_run_at_all(self, markers, mode):
+ # Every other earner there reaches its WebUI, and nothing keeps it off the LAN.
+ running, stopped = _node(network_mode="bridge"), _node(status="exited", network_mode="bridge")
+ held, apply = _guard([running, stopped], problem=lan_fence.mode_problem(mode), mode=mode)
+ running.stop.assert_called_once()
+ stopped.start.assert_not_called()
+ apply.assert_not_called()
+ assert "runs only behind the LAN fence" in held["cashpilot-mysterium"]
+
+
+class TestFenceProblem:
+ def _read(self, result, mode=lan_fence.MODE_ON):
+ with (
+ patch.object(lan_fence, "resolve_mode", return_value=mode),
+ patch.object(lan_fence, "read_fence", return_value=result),
+ ):
+ return lan_fence.fence_problem(MagicMock())
+
+ def test_a_complete_table_is_no_problem(self):
+ assert self._read({"present": True, "matches": True}) is None
+
+ def test_a_table_missing_a_rule_is(self):
+ assert self._read({"present": True, "matches": False})
+
+ def test_no_table_is(self):
+ assert self._read({"present": False, "matches": False})
+
+ def test_no_network_is(self):
+ assert self._read(None)
+
+ def test_on_the_operators_network_nothing_is_waited_for(self):
+ assert self._read(None, mode=lan_fence.MODE_OPERATOR) is None
+
+ @pytest.mark.parametrize("mode", [lan_fence.MODE_LEGACY, lan_fence.MODE_OFF, lan_fence.MODE_UNSUPPORTED])
+ def test_a_mode_without_a_fence_is_a_problem(self, mode):
+ assert "runs only behind the LAN fence" in self._read(None, mode=mode)
+
+
+# ---------------------------------------------------------------------------
+# (i) A deploy and the guard never act on the same node at once
+# ---------------------------------------------------------------------------
+
+
+class TestTheGuardDuringADeploy:
+ def test_it_starts_nothing_while_a_deploy_replaces_the_node(self, markers):
+ node = _node(status="exited")
+ with orchestrator.GATED_DEPLOY_LOCK:
+ _guard([node])
+ node.start.assert_not_called()
+
+ def test_it_does_not_clear_the_record_by_reading_the_old_node(self, markers):
+ # deploy_raw recorded the old node's identity and is about to stop it.
+ orchestrator._remember_identity("cashpilot-mysterium", "0x" + "a" * 40)
+ old = _node()
+ old.exec_run.return_value = (0, b'{"id":"0x' + b"a" * 40 + b'"}')
+ with orchestrator.GATED_DEPLOY_LOCK:
+ _guard([old])
+ assert (markers / "identity" / "cashpilot-mysterium").exists(), "the new node would never be compared"
+
+ def test_it_still_stops_a_node_when_the_fence_goes(self, markers):
+ node = _node()
+ with orchestrator.GATED_DEPLOY_LOCK:
+ _guard([node], problem="the LAN fence is not complete on this host")
+ node.stop.assert_called_once()
+
+ def test_control_once_the_deploy_is_done_it_starts_the_node(self, markers):
+ node = _node(status="exited")
+ _guard([node])
+ node.start.assert_called_once()
+ assert not orchestrator.GATED_DEPLOY_LOCK.locked()
+
+ def test_deploy_raw_holds_the_lock_while_it_replaces_the_node(self, markers):
+ seen = []
+ old = MagicMock()
+ old.name = "cashpilot-mysterium"
+ old.status = "exited"
+ old.labels = dict(LABELS)
+ old.attrs = {"Mounts": []}
+ old.stop.side_effect = lambda **_: seen.append(orchestrator.GATED_DEPLOY_LOCK.locked())
+ _raw_deploy(old=old)
+ assert seen == [True]
+ assert not orchestrator.GATED_DEPLOY_LOCK.locked()
+
+ def test_control_an_ordinary_earner_does_not_take_it(self, markers):
+ seen = []
+ old = MagicMock()
+ old.name = "cashpilot-honeygain"
+ old.status = "running"
+ old.labels = {"cashpilot.service": "honeygain", "cashpilot.managed": "true"}
+ old.attrs = {"Mounts": []}
+ old.stop.side_effect = lambda **_: seen.append(orchestrator.GATED_DEPLOY_LOCK.locked())
+ _raw_deploy(old=old, slug="honeygain")
+ assert seen == [False]
+
+
+# ---------------------------------------------------------------------------
+# (j) Start and Restart refuse a node the guard holds stopped
+# ---------------------------------------------------------------------------
+
+
+def _start(node, action="start_service"):
+ with (
+ patch.object(orchestrator, "_find_container", return_value=node),
+ patch.object(orchestrator, "_get_client", return_value=MagicMock()),
+ patch.object(lan_fence, "fence_problem", return_value=None),
+ patch.object(lan_fence, "resolve_mode", return_value=lan_fence.MODE_ON),
+ ):
+ getattr(orchestrator, action)("mysterium")
+
+
+class TestStartRefusesAHeldNode:
+ def test_a_node_held_for_another_identity_is_not_started(self, markers):
+ orchestrator._remember_identity("cashpilot-mysterium", "MISMATCH its identity is 0xb, not 0xa")
+ node = _node(status="exited")
+ with pytest.raises(orchestrator.ContainerNetworkError, match="identity"):
+ _start(node)
+ node.start.assert_not_called()
+
+ @pytest.mark.parametrize("action", ["start_service", "restart_service"])
+ def test_a_node_with_a_capability_the_catalog_does_not_grant_is_not_started(self, markers, action):
+ node = _node(status="exited", caps=("NET_ADMIN", "SETUID", "SETGID", "SYS_ADMIN"))
+ with pytest.raises(orchestrator.ContainerNetworkError, match="SYS_ADMIN"):
+ _start(node, action)
+ node.start.assert_not_called()
+ node.restart.assert_not_called()
+
+ def test_control_a_node_as_the_catalog_says_starts(self, markers):
+ orchestrator._mark_stopped("cashpilot-mysterium", True)
+ node = _node(status="exited")
+ _start(node)
+ node.start.assert_called_once()
+ assert "cashpilot-mysterium" not in orchestrator.user_stopped()
+
+
+# ---------------------------------------------------------------------------
+# (k) A node from 2.0.6 or earlier keeps running untouched until its redeploy
+# ---------------------------------------------------------------------------
+
+
+def _old_node(status="running"):
+ """What every Mysterium through 2.0.6 is: host network, unless-stopped, an unpinned image."""
+ c = _node(status=status, network_mode="host", image="mysteriumnetwork/myst")
+ c.attrs["HostConfig"]["RestartPolicy"] = {"Name": "unless-stopped"}
+ return c
+
+
+class TestAnOldHostNetworkNode:
+ @pytest.mark.parametrize("mode", [lan_fence.MODE_ON, lan_fence.MODE_LEGACY, lan_fence.MODE_OFF])
+ def test_the_guard_leaves_a_running_one_alone(self, markers, mode):
+ node = _old_node()
+ held, _ = _guard([node], problem=lan_fence.mode_problem(mode), mode=mode)
+ node.stop.assert_not_called()
+ node.start.assert_not_called()
+ node.restart.assert_not_called()
+ node.exec_run.assert_not_called()
+ assert held == {}
+
+ def test_the_guard_leaves_it_alone_when_the_fence_goes_too(self, markers):
+ node = _old_node()
+ _guard([node], problem="the LAN fence is not complete on this host")
+ node.stop.assert_not_called()
+
+ def test_the_guard_does_not_start_a_stopped_one(self, markers):
+ node = _old_node(status="exited")
+ _guard([node])
+ node.start.assert_not_called()
+
+ @pytest.mark.parametrize("action", ["start_service", "restart_service"])
+ def test_start_and_restart_work_as_before_in_legacy_mode(self, markers, action):
+ node = _old_node(status="exited")
+ with (
+ patch.object(orchestrator, "_find_container", return_value=node),
+ patch.object(orchestrator, "_get_client", return_value=MagicMock()),
+ patch.object(lan_fence, "resolve_mode", return_value=lan_fence.MODE_LEGACY),
+ ):
+ getattr(orchestrator, action)("mysterium")
+ assert node.start.called or node.restart.called
+
+ def test_stop_records_nothing_for_the_guard(self, markers):
+ node = _old_node()
+ with patch.object(orchestrator, "_find_container", return_value=node):
+ orchestrator.stop_service("mysterium")
+ node.stop.assert_called_once()
+ assert orchestrator.user_stopped() == set()
+
+ def test_a_redeploy_in_legacy_mode_is_refused_and_says_what_to_do(self, markers):
+ old = _old_node()
+ with pytest.raises(orchestrator.ContainerNetworkError) as refused:
+ _raw_deploy(old=old, mode=lan_fence.MODE_LEGACY)
+ # The catalog's name, the one on the dashboard card the owner redeploys from.
+ assert f"Switch isolation on in Fleet, then redeploy {MYST['name']}." in str(refused.value)
+ old.stop.assert_not_called()
+ old.remove.assert_not_called()
+
+ def test_drift_still_flags_it_and_gives_the_order(self):
+ found = " ".join(drift.container_drift(MYST, _shape(network_mode="host"), HOST_RECORD))
+ assert f"Switch isolation on in Fleet, then redeploy {MYST['name']} once" in found
+
+ def test_control_a_fence_gated_one_in_legacy_mode_is_stopped(self, markers):
+ node = _node(network_mode="bridge")
+ _guard([node], problem=lan_fence.mode_problem(lan_fence.MODE_LEGACY), mode=lan_fence.MODE_LEGACY)
+ node.stop.assert_called_once()
+
+
+# ---------------------------------------------------------------------------
+# (l) Review round: the guard fails closed, Stop stays a stop, nothing escapes it
+# ---------------------------------------------------------------------------
+
+
+class TestStopStaysAStop:
+ def test_stop_holds_the_guards_lock_and_records_first(self, markers):
+ node = _node()
+ seen = {}
+
+ def stopping(**_):
+ seen["locked"] = orchestrator.GATED_DEPLOY_LOCK.locked()
+ seen["recorded"] = "cashpilot-mysterium" in orchestrator.user_stopped()
+
+ node.stop.side_effect = stopping
+ with patch.object(orchestrator, "_find_container", return_value=node):
+ orchestrator.stop_service("mysterium")
+ assert seen == {"locked": True, "recorded": True}
+ assert not orchestrator.GATED_DEPLOY_LOCK.locked()
+
+ def test_a_guard_pass_that_read_the_records_first_cannot_start_it_after(self, markers):
+ """The guard holds the lock from reading the records to its start, so Stop waits for it."""
+ node = _node(status="exited")
+ with patch.object(orchestrator, "_find_container", return_value=node):
+ orchestrator.GATED_DEPLOY_LOCK.acquire()
+ try:
+ import threading
+
+ stopper = threading.Thread(target=orchestrator.stop_service, args=("mysterium",))
+ stopper.start()
+ stopper.join(0.3)
+ assert stopper.is_alive(), "Stop must wait for the guard pass"
+ assert orchestrator.user_stopped() == set()
+ finally:
+ orchestrator.GATED_DEPLOY_LOCK.release()
+ stopper.join(5)
+ assert orchestrator.user_stopped() == {"cashpilot-mysterium"}
+
+ def test_a_stop_that_cannot_be_recorded_fails_and_says_so(self, markers, monkeypatch):
+ monkeypatch.setattr(orchestrator, "STOPPED_MARKER", str(markers / "missing-dir" / "stopped"))
+ node = _node()
+ with (
+ patch.object(orchestrator, "_find_container", return_value=node),
+ pytest.raises(orchestrator.ContainerRuntimeError, match="could not be recorded"),
+ ):
+ orchestrator.stop_service("mysterium")
+ node.stop.assert_not_called()
+
+ def test_control_an_ordinary_earner_stops_without_a_record(self, markers, monkeypatch):
+ monkeypatch.setattr(orchestrator, "STOPPED_MARKER", str(markers / "missing-dir" / "stopped"))
+ node = MagicMock()
+ node.name = "cashpilot-honeygain"
+ with patch.object(orchestrator, "_find_container", return_value=node):
+ orchestrator.stop_service("honeygain")
+ node.stop.assert_called_once()
+
+
+class TestTheGuardFailsClosed:
+ def test_a_mode_it_cannot_read_stops_the_node(self, markers):
+ node = _node()
+ client = MagicMock()
+ client.containers.list.return_value = [node]
+ with (
+ patch.object(orchestrator, "docker_available", return_value=True),
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch.object(lan_fence, "resolve_mode", side_effect=RuntimeError("docker info failed")),
+ patch.object(lan_fence, "apply_fence") as apply,
+ ):
+ held = lan_fence.guard()
+ node.stop.assert_called_once()
+ node.start.assert_not_called()
+ apply.assert_not_called()
+ assert "could not be read" in held["cashpilot-mysterium"]
+
+ def test_control_a_mode_it_reads_leaves_a_fenced_node_running(self, markers):
+ node = _node()
+ _guard([node])
+ node.stop.assert_not_called()
+
+ def test_a_node_also_attached_to_another_network_is_stopped(self, markers):
+ node = _node()
+ node.attrs["NetworkSettings"] = {"Networks": {"cashpilot-isolated": {}, "bridge": {}}}
+ held, _ = _guard([node])
+ node.stop.assert_called_once()
+ assert "bridge" in held["cashpilot-mysterium"]
+
+ def test_control_a_node_on_its_one_network_is_left_alone(self, markers):
+ node = _node()
+ node.attrs["NetworkSettings"] = {"Networks": {"cashpilot-isolated": {}}}
+ _guard([node])
+ node.stop.assert_not_called()
+
+
+class TestTheFenceReadChecksTheNetwork:
+ def _net(self, **options):
+ opts = dict(lan_fence._REQUIRED_OPTIONS, **options)
+ return SimpleNamespace(attrs={"Driver": "bridge", "Options": opts})
+
+ def test_a_network_recreated_on_another_interface_is_not_a_fence(self):
+ net = self._net(**{"com.docker.network.bridge.name": "br-other"})
+ with (
+ patch.object(lan_fence, "resolve_mode", return_value=lan_fence.MODE_ON),
+ patch.object(lan_fence, "managed_network", return_value=net),
+ patch.object(lan_fence, "run_helper", return_value={"present": True, "matches": True}) as helper,
+ ):
+ problem = lan_fence.fence_problem(MagicMock())
+ assert problem and "bridge.name" in problem
+ helper.assert_not_called()
+
+ def test_control_the_network_as_created_reads_the_table(self):
+ with (
+ patch.object(lan_fence, "resolve_mode", return_value=lan_fence.MODE_ON),
+ patch.object(lan_fence, "managed_network", return_value=self._net()),
+ patch.object(lan_fence, "_subnet", return_value="172.30.0.0/24"),
+ patch.object(lan_fence, "run_helper", return_value={"present": True, "matches": True}) as helper,
+ ):
+ assert lan_fence.fence_problem(MagicMock()) is None
+ helper.assert_called_once()
+
+
+class TestTheDeployKeepsWhatTheGuardReliesOn:
+ def test_a_caller_cannot_relabel_the_node_out_of_the_guards_sight(self, markers):
+ client, _ = _raw_deploy(labels={"cashpilot.service": "other", "cashpilot.managed": "false", "x": "1"})
+ labels = client.containers.create.call_args.kwargs["labels"]
+ assert labels["cashpilot.service"] == "mysterium"
+ assert labels["cashpilot.managed"] == "true"
+ assert labels["x"] == "1"
+
+ def _replacement(self, status):
+ old = MagicMock()
+ old.name = "cashpilot-mysterium"
+ old.status = status
+ old.labels = dict(LABELS)
+ old.attrs = {"Mounts": []}
+ old.exec_run.return_value = (1, b"")
+ return old
+
+ def test_a_retry_over_an_unconfirmed_replacement_keeps_the_identity_to_match(self, markers):
+ orchestrator._remember_identity("cashpilot-mysterium", "0xa")
+ _raw_deploy(old=self._replacement("created"))
+ assert orchestrator._pending_identity("cashpilot-mysterium") == "0xa"
+
+ def test_a_retry_over_a_running_replacement_keeps_it_too(self, markers):
+ orchestrator._remember_identity("cashpilot-mysterium", "0xa")
+ _raw_deploy(old=self._replacement("running"))
+ assert orchestrator._pending_identity("cashpilot-mysterium") == "0xa"
+
+ def test_control_a_redeploy_over_a_held_mismatch_clears_it(self, markers):
+ """The owner's way out once they restored the volume, as the hold says."""
+ orchestrator._remember_identity("cashpilot-mysterium", "MISMATCH its identity is 0xb, not 0xa")
+ _raw_deploy(old=self._replacement("exited"))
+ assert orchestrator.identity_problem(self._replacement("exited")) is None
+
+ def test_control_a_redeploy_over_a_stopped_node_with_no_record_records_nothing(self, markers):
+ _raw_deploy(old=self._replacement("exited"))
+ assert orchestrator._pending_identity("cashpilot-mysterium") is None
+
+
+class TestNoComposeExport:
+ def test_mysterium_has_no_compose_export(self):
+ from app import compose_generator
+
+ with pytest.raises(ValueError, match="LAN fence"):
+ compose_generator.generate_compose_single("mysterium")
+
+ def test_a_multi_export_leaves_it_out(self):
+ from app import compose_generator
+
+ exported = yaml.safe_load(compose_generator.generate_compose_multi(["mysterium", "bitping"]))
+ assert list(exported["services"]) == ["cashpilot-bitping"]
+
+
+class TestTheMessagesNameTheService:
+ def test_the_mode_refusal_names_the_service_it_is_given(self):
+ problem = lan_fence.mode_problem(lan_fence.MODE_LEGACY, "Other")
+ assert "Other runs only behind the LAN fence" in problem
+ assert problem.endswith("Switch isolation on in Fleet, then redeploy Other.")
+
+ def test_drift_names_the_service_and_claims_no_capability_it_lacks(self):
+ other = {**MYST, "name": "Other", "docker": {**MYST["docker"], "cap_add": []}}
+ shape = _shape(network_mode="host")
+ shape["cap_add"] = []
+ found = " ".join(drift.container_drift(other, shape, None))
+ assert "redeploy Other once" in found
+ assert "NET_ADMIN" not in found
+
+ def test_drift_flags_a_node_that_publishes_its_webui_but_not_its_range(self):
+ found = drift.container_drift(MYST, _shape(udp=False), None)
+ assert any("does not publish port" in f and "56000/udp" in f for f in found)
+
+ def test_drift_judges_the_range_on_record(self):
+ record = {"env": {"UDP_PORTS": "56000:56100"}}
+ found = drift.container_drift(MYST, _shape(), record)
+ assert any("56100/udp" in f for f in found)
+
+ def test_the_container_down_alert_does_not_promise_a_restart_policy(self):
+ from datetime import UTC, datetime
+
+ from app import main as main_mod
+
+ copies = [{"_down_since": datetime.now(UTC), "_node": "w1", "restart_policy": "no", "exit_code": 0}]
+ message = main_mod._container_down_message("mysterium", copies)
+ assert "unless-stopped" not in message
+ assert "LAN fence" in message
+ control = main_mod._container_down_message("honeygain", copies)
+ assert "sets unless-stopped" in control
+
+
+def test_a_ui_default_on_any_loopback_address_is_no_inbound_exception():
+ from app import lan_isolation
+
+ def svc(default):
+ return {"docker": {"ui_port": {"env": "UI", "port": 4449}, "env": [{"key": "UI", "default": default}]}}
+
+ assert lan_isolation.published_ports(svc("127.0.0.2")) == []
+ assert lan_isolation.published_ports(svc("::1,localhost")) == []
+ assert lan_isolation.published_ports(svc("192.168.1.5")) == ["192.168.1.5:4449"]
diff --git a/tests/test_mysterium_sudo_capabilities.py b/tests/test_mysterium_sudo_capabilities.py
index 8904073d..ca8a26ce 100644
--- a/tests/test_mysterium_sudo_capabilities.py
+++ b/tests/test_mysterium_sudo_capabilities.py
@@ -61,9 +61,14 @@ def _run_kwargs(**deploy_kwargs):
client = MagicMock()
client.containers.get.side_effect = orchestrator.NotFound("nope")
client.containers.run.return_value = container
- with patch.object(orchestrator, "_get_client", return_value=client):
+ client.containers.create.return_value = container
+ # Mysterium is fence-gated: created, then started once the fence reads complete.
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value=None),
+ ):
orchestrator.deploy_raw(**deploy_kwargs)
- return client.containers.run.call_args.kwargs
+ return (client.containers.create.call_args or client.containers.run.call_args).kwargs
class TestTheNodeCanRunSudo:
@@ -78,7 +83,7 @@ def test_the_worker_accepts_the_catalog_entry_it_ships(self, mysterium, real_cat
image=docker["image"],
cap_add=docker["cap_add"],
devices=docker["devices"],
- network_mode=docker["network_mode"],
+ network_mode=docker.get("network_mode"),
),
"mysterium",
)
diff --git a/tests/test_mysterium_udp_ports.py b/tests/test_mysterium_udp_ports.py
index 51baea30..96201636 100644
--- a/tests/test_mysterium_udp_ports.py
+++ b/tests/test_mysterium_udp_ports.py
@@ -10,10 +10,9 @@
from __future__ import annotations
import pytest
-import yaml
from fastapi.testclient import TestClient
-from app import compose_generator
+from app import catalog, compose_generator
from tests.test_mysterium_ui_address import _deploy, _record, app
@@ -23,30 +22,39 @@ def client():
yield c
+def _udp(sent):
+ return sorted(k for k in sent["ports"] if k.endswith("/udp"))
+
+
def test_a_first_deploy_uses_the_range_the_guide_says_to_forward(client):
resp, sent = _deploy(client, {"env": {}})
assert resp.status_code == 200, resp.text
- assert " --udp.ports=56000:56100 service " in sent["command"]
+ assert " --udp.ports=56000:56029 service " in sent["command"]
+ # Published 1:1 on the bridge, so the router's forward reaches the node unchanged.
+ assert _udp(sent) == [f"{p}/udp" for p in range(56000, 56030)]
+ assert all(sent["ports"][f"{p}/udp"] == p for p in range(56000, 56030))
def test_a_node_deployed_before_the_setting_existed_gets_the_default(client):
resp, sent = _deploy(client, {"env": {}}, recorded=_record({"UI_ADDRESS": "127.0.0.1"}))
assert resp.status_code == 200, resp.text
- assert " --udp.ports=56000:56100 service " in sent["command"]
+ assert " --udp.ports=56000:56029 service " in sent["command"]
def test_a_typed_range_reaches_the_node(client):
resp, sent = _deploy(client, {"env": {"UDP_PORTS": "41000:41050"}})
assert resp.status_code == 200, resp.text
assert " --udp.ports=41000:41050 service " in sent["command"]
+ assert _udp(sent) == [f"{p}/udp" for p in range(41000, 41051)]
def test_a_blank_redeploy_keeps_the_range_this_machine_was_given(client):
recorded = _record({"UDP_PORTS": "41000:41050"})
- for env in ({}, {"UDP_PORTS": ""}, {"UDP_PORTS": "56000:56100"}):
+ for env in ({}, {"UDP_PORTS": ""}, {"UDP_PORTS": "56000:56029"}):
resp, sent = _deploy(client, {"env": env}, recorded=recorded)
assert resp.status_code == 200, resp.text
assert " --udp.ports=41000:41050 service " in sent["command"]
+ assert _udp(sent) == [f"{p}/udp" for p in range(41000, 41051)]
@pytest.mark.parametrize(
@@ -73,6 +81,7 @@ def test_a_recorded_reversed_range_is_refused_too(client):
assert not sent
-def test_the_exported_compose_file_uses_the_default_range():
- exported = yaml.safe_load(compose_generator.generate_compose_single("mysterium"))
- assert " --udp.ports=56000:56100 service " in exported["services"]["cashpilot-mysterium"]["command"]
+def test_the_compose_rendering_uses_the_default_range():
+ service = compose_generator._service_to_compose(catalog.get_service("mysterium"), None, None, None)
+ assert " --udp.ports=56000:56029 service " in service["command"]
+ assert "56000-56029:56000-56029/udp" in service["ports"]
diff --git a/tests/test_mysterium_ui_address.py b/tests/test_mysterium_ui_address.py
index 5b87e5c9..9a1b4ec7 100644
--- a/tests/test_mysterium_ui_address.py
+++ b/tests/test_mysterium_ui_address.py
@@ -2,10 +2,11 @@
1.37.4 bound the node's WebUI to 127.0.0.1 so host networking stopped
publishing it on every interface. That also cut off any reverse proxy in front
-of it, and the command had no way to say "loopback, plus the one address my
-proxy connects to". UI_ADDRESS is that way: the default is unchanged, and the
-value is checked before it reaches the command line, where a stray space turns
-into an argument and the node refuses to start.
+of it, so UI_ADDRESS said "loopback, plus the one address my proxy connects
+to". After 2.0.6 the node runs in its own namespace on the fenced bridge: it
+listens on 0.0.0.0 there, which reaches only that namespace, and UI_ADDRESS is
+the host addresses port 4449 is published on. The value is still checked
+before a deploy, and the default is still loopback.
"""
from __future__ import annotations
@@ -17,7 +18,6 @@
os.environ.setdefault("CASHPILOT_API_KEY", "test-fleet-key")
import pytest
-import yaml
from fastapi.testclient import TestClient
from app import catalog, compose_generator
@@ -27,6 +27,9 @@
"--firewall.protected.networks=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8,100.64.0.0/10,169.254.0.0/16"
)
LOOPBACK = f"--ui.address=127.0.0.1 --tequilapi.address=127.0.0.1 {PROTECTED} --udp.ports=56000:56100 service --agreed-terms-and-conditions"
+#: What every node runs after 2.0.6: the WebUI on its own namespace, published by Docker.
+COMMAND = f"--ui.address=0.0.0.0 --tequilapi.address=127.0.0.1 {PROTECTED} --udp.ports=56000:56029 service --agreed-terms-and-conditions"
+UI = "4449/tcp"
@asynccontextmanager
@@ -45,7 +48,13 @@ def client():
def _deploy(client, body, recorded=None):
"""Deploy the REAL catalog entry; return (response, spec sent to the worker)."""
- worker = {"id": 1, "name": "w1", "status": "online", "url": "http://192.168.1.10:8081"}
+ worker = {
+ "id": 1,
+ "name": "w1",
+ "status": "online",
+ "url": "http://192.168.1.10:8081",
+ "system_info": '{"features": ["fence_gated"]}',
+ }
sent: dict = {}
async def _fake_deploy(worker_id, slug, spec):
@@ -73,45 +82,48 @@ def _record(env):
def test_a_first_deploy_stays_on_loopback(client):
resp, sent = _deploy(client, {"env": {}})
assert resp.status_code == 200, resp.text
- assert sent["command"] == LOOPBACK
+ assert sent["command"] == COMMAND
+ assert sent["ports"][UI] == ["127.0.0.1", 4449]
def test_the_form_posting_its_prefilled_default_stays_on_loopback(client):
resp, sent = _deploy(client, {"env": {"UI_ADDRESS": "127.0.0.1"}})
assert resp.status_code == 200, resp.text
- assert sent["command"] == LOOPBACK
+ assert sent["ports"][UI] == ["127.0.0.1", 4449]
def test_a_node_deployed_before_the_setting_existed_stays_on_loopback(client):
"""The 1.37.4 fleet: recorded env has no UI_ADDRESS at all."""
resp, sent = _deploy(client, {"env": {}}, recorded=_record({}))
assert resp.status_code == 200, resp.text
- assert sent["command"] == LOOPBACK
+ assert sent["command"] == COMMAND
+ assert sent["ports"][UI] == ["127.0.0.1", 4449]
def test_an_added_address_reaches_the_webui_and_never_the_api(client):
resp, sent = _deploy(client, {"env": {"UI_ADDRESS": "127.0.0.1,172.18.0.1"}})
assert resp.status_code == 200, resp.text
- assert sent["command"] == (
- f"--ui.address=127.0.0.1,172.18.0.1 --tequilapi.address=127.0.0.1 {PROTECTED} --udp.ports=56000:56100 service --agreed-terms-and-conditions"
- )
+ assert sent["ports"][UI] == [["127.0.0.1", 4449], ["172.18.0.1", 4449]]
+ # The API is never published: it stays on the node's own loopback.
+ assert "4050/tcp" not in sent["ports"]
+ assert "--tequilapi.address=127.0.0.1 " in sent["command"]
def test_a_blank_redeploy_keeps_the_address_this_machine_was_given(client):
resp, sent = _deploy(client, {"env": {}}, recorded=_record({"UI_ADDRESS": "127.0.0.1,172.18.0.1"}))
assert resp.status_code == 200, resp.text
- assert "--ui.address=127.0.0.1,172.18.0.1 " in sent["command"]
+ assert sent["ports"][UI] == [["127.0.0.1", 4449], ["172.18.0.1", 4449]]
def test_localhost_takes_a_machine_back_to_loopback(client):
"""The way out: the prefilled 127.0.0.1 is not a decision, so it cannot undo one."""
recorded = _record({"UI_ADDRESS": "127.0.0.1,172.18.0.1"})
resp, sent = _deploy(client, {"env": {"UI_ADDRESS": "127.0.0.1"}}, recorded=recorded)
- assert "--ui.address=127.0.0.1,172.18.0.1 " in sent["command"]
+ assert sent["ports"][UI] == [["127.0.0.1", 4449], ["172.18.0.1", 4449]]
resp, sent = _deploy(client, {"env": {"UI_ADDRESS": "localhost"}}, recorded=recorded)
assert resp.status_code == 200, resp.text
- assert sent["command"].startswith("--ui.address=localhost --tequilapi.address=127.0.0.1 ")
+ assert sent["ports"][UI] == ["127.0.0.1", 4449]
@pytest.mark.parametrize(
@@ -135,10 +147,12 @@ def test_anything_but_addresses_is_refused_before_a_deploy(client, value):
assert not sent, "nothing may reach the worker"
-def test_the_exported_compose_file_stays_on_loopback():
- exported = yaml.safe_load(compose_generator.generate_compose_single("mysterium"))
- service = exported["services"]["cashpilot-mysterium"]
- assert service["command"] == LOOPBACK
+def test_the_compose_rendering_stays_on_loopback():
+ """Mysterium has no Compose export (it is fence-gated); the rendering is kept for a ui_port service."""
+ service = compose_generator._service_to_compose(catalog.get_service("mysterium"), None, None, None)
+ assert service["command"] == COMMAND
+ assert "127.0.0.1:4449:4449" in service["ports"]
+ assert "network_mode" not in service
def test_a_pattern_that_does_not_compile_is_a_catalog_error(tmp_path):
diff --git a/tests/test_owner_volumes.py b/tests/test_owner_volumes.py
index cf8029bb..34e583a4 100644
--- a/tests/test_owner_volumes.py
+++ b/tests/test_owner_volumes.py
@@ -21,11 +21,16 @@ def _client(old=None):
else:
client.containers.get.return_value = old
client.containers.run.return_value = MagicMock(id="cid")
+ client.containers.create.return_value = MagicMock(id="cid")
return client
def _deploy(client, **kwargs):
- with patch.object(orchestrator, "_get_client", return_value=client):
+ # Mysterium is fence-gated: created, then started once the fence reads complete.
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value=None),
+ ):
return orchestrator.deploy_raw(slug="mysterium", image="img", **kwargs)
@@ -34,7 +39,7 @@ def test_a_missing_named_volume_is_created_before_the_container():
client.volumes.get.side_effect = orchestrator.NotFound("no volume")
calls = []
client.volumes.create.side_effect = lambda name: calls.append(("create", name))
- client.containers.run.side_effect = lambda **kw: calls.append(("run", kw["name"])) or MagicMock(id="cid")
+ client.containers.create.side_effect = lambda **kw: calls.append(("run", kw["name"])) or MagicMock(id="cid")
_deploy(client, volumes=VOLUMES)
# Only the named volume: a host folder is not a volume.
assert calls == [("create", "mysterium-data"), ("run", "cashpilot-mysterium")]
@@ -61,7 +66,7 @@ def test_a_volume_the_proxy_will_not_let_the_worker_use_stops_the_redeploy_befor
assert "cashpilot-migrate" in str(err.value)
old.stop.assert_not_called()
old.remove.assert_not_called()
- client.containers.run.assert_not_called()
+ client.containers.create.assert_not_called()
def test_behind_a_proxy_from_an_older_compose_file_docker_still_makes_the_volume():
@@ -72,7 +77,7 @@ def test_behind_a_proxy_from_an_older_compose_file_docker_still_makes_the_volume
403, {**SOCKGUARD_403, "reason": "the CashPilot worker does not use this Docker API call"}
)
_deploy(client, volumes=VOLUMES)
- client.containers.run.assert_called_once()
+ client.containers.create.assert_called_once()
def test_any_other_docker_error_is_not_dressed_up_as_a_proxy_refusal():
diff --git a/tests/test_redeploy_keeps_live_mounts.py b/tests/test_redeploy_keeps_live_mounts.py
index 3d1eaa85..5055d577 100644
--- a/tests/test_redeploy_keeps_live_mounts.py
+++ b/tests/test_redeploy_keeps_live_mounts.py
@@ -49,10 +49,16 @@ def _deploy(old, **kwargs):
else:
client.containers.get.return_value = old
client.containers.run.return_value = container
+ client.containers.create.return_value = container
kept: list[dict[str, str]] = []
- with patch.object(orchestrator, "_get_client", return_value=client):
+ # Mysterium is fence-gated: created, then started once the fence reads complete.
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value=None),
+ ):
orchestrator.deploy_raw(slug="mysterium", image="img", kept_mounts=kept, **kwargs)
- return client.containers.run.call_args.kwargs["volumes"], kept
+ call = client.containers.create.call_args or client.containers.run.call_args
+ return call.kwargs["volumes"], kept
class TestTheRunningContainerDecidesWhereItsDataLives:
@@ -194,7 +200,7 @@ def _mysterium_spec():
return {
"image": docker["image"],
"volumes": CATALOG_VOLUMES,
- "network_mode": docker["network_mode"],
+ "network_mode": docker.get("network_mode"),
"cap_add": docker["cap_add"],
"devices": docker["devices"],
}
diff --git a/tests/test_socket_proxy.py b/tests/test_socket_proxy.py
index ba45cd62..d7bb18d9 100644
--- a/tests/test_socket_proxy.py
+++ b/tests/test_socket_proxy.py
@@ -82,10 +82,13 @@ def policy_problems(policy: dict, catalog: list[dict]) -> list[str]:
for key in ("allow_host_pid", "allow_host_ipc", "allow_host_userns"):
if create.get(key):
problems.append(f"{key} on")
- # The Activity reader of a host-network earner joins the host cgroup
- # namespace to tell that earner's sockets from the host's (app/activity.py).
- if not create.get("allow_host_cgroupns"):
- problems.append("allow_host_cgroupns off: the host-network Activity probe needs it")
+ # Only the Activity reader of a host-network earner joined the host cgroup
+ # namespace (app/activity.py). No catalog earner shares the host's network
+ # since Mysterium moved onto the fenced bridge, so nothing needs it.
+ if create.get("allow_host_cgroupns"):
+ problems.append("allow_host_cgroupns on: no earner shares the host's network")
+ if any((svc.get("docker") or {}).get("network_mode") == "host" for svc in catalog):
+ problems.append("a catalog earner shares the host's network")
# The LAN fence creates its bridge with driver options and moves earners
# with a forced disconnect (app/lan_fence.py); nothing else may open up.
network = policy["request_body"].get("network") or {}
@@ -328,7 +331,7 @@ def test_the_policy_allows_every_call_the_worker_makes():
"privileged": lambda p: p["request_body"]["container_create"].update(allow_privileged=True),
"no nnp": lambda p: p["request_body"]["container_create"].update(require_no_new_privileges=False),
"host pid": lambda p: p["request_body"]["container_create"].update(allow_host_pid=True),
- "no host cgroupns": lambda p: p["request_body"]["container_create"].update(allow_host_cgroupns=False),
+ "host cgroupns": lambda p: p["request_body"]["container_create"].update(allow_host_cgroupns=True),
"no network driver options": lambda p: p["request_body"]["network"].update(allow_driver_options=False),
"no forced disconnect": lambda p: p["request_body"]["network"].pop("allow_disconnect_force"),
"custom network drivers": lambda p: p["request_body"]["network"].update(allow_custom_drivers=True),
@@ -706,23 +709,32 @@ def test_proxy_bind_mounts_also_open_the_worker_allowlist():
MYST_TARGET = "/var/lib/mysterium-node"
-def _redeploy(monkeypatch, old_mounts, **kwargs):
- """deploy_raw over a running mysterium container; returns (client, old, error)."""
+def _redeploy(monkeypatch, old_mounts, slug="mysterium", **kwargs):
+ """deploy_raw over a running container of ``slug``; returns (client, old, error)."""
old = MagicMock()
- old.labels = {"cashpilot.service": "mysterium"}
+ old.labels = {"cashpilot.service": slug}
old.attrs = {"Mounts": old_mounts}
client = MagicMock()
client.containers.get.return_value = old
client.containers.run.return_value = MagicMock(id="cid")
+ client.containers.create.return_value = MagicMock(id="cid")
error = None
- with patch.object(orchestrator, "_get_client", return_value=client):
+ # Mysterium is fence-gated: created, then started once the fence reads complete.
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value=None),
+ ):
try:
- orchestrator.deploy_raw(slug="mysterium", image="img", **kwargs)
+ orchestrator.deploy_raw(slug=slug, image="img", **kwargs)
except orchestrator.ContainerRuntimeError as exc:
error = exc
return client, old, error
+def _created(client):
+ return (client.containers.create.call_args or client.containers.run.call_args).kwargs
+
+
def test_proxied_a_kept_host_folder_the_proxy_refuses_leaves_the_node_running(monkeypatch):
_proxied_env(monkeypatch, bind_mounts="/srv/allowed")
legacy = {"Type": "bind", "Source": "/home/me/legacy", "Destination": MYST_TARGET}
@@ -750,7 +762,7 @@ def test_proxied_a_kept_host_folder_inside_an_allowed_one_redeploys(monkeypatch)
)
assert error is None
old.remove.assert_called_once()
- assert client.containers.run.call_args.kwargs["volumes"] == [f"/srv/allowed/myst:{MYST_TARGET}:rw"]
+ assert _created(client)["volumes"] == [f"/srv/allowed/myst:{MYST_TARGET}:rw"]
def test_with_the_raw_socket_a_kept_host_folder_is_not_the_proxys_business(monkeypatch):
@@ -769,7 +781,8 @@ def test_with_the_raw_socket_a_kept_host_folder_is_not_the_proxys_business(monke
@pytest.mark.parametrize("runtime", ["io.containerd.runc.v2", "nvidia", "crun", "sysbox-runc"])
def test_proxied_a_runtime_the_proxy_refuses_leaves_the_service_running(monkeypatch, runtime):
_proxied_env(monkeypatch)
- client, old, error = _redeploy(monkeypatch, [], runtime=runtime)
+ # Not Mysterium: its catalog entry requires runc, which wins over any request.
+ client, old, error = _redeploy(monkeypatch, [], slug="honeygain", runtime=runtime)
assert error is not None and runtime in str(error)
old.stop.assert_not_called()
old.remove.assert_not_called()
diff --git a/tests/test_worker_api_coverage.py b/tests/test_worker_api_coverage.py
index 6b3a3733..1650a2dd 100644
--- a/tests/test_worker_api_coverage.py
+++ b/tests/test_worker_api_coverage.py
@@ -159,11 +159,7 @@ def test_disallowed_network_mode_rejected(self):
_validate_deploy_spec(spec)
assert ei.value.status_code == 403
- def test_host_network_mode_allowed_for_mysterium(self):
- # mysterium.yml is the only catalog service declaring network_mode: host.
- _validate_deploy_spec(DeploySpec(image="x", network_mode="host"), slug="mysterium") # must not raise
-
- @pytest.mark.parametrize("slug", ["honeygain", "storj", "not-a-real-slug", None])
+ @pytest.mark.parametrize("slug", ["mysterium", "honeygain", "storj", "not-a-real-slug", None])
def test_host_network_mode_rejected_for_non_allowlisted_slug(self, slug):
spec = DeploySpec(image="x", network_mode="host")
with pytest.raises(HTTPException) as ei:
diff --git a/tests/test_worker_resources.py b/tests/test_worker_resources.py
index 3d1019f2..7e6783e6 100644
--- a/tests/test_worker_resources.py
+++ b/tests/test_worker_resources.py
@@ -195,7 +195,7 @@ def test_ports_suppressed_in_host_network(self):
client = self._mock_client()
with patch.object(orchestrator, "_get_client", return_value=client):
orchestrator.deploy_raw(
- slug="mysterium",
+ slug="hostnode",
image="img",
ports={"4449/tcp": 4449},
network_mode="host",
@@ -256,8 +256,16 @@ def test_hardening_flags_always_applied(self):
def test_catalog_declared_capability_is_added_back(self):
# mysterium is the only catalog service declaring a cap; dropping ALL must not
- # break it — NET_ADMIN is re-added on top of the drop.
- kwargs = self._run_kwargs(slug="mysterium", image="img", cap_add=["NET_ADMIN"])
+ # break it — NET_ADMIN is re-added on top of the drop. It is fence-gated, so
+ # it is created and started apart, once the fence reads complete.
+ client = self._mock_client()
+ client.containers.create.return_value = client.containers.run.return_value
+ with (
+ patch.object(orchestrator, "_get_client", return_value=client),
+ patch("app.lan_fence.fence_problem", return_value=None),
+ ):
+ orchestrator.deploy_raw(slug="mysterium", image="img", cap_add=["NET_ADMIN"])
+ kwargs = client.containers.create.call_args.kwargs
assert kwargs["cap_drop"] == ["ALL"]
assert kwargs["cap_add"] == ["NET_ADMIN"]