From 3e3e28cd16864b209fa5f1e1cb803440d1bd902f Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 13:16:47 +0100 Subject: [PATCH 1/7] docs: warn operators about the traps that stop Mysterium and proxied copies from earning Moving a Mysterium node behind the fence, choosing a UDP range Docker can always reserve, probing an earner's servers through a commercial proxy, using a second site as an exit node, and measuring what each earner costs on a small box. --- docs/faq.md | 14 ++- docs/fleet.md | 37 ++++++ docs/guides/mysterium.md | 7 ++ docs/mysterium-fence.md | 251 +++++++++++++++++++++++++++++++++++++++ docs/proxies.md | 112 +++++++++++++++++ zensical.toml | 1 + 6 files changed, 421 insertions(+), 1 deletion(-) create mode 100644 docs/mysterium-fence.md diff --git a/docs/faq.md b/docs/faq.md index 4226b6e3..8f325ba8 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -38,10 +38,22 @@ If CashPilot cannot write the key to disk at all, because `/data` is unwritable **What about security?** -Every service CashPilot deploys runs in its own container with every Linux capability dropped (a few services add back only the ones they declare) and `--security-opt no-new-privileges` set, so a process inside cannot gain privileges. Containers cannot see your host filesystem beyond the volumes a service declares. On a new install the worker also fences them off your LAN, the host, your tailnet and cloud metadata, so they reach the internet and nothing local ([LAN isolation](isolation.md)). On Podman the fence is not available: earners run on Podman's default network, which can reach your LAN, and the Fleet page says so ([LAN isolation](isolation.md) lists the options). An install upgraded from a release before 1.70 keeps its earners on Docker's default bridge, which can reach your LAN, until you click **Turn on isolation** on the Fleet page. The fence does not cover a service on host networking (Mysterium, today); [the isolation page](isolation.md) says what to do about those, and [Protecting Your Home Network](home-network-security.md) covers firewalling your own bridge instead. Service credentials are encrypted at rest using Fernet symmetric encryption. The UI has no Docker access. With the shipped compose files the worker is never given the raw Docker socket: it reaches Docker through `cashpilot-socket-proxy`, which refuses privileged containers and binds of `/` or the socket (unless you put one of them, or a folder above the socket, in `CASHPILOT_PROXY_BIND_MOUNTS` or `CASHPILOT_COMPOSE_DIR`: the proxy takes those as given), and limits the worker to containers, volumes and networks with its owner label ([security defaults](security-defaults.md)). The Unraid templates and the TrueNAS app still mount the raw socket in the worker, which is root on the host ([app stores](app-stores.md)). +Every service CashPilot deploys runs in its own container with every Linux capability dropped (a few services add back only the ones they declare) and `--security-opt no-new-privileges` set, so a process inside cannot gain privileges. Containers cannot see your host filesystem beyond the volumes a service declares. On a new install the worker also fences them off your LAN, the host, your tailnet and cloud metadata, so they reach the internet and nothing local ([LAN isolation](isolation.md)). On Podman the fence is not available: earners run on Podman's default network, which can reach your LAN, and the Fleet page says so ([LAN isolation](isolation.md) lists the options). An install upgraded from a release before 1.70 keeps its earners on Docker's default bridge, which can reach your LAN, until you click **Turn on isolation** on the Fleet page. Since 2.0.7 Mysterium runs behind the fence too; a node deployed through 2.0.6 stays on the host's network until you redeploy it ([Moving your Mysterium node behind the fence](mysterium-fence.md)). [Protecting Your Home Network](home-network-security.md) covers firewalling your own bridge instead. Service credentials are encrypted at rest using Fernet symmetric encryption. The UI has no Docker access. With the shipped compose files the worker is never given the raw Docker socket: it reaches Docker through `cashpilot-socket-proxy`, which refuses privileged containers and binds of `/` or the socket (unless you put one of them, or a folder above the socket, in `CASHPILOT_PROXY_BIND_MOUNTS` or `CASHPILOT_COMPOSE_DIR`: the proxy takes those as given), and limits the worker to containers, volumes and networks with its owner label ([security defaults](security-defaults.md)). The Unraid templates and the TrueNAS app still mount the raw socket in the worker, which is root on the host ([app stores](app-stores.md)). That said, no setup is bulletproof. You are still running third-party software that routes external traffic through your network. Docker isolation significantly reduces the attack surface compared to running these services directly on your host, but it does not eliminate all risk. We recommend running CashPilot on a dedicated machine or VLAN, keeping Docker and your host OS up to date, and reviewing the open-source code of any service before deploying it. **What happens if a service container crashes?** CashPilot monitors container health continuously. If a service container exits unexpectedly, it is automatically restarted. The dashboard shows uptime and health status for every running service. + +**My Mysterium node will not start after a redeploy. Why?** + +The most common cause is a port in its UDP range that another program on the host already holds. Docker must reserve every port in the range, so one busy port stops the start. [Choosing the UDP port range](mysterium-fence.md#choosing-the-udp-port-range) explains how to pick a range that stays free. + +**My proxied copy is running but earns nothing. Why?** + +The proxy may refuse to connect to that earner's servers, or allow only ports 80 and 443. CashPilot's **Test** checks only that the proxy reaches the internet. [Probe an earner's servers through the proxy](proxies.md#probe-an-earners-servers-through-the-proxy) shows how to check the earner's own host and port. + +**Which services are heavy, and where should they run?** + +Storage nodes, relays and VPN exits, and GPU work use the most. [Budgeting a small box](fleet.md#budgeting-a-small-box) lists them and shows how to measure each container before you decide. diff --git a/docs/fleet.md b/docs/fleet.md index 6dc265b9..1c550b49 100644 --- a/docs/fleet.md +++ b/docs/fleet.md @@ -196,6 +196,43 @@ The **Fleet** page lists every box: Starting, stopping, restarting and viewing logs happen on the **Dashboard**. There, a service running on several workers shows an expandable row with one sub-row per instance; the main row shows the average CPU and memory (prefixed with `~`), and each sub-row its own figures and buttons. +## Budgeting a small box + +Earners differ a lot in what they use. Measure before you decide what runs +where, rather than spreading every service over every box. + +**The usual heavy ones.** The catalog's memory caps (`docker.resources.mem_limit` +in each `services/*/*.yml`) are limits, not usage, but they show the spread: + +| Kind | Examples | What it costs | +|---|---|---| +| Storage nodes | Sia (hostd), Storj | The highest memory caps (4 GB and 2 GB), steady disk I/O, a large data disk | +| Relays and VPN exits | Anyone Protocol, Mysterium | Memory caps of 1.5 GB and 768 MB, CPU that rises with traffic. Mysterium's published UDP ports also run `docker-proxy` processes outside its container ([what each port costs](mysterium-fence.md#what-each-port-costs)) | +| GPU compute | Salad, Vast.ai, io.net | The whole GPU, plus CPU and memory while a job runs | +| Most bandwidth earners | Honeygain, EarnApp, Traffmonetizer, Repocket | Memory caps of 128 to 256 MB | + +A [proxied copy](proxies.md) adds a tunnel container next to each earner. + +**Measure on the box.** The **Dashboard** shows each container's CPU and +memory, refreshed every few minutes. For a snapshot of everything at once: + +```bash +docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' \ + $(docker ps -q --filter name=cashpilot-) +``` + +CPU is per core: `100%` is one full core. Take a few snapshots at different +times of day, since bandwidth earners and exits are busiest when someone is +buying traffic. `docker stats` counts only what runs inside containers, so +also look at the host as a whole (`free -m`, `uptime`) and at the CashPilot UI +and worker themselves ([Memory](configuration.md#memory)). + +**Then decide.** Put storage nodes and GPU work on the machine with the disk +or the card. Keep a small box to the light earners. When what you want from a +small box at another site is its IP rather than its hardware, make it an exit +node instead of a worker +([A second site as an exit node](proxies.md#a-second-site-as-an-exit-node-instead-of-a-worker)). + ## Cross-Subnet Workers If the worker and UI are on different subnets (e.g., connected via Tailscale): diff --git a/docs/guides/mysterium.md b/docs/guides/mysterium.md index a3a6f724..4d0df99f 100644 --- a/docs/guides/mysterium.md +++ b/docs/guides/mysterium.md @@ -80,6 +80,13 @@ range. Create one rule on your router: | Internal ports | the same, 56000-56029 | | Internal IP | the LAN address of the machine running the node | +Before you pick a range, read +[Choosing the UDP port range](../mysterium-fence.md#choosing-the-udp-port-range): +one port in it held by another program stops the node from starting, and a +range inside Linux's ephemeral port range can be taken at random. Moving a +node deployed through 2.0.6? Follow +[Moving your Mysterium node behind the fence](../mysterium-fence.md). + Give that machine a fixed LAN address (a DHCP reservation), or the rule points at the wrong host after a lease change. UPnP does not work: the node sits behind the LAN fence, which drops the router, so it cannot ask the router for a diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md new file mode 100644 index 00000000..bab0b626 --- /dev/null +++ b/docs/mysterium-fence.md @@ -0,0 +1,251 @@ +# Moving your Mysterium node behind the fence + +Through 2.0.6, CashPilot ran a [MystNodes](https://mystnodes.co/?referral_code=do7v7YOoBBpbOstKQovX2pUvZYKia4ZhH3QIdNtE) +node on the host's network. From 2.0.7 it runs in its own network namespace +on the fenced earner bridge, like every other earner, and the worker starts it +only once the LAN fence is confirmed. +[Mysterium on the fenced bridge](isolation.md#mysterium-on-the-fenced-bridge) +explains why and how. This page is the step-by-step move for a node you +already run, and how to prove afterwards that it worked. + +A node deployed through 2.0.6 keeps running exactly as it is until you +redeploy it. Nothing moves on its own, so you choose when. + +## What changes for the node + +- It leaves the host's network and joins `cashpilot-isolated`. Its `NET_ADMIN` + then reaches only its own namespace, and the host's fence drops anything it + sends to your LAN, the host, a tailnet or cloud metadata. +- Its UDP range (`UDP_PORTS`) is published 1:1 on the host, so the forward on + your router stays the same. Read [Choosing the UDP port range](#choosing-the-udp-port-range) + before you redeploy: on the host's network a busy port was skipped; now a + busy port stops the whole start. +- Docker never starts it by itself. The worker starts it once it has read the + fence back complete, and a guard stops it within 30 seconds if a fence rule + goes missing or the container's shape changes. +- On a redeploy of a running node, the worker reads the node's identity before + it removes the old container and again from the new one. If they differ, + the new node is kept stopped. +- UPnP no longer works for it, since the fence drops your router. Forward the + range by hand. + +## Before you start + +### 1. Back up the keystore + +The node's identity lives in its data volume. If a redeploy ever comes up on +an empty volume, it creates a new identity, and the old one's reputation and +unsettled earnings stay with the old keys. The identity check above catches +that case only when it can read both identities. Take a backup first: +[Backing up node identities](backup.md). + +### 2. Write down the current identity + +On the machine running the node: + +```bash +docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities +``` + +Keep the `0x...` address it prints. You compare against it after the move. + +### 3. Check the isolation mode + +The node runs only in two modes: + +| Mode | Where the node runs | +|---|---| +| `on` | behind CashPilot's fence, on `cashpilot-isolated` | +| your own network (`CASHPILOT_CONTAINER_NETWORK` set) | on your network, under your firewall rules | + +In any other mode the worker refuses the deploy before it touches the running +node, and the message says what to do: + +| The message ends with | What to do | +|---|---| +| "isolation is legacy on this worker ... Switch isolation on in Fleet, then redeploy MystNodes." | On the **Fleet** page, press **Turn on isolation** for that worker. | +| "isolation is off on this worker ... Set CASHPILOT_ISOLATION=auto on this worker and restart it, then redeploy MystNodes." | Change the worker's environment and recreate the worker. | +| "isolation is unsupported on this worker ... Run it on Docker with isolation on, or on your own network (CASHPILOT_CONTAINER_NETWORK)." | The worker runs on Podman, which CashPilot cannot fence. Move the node to a Docker host, or bring your own network and rules. | + +The Fleet page shows each worker's mode. [Modes](isolation.md#modes) has the +full table. + +### 4. Upgrade the UI and the worker + +Both, and the worker that runs Mysterium in particular. The dashboard refuses +to redeploy Mysterium onto an older worker. + +### 5. Check the UDP range is free + +See [Choosing the UDP port range](#choosing-the-udp-port-range) below. One port +in the range held by another program on the host is enough to stop the node +from starting. + +## Move it + +Redeploy Mysterium on that machine from the dashboard. It keeps its data +volume, so it keeps its identity and its earnings. The +[upgrade notes](https://github.com/GeiserX/CashPilot/blob/main/UPGRADING.md) +cover the same move, including going back to 2.0.6. + +## Check that it worked + +### Same identity + +```bash +docker exec cashpilot-mysterium wget -qO- http://127.0.0.1:4050/identities +``` + +It must print the address you wrote down. If the worker saw a different one, +it stops the node and the Fleet page says why: "its identity is ..., not ... +as before the redeploy, so it is not running on the data volume that holds +its keystore". Restore the volume from your backup, then redeploy. + +### Check isolation is clean + +On the **Fleet** page, press **Check isolation** on the worker. It should +read **Isolated**, with no line about containers that "hold firewall +capabilities in this host's own network namespace". That line means a +Mysterium container is still on the host's network. [Reading the result](isolation.md#reading-the-result) +explains each line. + +### Probe from inside the node + +The node's own namespace should reach the internet and Mysterium's broker, +and nothing on your LAN. A target nothing listens on reads blocked with or +without a fence, so first pick targets that answer from the host itself: your +router's web page, another machine's SSH, a NAS share. Replace the addresses +below with yours. + +```bash +# Control, from the host: each LAN target here must answer. +T='192.168.1.1:80 192.168.1.20:22' +for t in $T; do nc -z -w 3 ${t%:*} ${t#*:} && echo "host reaches $t" || echo "host cannot reach $t"; done + +# From inside the node: each of the same targets must be blocked. +docker exec cashpilot-mysterium sh -c 'for t in 192.168.1.1:80 192.168.1.20:22; do + nc -z -w 3 ${t%:*} ${t#*:} && echo "REACHED $t" || echo "blocked $t"; done' + +# From inside the node: the internet and the broker must work. +docker exec cashpilot-mysterium nc -z -w 5 1.1.1.1 443 && echo "internet ok" +docker exec cashpilot-mysterium nc -z -w 5 broker.mysterium.network 4222 && echo "broker ok" +``` + +Only a target the host reached counts. A `REACHED` line from inside the node +means the fence is not doing its job on that path: run **Check isolation** and +follow its fix. [Test your isolation](isolation-check.md#mysterium) has a +version of this probe that runs in a separate container with no capabilities, +and how to read the fence's drop counters. + +### The node can reach its forwarded ports + +Each session first tries the forwarded range. The node checks those ports +from outside and logs the result: + +```bash +docker logs cashpilot-mysterium 2>&1 | grep -E 'All ports .* are reachable|Failed to check manual ports' | tail -5 +``` + +`All ports [56000 56001] are reachable` (your ports) means the router forward +works. `Failed to check manual ports ... globally` means it does not. Both +lines appear only when a session starts, so wait for some traffic. +[Port forwarding](guides/mysterium.md#port-forwarding) in the MystNodes guide +has the per-session count and the dashboard check that flags a node whose +sessions never connect directly. + +## If the node does not start + +- **The Fleet page says the node is held stopped.** The worker could not read + the fence back complete, or the container changed shape (privileged, a + capability or image its catalog entry does not give it, a restart policy, the + wrong network). The message names which. The worker writes the fence again + and starts the node on the next check once it reads complete. +- **"address already in use" on a UDP port.** Another program on the host + holds a port in the range. See the next section. +- **An identity mismatch.** See [Same identity](#same-identity). + +## Choosing the UDP port range + +### Why one busy port now matters + +On the host's network, Mysterium picked ports from its range by itself and +skipped any port another program held. On the bridge, Docker publishes the +range for the node, and it must reserve every port in it. If one port is +held by any process on the host, the whole start fails. + +Linux hands out ports from its ephemeral range to any program that opens a +socket without choosing a port: DNS lookups, discovery daemons, NAS management +tools. Your host's range is in: + +```bash +cat /proc/sys/net/ipv4/ip_local_port_range +``` + +It is commonly `32768 60999`. A Mysterium range inside it can be free today and +taken by some program at random tomorrow, and the node then fails on its next +start. + +### The rule + +1. **Pick a range below the start of the ephemeral range**, so the kernel never + gives those ports to anyone else. If you must stay inside it, reserve the + ports so the kernel never hands them out: + `sysctl -w net.ipv4.ip_local_reserved_ports=-` (add it to + `/etc/sysctl.conf` or a file in `/etc/sysctl.d/` so it survives a reboot). +2. **Check nothing holds the range now**, before you deploy: + + ```bash + sudo ss -uanp '( sport >= : and sport <= : )' + ``` + + No output means the range is free. Once the node runs, its own + `docker-proxy` processes show here, which is expected. + +3. **Keep it small.** Each session uses about two ports, so 20 to 30 ports + carry 10 to 15 sessions at once, more than a home node usually gets. See the + memory cost below. +4. **Forward the same range on the router**, UDP, external and internal ports + the same, to the machine's fixed LAN address. +5. **Several hosts behind one public IP need one range each.** A router + forwards a port to one LAN address only, so two nodes in one house cannot + share a range. Give each host its own range with no overlap, and one + forward per host. + +The catalog's default range is the `UDP_PORTS` field in +[services/bandwidth/mysterium.yml](https://github.com/GeiserX/CashPilot/blob/main/services/bandwidth/mysterium.yml). +Compare it with your host's ephemeral range before you accept it; set your own +in the deploy form when they overlap. + +### What each port costs + +Docker runs its port proxy, `docker-proxy`, as a separate host process per +published port and address family, so a range published on IPv4 and IPv6 runs +two processes per port. These run on the host, outside the node's container, +so `docker stats` does not count them and the node's memory limit does not +cap them. + +Measured with 101 ports: 205 processes. By resident memory that was about +2.3 MB per process, or about 460 MB in total. By proportional set size, which +splits pages the processes share, it was about 250 MB +([Limits](isolation.md#limits)). Either way the cost grows with every port. A +30-port range runs about 60 of these processes. + +To see what they cost on your host: + +```bash +ps -C docker-proxy -o rss= | awk '{s+=$1} END {printf "%d processes, %.0f MB resident\n", NR, s/1024}' +``` + +### Changing the range of a running node + +The node can only use the range it was started with, and the router can only +send sessions to the range it forwards. Change both in this order, so the node +always has a working forward: + +1. Add a router forward for the new range, next to the old one. +2. Set `UDP_PORTS` to the new range in the deploy form, and redeploy the node + on that machine. +3. Confirm sessions reach it: `All ports [...] are reachable` with the new + ports in the node's log, or `Method:manual Success:true` lines + ([Port forwarding](guides/mysterium.md#port-forwarding)). +4. Delete the old forward. diff --git a/docs/proxies.md b/docs/proxies.md index ed5757cc..47293007 100644 --- a/docs/proxies.md +++ b/docs/proxies.md @@ -191,6 +191,71 @@ A Tailscale exit node on a connection you own has none of the proxy service's limits: it carries any port, UDP included. The exit's own network can still block a port, the same as for an earner running there. +#### Probe an earner's servers through the proxy + +Do this before you rely on a copy, and again whenever one earns nothing. You +need the host and port the earner connects to. Its log usually names them +(`docker logs cashpilot--p`), or the provider's docs do. + +**Before you deploy, from any machine.** Ask the proxy directly. curl prints +the proxy's answer to its `CONNECT` request: + +```bash +# HTTP CONNECT proxy +curl -v -m 15 -o /dev/null -x 'http://USER:PASS@gate.example.net:8080' https://earner-host.example:443/ +# SOCKS5 proxy, name resolved by the proxy +curl -v -m 15 -o /dev/null -x 'socks5h://USER:PASS@gate.example.net:7000' https://earner-host.example:443/ +``` + +A `403`, `Access violation`, `connection not allowed by ruleset` or a closed +connection right after the `CONNECT` means the proxy refuses that host or +port. Try the earner's real port, not only 443: a proxy that allows 443 and +nothing else passes a 443 test. + +**After you deploy, through the copy's own tunnel.** Join the tunnel's +network namespace with a throwaway container. Run the same probe once from +the box's own network as the control, so you know what an answer looks like: + +```bash +TUN=cashpilot-tunnel--p +URL=https://earner-host.example:443/ + +# Control: straight from the box +docker run --rm --cap-drop ALL curlimages/curl:8.10.1 -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" +# Through the proxy +docker run --rm --cap-drop ALL --network "container:$TUN" curlimages/curl:8.10.1 -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" + +# What the proxy answered, if it refused +docker logs "$TUN" 2>&1 | grep -i 'replied with\|error' | tail -5 +``` + +Any HTTP status from the far server, even `404`, means the path works. A +TLS error such as `SSL_ERROR_SYSCALL`, `Connection reset by peer` or `Empty +reply from server` through the tunnel, where the control got a status, means +the proxy closed the connection. The tunnel's log then has a line such as +`Server replied with 403 [Reason: Access violation]`. It names the port; the +address in it is a placeholder from the tunnel's DNS, not the real one. + +!!! warning "A plain port check proves nothing through the tunnel" + + `nc -z` and other connect-only checks report the port open through a + tunnel even when the proxy refused, and even for an address where nothing + exists. The tunnel accepts the connection locally first, then asks the + proxy. Use a probe that needs an answer from the far server, as above. + +#### A new device the provider accepts only later + +Some providers add a new device to your account only after its client has +been connecting for a while. [ProxyRack](https://peer.proxyrack.com/ref/mpwiok3xlaxeycnn5znqlg7ipjeutxyxr6xl7vmn) +knows a new device about five minutes after it first connects, so an add +right at deploy is refused. This holds for a proxied copy as much as for a +deploy on the box. When the provider says "not yet", CashPilot keeps the +device and tries again: first after 6 minutes, then at doubling intervals up +to 4 hours apart. After 24 hours without success it stops and sends a notice, +to the bell and to your notification channel, with the manual step. It needs that provider's credentials saved in CashPilot. +While it is still trying, the dashboard does not ask you to add the device by +hand. A redeploy starts a new 24 hours. + ## Exit IPs and grouping The exit IP a copy shows is the one the tunnel saw when it came up. Copies are @@ -302,3 +367,50 @@ start fails the same way. Create a new key, paste it into the proxy's settings, then press **Start** on each copy (see [Editing a proxy](#editing-a-proxy)). A leaked reusable key lets anyone join your tailnet as that tag, which is why it must be ephemeral, tagged and used for nothing else. + +### A second site as an exit node instead of a worker + +If you have a machine at a second site (a relative's house, a holiday flat) +you can run a worker and earners there, or only make it a Tailscale exit node +and run the earners elsewhere as proxied copies that leave through it. The +second way suits a small box: an exit node only forwards packets, while each +earner is a container with its own memory and CPU. + +It makes sense when: + +- the box is short on memory or CPU, or does other work you do not want to + slow down; +- what you want from that site is its residential IP, not its hardware; +- you would rather keep containers and updates on one machine you look after. + +What changes: + +- **Only services that can run as a proxied copy move.** Anything that needs + published ports, volumes, devices or extra capabilities stays where it runs + ([What cannot run through a proxy](#what-cannot-run-through-a-proxy)). + That rules out Mysterium and the storage nodes, for example. Earners whose + terms forbid proxies or VPNs are refused too. +- **The per-IP limits move with the IP.** A copy through the exit node counts + against that site's public IP. If the box there still runs a service with a + limit of one device per IP, a copy of the same service through its exit + earns nothing; the page warns before you start it + ([Exit IPs and grouping](#exit-ips-and-grouping)). Move each service, do + not double it. +- **The traffic crosses both connections.** The earner's traffic goes from + the box running the copy, over the tailnet, to the exit node, and out from + there. The exit's connection carries it in both directions, and the copy's + own box carries it too, encrypted. Check both upload allowances. +- **The exit node still does work.** It encrypts and forwards every packet, so + its CPU rises with the traffic. Measure it under load (see + [Budgeting a small box](fleet.md#budgeting-a-small-box)). +- **Each copy is two containers on the box that runs it**, the tunnel and the + earner, and each joins the tailnet as its own ephemeral node. +- **The path to the exit node is over the internet.** With + [LAN isolation](isolation.md#lan-isolation-by-default) on, the tunnel reaches + the exit node over its public endpoints or a DERP relay, never a LAN path. A + relay carries the traffic further and slower; a direct connection between + the two sites avoids it. + +Set it up with the steps in [Tailscale exit nodes](#tailscale-exit-nodes), +then add one copy per service you move, and remove that service from the +second site's worker if one still runs there. diff --git a/zensical.toml b/zensical.toml index 8291bed4..13c0028c 100644 --- a/zensical.toml +++ b/zensical.toml @@ -132,6 +132,7 @@ nav = [ { "Protecting your home network" = "home-network-security.md" }, { "Network isolation" = "isolation.md" }, { "Test your isolation" = "isolation-check.md" }, + { "Moving Mysterium behind the fence" = "mysterium-fence.md" }, { "Activity per container" = "activity.md" }, ] }, { Operations = [ From 0be1edcae322c58ebf141f0205bec65bb094fc97 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 16:40:24 +0100 Subject: [PATCH 2/7] docs: address review on the operator guides and match the one-click Mysterium move Keep existing reserved ports when adding a range, stop curl's NO_PROXY from bypassing the probe, do not blame every reset on the proxy, list every container in the docker stats snapshot, point to Mysterium's per-IP warning, and describe the Fleet page's move button and the identity refusal from #670. --- docs/fleet.md | 6 +++--- docs/mysterium-fence.md | 41 ++++++++++++++++++++++++++++++++--------- docs/proxies.md | 13 ++++++++----- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/docs/fleet.md b/docs/fleet.md index 1c550b49..0df34aec 100644 --- a/docs/fleet.md +++ b/docs/fleet.md @@ -217,11 +217,11 @@ A [proxied copy](proxies.md) adds a tunnel container next to each earner. memory, refreshed every few minutes. For a snapshot of everything at once: ```bash -docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' \ - $(docker ps -q --filter name=cashpilot-) +docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' ``` -CPU is per core: `100%` is one full core. Take a few snapshots at different +It lists every running container, including earners you started outside +CashPilot. CPU is per core: `100%` is one full core. Take a few snapshots at different times of day, since bandwidth earners and exits are busiest when someone is buying traffic. `docker stats` counts only what runs inside containers, so also look at the host as a whole (`free -m`, `uptime`) and at the CashPilot UI diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index bab0b626..13b476a2 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -35,8 +35,9 @@ redeploy it. Nothing moves on its own, so you choose when. The node's identity lives in its data volume. If a redeploy ever comes up on an empty volume, it creates a new identity, and the old one's reputation and -unsettled earnings stay with the old keys. The identity check above catches -that case only when it can read both identities. Take a backup first: +unsettled earnings stay with the old keys. The identity check above refuses +to replace a running node whose identity it cannot read, but a node that is +already stopped is replaced without the check. Take a backup first: [Backing up node identities](backup.md). ### 2. Write down the current identity @@ -83,8 +84,19 @@ from starting. ## Move it -Redeploy Mysterium on that machine from the dashboard. It keeps its data -volume, so it keeps its identity and its earnings. The +On the **Fleet** page, a node still on the host's network shows under its +worker with a **Redeploy MystNodes to move behind the fence** button. Press +it, or redeploy Mysterium on that machine from the dashboard; both do the +same. With isolation off, the page says to switch it on first. The node keeps +its data volume, so it keeps its identity and its earnings. + +If the running node does not answer when the worker reads its identity, the +move is refused and the old container is left as it was: "its identity could +not be read from the running node ... Try again in a minute." A node that is +broken for good can be stopped from the dashboard first; a stopped node is +replaced without the check, so have your backup ready. + +The [upgrade notes](https://github.com/GeiserX/CashPilot/blob/main/UPGRADING.md) cover the same move, including going back to 2.0.6. @@ -146,7 +158,7 @@ from outside and logs the result: docker logs cashpilot-mysterium 2>&1 | grep -E 'All ports .* are reachable|Failed to check manual ports' | tail -5 ``` -`All ports [56000 56001] are reachable` (your ports) means the router forward +`All ports [ ] are reachable`, with ports from your range, means the router forward works. `Failed to check manual ports ... globally` means it does not. Both lines appear only when a session starts, so wait for some traffic. [Port forwarding](guides/mysterium.md#port-forwarding) in the MystNodes guide @@ -189,9 +201,17 @@ start. 1. **Pick a range below the start of the ephemeral range**, so the kernel never gives those ports to anyone else. If you must stay inside it, reserve the - ports so the kernel never hands them out: - `sysctl -w net.ipv4.ip_local_reserved_ports=-` (add it to - `/etc/sysctl.conf` or a file in `/etc/sysctl.d/` so it survives a reboot). + ports so the kernel never hands them out. The setting is one list, and + writing it replaces what is there, so read it first and add your range to + it: + + ```bash + sysctl net.ipv4.ip_local_reserved_ports # empty, or e.g. 8080,9000-9010 + sudo sysctl -w net.ipv4.ip_local_reserved_ports='8080,9000-9010,-' + ``` + + Put the same full list in a file under `/etc/sysctl.d/` so it survives a + reboot, and check no other file there sets it too. 2. **Check nothing holds the range now**, before you deploy: ```bash @@ -209,7 +229,10 @@ start. 5. **Several hosts behind one public IP need one range each.** A router forwards a port to one LAN address only, so two nodes in one house cannot share a range. Give each host its own range with no overlap, and one - forward per host. + forward per host. Separate ranges only stop the ports clashing. Whether a + second node on the same public IP gets any traffic is up to Mysterium: + read the warning in the [MystNodes guide](guides/mysterium.md) before you + count on it. The catalog's default range is the `UDP_PORTS` field in [services/bandwidth/mysterium.yml](https://github.com/GeiserX/CashPilot/blob/main/services/bandwidth/mysterium.yml). diff --git a/docs/proxies.md b/docs/proxies.md index 47293007..af9530cb 100644 --- a/docs/proxies.md +++ b/docs/proxies.md @@ -202,9 +202,9 @@ the proxy's answer to its `CONNECT` request: ```bash # HTTP CONNECT proxy -curl -v -m 15 -o /dev/null -x 'http://USER:PASS@gate.example.net:8080' https://earner-host.example:443/ +curl -v -m 15 -o /dev/null --noproxy '' -x 'http://USER:PASS@gate.example.net:8080' https://earner-host.example:443/ # SOCKS5 proxy, name resolved by the proxy -curl -v -m 15 -o /dev/null -x 'socks5h://USER:PASS@gate.example.net:7000' https://earner-host.example:443/ +curl -v -m 15 -o /dev/null --noproxy '' -x 'socks5h://USER:PASS@gate.example.net:7000' https://earner-host.example:443/ ``` A `403`, `Access violation`, `connection not allowed by ruleset` or a closed @@ -232,9 +232,12 @@ docker logs "$TUN" 2>&1 | grep -i 'replied with\|error' | tail -5 Any HTTP status from the far server, even `404`, means the path works. A TLS error such as `SSL_ERROR_SYSCALL`, `Connection reset by peer` or `Empty reply from server` through the tunnel, where the control got a status, means -the proxy closed the connection. The tunnel's log then has a line such as -`Server replied with 403 [Reason: Access violation]`. It names the port; the -address in it is a placeholder from the tunnel's DNS, not the real one. +something closed the connection. If the proxy refused it, the tunnel's log has +a line such as `Server replied with 403 [Reason: Access violation]`, naming the +port; the address in it is a placeholder from the tunnel's DNS, not the real +one. With no such line, the earner's server may be refusing the proxy's exit +IP rather than the proxy refusing the server, since the control and the +tunnel leave from different addresses. !!! warning "A plain port check proves nothing through the tunnel" From f7bed10eca56ac9197be90b0358fb6a5fb46f0af Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 16:52:05 +0100 Subject: [PATCH 3/7] docs: say how to keep reserved ports on Unraid, and that an operator network has no CashPilot fence --- docs/mysterium-fence.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index 13b476a2..2714e5b7 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -57,7 +57,7 @@ The node runs only in two modes: | Mode | Where the node runs | |---|---| | `on` | behind CashPilot's fence, on `cashpilot-isolated` | -| your own network (`CASHPILOT_CONTAINER_NETWORK` set) | on your network, under your firewall rules | +| your own network (`CASHPILOT_CONTAINER_NETWORK` set) | on your network. CashPilot writes no fence there, so your firewall rules are the only thing keeping it off your LAN | In any other mode the worker refuses the deploy before it touches the running node, and the message says what to do: @@ -211,7 +211,9 @@ start. ``` Put the same full list in a file under `/etc/sysctl.d/` so it survives a - reboot, and check no other file there sets it too. + reboot, and check no other file there sets it too. On Unraid, `/etc` is + rebuilt at every boot: add the `sysctl -w` line to `/boot/config/go`, + above the line that starts `emhttp`, so it runs before Docker starts. 2. **Check nothing holds the range now**, before you deploy: ```bash From 7a759940838e4c77a0e43d376fd7d284b80d75a2 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:03:31 +0100 Subject: [PATCH 4/7] docs: warn about the WebUI's LAN exposure before the move, and tell the old node's ports from a real conflict --- docs/mysterium-fence.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index 2714e5b7..a00b4579 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -76,7 +76,15 @@ full table. Both, and the worker that runs Mysterium in particular. The dashboard refuses to redeploy Mysterium onto an older worker. -### 5. Check the UDP range is free +### 5. Check how you reach the WebUI + +If `UI_ADDRESS` holds only `127.0.0.1`, skip this. If it holds a LAN address +for a reverse proxy on another machine, know that after the move port 4449 is +published through Docker's forwarding, so a host `INPUT` rule no longer limits +who reaches it. Add a `DOCKER-USER` rule or keep the proxy on the same machine +before you move ([WebUI address](guides/mysterium.md#webui-address)). + +### 6. Check the UDP range is free See [Choosing the UDP port range](#choosing-the-udp-port-range) below. One port in the range held by another program on the host is enough to stop the node @@ -220,8 +228,10 @@ start. sudo ss -uanp '( sport >= : and sport <= : )' ``` - No output means the range is free. Once the node runs, its own - `docker-proxy` processes show here, which is expected. + No output means the range is free. Two holders are expected: an old + node still on the host's network shows as `myst`, and its sockets close + when the move stops it; a node already on the bridge shows as + `docker-proxy`. Any other program in the list means a different range. 3. **Keep it small.** Each session uses about two ports, so 20 to 30 ports carry 10 to 15 sessions at once, more than a home node usually gets. See the From 4f55aef6eac9cc97135fea68ecc99b476ca4c4a1 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:13:58 +0100 Subject: [PATCH 5/7] docs: take the LAN address out of UI_ADDRESS when the proxy moves, and check which container holds a port --- docs/mysterium-fence.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index a00b4579..d7e3d6c2 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -81,8 +81,11 @@ to redeploy Mysterium onto an older worker. If `UI_ADDRESS` holds only `127.0.0.1`, skip this. If it holds a LAN address for a reverse proxy on another machine, know that after the move port 4449 is published through Docker's forwarding, so a host `INPUT` rule no longer limits -who reaches it. Add a `DOCKER-USER` rule or keep the proxy on the same machine -before you move ([WebUI address](guides/mysterium.md#webui-address)). +who reaches it. Before you move, either add a `DOCKER-USER` rule, or move the +proxy to the same machine and also take the LAN address out of `UI_ADDRESS` +in the deploy form (enter `localhost` for loopback only). Moving the proxy +alone leaves the LAN address published +([WebUI address](guides/mysterium.md#webui-address)). ### 6. Check the UDP range is free @@ -231,7 +234,10 @@ start. No output means the range is free. Two holders are expected: an old node still on the host's network shows as `myst`, and its sockets close when the move stops it; a node already on the bridge shows as - `docker-proxy`. Any other program in the list means a different range. + `docker-proxy`. A `docker-proxy` can also belong to another container, + so check which one publishes the ports with + `docker ps --format '{{.Names}} {{.Ports}}' | grep udp`. Any holder but + the node you are moving means a different range. 3. **Keep it small.** Each session uses about two ports, so 20 to 30 ports carry 10 to 15 sessions at once, more than a home node usually gets. See the From 101ccf80c1453960602e49ac3eff1b873206d03a Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:25:00 +0100 Subject: [PATCH 6/7] docs: warn that Docker before 28.0 exposes loopback-published ports to the LAN segment --- docs/mysterium-fence.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index d7e3d6c2..a916440d 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -78,7 +78,11 @@ to redeploy Mysterium onto an older worker. ### 5. Check how you reach the WebUI -If `UI_ADDRESS` holds only `127.0.0.1`, skip this. If it holds a LAN address +On a Docker engine older than 28.0, a port published on `127.0.0.1` can still +be reached by other machines on the same LAN segment, so upgrade Docker or +block port 4449 with a `DOCKER-USER` rule first (`docker version` shows the +engine). If `UI_ADDRESS` holds only `127.0.0.1` on Docker 28.0 or later, skip +the rest of this step. If it holds a LAN address for a reverse proxy on another machine, know that after the move port 4449 is published through Docker's forwarding, so a host `INPUT` rule no longer limits who reaches it. Before you move, either add a `DOCKER-USER` rule, or move the From 54e11a633474ff0135cfbca070230b42178aeba9 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 17:37:07 +0100 Subject: [PATCH 7/7] docs: address the second review round on the operator guides Say the fence claims hold in the on mode only, that the broker probe proves TCP only, that a redeploy restarts the retry window only after CashPilot gave up, how to go back to the default range, what the docker-proxy total counts, and that a failed replacement now puts the old node back (#675). --- docs/fleet.md | 4 +++- docs/mysterium-fence.md | 39 +++++++++++++++++++++++++++++---------- docs/proxies.md | 31 +++++++++++++++++++------------ 3 files changed, 51 insertions(+), 23 deletions(-) diff --git a/docs/fleet.md b/docs/fleet.md index 0df34aec..cf8c10a2 100644 --- a/docs/fleet.md +++ b/docs/fleet.md @@ -214,7 +214,9 @@ in each `services/*/*.yml`) are limits, not usage, but they show the spread: A [proxied copy](proxies.md) adds a tunnel container next to each earner. **Measure on the box.** The **Dashboard** shows each container's CPU and -memory, refreshed every few minutes. For a snapshot of everything at once: +memory, refreshed every few minutes, for the services it reports; earners +you started outside CashPilot may not show there. For a snapshot of everything +at once: ```bash docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' diff --git a/docs/mysterium-fence.md b/docs/mysterium-fence.md index a916440d..2a077c07 100644 --- a/docs/mysterium-fence.md +++ b/docs/mysterium-fence.md @@ -3,7 +3,10 @@ Through 2.0.6, CashPilot ran a [MystNodes](https://mystnodes.co/?referral_code=do7v7YOoBBpbOstKQovX2pUvZYKia4ZhH3QIdNtE) node on the host's network. From 2.0.7 it runs in its own network namespace on the fenced earner bridge, like every other earner, and the worker starts it -only once the LAN fence is confirmed. +only once the LAN fence is confirmed. That is the `on` isolation mode. On a +network of your own (`CASHPILOT_CONTAINER_NETWORK`) CashPilot writes and checks +no fence, so your firewall rules, tested from inside the node, are what keep it +off your LAN. [Mysterium on the fenced bridge](isolation.md#mysterium-on-the-fenced-bridge) explains why and how. This page is the step-by-step move for a node you already run, and how to prove afterwards that it worked. @@ -155,10 +158,12 @@ docker exec cashpilot-mysterium sh -c 'for t in 192.168.1.1:80 192.168.1.20:22; # From inside the node: the internet and the broker must work. docker exec cashpilot-mysterium nc -z -w 5 1.1.1.1 443 && echo "internet ok" -docker exec cashpilot-mysterium nc -z -w 5 broker.mysterium.network 4222 && echo "broker ok" +docker exec cashpilot-mysterium nc -z -w 5 broker.mysterium.network 4222 && echo "broker port reachable" ``` -Only a target the host reached counts. A `REACHED` line from inside the node +The broker line proves only that the TCP port answers, not that the node's +session with the broker works; the node's log shows that. Only a target the +host reached counts. A `REACHED` line from inside the node means the fence is not doing its job on that path: run **Check isolation** and follow its fix. [Test your isolation](isolation-check.md#mysterium) has a version of this probe that runs in a separate container with no capabilities, @@ -185,10 +190,15 @@ sessions never connect directly. - **The Fleet page says the node is held stopped.** The worker could not read the fence back complete, or the container changed shape (privileged, a capability or image its catalog entry does not give it, a restart policy, the - wrong network). The message names which. The worker writes the fence again - and starts the node on the next check once it reads complete. + wrong network). The message names which. For a missing fence rule, the + worker writes the fence again and starts the node on a later check once it + reads complete. For a container that changed shape, redeploy the node from + the dashboard. - **"address already in use" on a UDP port.** Another program on the host - holds a port in the range. See the next section. + holds a port in the range. In releases after 2.1.0 the worker puts the old node back and + starts it again when Docker refuses the new one, so you keep earning while + you pick a free range; on an older worker the old container is already + gone. See the next section. - **An identity mismatch.** See [Same identity](#same-identity). ## Choosing the UDP port range @@ -226,7 +236,9 @@ start. ``` Put the same full list in a file under `/etc/sysctl.d/` so it survives a - reboot, and check no other file there sets it too. On Unraid, `/etc` is + reboot, and check no other sysctl file (`/etc/sysctl.conf`, + `/usr/lib/sysctl.d/`, `/run/sysctl.d/`) sets it too. After the next + reboot, read the value again. On Unraid, `/etc` is rebuilt at every boot: add the `sysctl -w` line to `/boot/config/go`, above the line that starts `emhttp`, so it runs before Docker starts. 2. **Check nothing holds the range now**, before you deploy: @@ -235,7 +247,7 @@ start. sudo ss -uanp '( sport >= : and sport <= : )' ``` - No output means the range is free. Two holders are expected: an old + No rows under the header means the range is free. Two holders are expected: an old node still on the host's network shows as `myst`, and its sockets close when the move stops it; a node already on the bridge shows as `docker-proxy`. A `docker-proxy` can also belong to another container, @@ -263,7 +275,8 @@ in the deploy form when they overlap. ### What each port costs -Docker runs its port proxy, `docker-proxy`, as a separate host process per +With Docker's default settings (the userland proxy on), Docker runs its port +proxy, `docker-proxy`, as a separate host process per published port and address family, so a range published on IPv4 and IPv6 runs two processes per port. These run on the host, outside the node's container, so `docker stats` does not count them and the node's memory limit does not @@ -275,7 +288,8 @@ splits pages the processes share, it was about 250 MB ([Limits](isolation.md#limits)). Either way the cost grows with every port. A 30-port range runs about 60 of these processes. -To see what they cost on your host: +To see what they cost on your host (this counts every container's published +ports, not only Mysterium's): ```bash ps -C docker-proxy -o rss= | awk '{s+=$1} END {printf "%d processes, %.0f MB resident\n", NR, s/1024}' @@ -294,3 +308,8 @@ always has a working forward: ports in the node's log, or `Method:manual Success:true` lines ([Port forwarding](guides/mysterium.md#port-forwarding)). 4. Delete the old forward. + +To go back to the catalog's default range, typing it in is not enough: the +form treats the prefilled default as untouched and keeps the range on record. +Remove the service from that machine (its data volume stays) and deploy it +again. diff --git a/docs/proxies.md b/docs/proxies.md index af9530cb..a21a5041 100644 --- a/docs/proxies.md +++ b/docs/proxies.md @@ -201,16 +201,18 @@ need the host and port the earner connects to. Its log usually names them the proxy's answer to its `CONNECT` request: ```bash -# HTTP CONNECT proxy -curl -v -m 15 -o /dev/null --noproxy '' -x 'http://USER:PASS@gate.example.net:8080' https://earner-host.example:443/ +# HTTP CONNECT proxy; curl asks for the password, so it stays out of your shell history +curl -v -m 15 -o /dev/null --noproxy '' -x 'http://gate.example.net:8080' --proxy-user USER https://earner-host.example:443/ # SOCKS5 proxy, name resolved by the proxy -curl -v -m 15 -o /dev/null --noproxy '' -x 'socks5h://USER:PASS@gate.example.net:7000' https://earner-host.example:443/ +curl -v -m 15 -o /dev/null --noproxy '' -x 'socks5h://gate.example.net:7000' --proxy-user USER https://earner-host.example:443/ ``` A `403`, `Access violation`, `connection not allowed by ruleset` or a closed connection right after the `CONNECT` means the proxy refuses that host or -port. Try the earner's real port, not only 443: a proxy that allows 443 and -nothing else passes a 443 test. +port. A `200` to the `CONNECT` means the proxy let it through, so a TLS +error after that comes from the far server or the protocol, not the proxy. +Try the earner's real port, not only 443: a proxy that allows 443 and nothing +else passes a 443 test. **After you deploy, through the copy's own tunnel.** Join the tunnel's network namespace with a throwaway container. Run the same probe once from @@ -221,12 +223,12 @@ TUN=cashpilot-tunnel--p URL=https://earner-host.example:443/ # Control: straight from the box -docker run --rm --cap-drop ALL curlimages/curl:8.10.1 -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" +docker run --rm --cap-drop ALL curlimages/curl:8.10.1 --noproxy '*' -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" # Through the proxy -docker run --rm --cap-drop ALL --network "container:$TUN" curlimages/curl:8.10.1 -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" +docker run --rm --cap-drop ALL --network "container:$TUN" curlimages/curl:8.10.1 --noproxy '*' -sS -m 10 -o /dev/null -w '%{http_code}\n' "$URL" -# What the proxy answered, if it refused -docker logs "$TUN" 2>&1 | grep -i 'replied with\|error' | tail -5 +# What the proxy answered, if it refused, in the last minute only +docker logs --since 1m "$TUN" 2>&1 | grep -i 'replied with\|error' ``` Any HTTP status from the far server, even `404`, means the path works. A @@ -235,7 +237,8 @@ reply from server` through the tunnel, where the control got a status, means something closed the connection. If the proxy refused it, the tunnel's log has a line such as `Server replied with 403 [Reason: Access violation]`, naming the port; the address in it is a placeholder from the tunnel's DNS, not the real -one. With no such line, the earner's server may be refusing the proxy's exit +one. The earner's own traffic shares this log, so match the port and the time +of your probe. With no such line, the earner's server may be refusing the proxy's exit IP rather than the proxy refusing the server, since the control and the tunnel leave from different addresses. @@ -257,7 +260,8 @@ device and tries again: first after 6 minutes, then at doubling intervals up to 4 hours apart. After 24 hours without success it stops and sends a notice, to the bell and to your notification channel, with the manual step. It needs that provider's credentials saved in CashPilot. While it is still trying, the dashboard does not ask you to add the device by -hand. A redeploy starts a new 24 hours. +hand. A redeploy after CashPilot gave up starts a new 24 hours; a redeploy +while it is still trying keeps the current window. ## Exit IPs and grouping @@ -414,6 +418,9 @@ What changes: relay carries the traffic further and slower; a direct connection between the two sites avoids it. -Set it up with the steps in [Tailscale exit nodes](#tailscale-exit-nodes), +On a Linux machine, the exit node also needs IP forwarding turned on before +Tailscale lets it advertise itself; Tailscale's +[exit node guide](https://tailscale.com/kb/1103/exit-nodes) has the commands +per system. Then set it up with the steps in [Tailscale exit nodes](#tailscale-exit-nodes), then add one copy per service you move, and remove that service from the second site's worker if one still runs there.