From 38575a810d14855c28de4e7fa11368551466d283 Mon Sep 17 00:00:00 2001 From: Josh Poole Date: Wed, 26 Aug 2026 13:46:07 +0100 Subject: [PATCH] 20260826 - Pin resolvers on the two services that reach the internet Docker copies the host's /etc/resolv.conf into a container once, at creation, and never refreshes it: not on restart, not on reboot. There is no loopback stripping for network_mode: host, correctly, because the container shares the host's network namespace. So a container created while that file was still wrong keeps a dead resolver for its entire life. That stranded five boards. Their telemetry containers were created in the seconds before NetworkManager wrote a usable resolv.conf, so they pointed at 127.0.0.53, where nothing listens on these boards. The hosts resolved fine throughout. The boards checked in to Mender, ran blah2, collected detections and reported status active while sending nothing at all, and every check that ran outside the container reported them healthy. Sacremento sat like that for five days. See ClickUp 86cba42d8. Setting dns here makes Docker write the file from these values and ignore the host's copy, so creation timing stops mattering. Docker records it in the file it writes, as "Overrides: [nameservers]". Only two services resolve anything off-box: retina-telemetry for api.retina.fm and retina-spectrum for tower-finder.retina.fm. Everything else talks over 127.0.0.1, and tar1090 is on the bridge network where Docker's embedded resolver tracks the host file dynamically and never had this problem. Verified on ret824685c9 with Compose v5.1.3, against the real deployed manifest: - renders as [1.1.1.1, 8.8.8.8, 2606:4700:4700::1111], and under the spectrum profile for retina-spectrum - NODE_DNS_* overrides render correctly and mix with the defaults - with the host's resolv.conf set to the exact broken content from the incident, a host-network container created without dns: inherits 127.0.0.53 and cannot resolve, while retina-telemetry recreated with this block resolves normally NODE_DNS_* is overridable in .env for standalone deployments only. On a Mender node config-merger regenerates the manifests .env wholesale on every run, so a value added there by hand does not survive. A site that blocks outbound DNS to public resolvers needs config-merger taught to emit these first. Co-Authored-By: Claude Opus 5 --- .env.example | 17 +++++++++++++++++ docker-compose.yml | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/.env.example b/.env.example index c385d33..ed1e190 100644 --- a/.env.example +++ b/.env.example @@ -32,3 +32,20 @@ ADSBLOL_RADIUS=40 # retina-telemetry logging. INFO is quiet by design: it logs starting, # registering, and the failures that need an operator. DEBUG logs every request. # TELEMETRY_LOG_LEVEL=DEBUG + +# --- DNS --- +# retina-telemetry and retina-spectrum pin their resolvers instead of inheriting +# the host's, because a container keeps the copy of /etc/resolv.conf it was +# given at creation for life, and one created before the network settled would +# otherwise never resolve anything again. +# +# Override these where the site blocks outbound DNS to public resolvers, or +# where policy requires the local resolver. Point them at the LAN gateway or +# the site's own servers. glibc reads at most three nameservers, so a fourth +# would be ignored. +# +# Standalone only. On a Mender-managed node config-merger regenerates the +# manifests .env on every run, so setting these there does not survive. +# NODE_DNS_1=192.168.1.1 +# NODE_DNS_2=8.8.8.8 +# NODE_DNS_3=2606:4700:4700::1111 diff --git a/docker-compose.yml b/docker-compose.yml index 13201e0..7ff9a42 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -149,6 +149,13 @@ services: restart: "no" image: ghcr.io/offworldlabs/retina-spectrum:${SPECTRUM_V:-v0.1.1} network_mode: host + # Pinned for the same reason as retina-telemetry: this resolves + # tower-finder.retina.fm, so it must not depend on the host's resolv.conf + # having been correct at the instant the container happened to be created. + dns: + - ${NODE_DNS_1:-1.1.1.1} + - ${NODE_DNS_2:-8.8.8.8} + - ${NODE_DNS_3:-2606:4700:4700::1111} pid: "host" privileged: true environment: @@ -188,6 +195,27 @@ services: # network_mode: host to reach blah2_api on 127.0.0.1:3000. It binds no # listening ports of its own — every input here is a poll or a file read. network_mode: host + # Pinned resolvers rather than the host's. Docker copies the host's + # /etc/resolv.conf into a container once, at creation, and never refreshes + # it: not on restart, not on reboot. A container created while that file was + # still wrong therefore stays wrong for its entire life. That is what + # stranded five boards in August 2026. They ran blah2, collected detections + # and reported active while resolving nothing, because their telemetry + # container was created in the seconds before NetworkManager wrote a usable + # resolv.conf. Setting dns here makes Docker write the file from these + # values and ignore the host's copy entirely, so creation timing stops + # mattering. Docker confirms it in the file it writes, as "Overrides: + # [nameservers]". + # + # NODE_DNS_* can be set in .env for standalone deployments. On a Mender + # node it cannot, yet: config-merger rewrites the manifests .env wholesale + # on every run, so a value added there by hand is lost at the next config + # merge. A site that blocks outbound DNS to public resolvers therefore + # needs config-merger taught to emit these before the override is usable. + dns: + - ${NODE_DNS_1:-1.1.1.1} + - ${NODE_DNS_2:-8.8.8.8} + - ${NODE_DNS_3:-2606:4700:4700::1111} # No depends_on, unlike every other service. This is the one that must keep # reporting while the rest of the stack is crash-looping, which is the whole # point of it: gating it on config-merger would leave it dead exactly when