Pin resolvers on the two services that reach the internet

[Purple10101]

Fixes the stale-container-resolver fault behind ClickUp 86cba42d8.

The fault

Docker copies the host's /etc/resolv.conf into a container once, at creation, and never refreshes it: not on restart, not on reboot. There is no loopback stripping under network_mode: host, correctly, because the container shares the host's network namespace. A container created while that file was still wrong therefore keeps a dead resolver for its entire life.

Five boards were in that state. Their telemetry containers had been created in the seconds before NetworkManager wrote a usable resolv.conf, so they pointed at 127.0.0.53, where nothing listens on these boards. The hosts resolved fine throughout. The boards checked in to Mender, ran blah2, collected detections and reported status: active while sending nothing, and every check running outside the container reported them healthy. Sacremento sat like that for five days.

A single board showed it cleanly. Same host, same daemon, containers created at different moments:

container	network	created	resolver	resolves
retina-telemetry	host	Aug 19, 11:27:36	127.0.0.53	no
adsb2dd	host	Aug 19, 11:27:36	127.0.0.53	no
tar1090	bridge	Aug 19, 11:32:53	127.0.0.11	yes
blah2-api	host	Aug 24, 19:02:28	192.168.1.1	yes

Boot was 11:32:29. NetworkManager wrote the correct file at 11:32:53. Only the clock differed.

The change

dns: on the two services that resolve anything off-box: retina-telemetry for api.retina.fm, retina-spectrum for tower-finder.retina.fm. Those are the only external URLs in the compose. Everything else talks over 127.0.0.1, and tar1090 is on the bridge network where Docker's embedded resolver tracks the host file dynamically and never had this problem.

Docker then writes the container's resolv.conf from these values and ignores the host's copy, so creation timing stops mattering. It records this in the file itself, as Overrides: [nameservers].

Three entries, because glibc reads at most three nameservers and ignores the rest. Two providers rather than two addresses from one, so a single provider outage doesn't take fleet telemetry with it.

Verification

On ret824685c9 with Compose v5.1.3, against the real deployed manifest:

• Renders as [1.1.1.1, 8.8.8.8, 2606:4700:4700::1111], and the same for retina-spectrum under its profile.
• NODE_DNS_* overrides render correctly and mix with the remaining defaults.
• With the host's resolv.conf set to the exact broken content from the incident, a host-network container created without dns: inherits 127.0.0.53 and cannot resolve, while retina-telemetry recreated with this block resolves normally. The fault reproduces without the change and does not occur with it.

The board's telemetry error count dropped from 4 to 1, the survivor being a server-side 400: tx_lat, so it is reaching the API.

Note that CI does not validate the compose file (ruff, vulture and the config-merger suite only), so the hardware check above is the evidence for this one.

Known limitation

NODE_DNS_* is overridable in .env for standalone deployments only. On a Mender node, config-merger regenerates the manifests .env wholesale on every run, so a value added there by hand does not survive. A site that blocks outbound DNS to public resolvers needs config-merger taught to emit these first. Documented in both files rather than left to be discovered.

🤖 Generated with Claude Code