Repository navigation
build(docker): Alpine 3.24.1 runtime with Mosquitto 2.1 compatibility - #57
Merged
Merged
Conversation
Move the runtime stage of Dockerfile and Dockerfile.go from the EOL alpine:3.20 to alpine:3.24.1, pinned by digest. The Go builder (golang:1.27.1-alpine3.24, digest-pinned) is unchanged, and the Go binaries are byte-identical to the Alpine 3.20 image per architecture. Alpine 3.24 ships Mosquitto 2.1.2, which changed two behaviours the in-container broker relied on: - allow_duplicate_messages now defaults to true. The ingestor can hold overlapping subscriptions (meshcore/+/+/packets + meshcore/#), so each packet was delivered twice and packetsTotal, tx_dupes, observer upserts and observation rows doubled. docker/mosquitto.conf sets it back to false. Mosquitto 2.0.18 and 2.1.2 both honour it and both log a deprecation warning; upstream plans to remove it in Mosquitto 3.0. - PUID/PGID are honoured for privilege dropping. entrypoint-go.sh exports /app/data/.env, so a PUID there made the broker run as that uid and fail to write /var/lib/mosquitto/mosquitto.db. The supervisord programs that start the broker now strip PUID/PGID with `env -u` for mosquitto only; every other process keeps its environment and user. scripts/test-docker-mosquitto-compat.sh is a behavioural regression test for both. It runs a built image with synthetic data and --network none, publishes to the in-container broker and checks deliveries, ingestor counters, the broker's uid/gid and persistence across a broker restart. It fails on the unfixed Alpine 3.24.1 image and passes with this change. It is not registered in CI. Dockerfile.go still fails at `go mod download` because of missing COPY internal/ lines; that pre-existing issue is out of scope here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the independent review of scripts/test-docker-mosquitto-compat.sh. Only the test changes; the image, broker config and supervisord files are untouched and the existing 28 functional checks and their expected values stay as they were. - Cleanup: containers get a unique com.corescope.test.run=<RUN_ID> label and are registered right after `docker create`. On pass, failure, start errors and SIGINT/SIGTERM/SIGHUP the test keeps their logs, removes only containers carrying its own run label together with their anonymous volumes (docker rm -f -v, with a checked fallback) and verifies both are gone. A failing run keeps its non-zero status; a cleanup failure turns a passing run into exit 3. - No registry contact: the image reference must exist locally, is resolved once to its image ID, and every container is created from that ID with --pull never. A missing image fails before any container is created. - Wall-clock limits: TOTAL_S (whole run, with a cleanup reserve), WAIT_S (each wait) and EXEC_S (each docker command) replace iteration-counted loops; wget uses -T. SIGKILL still cannot run the cleanup, so the header keeps recommending an outer timeout. - Completion: every former fire-and-forget wait is now a precondition that stops the run when it does not hold. Exit 0 requires all 28 checks and all 15 preconditions to have run and passed; the image ID and the Mosquitto version are printed. The test still covers only the default supervisord start path and says nothing about Caddy start-up crashes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…osed Follow-up to the independent review of 3c1b7ca (N1, N3, N5, N8). Only scripts/test-docker-mosquitto-compat.sh changes; the 28 checks, the 15 preconditions and their expected values are unchanged. - N1: stdout/stderr are saved on fds 3/4 at start and restored at the top of the signal and exit traps and of finish, so signal, RESULT and cleanup lines are printed even when a signal lands inside a redirected docker call. Child processes get fds 3/4 closed. - N3: if the case 1 container cannot be verifiably stopped and removed, the run stops with a non-zero status before case 2 is created. A later successful cleanup pass never clears that earlier failure. - N5: the exit path no longer waits unbounded on a background child. Own children get TERM, then KILL after a short grace period, and only while their PID still has the parent PID and start time recorded at spawn, so a recycled PID is never signalled. - N8: containers and volumes are present, absent or unknown. Absence is only concluded from a successful docker listing that no longer contains the object; failed or timed-out listings, inspects and post-removal checks are reported as CLEANUP UNRESOLVED and make the run non-zero, without any broader fallback deletion. Cleanup now runs before the RESULT line, and a verified cleanup is part of exit 0. SIGKILL still cannot run any of this, so the header keeps recommending an outer timeout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tate Follow-up to the independent review of dcbdd90 (blocker B1). Only scripts/test-docker-mosquitto-compat.sh changes; the 28 checks, the 15 preconditions and their expected values are unchanged. container_state and volume_state matched with printf ... | grep -qxF. grep -q exits at the first hit, the writing printf then gets SIGPIPE, and pipefail turns that hit into a non-zero pipeline, so on a host whose docker ps -a -q or docker volume ls -q listing is large enough an existing container or volume was reported as absent: cleanup skipped it, "already gone" was printed for a container that was still running, and case 2 could be started on top of case 1. The match now reads the whole listing, and the two status sources are kept apart: writer 0 and grep 0 is present, writer 0 and grep 1 is absent, and anything else - a failed writer or a grep error - is unknown, which stays CLEANUP UNRESOLVED and makes the run non-zero. A failed or timed-out listing is still unknown as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the Go runtime image (
Dockerfile,Dockerfile.go) fromalpine:3.20toalpine:3.24.1@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b,and adds the two Mosquitto 2.1 compatibility fixes the upgrade needs. It also adds a
regression test for both Mosquitto behaviours.
The Go builder is unchanged:
golang:1.27.1-alpine3.24@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125.Relation to #22: #22 bumps only the Alpine base in
Dockerfile. On its own that brings inMosquitto 2.1.2, which changes two defaults this image depends on (see below). This PR is meant as
the safer alternative: the same base bump, pinned by digest, plus the compatibility fixes and a
test. #22 is left untouched.
Changes (6 files, 4 commits)
Dockerfile,Dockerfile.goFROM alpine:3.20→alpine:3.24.1@sha256:28bd5fe8…docker/mosquitto.confallow_duplicate_messages falsedocker/supervisord-go.conf,docker/supervisord-go-no-caddy.conf/usr/bin/env -u PUID -u PGID /usr/sbin/mosquitto …scripts/test-docker-mosquitto-compat.shCommits:
079a96b0(runtime and config), then3c1b7ca1,dcbdd90band2f3f03dd, which changeonly the test script. The production files are unchanged since
079a96b0.Mosquitto 2.1 compatibility
Duplicate delivery. Mosquitto 2.1.0 changed the default of
allow_duplicate_messagestotrue. The ingestor's overlapping default subscriptions (meshcore/+/+/packetsandmeshcore/#) then receive each message twice. In a controlled arm64 run with one publish,the unfixed candidate compared with baseline showed:
tx_dupes: 0 → 1DB dedup did not catch the second delivery. The two observation rows got
observer_idxNULLand 1, which the unique index treats as distinct. Setting
allow_duplicate_messages falserestores the 2.0 behaviour for the built-in broker.
PUID/PGID. Mosquitto 2.1 drops privileges to
PUID/PGIDwhen they are set.entrypoint-go.shexports/app/data/.env, so a.envcontainingPUID=1000made the brokerrun as 1000:1000. It then failed to write
/var/lib/mosquitto/mosquitto.db(Permission denied), and retained messages were lost across restarts. The variables are now removed forthe broker process only. Caddy, the ingestor and the server still inherit them.
Relevant package changes (Alpine 3.20.10 → 3.24.1, 51 → 62 packages)
New packages: alpine-release, brotli-libs, gmp, gnutls, libapk, libedit, libmicrohttpd, libtasn1,
nettle, p11-kit, zstd-libs.
Unchanged: the
corescope-server,corescope-ingestorandcorescope-decryptbinaries arebyte-identical between the baseline and candidate images (go1.27.1, CGO disabled).
Evidence
A. Earlier image evidence (images built from the original base
c5d472ba)These images were built from the original application base
c5d472ba, not from theintegration with current master. Image IDs:
bdf96341027e…, alpine 3.20.10f3f0042d5abd…, this PR's runtime change342f149535e9…bc398d24dfca…, arm642009c2e6a722…Build logs record
alpine:3.24.1@sha256:28bd5fe8…and the builder digest above.Regression test, fail-before/pass-after
079a96b0: candidate FAIL, candfix PASS.2f3f03dd(sha256b6058a17…):Caddy init, native amd64: 30 starts each for baseline, candidate and candfix, plus 100
extra candfix starts. That is 130 native candfix starts with 0 non-zero exits and 0
panics/fatal errors.
Full stack, native amd64: 20 baseline + 20 candfix starts, each observed for 60 s.
All 40 were ready in about 3.7–3.9 s and exited 0. No OOM, 0 panics. Every run had
tx_ins=1 tx_dup=0 obs_ins=1, and the broker ran as 101:102.No-Caddy variant, native amd64: baseline and candfix both exit 0 with
tx_inserted=1 obs_inserted=1 tx_dupes=0 write_errors=0.Persistence: retained messages survive a broker restart, and the persistence DB is owned
by the broker user. Both are covered by the regression test.
Synthetic TLS (isolated network, test CA): on baseline and candfix the ingestor subscribes
with the test CA. It does not connect with an untrusted CA or a wrong hostname (0 subscriptions,
0 rows). A separate probe confirmed the rejection reasons (verify codes 19 and 62).
Rollback baseline → candfix → baseline on a shared data volume: schema and migration hashes
are stable, and rows are kept in both directions (counts 2/3/3 as expected, integrity ok).
Real endpoints, 6 credential-free TLS handshakes from native amd64 (2026-09-16 11:51 UTC):
one per endpoint per image, all verified with both images.
store was used, with full chain and hostname verification.
DefaultGODEBUGas theingestor (go directive
1.22).mqtt.meshview.dk:888369729b8e…, ISRG Root X196bcec06…acme-v02.api.letsencrypt.org:443acme.zerossl.com:4437bb647a6…, USERTrust RSAe793c9b0…Intermediates and roots were identical between baseline and candfix. None of the 26 roots
removed in the new CA bundle appear in these chains.
B. Fresh integration check against current master (2026-09-16)
12cc30ffdb1bd46cf847da263ca4e92f6b5d935bis 8 commits ahead ofc5d472ba. Thosecommits touch
public/app.js, two JS tests,test-all.shand one added test line in.github/workflows/deploy.yml. There is no overlap with this PR's files and nothing in build,runtime, config or TLS.
git merge-tree --write-tree: no conflicts. Against master the diff is exactly the 6 files,byte-identical to the feature diff. Master's own changes, including
deploy.yml, are preservedunchanged.
DockerfileandDockerfile.goare byte-identical to the build context of the testedcandidate images.
bash -non the test script: okgit diff --check: cleanscripts/check-dockerfile-internal-pkgs.sh: okgo test -run TestForkGuard ./cmd/server(go1.27.1, offline): 3/3 PASSsupervisord-go*.conffiles parse with supervisor 4.3.0.env -u PUID -u PGIDremoves both variables.mosquitto.confwith 0 errors and runs asmosquitto, evenwith
PUID/PGID=1000set.Limitations
proven to be the cause. Native amd64 (above) showed no crashes.
corescope-serverexits once withdb-not-readyand is restarted bysupervisor. This happens in all stack runs, baseline included, so it is not introduced here.
allow_duplicate_messages: deprecated upstream (removal planned for Mosquitto 3.0; thebroker logs a warning at startup). It only affects the built-in broker.
cover. The underlying DB dedup gap (
observer_idxNULL vs 1 inidx_observations_dedup) isnot fixed here.
DigiCert Global Root CA, DigiCert High Assurance EV Root CA, Entrust, GTS Root R2 and COMODO).
The handshake results are a snapshot; certificate chains can change.
ACME account registration, issuance or renewal. A verified handshake only shows the endpoint's
certificate is trusted.
rejectUnauthorizedand broker URLs.Dockerfile.gobuild failure and theDockerfile.nodelimitation.
scripts/test-docker-mosquitto-compat.shis not registered in CI.🤖 Generated with Claude Code