ops: bundled Mosquitto opt-in, normalised DISABLE_MOSQUITTO, pprof loopback default (#441)

.env.example

@@ -42,10 +42,15 @@ PROD_MQTT_PORT=1883
 # Used by: docker compose
 PROD_MQTT_BIND=
 
-# Disable internal Mosquitto broker (set true to use external MQTT only)
-# Default: false
-# Used by: manage.sh + docker compose overrides
-DISABLE_MOSQUITTO=false
+# Disable the bundled Mosquitto broker. The bundled broker is OPT-IN (#441):
+# leave this true and point mqttSources in config.json (or MQTT_BROKER below)
+# at your own broker, or set it to false to run the bundled broker.
+# true/1/yes/on and false/0/no/off are accepted, trimmed and in any case, by
+# manage.sh, docker compose + the container entrypoint and the ingestor
+# alike; an unset, empty or unrecognised value means true.
+# Default: true
+# Used by: manage.sh + docker compose overrides + the container entrypoint
+DISABLE_MOSQUITTO=true
 
 # --- Staging (HTTP only, no HTTPS) ---
 # Data directory
@@ -80,7 +85,7 @@ MQTT_BROKER=
 MQTT_TOPIC=
 
 # Refuse to start the ingestor when the broker setup cannot ingest anything,
-# instead of only logging it. With DISABLE_MOSQUITTO=true and no mqttSources
+# instead of only logging it. With the bundled broker off and no mqttSources
 # (or only loopback ones) the ingestor connects to a broker that is not
 # running; this turns that into a hard failure at startup.
 # Accepted truthy values: 1 true TRUE yes YES y Y on ON
@@ -96,22 +101,24 @@ MQTT_SOURCE_STRICT=false
 ENABLE_PPROF=false
 
 # Host interface the pprof HOST PORTS are published on, for a container whose
-# compose file maps them (docker-compose.staging*.yml). EMPTY (the default)
-# publishes them with no host interface, i.e. both IPv4 and IPv6, which is
-# what those files have always done; 127.0.0.1 requires an SSH tunnel to
-# reach the profiler. Writing 0.0.0.0 here is NOT the same as leaving it
-# empty: it publishes IPv4 only.
-# Default: (empty - every interface, IPv4 and IPv6)
+# compose file maps them (docker-compose.staging*.yml). Empty means the
+# default, 127.0.0.1 (#441): /debug/pprof/ is unauthenticated, so reach it
+# through an SSH tunnel —
+#   ssh -L 6060:127.0.0.1:6060 -L 6061:127.0.0.1:6061 you@host
+# Set 0.0.0.0 to publish it on every IPv4 interface again (remote access).
+# Default: 127.0.0.1
 # Used by: docker compose
 PPROF_BIND=
 
-# Host the pprof listener INSIDE the process binds to. Empty means every
-# interface, which is what the server and ingestor have always done. Set it
-# to 127.0.0.1 for a process that runs without a published port — a loopback
-# listener is NOT reachable through a docker port mapping, so use PPROF_BIND
-# for a container instead.
+# Host the pprof listener INSIDE the process binds to. Empty means the
+# default, 127.0.0.1 (#441). Set 0.0.0.0 (or *) to listen on every interface
+# — needed for a process whose pprof port is reached from outside it without
+# a compose file that handles this: a bare binary profiled remotely, or a
+# `docker run -p ...:6060:6060`, since a loopback listener is NOT reachable
+# through a docker port mapping. The staging compose files pin it to 0.0.0.0
+# themselves and narrow the published port with PPROF_BIND instead.
 # Per-role overrides: PPROF_HOST_SERVER, PPROF_HOST_INGESTOR below.
-# Default:
+# Default: 127.0.0.1
 # Used by: the server + the ingestor
 PPROF_HOST=
 

DEPLOY.md

@@ -41,10 +41,10 @@ Settings can be overridden via environment variables:
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `DISABLE_CADDY` | `false` | Skip internal Caddy (set `true` behind a reverse proxy) |
-| `DISABLE_MOSQUITTO` | `true` in `docker-compose.staging.yml`; `false` elsewhere | Skip internal MQTT broker. **Which values count depends on the launcher.** `./manage.sh` resolves it with `is_true()` (`1`/`true`/`TRUE`/`yes`/`YES`/`y`/`Y`/`on`/`ON`) and then runs `docker-compose.no-mosquitto.yml`, which hard-codes `DISABLE_MOSQUITTO=true` in the container — so any of those values gives an **external** broker on a manage.sh-managed host. A bare `docker compose up` against `docker-compose.yml` passes the value straight through, and the entrypoint disables the broker only on the exact `true`, so there `yes` leaves the bundled broker **running**. `./manage.sh broker-check` reports the mode this script will actually select, and warns when the value falls in that gap. Default flipped to `true` for the staging deploy in v3.7+ because a standalone `mqtt-broker` container owns MQTT on that host — see "Standalone MQTT broker (staging)" below. |
+| `DISABLE_MOSQUITTO` | `true` | Skip the bundled MQTT broker. The bundled broker is **opt-in** (#441): set `false` to run it. `./manage.sh`, the container entrypoint and the ingestor read the value the same way — `true`/`1`/`yes`/`on` and `false`/`0`/`no`/`off`, trimmed, any case; unset, empty or anything else means `true` (and an unrecognised value is reported). `./manage.sh broker-check` prints the effective mode. |
 | `PROD_MQTT_BIND` | *(empty — every interface, IPv4 and IPv6)* | Host interface the bundled broker's `1883` is published on. Empty renders the mapping with no host interface, which is the unchanged dual-stack publish; `127.0.0.1` makes the plaintext broker reachable from the host only. An explicit `0.0.0.0` is **IPv4 only** — not the same as empty. |
-| `PPROF_BIND` | *(empty — every interface, IPv4 and IPv6)* | Host interface the staging pprof ports `6060`/`6061` are published on. Empty is the unchanged dual-stack publish; `127.0.0.1` requires an SSH tunnel to profile. An explicit `0.0.0.0` is IPv4 only. |
-| `PPROF_HOST` | *(every interface)* | Host the in-process pprof listener binds. **Not** a substitute for `PPROF_BIND`: a loopback listener inside a container is unreachable through a docker port mapping. |
+| `PPROF_BIND` | `127.0.0.1` | Host interface the staging pprof ports `6060`/`6061` are published on (#441). Profile through an SSH tunnel (`ssh -L 6060:127.0.0.1:6060 -L 6061:127.0.0.1:6061 you@host`), or set `0.0.0.0` for remote access over IPv4. |
+| `PPROF_HOST` | `127.0.0.1` | Host the in-process pprof listener binds (#441); `0.0.0.0` or `*` for every interface. **Not** a substitute for `PPROF_BIND`: a loopback listener inside a container is unreachable through a docker port mapping, which is why the staging compose files pin it to `0.0.0.0`. |
 | `PPROF_PORT_SERVER` / `PPROF_PORT_INGESTOR` | `6060` / `6061` | Per-process pprof ports; they must stay distinct. Override the shared `PPROF_PORT`. |
 | `MQTT_SOURCE_STRICT` | `false` | Make the ingestor refuse to start when its broker setup cannot ingest, instead of only logging it. |
 | `MQTT_BROKER` / `MQTT_TOPIC` | *(empty — `config.json` decides)* | External broker URL and topic for the ingestor; `MQTT_BROKER` replaces the `mqttSources` in `config.json`. |
@@ -53,7 +53,12 @@ Settings can be overridden via environment variables:
 `PPROF_HOST*` / `PPROF_PORT*` settings are read by the server and the ingestor,
 not by docker compose, so every app compose file forwards them into the
 container through its `environment:` block with an empty default. Setting one
-in `.env` therefore does reach the process; leaving it unset is inert.
+in `.env` therefore does reach the process; leaving it unset is inert. The
+staging compose files pin `PPROF_HOST=0.0.0.0` instead, because they publish
+the pprof ports.
+
+Upgrading past #441 changes two defaults — the bundled broker and the pprof
+bind — see [docs/deployment.md, "Upgrading past #441"](docs/deployment.md#upgrading-past-441--bundled-broker-opt-in-pprof-on-loopback).
 
 One caveat: `MQTT_BROKER` **overrides** `mqttSources` in `config.json`. Before
 the compose files forwarded it, a value in `.env` was ignored by the container;
@@ -309,8 +314,8 @@ not in `/tmp`, which is ephemeral by definition.
 
 ## Broker preflight (`./manage.sh broker-check`)
 
-`DISABLE_MOSQUITTO=true` turns the bundled broker off; it does not configure
-a replacement. With no `mqttSources` the ingestor falls back to its built-in
+The bundled broker is opt-in, and turning it off (the default,
+`DISABLE_MOSQUITTO=true`) does not configure a replacement. With no `mqttSources` the ingestor falls back to its built-in
 `mqtt://localhost:1883` — the bundled broker's own address, where nothing is
 then listening — and ingests nothing while the container still reports
 healthy.

README.md

@@ -99,13 +99,14 @@ docker run -d --name corescope-meshview \
   --restart=unless-stopped \
   -p 127.0.0.1:8080:80 \
   -p 127.0.0.1:1883:1883 \
+  -e DISABLE_MOSQUITTO=false \
   --mount "type=bind,source=$(pwd)/data,target=/app/data" \
   corescope-meshview:local
 ```
 
 Open [http://localhost:8080](http://localhost:8080). The included Caddy proxy forwards HTTP requests to the Go server. The database and configuration persist in `./data`; packets appear once an observer publishes to the local MQTT broker.
 
-This example exposes HTTP and MQTT on the Docker host's loopback interface. A publisher on that host can connect to `mqtt://localhost:1883`. To collect from an external broker, replace `mqttSources` with that broker's connection settings and restart the container. You can then disable the included broker with `-e DISABLE_MOSQUITTO=true` and omit the MQTT port mapping.
+This example exposes HTTP and MQTT on the Docker host's loopback interface. The included broker is opt-in, which is what `-e DISABLE_MOSQUITTO=false` does; a publisher on that host can then connect to `mqtt://localhost:1883`. To collect from an external broker, replace `mqttSources` with that broker's connection settings, drop `-e DISABLE_MOSQUITTO=false` and the MQTT port mapping, and recreate the container.
 
 ```bash
 docker logs -f corescope-meshview

cmd/ingestor/broker_mode.go

@@ -2,30 +2,25 @@ package main
 
 // Broker-mode diagnostics (#372).
 //
-// CoreScope ships a bundled mosquitto that an operator can turn off with
-// DISABLE_MOSQUITTO=true to point the ingestor at their own broker. Two
-// things could go wrong silently in that arrangement:
+// CoreScope ships a bundled mosquitto. It is opt-in (#441): it runs only
+// when DISABLE_MOSQUITTO is a recognised false value, and otherwise the
+// ingestor is expected to reach the operator's own broker. Two things could
+// go wrong silently in that arrangement:
 //
 //  1. Nothing validates the external source. LoadConfig falls back to
 //     mqtt://localhost:1883 when no mqttSources are configured, which is
 //     the bundled broker's address — with the bundled broker off, the
 //     ingestor then connects to nothing and ingest stops with no explanation
 //     beyond a connection error in the log.
-//  2. The value of DISABLE_MOSQUITTO is read differently by different
-//     launchers. docker/entrypoint-go.sh — the only reader inside this
-//     container — disables mosquitto on exactly "true". manage.sh's
-//     is_true() also accepts 1/TRUE/yes/y/on, but it resolves the value
-//     itself: dc_prod/dc_staging swap in docker-compose.no-mosquitto.yml,
-//     which hard-codes DISABLE_MOSQUITTO=true in the container. So a
-//     container that can see "yes" here was started WITHOUT manage.sh, by
-//     a bare `docker compose up` against docker-compose.yml, and the
-//     bundled broker is running even though the operator asked for an
-//     external one.
+//  2. A DISABLE_MOSQUITTO value nobody recognises. docker/entrypoint-go.sh,
+//     manage.sh and this file read the value with one table
+//     (test-fixtures/disable-mosquitto-cases.json), so they agree on every
+//     value, but a typo silently takes the default.
 //
 // This file resolves the mode the way the entrypoint does and reports both
 // hazards as diagnostics. They are diagnostics, not refusals: an install
-// that merely updates must behave exactly as before. MQTT_SOURCE_STRICT
-// turns the errors into a refusal to start for operators who want that.
+// that merely updates must keep starting. MQTT_SOURCE_STRICT turns the
+// errors into a refusal to start for operators who want that.
 //
 // Every broker URL reaches a diagnostic through brokerurl.Mask, and hosts
 // are read out of the masked form, so no password, user name or query token
@@ -61,32 +56,40 @@ type BrokerMode struct {
 	// decided exactly as docker/entrypoint-go.sh decides it.
 	Bundled bool
 	// Raw is the DISABLE_MOSQUITTO value exactly as the environment carries
-	// it — untrimmed, because the entrypoint compares it untrimmed.
+	// it, for diagnostics.
 	Raw string
-	// Ambiguous is true when Raw is neither the exact "true" the entrypoint
-	// requires nor a recognised false value. Such a value leaves the
-	// bundled broker running in this container even though the operator
-	// who typed it probably meant to disable it.
-	Ambiguous bool
+	// Unrecognised is true when Raw is neither a true nor a false spelling,
+	// so the default (bundled broker off) applied.
+	Unrecognised bool
 }
 
 // ResolveBrokerMode reads DISABLE_MOSQUITTO through getenv (os.Getenv in
-// production, a map in tests).
-//
-// The value is compared RAW, not trimmed: docker/entrypoint-go.sh does
-// `[ "${DISABLE_MOSQUITTO:-false}" = "true" ]`, so " true" is not "true" to
-// it and the bundled broker keeps running. Trimming here would report the
-// opposite mode to the one this container is actually in. Such a value can
-// only reach the container through the process environment — manage.sh's
-// own .env reader and docker compose's dotenv parser both trim — which is
-// exactly the case Ambiguous exists to flag.
+// production, a map in tests). Trimmed of POSIX whitespace and compared
+// case-insensitively, true/1/yes/on disable the bundled broker and
+// false/0/no/off enable it; unset, empty and anything else take the
+// default, disabled. docker/entrypoint-go.sh and manage.sh's
+// parse_disable_mosquitto apply the same table.
 func ResolveBrokerMode(getenv func(string) string) BrokerMode {
 	raw := getenv("DISABLE_MOSQUITTO")
-	mode := BrokerMode{Raw: raw, Bundled: raw != "true"}
-	mode.Ambiguous = mode.Bundled && !isClearlyFalse(raw)
+	mode := BrokerMode{Raw: raw}
+	switch strings.ToLower(strings.Trim(raw, " \t\n\v\f\r")) {
+	case "false", "0", "no", "off":
+		mode.Bundled = true
+	case "true", "1", "yes", "on", "":
+	default:
+		mode.Unrecognised = true
+	}
 	return mode
 }
 
+// setting renders the DISABLE_MOSQUITTO value a diagnostic is about.
+func (m BrokerMode) setting() string {
+	if m.Raw == "" {
+		return "DISABLE_MOSQUITTO unset (default true)"
+	}
+	return fmt.Sprintf("DISABLE_MOSQUITTO=%q", m.Raw)
+}
+
 // StrictBrokerSources reports whether MQTT_SOURCE_STRICT asks for broker
 // errors to stop the process instead of being logged. It is a new setting
 // with no legacy spelling to preserve, so it accepts the same truthy values
@@ -95,7 +98,8 @@ func StrictBrokerSources(getenv func(string) string) bool {
 	return isShellTruthy(strings.TrimSpace(getenv("MQTT_SOURCE_STRICT")))
 }
 
-// isShellTruthy mirrors manage.sh's is_true().
+// isShellTruthy is the truthy set .env.example documents for
+// MQTT_SOURCE_STRICT.
 func isShellTruthy(v string) bool {
 	switch v {
 	case "1", "true", "TRUE", "yes", "YES", "y", "Y", "on", "ON":
@@ -104,18 +108,6 @@ func isShellTruthy(v string) bool {
 	return false
 }
 
-// isClearlyFalse reports whether a value unambiguously means "keep the
-// bundled broker". Anything outside both this set and the exact "true" is
-// ambiguous: the entrypoint keeps the bundled broker, but the operator who
-// typed it may well have meant the opposite.
-func isClearlyFalse(v string) bool {
-	switch v {
-	case "", "false", "FALSE", "False", "0", "no", "NO", "n", "N", "off", "OFF":
-		return true
-	}
-	return false
-}
-
 // brokerSchemes are the schemes paho reaches a broker with, after
 // Config.ResolvedSources maps mqtt:// and mqtts:// onto tcp:// and ssl://.
 var brokerSchemes = map[string]bool{
@@ -131,14 +123,12 @@ var brokerSchemes = map[string]bool{
 func CheckBrokerSources(mode BrokerMode, sources []MQTTSource, sourcesDefaulted bool) []BrokerDiagnostic {
 	var diags []BrokerDiagnostic
 
-	if mode.Ambiguous {
+	if mode.Unrecognised {
 		diags = append(diags, BrokerDiagnostic{
 			Severity: SeverityWarn,
-			Message: fmt.Sprintf("DISABLE_MOSQUITTO=%q is neither the exact %q the container entrypoint requires nor a recognised false value, "+
-				"so the bundled broker WILL start in this container. A container that sees this value was started by a bare "+
-				"`docker compose up` against docker-compose.yml: ./manage.sh resolves the value itself and swaps in "+
-				"docker-compose.no-mosquitto.yml, which passes the exact %q instead. Set it to %q or %q so every launcher reads it the same way.",
-				mode.Raw, "true", "true", "true", "false"),
+			Message: fmt.Sprintf("DISABLE_MOSQUITTO=%q is not a recognised value (true/1/yes/on or false/0/no/off), "+
+				"so the default applies and the bundled broker is off. Set it to %q to run the bundled broker, or to %q.",
+				mode.Raw, "false", "true"),
 		})
 	}
 
@@ -170,23 +160,25 @@ func CheckBrokerSources(mode BrokerMode, sources []MQTTSource, sourcesDefaulted
 	case sourcesDefaulted:
 		diags = append(diags, BrokerDiagnostic{
 			Severity: SeverityError,
-			Message: "DISABLE_MOSQUITTO=true selects an external MQTT broker, but no mqttSources are configured: " +
+			Message: mode.setting() + " selects an external MQTT broker, but no mqttSources are configured: " +
 				"the ingestor fell back to its built-in mqtt://localhost:1883, which is the bundled broker's address and " +
-				"nothing is listening there. Configure mqttSources in config.json (or set MQTT_BROKER) to the external broker.",
+				"nothing is listening there. Configure mqttSources in config.json (or set MQTT_BROKER) to the external broker, " +
+				"or set DISABLE_MOSQUITTO=false to run the bundled broker (it is opt-in).",
 		})
 	case usable == 0:
 		// Every source was malformed; the errors above already say so.
 	case loopback == usable:
 		diags = append(diags, BrokerDiagnostic{
 			Severity: SeverityError,
-			Message: fmt.Sprintf("DISABLE_MOSQUITTO=true disables the bundled broker, but all %d configured mqtt source(s) point at loopback (%s): "+
-				"nothing is listening there. Point them at the external broker's address.", usable, maskedBrokerList(sources)),
+			Message: fmt.Sprintf("%s disables the bundled broker, but all %d configured mqtt source(s) point at loopback (%s): "+
+				"nothing is listening there. Point them at the external broker's address, or set DISABLE_MOSQUITTO=false to run the bundled broker.",
+				mode.setting(), usable, maskedBrokerList(sources)),
 		})
 	case loopback > 0:
 		diags = append(diags, BrokerDiagnostic{
 			Severity: SeverityWarn,
-			Message: fmt.Sprintf("DISABLE_MOSQUITTO=true disables the bundled broker, so %d of %d mqtt source(s) pointing at loopback cannot connect (%s); "+
-				"the remaining source(s) still ingest.", loopback, usable, maskedBrokerList(sources)),
+			Message: fmt.Sprintf("%s disables the bundled broker, so %d of %d mqtt source(s) pointing at loopback cannot connect (%s); "+
+				"the remaining source(s) still ingest.", mode.setting(), loopback, usable, maskedBrokerList(sources)),
 		})
 	}
 	return diags