diff --git a/deploy/compose.observability.yml b/deploy/compose.observability.yml index ce57b79..ce07169 100644 --- a/deploy/compose.observability.yml +++ b/deploy/compose.observability.yml @@ -76,6 +76,14 @@ services: environment: GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER is required} GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD is required} + # Slack destination for the provisioned MaxScale-down alert. Deliberately + # NOT `:?required` -- the alert rule and its Grafana UI state are useful + # on their own, and a missing webhook must not stop the whole stack from + # starting. The default is a syntactically valid but non-functional URL: + # Grafana's provisioning rejects an empty one, so a placeholder is what + # lets this deploy before the webhook exists. Delivery then fails and is + # logged until it is replaced, which is visible rather than silent. + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL:-https://hooks.slack.com/services/REPLACE-ME} ports: - "127.0.0.1:3000:3000" volumes: @@ -84,6 +92,29 @@ services: # datasource or dashboard is a file drop rather than a Compose edit. - ../observability/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro,z - ../observability/grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro,z + - ../observability/grafana/provisioning/alerting:/etc/grafana/provisioning/alerting:ro,z + networks: + - observability_net + + # Probes MaxScale's client listener from Delta. This is the only thing in the + # stack that can report the database tier being unreachable: MaxScale's own + # alert script cannot, because a dead MaxScale sends nothing. See + # ../observability/blackbox/blackbox.yml for why it is a TCP connect on :4006 + # rather than the admin API. + blackbox: + image: prom/blackbox-exporter:v0.25.0 + restart: unless-stopped + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + command: ["--config.file=/etc/blackbox/blackbox.yml"] + volumes: + - ../observability/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro,z + # No ports published: only Prometheus talks to it, over the internal + # network. It probes OUTWARD to the MaxScale host, which needs no inbound + # exposure here. networks: - observability_net diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index e1e728b..70f6870 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -6,25 +6,32 @@ The CMS opens a single connection to the endpoint configured as which splits writes to the primary and reads to the replica. ``` - CMS / app host (Delta) DB primary host DB replica host - ┌───────────────────────┐ ┌────────────────┐ ┌────────────────┐ - │ cms-blue cms-green │ writes │ DB1 │ GTID│ DB2 (planned) │ - │ │ │ │────────▶│ (server_id 1) │────▶│ (server_id 2) │ - │ └────┬───┘ │ │ binlog+ACID │async│ read_only │ - └───────────┼───────────┘ reads └────────────────┘ └────────────────┘ - └──────────▶ MaxScale (rwsplit :4006) ──────────────▶ + CMS / app host (Delta) DB primary host DB replica host + ┌───────────────────────┐ ┌────────────────┐ ┌────────────────┐ + │ cms-blue cms-green │ writes │ DB1 │ GTID │ DB2 │ + │ │ │ │────────▶│ (server_id 1) │──────▶│ (server_id 2) │ + │ └────┬───┘ │ │ binlog+ACID │◀──ack─│ read_only │ + └───────────┼───────────┘ reads └────────────────┘ semi- └────────────────┘ + └──────────▶ MaxScale (rwsplit :4006) ─────sync──────▶ + mariadbmon: auto_failover ``` -| Host | Address | Role | Software | -| --- | --- | --- | --- | -| `THETRIANGLE-DB1-LXC` | `10.248.40.154` | primary | MariaDB 11.8 LTS | -| `THETRIANGLE-MAXSCALE` | `10.248.40.183` | proxy | MaxScale 24.02 | -| DB2 | not provisioned | replica | — | +| Host | Address | CT | Role | Software | +| --- | --- | --- | --- | --- | +| `THETRIANGLE-DB1-LXC` | `10.248.40.154` | 108 | primary | MariaDB 11.8 LTS | +| `THETRIANGLE-DB2-LXC` | `10.248.40.155` | 111 | replica / failover target | MariaDB 11.8 LTS | +| `THETRIANGLE-MAXSCALE` | `10.248.40.183` | 109 | proxy | MaxScale 24.02 | -Both are unprivileged **LXC containers**, 4 vCPU / 4 GB / 63 GB, installed from -apt. Docker was deliberately not used: it needs Proxmox-side nesting on an +All three are unprivileged **LXC containers**, 4 vCPU / 4 GB / 63 GB, installed +from apt. Docker was deliberately not used: it needs Proxmox-side nesting on an unprivileged container and costs ~300 MB of a 4 GB budget. +> **DB2 was created ~2026-08-03 with DB1's address**, `10.248.40.154`, and both +> were live on the bridge — which host you reached depended on which ARP entry +> won on your path. It has since been renumbered to `10.248.40.155` +> (MAC `bc:24:11:71:f9:f8`). Confirm `hostname` before trusting the output of +> anything you send to a DB host. + ## What this directory is These `.cnf` files are the **source of truth for what is installed on those @@ -33,9 +40,10 @@ hosts** — edit here, copy up, restart the service. | File | Installed as | Host | | --- | --- | --- | | [primary.cnf](primary.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-primary.cnf` | DB1 | -| [replica.cnf](replica.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf` | DB2, when it exists | +| [replica.cnf](replica.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf` | DB2 | | [../maxscale/maxscale.cnf](../maxscale/maxscale.cnf) | `/etc/maxscale.cnf` | MaxScale | -| [setup-replica.sh](setup-replica.sh) | run once on DB2 | DB2, when it exists | +| [provision-db2.sh](provision-db2.sh) | run once on DB2 | DB2 | +| [setup-replica.sh](setup-replica.sh) | run once on DB2, after the above | DB2 | **The `70-` prefix is load-bearing.** MariaDB reads `mariadb.conf.d/` in lexical order and Ubuntu's stock `50-server.cnf` sets `bind-address = 127.0.0.1`. A file @@ -73,37 +81,201 @@ never left the boxes. ``` PRIMARY_HOST=10.248.40.154 - REPLICA_HOST=192.0.2.2 # RFC 5737 placeholder until DB2 exists + REPLICA_HOST=10.248.40.155 MARIADB_PORT=3306 MAXSCALE_USER=maxscale MAXSCALE_PASSWORD=... + REPL_USER=repl + REPL_PASSWORD=... # what mariadbmon writes into CHANGE MASTER ``` + All six are required. Removing any one leaves an unsubstituted `$VAR` in the + config and MaxScale will not start — worth remembering when editing this file + with `sed`, where a range delete can silently take an adjacent line with it. + ## Accounts on DB1 | Account | Grants | Why | | --- | --- | --- | -| `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | the app, via MaxScale | -| `triangle_user@10.248.40.168` | `ALL PRIVILEGES ON triangle.*` | Delta direct; cutover verification only, droppable | -| `maxscale@10.248.40.183` | monitor + account-table reads | no failover privileges | -| `repl@'%'` | `REPLICATION SLAVE` | pending DB2 | +| `triangle_user@10.248.40.168` | `ALL PRIVILEGES ON triangle.*` | **client-side auth**: lets Delta log in *through* MaxScale | +| `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | **backend-side auth**: lets MaxScale open the backend connection | +| `maxscale@10.248.40.183` | monitor, account reads, **+ failover admin** | promotes/demotes on failover | +| `repl@10.248.40.155` | `REPLICATION SLAVE` | DB2's replication link | +| `repl@10.248.40.154` | `REPLICATION SLAVE` | reverse link, for `auto_rejoin` after failover | `ALL PRIVILEGES` rather than DML-only because the CMS runs additive DDL (`ADD COLUMN IF NOT EXISTS`) at startup. -## Adding the replica (DB2) +**Both `triangle_user` rows are required — do not "clean up" the Delta-scoped +one.** MaxScale authenticates a client against the backend user table by the +**client's own source address**, then connects to the backend from its own. Drop +either and the CMS gets `Error 1045`. See the warning in step 6. + +**Every one of these must exist on DB2 as well**, because on promotion DB2 serves +the application and MaxScale re-authenticates everything against *its* user +table. **`setup-replica.sh` does NOT copy them** — it dumps `--databases +triangle`, which excludes `mysql.*` — so accounts created *before* replication +started are absent on DB2, and only those created *after* replicate. Verify with +`SELECT CONCAT(user,'@',host) FROM mysql.user` on both hosts; a mismatch here +means a "successful" failover promotes a server nothing can log in to. To backfill +without polluting the replication stream, apply them on DB2 under +`SET SESSION sql_log_bin=0`. + +## Bringing up DB2 + +Ordered so the cluster is never in a state where automated failover could fire +at a replica that is not ready. **Do not enable `auto_failover` before step 6.** + +**Prerequisite — `tadmin` has no passwordless sudo on DB2.** It is in the `sudo` +group but no `NOPASSWD` rule exists, unlike DB1 and MaxScale. Either add one to +match the other two hosts, or run steps 1–3 from the Proxmox console. + +0. **Take a backup of DB1 first.** DB1 is still the only copy of the data (see + Notes). Automated failover is a mechanism for *promoting* a server, not a + substitute for being able to restore one. + ✅ *Done 2026-08-05: `/var/backups/triangle/triangle-predb2-20260805-005529.sql.gz` + on DB1 — 36.7 MB gzipped, 97.6 MB raw, 16 tables, `gzip -t` clean.* + +1–2. **Install MariaDB 11.8 and the replica config** — run + [provision-db2.sh](provision-db2.sh) on DB2 as root. It adds the + `deb.mariadb.org` 11.8 repo (same series as DB1, so a promotion is not also a + version change), installs `replica.cnf` as + `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf`, restarts, and verifies + `server_id=2`, the bind address, and durability. It refuses to run unless the + hostname is `THETRIANGLE-DB2-LXC`. + + *Verified reachable from DB2: DB1:3306, MaxScale:4006, deb.mariadb.org:443.* + +3. **Create the replication accounts on DB1.** + ✅ *Done 2026-08-05: `repl@10.248.40.155` and `repl@10.248.40.154` both + created with `REPLICATION SLAVE`, password from + `/root/triangle-db-credentials.env`.* The `.154` one is so the **old primary + can replicate back** from DB2 after `auto_rejoin`; without it, a rejoin fails + on access denied. + + A pre-existing **`repl@'%'` is still present** and should be dropped — a + wildcard replication account defeats the point of host-scoping, and DB1's + 3306 is reachable from the whole subnet until the firewall step below. + +4. **Run [setup-replica.sh](setup-replica.sh) on DB2** (not on DB1). It takes a + GTID-consistent `--single-transaction --master-data=1` dump from DB1, restores + it, and starts replication. DB1 keeps serving throughout; the only lock is a + brief global read lock at the start, held just long enough to read the binlog + position: + + ``` + PRIMARY_HOST=10.248.40.154 \ + DUMP_USER=root DUMP_PASSWORD=... \ + REPL_USER=repl REPL_PASSWORD=... \ + sudo -E sh setup-replica.sh + ``` + + `DUMP_USER` must be a privileged account, **not** `repl`: `REPLICATION SLAVE` + grants the binlog stream but no table reads, so dumping as `repl` fails with + "SELECT command denied". The script is idempotent — it exits early if + replication is already running. + + Confirm `Slave_IO_Running: Yes`, `Slave_SQL_Running: Yes`, and + `Seconds_Behind_Master: 0` before continuing. + +5. **Turn on semi-sync and confirm it engages.** The setting is persisted in the + `.cnf` files, but it is dynamic, so it can be applied without a restart: + + ```sql + -- DB1 + SET GLOBAL rpl_semi_sync_master_enabled = ON; + SET GLOBAL rpl_semi_sync_master_wait_point = AFTER_SYNC; + SET GLOBAL rpl_semi_sync_master_timeout = 1000; + SET GLOBAL rpl_semi_sync_master_wait_no_slave = OFF; + -- DB2 + SET GLOBAL rpl_semi_sync_slave_enabled = ON; + STOP SLAVE IO_THREAD; START SLAVE IO_THREAD; -- required: the slave only + -- registers as semi-sync when + -- the IO thread reconnects + ``` + + Then on DB1, **verify rather than assume**: + + ```sql + SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1 <- the real check + SHOW STATUS LIKE 'Rpl_semi_sync_master_yes_tx'; -- rises on acknowledged commits + SHOW STATUS LIKE 'Rpl_semi_sync_master_no_tx'; -- must NOT keep climbing + SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- ON, but see below + ``` -1. Provision the host, install MariaDB 11.8, install `replica.cnf` as - `70-triangle-replica.cnf` **with `bind-address` set** to DB2's NIC. -2. Run [setup-replica.sh](setup-replica.sh) on DB2 — it takes a GTID-consistent - `--single-transaction` dump from DB1 (non-blocking; DB1 keeps serving), - restores it, and starts replication. Pass `DUMP_USER`/`DUMP_PASSWORD` for a - privileged account: dumping as `repl` fails, because `REPLICATION SLAVE` - grants the binlog stream but no table reads. -3. Point `REPLICA_HOST` in MaxScale's `backend.env` at DB2 and restart MaxScale. - `maxscale.cnf` already carries the two-server topology, so no config edit is - needed. -4. Confirm `maxctrl list servers` shows `Master, Running` and `Slave, Running`. + **`clients` is the signal, not `status`.** With + `rpl_semi_sync_master_wait_no_slave=OFF`, `status` stays `ON` even while + nothing is acknowledging — verified by stopping DB2 on 2026-08-05: seven + commits completed unacknowledged (`no_tx` 0 → 7) with `status` still `ON` + and `clients` at 0. So `clients = 1` plus `yes_tx` rising is what shows the + guarantee is actually in force; `status = ON` on its own proves nothing. + +6. **Fence the write path — at the network, and ONLY at the network.** This is + what removes the split-brain risk, and it is a prerequisite for step 8, not + an optional hardening pass. Restrict 3306 on both DB hosts to MaxScale and + the DB peer: + + ```sh + ufw allow 22/tcp # BEFORE enabling + ufw allow from 10.248.40.183 to any port 3306 proto tcp # MaxScale + ufw allow from to any port 3306 proto tcp # replication + ufw --force enable + ``` + + Add the `allow` rules **before** `enable`, and do it **with Proxmox console + access open** — `pct enter ` gets you back in if you cut yourself off. + Do DB2 first: a mistake there costs replication, a mistake on DB1 costs the + site. + + > ⚠️ **Do NOT "fence" this by dropping `triangle_user@10.248.40.168`.** + > It looks like a direct-bypass account and it is not. **MaxScale + > authenticates a client against the backend's user table using the + > CLIENT's own source address**, so `triangle_user@` is precisely + > what lets the CMS log in *through* MaxScale; `triangle_user@` + > is what lets MaxScale then open the backend connection. **Both are + > required.** Dropping the Delta-scoped one closes no bypass and takes the + > site down with `Error 1045 Access denied for user + > 'triangle_user'@'10.248.40.168'` on any DB-backed route, while + > `/v1/health` keeps returning 200 — so it looks fine until someone loads a + > page. This was done and reverted on 2026-08-05. The firewall above is what + > actually removes the bypass, because Delta is no longer permitted to reach + > 3306 at all. + +7. **Grant MaxScale the failover privileges**, on **both** hosts: + + ```sql + GRANT REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD, + BINLOG ADMIN, CONNECTION ADMIN, REPLICATION MASTER ADMIN, + READ_ONLY ADMIN, SHOW DATABASES + ON *.* TO 'maxscale'@'10.248.40.183'; + ``` + + **`BINLOG ADMIN` is the one that is easy to miss and it breaks `auto_rejoin` + outright.** MariaDB 10.5+ split the old catch-all `SUPER` into discrete + privileges, so holding `SUPER` no longer implies it. Without it mariadbmon + cannot run `SET @@session.sql_log_bin=0` while demoting a returning primary + and loops forever on: + + ``` + Failed to prepare (demote) standalone server 'primary' for rejoin. + ``` + + Apply it on the current replica under `SET SESSION sql_log_bin=0` as well — + a grant made only on the primary reaches the replica by replication, but the + node that needs it during a rejoin is the one that is *not* currently + replicating. + +8. **Point MaxScale at DB2 and enable failover.** Set + `REPLICA_HOST=10.248.40.155` in `/etc/maxscale.secrets.d/backend.env` + (it is the RFC 5737 placeholder `192.0.2.2` until this is done), copy up the + new `maxscale.cnf`, and `systemctl restart maxscale`. + + Confirm `maxctrl list servers` shows `Master, Running` **and** + `Slave, Running` — a replica showing `Down` here means MaxScale still has the + placeholder address. + +9. **Test the failover before trusting it**, at a quiet hour, with the backup + from step 0 in hand. See below. ## Verifying the split @@ -118,18 +290,204 @@ session — on the primary. `causal_reads=local` makes MaxScale wait for the replica to reach the write's GTID, so a session always sees its own writes despite lag and **no app-level split is needed**. -## Enabling automated failover (optional) +## Why automated failover is safe here + +`auto_failover` is **on**. The two standard objections to automating promotion +on a 2-node pair are real, and neither is answered by the failover setting +itself — they are answered by the replication mode and the network topology. + +**"Async replication loses the tail on promotion."** True, and it is why this +was left off originally. Replication is now **semi-synchronous with +`wait_point=AFTER_SYNC`**: DB1 does not commit to the storage engine, and +therefore never acknowledges to the CMS, until DB2 has the binlog event +durably. A promoted DB2 cannot be missing a write that the application was told +succeeded. `AFTER_COMMIT` — the MariaDB default — would *not* give this: it +makes the write visible to other sessions before the ack, which is exactly the +window that loses data. + +**"Two nodes can't tell 'primary is dead' from 'I can't see the primary'."** +Also true, and unfixable at that layer: a 2-node cluster has no quorum, so it +cannot vote. The risk is removed structurally instead, by making MaxScale the +**only** path to the databases — 3306 on both DB hosts is firewalled to +MaxScale and the DB peer, so nothing else can open a connection at all. Once +that holds, "MaxScale cannot reach DB1" implies "the CMS cannot reach DB1", so +promoting DB2 cannot result in two servers taking application writes. The +partition that would split-brain a quorum-less cluster instead just moves all +traffic to the promoted node, which is the desired outcome. + +Note this fencing is **purely a network property**. It is tempting to also +revoke the app's Delta-scoped grant as "a second write path", but that account +is not a bypass — it is how MaxScale authenticates the client — and removing it +only breaks the site. See step 6. + +**"The old primary comes back and clobbers things."** `gtid_strict_mode=ON` on +both nodes. A returning DB1 that diverged is *refused* by `auto_rejoin` and +sits there needing a human, rather than replicating conflicting history. + +### What is still not covered + +Honest limits, all of which need a third node to close: + +- **While DB2 is down, semi-sync degrades to async** and the zero-loss guarantee + lapses. This is deliberate — `rpl_semi_sync_master_wait_no_slave=OFF` — because + the alternative is DB1 stalling every commit for `rpl_semi_sync_master_timeout` + whenever DB2 is offline, turning a replica outage into a site outage. So + DB2-down-then-DB1-dies can still lose writes. **Alert on + `Rpl_semi_sync_master_clients == 0`, NOT on `Rpl_semi_sync_master_status`** — + with `wait_no_slave=OFF` the status stays `ON` while commits go + unacknowledged, so it is not evidence the guarantee holds. A `slave_down` / + `lost_slave` Slack alert (see Alerting) covers the same condition. +- **MaxScale is a single point of failure** and the sole arbiter. If it dies, + the CMS is down regardless of how healthy both databases are. Fencing the + write path to MaxScale deepens this dependency — that is the price of removing + split-brain without a quorum. +- **Failover is not backup.** Promotion protects against a host dying, not + against a bad migration or a `DROP TABLE`, both of which replicate to DB2 in + milliseconds. + +### Planned maintenance + +Use **switchover**, not failover — it demotes the old primary cleanly instead of +assuming it is dead: + +``` +maxctrl call command mariadbmon switchover MariaDB-Monitor replica primary +``` + +### Testing it + +Do this once, at a quiet hour, with a fresh DB1 backup, **before** relying on it: + +``` +maxctrl list servers # baseline: Master/Slave, Running +``` + +Stop MariaDB on DB1 (`systemctl stop mariadb`), then watch MaxScale's log at +`/var/log/maxscale/maxscale.log`. Within ~10s (`failcount` 5 × `monitor_interval` +2000ms) it should log `master_down`, promote DB2, and `maxctrl list servers` +should show DB2 as `Master, Running`. Confirm the CMS still serves and can write. + +Then start DB1 again and confirm `auto_rejoin` brings it back as +`Slave, Running`. **If it does not rejoin, that is often the safety net working +rather than a bug** — compare `@@gtid_current_pos` on both before forcing +anything. A returning node that is merely *behind* (its GTID is a prefix of the +new primary's) is cleanly rejoinable; one that is genuinely diverged is refused +by `gtid_strict_mode`, and that refusal is correct. + +Two non-divergence failures seen on the first real test, both worth recognising: + +- **`Failed to prepare (demote) standalone server for rejoin`, repeating every + monitor tick** — the monitor user is missing `BINLOG ADMIN`. See step 7. +- **Rejoin appears to succeed then instantly reverts** (`new_slave` followed by + `lost_slave` about two seconds later). The monitor built the replication link + with the wrong credentials; `replication_user`/`replication_password` default + to the *monitor* user, which is host-scoped to the MaxScale host and so does + not exist from the rejoining node. `maxscale.cnf` now sets them to `repl` + explicitly. MaxScale reports this only as `lost_slave` — the real error is on + the rejoining node, in `SHOW SLAVE STATUS` `Last_IO_Error` (1045). + **After fixing it, clear the stale connection** with + `STOP SLAVE; RESET SLAVE ALL;` on the rejoining node: while a replica + connection exists the node is no longer "standalone", so the monitor will not + rebuild it and simply leaves it broken. + +Finish by switching back with the `switchover` command above so DB1 is primary +again, and confirm semi-sync re-engages (`Rpl_semi_sync_master_clients = 1`) +on whichever node ends up primary. + +## Alerting + +`maxscale.cnf` sets `script=` on the monitor, so mariadbmon runs +[../maxscale/maxscale-alert.sh](../maxscale/maxscale-alert.sh) (installed as +`/usr/local/bin/maxscale-alert.sh`, 0755) as the `maxscale` user on each event +in `events=`. It fires within one monitor tick, rather than waiting for a +scrape, and it does not depend on Delta or the observability stack being up. + +It **always** appends to `/var/log/maxscale/failover-events.log` and posts to +Slack only if `/etc/maxscale.secrets.d/alert.env` (0640 `root:maxscale`) +supplies a webhook: + +``` +SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... +``` + +With that empty it degrades to log-only, so it is safe to install first. The +webhook must not go in `maxscale.cnf`, which is world-readable 0644. + +**Event names are not intuitive, and getting them wrong fails silently — you +only notice by the alert that never arrives.** In particular `new_slave` +(`[Running]→[Slave,Running]`, a standalone node joining) and `slave_up` +(`[Down]→[Slave,Running]`, a node returning from an outage) are different +transitions; listing only the former alerts on the way down and stays silent on +recovery. Confirm what actually fired with: + +```sh +grep "changed state" /var/log/maxscale/maxscale.log +``` + +### The MaxScale-down watchdog + +The script can only report events MaxScale is alive to observe, so it cannot +tell you MaxScale *itself* died — and because the write path is fenced to +MaxScale, that is a total outage. That gap is covered from **outside** the +database tier, by the observability stack on Delta: + +``` +blackbox_exporter --TCP connect--> 10.248.40.183:4006 + ^ | + | scrape v + Prometheus ----> Grafana alert ----> Slack + probe_success == 0 for 1m +``` + +- [../../observability/blackbox/blackbox.yml](../../observability/blackbox/blackbox.yml) + — a plain TCP connect, no MySQL login, so no credentials are needed. +- [../../observability/prometheus/prometheus.delta.yml](../../observability/prometheus/prometheus.delta.yml) + — job `blackbox-tcp`, with the usual exporter relabel indirection. +- [../../observability/grafana/provisioning/alerting/maxscale.yml](../../observability/grafana/provisioning/alerting/maxscale.yml) + — rule `maxscale-unreachable` plus the `slack-triangle` contact point. + +**Why `:4006` and not the admin API:** 4006 is the port the CMS actually uses, +so it tests the real dependency; the admin API (8989) would have to be opened to +Delta and it can reconfigure MaxScale, which is a much worse thing to expose. +MaxScale 24.02 serves no Prometheus endpoint anyway (`/metrics` and +`/v1/metrics` both 404). + +**DB1/DB2 are deliberately not probed** — their 3306 is firewalled to the +MaxScale host and the DB peer, so Delta cannot reach them by design and such a +target would alert forever. + +`noDataState: Alerting` is intentional: if the probe series disappears, nobody +is watching the database tier, which is worth waking someone for even though +the cause is Prometheus rather than MaxScale. + +> **The Slack webhook is supplied by `SLACK_WEBHOOK_URL` in +> `observability.env`**, and defaults to a non-functional placeholder so the +> stack still starts without it (Grafana's provisioning rejects an empty URL). +> Until it is set, the alert fires correctly in Grafana and delivery fails in +> the Grafana log — visible, not silent. Note this is a **second** place the +> webhook is needed: the MaxScale host has its own copy in +> `/etc/maxscale.secrets.d/alert.env`, because the two alert paths run on +> different machines by design. + +**Testing it by changing the probe target leaves a stale series behind.** The +old `instance` keeps its last value inside Prometheus's 5-minute instant-query +lookback, so the rule goes on firing for several minutes after you revert. That +is an artifact of the test, not of the alert: in normal operation the target +never changes, `probe_success` moves 1→0→1 on one series, and recovery is +immediate. -`auto_failover` is **off**. Promoting a replica on a 2-node async pair risks -split-brain and data loss, and should be a deliberate ops action. To enable: +**The observability stack does not live in the runner's checkout.** It runs from +`/home/tadmin/triangle-observability` on Delta — a hand-copied tree, not a git +clone — with `--env-file ../observability.env` (not `cms.env`, despite what +deploy/README.md says elsewhere). Changing any file above means copying it there +and restarting the affected service; note `docker compose up -d` will NOT +restart Prometheus for a config-file-only change, so `restart prometheus` +explicitly. -1. Grant `REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD` to - `MAXSCALE_USER` on the primary. -2. Set `auto_failover=true` in [maxscale.cnf](../maxscale/maxscale.cnf). -3. Prefer `maxctrl call command mariadbmon switchover` for planned maintenance. +### Manual failover, without MaxScale -Manual failover without MaxScale: on the replica `STOP SLAVE; RESET SLAVE ALL;`, -set `read_only=OFF`, repoint `PRIMARY_HOST`, rebuild the old primary as a replica. +On DB2: `STOP SLAVE; RESET SLAVE ALL;` then `SET GLOBAL read_only=OFF`. Repoint +`PRIMARY_HOST` and rebuild the old primary as a replica. ## Notes / gotchas @@ -150,10 +508,30 @@ set `read_only=OFF`, repoint `PRIMARY_HOST`, rebuild the old primary as a replic reachable from the whole subnet. It should be restricted to the MaxScale host (and DB2 later); this was left alone deliberately to avoid an SSH lockout on a remote host, so do it with console access available. -- **DB1 is the only copy of the data.** The old dev container and its volume are - gone. The pre-cutover dump on Delta at +- **`mariadb-dump --gtid` on its own records NOTHING.** It only changes the + *format* of the position emitted by `--master-data`/`--dump-slave`, so without + one of those the dump carries no replication start position at all. A replica + seeded from such a dump begins at its own empty `gtid_slave_pos` — i.e. the + start of the primary's binlogs, which expire after 7 days + (`binlog_expire_logs_seconds`) — so replication either dies with error 1236 or + replays history on top of the restored data. `setup-replica.sh` carried this + bug and now passes `--master-data=1`, and hard-fails if the dump comes out + without an active `SET GLOBAL gtid_slave_pos=` line. It must be `=1`: `=2` + emits the same line **commented out**. Verified against 11.8.8. +- **DB1 is the only copy of the data until DB2 is replicating.** The old dev + container and its volume are gone. The pre-cutover dump on Delta at `~/triangle-deploy/backups/triangle-precutover-20260729-2137.sql.gz` is a - point-in-time artifact, not a backup rotation — real backups are still owed. + point-in-time artifact, not a backup rotation — real backups are still owed, + and a replica is not one: a `DROP TABLE` reaches DB2 in milliseconds. +- **`tadmin` has no `NOPASSWD` sudo on DB2**, unlike DB1 and MaxScale. It is in + the `sudo` group, so an interactive password works, but every scripted step in + the bring-up runbook fails without a rule matching the other two hosts. Note + the failure mode is quiet: `ssh ... 'sudo -n ...' ` prints "sudo: a password is + required" to stderr while the pipeline reports success, because + `cmd | ssh ...` returns the *local* command's exit status. +- **DB2 has no `10-eth0-static.network` override.** Its address is pinned in the + CT config (`pct config 111`) but not inside the container, so it relies on a + single layer where DB1 and MaxScale have two. Add the override to match. - `server_id` must be unique per node (1 primary / 2 replica); `gtid_domain_id` must match (1 here). - Schema changes: the CMS runs additive, idempotent migrations at startup that diff --git a/deploy/mariadb/primary.cnf b/deploy/mariadb/primary.cnf index 987b5b8..1b2f620 100644 --- a/deploy/mariadb/primary.cnf +++ b/deploy/mariadb/primary.cnf @@ -74,3 +74,38 @@ binlog_expire_logs_seconds = 604800 # keep 7 days of binlogs for replica gtid_domain_id = 1 gtid_strict_mode = ON log_slave_updates = ON # lets the replica be chained / used for backups + +# --- Semi-synchronous replication: MASTER side -------------------------------- +# This is the setting that makes MaxScale's automated failover safe to turn on. +# Without it, replication is async: DB1 acknowledges a commit to the CMS before +# DB2 has seen it, so promoting DB2 silently discards the tail of the write +# stream. With it, a commit is not acknowledged until DB2 has the event. +rpl_semi_sync_master_enabled = ON +# AFTER_SYNC (not the AFTER_COMMIT default): wait for the replica's ack BEFORE +# committing to the storage engine, so a write is never visible to other +# sessions on DB1 until it is safe on DB2. AFTER_COMMIT makes the write visible +# first, which is precisely the window that loses data on promotion. +rpl_semi_sync_master_wait_point = AFTER_SYNC +# How long a commit waits for an ack before degrading to async, in ms. +rpl_semi_sync_master_timeout = 1000 +# OFF is load-bearing for availability. With the ON default, DB1 pays the full +# timeout above on EVERY commit while no replica is connected — so planned DB2 +# maintenance would add a second of latency to every write. OFF drops straight +# to async the moment there is no semi-sync replica, and back to semi-sync when +# DB2 reconnects. The tradeoff is real and must be understood: while DB2 is +# down, the zero-loss guarantee is NOT in force. +# +# MONITOR THIS WITH Rpl_semi_sync_master_clients, NOT ..._status. Verified by +# stopping DB2 on 2026-08-05: status stayed **ON** through seven unacknowledged +# commits while clients sat at 0 and no_tx climbed 0 -> 7. With +# wait_no_slave=OFF the master never enters the "off" state it would otherwise +# fall into, so status is not evidence the guarantee is holding. +# Rpl_semi_sync_master_clients == 0 -> nothing is acknowledging: ALERT +# Rpl_semi_sync_master_no_tx rising -> commits completing unacknowledged +# Rpl_semi_sync_master_yes_tx rising -> the guarantee is actually in force +rpl_semi_sync_master_wait_no_slave = OFF +# Slave side, inert while this node is the primary. Present so that after a +# failover and auto_rejoin — when this node comes back as a REPLICA of DB2 — it +# registers as a semi-sync client instead of silently leaving the new primary +# running asynchronously. Mirrors replica.cnf; both nodes carry both roles. +rpl_semi_sync_slave_enabled = ON diff --git a/deploy/mariadb/provision-db2.sh b/deploy/mariadb/provision-db2.sh new file mode 100755 index 0000000..6da526f --- /dev/null +++ b/deploy/mariadb/provision-db2.sh @@ -0,0 +1,108 @@ +#!/bin/sh +# One-time: install MariaDB 11.8 on DB2 and put the replica config in place. +# +# Run this ON DB2 (THETRIANGLE-DB2-LXC, 10.248.40.155) as root: +# +# sudo sh provision-db2.sh +# +# It does NOT start replication and does NOT touch MaxScale — it only gets a +# correctly-configured, correctly-bound MariaDB running. Run setup-replica.sh +# afterwards, then follow "Bringing up DB2" in README.md from step 5. +# +# Idempotent: safe to re-run. Existing repo/key/config are refreshed in place +# and apt skips packages already at the right version. +# +# NOTE ON ACCESS: tadmin has no NOPASSWD sudo on DB2 (unlike DB1 and MaxScale), +# so this cannot be driven over ssh non-interactively until such a rule exists: +# echo 'tadmin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/90-tadmin +# chmod 440 /etc/sudoers.d/90-tadmin +set -eu + +EXPECT_HOST=THETRIANGLE-DB2-LXC +EXPECT_ADDR=10.248.40.155 +SERIES=11.8 # LTS. deb.mariadb.org carries ONLY LTS lines. +CNF_SRC="$(dirname "$0")/replica.cnf" +CNF_DST=/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf + +# --- Guards ------------------------------------------------------------------ +# DB2 was briefly live on DB1's address (10.248.40.154) and ssh gives no +# host-key warning when the ARP winner changes underneath you. Never let this +# script run against the primary: it would overwrite the primary's config with +# a read_only replica config. +[ "$(id -u)" = 0 ] || { echo "must run as root" >&2; exit 1; } +if [ "$(hostname)" != "$EXPECT_HOST" ]; then + echo "REFUSING: hostname is '$(hostname)', expected '$EXPECT_HOST'." >&2 + echo "You are not on DB2. Check which host you actually reached." >&2 + exit 1 +fi +if ! ip -4 addr show | grep -q "inet ${EXPECT_ADDR}/"; then + echo "REFUSING: ${EXPECT_ADDR} is not configured on this host." >&2 + exit 1 +fi +[ -f "$CNF_SRC" ] || { echo "cannot find replica.cnf next to this script" >&2; exit 1; } + +echo "==> Host verified: $(hostname) / ${EXPECT_ADDR}" + +# --- MariaDB apt repo --------------------------------------------------------- +# Note this is the SERVER key. MaxScale uses a different key entirely and is not +# installed here — see README.md. +echo "==> Adding MariaDB ${SERIES} repository" +apt-get update -qq +apt-get install -y -qq curl gpg apt-transport-https ca-certificates + +install -d -m 0755 /etc/apt/keyrings +curl -fsSL https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY \ + | gpg --dearmor --yes -o /etc/apt/keyrings/mariadb.gpg +chmod 0644 /etc/apt/keyrings/mariadb.gpg + +. /etc/os-release +cat > /etc/apt/sources.list.d/mariadb.list < Installing mariadb-server" +DEBIAN_FRONTEND=noninteractive apt-get install -y -qq mariadb-server mariadb-client +mariadbd --version + +# --- Config ------------------------------------------------------------------- +# The 70- prefix is load-bearing: Ubuntu's stock 50-server.cnf sets +# bind-address = 127.0.0.1 and mariadb.conf.d is read in lexical order, so a +# file sorting before it cannot override the bind and the replica would be +# unreachable from both MaxScale and the primary. +echo "==> Installing ${CNF_DST}" +install -o root -g root -m 0644 "$CNF_SRC" "$CNF_DST" + +echo "==> Restarting mariadb" +systemctl enable --now mariadb +systemctl restart mariadb + +# --- Verify ------------------------------------------------------------------- +echo "==> Verifying" +mariadb -N -B -e "SELECT @@hostname, @@server_id, @@read_only, @@gtid_domain_id, @@gtid_strict_mode" + +# server_id must differ from DB1's (1) or replication refuses to start. +SID=$(mariadb -N -B -e "SELECT @@server_id") +[ "$SID" = "2" ] || { echo "FAIL: server_id is ${SID}, expected 2" >&2; exit 1; } + +# The whole point of the 70- prefix. If this shows 127.0.0.1, the config did not +# take and nothing downstream will work. +echo "--- listening sockets ---" +ss -ltnp 2>/dev/null | grep 3306 || echo "WARNING: nothing listening on 3306" +if ! ss -ltn 2>/dev/null | grep -q "${EXPECT_ADDR}:3306\|0.0.0.0:3306\|\*:3306"; then + echo "FAIL: not bound to ${EXPECT_ADDR}:3306 — check ${CNF_DST} ordering" >&2 + exit 1 +fi + +# Durability must match the primary: DB2 is a failover target, not a read cache. +echo "--- durability (must be 1 / 1) ---" +mariadb -N -B -e "SELECT @@innodb_flush_log_at_trx_commit, @@sync_binlog" + +echo +echo "OK. MariaDB ${SERIES} is installed, bound to ${EXPECT_ADDR}, and read_only." +echo "NEXT: run setup-replica.sh on this host to seed from DB1 and start" +echo "replication, then continue at README.md 'Bringing up DB2' step 5." diff --git a/deploy/mariadb/replica.cnf b/deploy/mariadb/replica.cnf index ec12a31..377df7a 100644 --- a/deploy/mariadb/replica.cnf +++ b/deploy/mariadb/replica.cnf @@ -1,19 +1,19 @@ [mysqld] # ============================================================================= -# Triangle CMS — MariaDB READ REPLICA (DB2) config. NOT YET DEPLOYED: DB2 does -# not exist as of 2026-07-30, so this is the config to install when it lands. -# Runs on its OWN host (separate from DB1), replicating asynchronously from the -# primary via GTID. Serves READ traffic only. +# Triangle CMS — MariaDB REPLICA + FAILOVER TARGET (DB2) config. +# Host: THETRIANGLE-DB2-LXC, 10.248.40.155, CT 111 (4 vCPU / 4 GB / 63 GB). +# Runs on its OWN host (separate from DB1), replicating from the primary via +# GTID with semi-synchronous acknowledgement. Serves READ traffic, and is +# promoted to primary by MaxScale's mariadbmon on failover. # Installed natively at /etc/mysql/mariadb.conf.d/70-triangle-replica.cnf — the # 70- prefix is load-bearing, see bind-address below. # ============================================================================= # --- Network ------------------------------------------------------------------ -# UNCOMMENT AND SET to DB2's internal NIC address before starting MariaDB. # Ubuntu's stock 50-server.cnf binds 127.0.0.1, which would leave the replica # unreachable from both MaxScale and the primary; this file must sort after it. # Firewall 3306 to the MaxScale and primary hosts only. -# bind-address = 10.248.40.xxx +bind-address = 10.248.40.155 # --- InnoDB memory ----------------------------------------------------------- # Sized to match the primary; see the rationale in primary.cnf. Adjust the two @@ -24,13 +24,21 @@ innodb_buffer_pool_size = 1G innodb_log_file_size = 256M innodb_log_buffer_size = 32M -# --- Durability: relaxed on the replica --------------------------------------- -# A read replica can re-fetch anything it loses on crash from the primary via -# GTID, so we trade per-commit fsyncs for throughput on apply. Never do this on -# the primary. -innodb_flush_log_at_trx_commit = 2 -sync_binlog = 0 +# --- Durability: FULL, same as the primary ------------------------------------ +# Deliberately NOT the relaxed (2 / 0) setting usually given to a read replica. +# Two reasons, both consequences of DB2 being a failover target rather than a +# pure read cache: +# 1. On promotion DB2 *becomes* the primary. Relaxed settings would silently +# leave production running without per-commit fsyncs until someone noticed. +# 2. Semi-sync's guarantee is only as strong as the ack. With sync_binlog=0 +# the replica acks once the event is in the OS page cache, so a power loss +# on DB2 discards writes the primary already told the client were durable. +# The CMS write volume is trivial (a newsroom, not a transaction processor), so +# the throughput this costs is not measurable here. +innodb_flush_log_at_trx_commit = 1 # fsync redo on every commit +sync_binlog = 1 # fsync binlog on every commit innodb_flush_method = O_DIRECT +innodb_doublewrite = ON # --- Concurrency / caches ---------------------------------------------------- max_connections = 300 # replica typically fields more read conns @@ -70,3 +78,22 @@ binlog_expire_logs_seconds = 604800 # Parallel apply keeps replication lag low under write bursts from the primary. slave_parallel_threads = 4 slave_parallel_mode = optimistic + +# --- Semi-synchronous replication --------------------------------------------- +# BOTH sides are enabled on BOTH nodes, because either node can hold either role +# after a failover. MariaDB only acts on the side matching its current role, so +# the master settings sit inert here until this node is promoted. +# +# Slave side: acknowledges each binlog event back to the primary, which is what +# lets automated failover promote DB2 without losing acknowledged commits. Must +# be ON here for the primary's AFTER_SYNC wait to ever be satisfied — if this is +# OFF, the primary just times out and degrades to async on every commit. +rpl_semi_sync_slave_enabled = ON +# Master side: inert while this node is a replica, load-bearing the moment it is +# promoted. Without it a failover silently drops to asynchronous replication — +# losing the zero-data-loss guarantee at exactly the moment you have just proven +# you need it. Values must match primary.cnf; see the rationale there. +rpl_semi_sync_master_enabled = ON +rpl_semi_sync_master_wait_point = AFTER_SYNC +rpl_semi_sync_master_timeout = 1000 +rpl_semi_sync_master_wait_no_slave = OFF diff --git a/deploy/mariadb/setup-replica.sh b/deploy/mariadb/setup-replica.sh index 19c4852..cb868a2 100755 --- a/deploy/mariadb/setup-replica.sh +++ b/deploy/mariadb/setup-replica.sh @@ -14,9 +14,9 @@ # (unix_socket auth is the default for root on a native apt install). # # Idempotency: it exits early if replication is already running, so re-running -# after a hiccup is safe. A GTID-consistent dump (--gtid) records the exact -# primary position, so START SLAVE ... MASTER_USE_GTID=slave_pos resumes with no -# gaps or duplicates. +# after a hiccup is safe. The dump records the exact primary position via +# --gtid --master-data=1 (BOTH are needed — see the dump step below), so +# START SLAVE ... MASTER_USE_GTID=slave_pos resumes with no gaps or duplicates. set -eu : "${PRIMARY_HOST:?PRIMARY_HOST is required (DB1's address)}" @@ -50,12 +50,34 @@ until mariadb -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP done echo "dumping ${MARIADB_DATABASE} from primary (GTID-consistent) ..." -# --gtid emits SET GLOBAL gtid_slave_pos=...; --single-transaction takes no lock -# on InnoDB, so the primary keeps serving throughout. +# --master-data=1 is REQUIRED and is not optional decoration. `--gtid` on its own +# emits NOTHING: in MariaDB it only changes the FORMAT of the position recorded +# by --master-data/--dump-slave, so `--gtid` without one of those produces a dump +# carrying no replication start position at all. The replica would then begin +# from its own empty gtid_slave_pos, i.e. from the very start of the primary's +# binlogs — which are expired after binlog_expire_logs_seconds (7 days), so +# replication either dies with error 1236 or replays history on top of the +# restored data. Verified against 11.8.8: --gtid alone emits no gtid line; +# --master-data=1 emits an ACTIVE `SET GLOBAL gtid_slave_pos='1-1-...';`, which +# is what the CHANGE MASTER ... MASTER_USE_GTID=slave_pos below consumes. +# It must be =1, not =2 — =2 comments that same line out. +# +# --single-transaction keeps the dump consistent without locking InnoDB tables, +# but note that combining it with --master-data does take a brief global read +# lock at the very start, just long enough to read the binlog position. It is +# milliseconds, not the length of the dump, and the primary serves throughout. mariadb-dump -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP_PASSWORD}" \ - --single-transaction --gtid --routines --triggers --events \ + --single-transaction --gtid --master-data=1 --routines --triggers --events \ --databases "${MARIADB_DATABASE}" > "${DUMP_FILE}" +# Fail loudly here rather than starting replication from a bogus position. +if ! grep -q "^SET GLOBAL gtid_slave_pos=" "${DUMP_FILE}"; then + echo "ERROR: dump contains no active 'SET GLOBAL gtid_slave_pos=' line." >&2 + echo "Replication would start from the wrong position. Check that" >&2 + echo "${DUMP_USER} holds RELOAD/BINLOG MONITOR on the primary." >&2 + exit 1 +fi + echo "loading dump into replica ..." # read_only=ON is set in replica.cnf; root is exempt (it holds SUPER), so the # restore lands without having to relax it. @@ -77,8 +99,43 @@ sleep 2 local_sql -e "SHOW SLAVE STATUS\G" | grep -E "Slave_IO_Running|Slave_SQL_Running|Seconds_Behind_Master|Last_.*Error" || true echo echo "done. Verify 'Slave_IO_Running: Yes' and 'Slave_SQL_Running: Yes' above." -echo "maxscale.cnf already carries a [replica] server, so on the MaxScale host" -echo "just point REPLICA_HOST in /etc/maxscale.secrets.d/backend.env at DB2" -echo "(it is currently the RFC 5737 placeholder 192.0.2.2), then" -echo "'systemctl restart maxscale' and confirm 'maxctrl list servers' shows" -echo "the replica as Slave, Running." +echo +echo "NEXT, in order — see 'Bringing up DB2' in README.md. Do NOT enable" +echo "auto_failover until semi-sync is confirmed engaged and the write path is" +echo "fenced; promoting an async replica loses acknowledged writes." +echo +echo " 1. Enable semi-sync on the PRIMARY (dynamic, no restart needed):" +echo " SET GLOBAL rpl_semi_sync_master_enabled = ON;" +echo " SET GLOBAL rpl_semi_sync_master_wait_point = AFTER_SYNC;" +echo " SET GLOBAL rpl_semi_sync_master_timeout = 1000;" +echo " SET GLOBAL rpl_semi_sync_master_wait_no_slave = OFF;" +echo " replica.cnf already sets rpl_semi_sync_slave_enabled=ON here, and the" +echo " CHANGE MASTER above reconnected the IO thread, so this node registers" +echo " as a semi-sync client as soon as the primary is enabled." +echo +echo " 2. CONFIRM it engaged, on the primary — do not assume:" +echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- must be ON" +echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1" +echo " Healthy replication with status OFF means it degraded to async." +echo +echo " 3. Copy the app/monitor/repl accounts to THIS host. The dump above is" +echo " --databases triangle, which EXCLUDES mysql.*, so any account created" +echo " before replication started is missing here and a promotion would" +echo " yield a server nothing can log in to. Apply them under" +echo " SET SESSION sql_log_bin=0 so they stay out of the binlog." +echo +echo " 4. Fence the write path AT THE NETWORK: firewall 3306 to the MaxScale" +echo " host and the DB peer, adding 'ufw allow 22/tcp' BEFORE enabling." +echo " Do NOT drop triangle_user@ to 'close a bypass' — MaxScale" +echo " authenticates clients by their own source address, so that account" +echo " is how the CMS logs in through MaxScale. Dropping it just breaks" +echo " the site. Have console access open." +echo +echo " 5. On the MaxScale host, set REPLICA_HOST in" +echo " /etc/maxscale.secrets.d/backend.env to this node (it is the RFC 5737" +echo " placeholder 192.0.2.2 until then), grant the maxscale user" +echo " REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD," +echo " copy up maxscale.cnf, and 'systemctl restart maxscale'." +echo +echo " 6. Confirm 'maxctrl list servers' shows Master, Running and Slave," +echo " Running — then test a real failover before relying on it." diff --git a/deploy/maxscale/maxscale-alert.sh b/deploy/maxscale/maxscale-alert.sh new file mode 100755 index 0000000..4bed0e4 --- /dev/null +++ b/deploy/maxscale/maxscale-alert.sh @@ -0,0 +1,99 @@ +#!/bin/sh +# Alert hook for MaxScale's mariadbmon, invoked via `script=` in maxscale.cnf. +# Installed on THETRIANGLE-MAXSCALE as /usr/local/bin/maxscale-alert.sh (0755). +# +# MaxScale runs this as the `maxscale` user on every event listed in `events=`, +# substituting $EVENT/$INITIATOR/$NODELIST/$PARENT before exec. It is bounded by +# script_timeout (90s) — if it hangs, the monitor blocks, so every outbound call +# here MUST have its own timeout. +# +# Design notes: +# - It ALWAYS writes the local log first and posts to Slack second. A failover +# that happened is a fact worth keeping even if Slack is unreachable, and +# the log is what you correlate against maxscale.log afterwards. +# - It exits 0 unconditionally. A non-zero exit here is noise in maxscale.log +# and there is nothing MaxScale can usefully do about a failed notification. +# - Until SLACK_WEBHOOK_URL is configured it degrades to log-only rather than +# failing, so it is safe to install before the webhook exists. +# +# The webhook lives in /etc/maxscale.secrets.d/alert.env (0640 root:maxscale), +# NOT here and NOT in maxscale.cnf, which is world-readable 0644: +# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... +set -u + +EVENT="${1:-unknown}" +INITIATOR="${2:-unknown}" +NODELIST="${3:-}" +PARENT="${4:-}" + +TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LOG=/var/log/maxscale/failover-events.log + +# --- Always record locally, before anything that can fail -------------------- +printf '%s event=%s initiator=%s nodes=%s parent=%s\n' \ + "$TS" "$EVENT" "$INITIATOR" "$NODELIST" "$PARENT" >> "$LOG" 2>/dev/null + +# --- Classify ----------------------------------------------------------------- +# new_master is the one that means an automated failover actually promoted a +# node. lost_slave/slave_down matter more than they look: with only two nodes, +# losing the replica means semi-sync degrades to async +# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-data-loss guarantee +# stops holding until it returns. +case "$EVENT" in + new_master) + ICON=":rotating_light:"; SEV="FAILOVER" + NOTE="A node was promoted. The old primary needs checking before it is trusted again." ;; + master_down|lost_master) + ICON=":rotating_light:"; SEV="PRIMARY DOWN" + NOTE="Writes are failing until a promotion completes." ;; + slave_down|lost_slave) + ICON=":warning:"; SEV="REPLICA DOWN" + NOTE="Semi-sync has degraded to async: writes are no longer guaranteed durable on two nodes." ;; + master_up|slave_up|new_slave|server_up) + # slave_up ([Down]->[Slave,Running]) is the recovery counterpart of + # slave_down; new_slave ([Running]->[Slave,Running]) is a standalone node + # joining. Both belong here or an outage never reports that it ended. + ICON=":white_check_mark:"; SEV="RECOVERED" + NOTE="Confirm with 'maxctrl list servers' and check Rpl_semi_sync_master_clients=1 on the primary." ;; + server_down) + ICON=":warning:"; SEV="SERVER DOWN" + NOTE="A backend stopped responding to the monitor." ;; + *) + ICON=":information_source:"; SEV="EVENT" + NOTE="" ;; +esac + +# --- Post to Slack, if configured -------------------------------------------- +[ -r /etc/maxscale.secrets.d/alert.env ] && . /etc/maxscale.secrets.d/alert.env +[ -n "${SLACK_WEBHOOK_URL:-}" ] || exit 0 + +# Build the message with RAW values, then escape exactly once when it becomes +# JSON. Escaping the fields individually and then escaping the whole string +# again would double every backslash. +TEXT="${ICON} *MaxScale ${SEV}* — \`${EVENT}\` +*initiator:* ${INITIATOR} +*nodes:* ${NODELIST:-n/a} +*parent:* ${PARENT:-n/a} +*time:* ${TS} +${NOTE}" + +# JSON string escaping: backslash and quote, then fold the real newlines into +# \n. Literal newlines inside a JSON string are invalid and Slack rejects the +# payload outright, so the multi-line message above must be collapsed here. +# The ':a;N;$!ba' idiom slurps the whole input before substituting. +json_esc() { + printf '%s' "$1" \ + | sed 's/\\/\\\\/g; s/"/\\"/g' \ + | sed ':a;N;$!ba;s/\n/\\n/g' +} + +# --max-time well under script_timeout so a slow Slack cannot stall the monitor. +# Output is discarded: an error response body can echo the URL back. +curl -sS -X POST \ + --max-time 10 \ + -H 'Content-Type: application/json' \ + --data "{\"text\":\"$(json_esc "$TEXT")\"}" \ + "$SLACK_WEBHOOK_URL" >/dev/null 2>&1 \ + || printf '%s event=%s slack_post_failed\n' "$TS" "$EVENT" >> "$LOG" 2>/dev/null + +exit 0 diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf index b5528a6..f638d84 100644 --- a/deploy/maxscale/maxscale.cnf +++ b/deploy/maxscale/maxscale.cnf @@ -44,13 +44,85 @@ module=mariadbmon servers=primary,replica user=$MAXSCALE_USER password=$MAXSCALE_PASSWORD +# Credentials mariadbmon writes into CHANGE MASTER when it promotes, demotes or +# rejoins a node. MUST be set explicitly: when omitted these DEFAULT TO THE +# MONITOR USER above, and mariadbmon then builds the replication link as +# maxscale@ — an account that does not exist, because +# the monitor user is host-scoped to the MaxScale host. The symptom is a rejoin +# that appears to succeed and then immediately flips back +# ("new_slave" followed by "lost_slave" ~2s later), with the real cause only +# visible on the rejoining node as SHOW SLAVE STATUS Last_IO_Error 1045. +# `repl` is host-scoped to BOTH DB hosts precisely so either direction works. +replication_user=$REPL_USER +replication_password=$REPL_PASSWORD monitor_interval=2000ms -# Keep replicas read-only automatically. Automated failover is OFF by default: -# promoting a replica on a 2-node async pair risks split-brain/data loss and -# should be a deliberate ops action. See deploy/mariadb/README.md to enable. -enforce_read_only_slaves=true + +# --- Alerting ---------------------------------------------------------------- +# Fired by the monitor on each event below, as the `maxscale` user. See +# deploy/maxscale/maxscale-alert.sh — it always writes +# /var/log/maxscale/failover-events.log and posts to Slack only if +# /etc/maxscale.secrets.d/alert.env supplies a webhook, so it is safe to install +# before that exists. The script must stay well inside script_timeout or it +# blocks the monitor; the curl inside it is capped at 10s. +# +# NOTE this can only report what MaxScale is alive to observe — it cannot tell +# you MaxScale itself has died. That needs an external check. +script=/usr/local/bin/maxscale-alert.sh $EVENT $INITIATOR $NODELIST $PARENT +script_timeout=90000ms +# Explicit rather than the default "all": these are the transitions worth waking +# someone for. slave_down/lost_slave are included deliberately — on a two-node +# pair, losing the replica silently degrades semi-sync to async, so it is the +# moment the zero-data-loss guarantee stops holding. +# `slave_up` and `new_slave` are BOTH needed and are not the same transition: +# new_slave is [Running]->[Slave,Running] (a standalone node joining), slave_up +# is [Down]->[Slave,Running] (a node coming back from an outage). Listing only +# new_slave means an outage alerts on the way down and stays silent on recovery. +events=master_down,master_up,lost_master,new_master,slave_down,slave_up,lost_slave,new_slave,server_down,server_up + +# --- Automated failover ------------------------------------------------------ +# ON. This was previously off because promoting a replica on a 2-node ASYNC pair +# loses the un-replicated tail of the write stream and can split-brain. Both of +# those are addressed elsewhere rather than by leaving failover off: +# +# Data loss -> replication is now SEMI-SYNCHRONOUS with wait_point=AFTER_SYNC +# (see deploy/mariadb/primary.cnf). DB1 does not acknowledge a +# commit until DB2 holds the event durably, so a promotion +# cannot discard an acknowledged write. +# Split-brain -> a 2-node cluster has no quorum and therefore cannot VOTE on +# who is alive. Instead the write path is fenced AT THE NETWORK: +# 3306 on both DB hosts is firewalled to this host and the DB +# peer only, so MaxScale is the ONLY route to the databases. +# "MaxScale cannot see DB1" therefore also means "the CMS cannot +# see DB1", and promoting DB2 cannot produce two servers taking +# application writes. +# Do NOT try to fence this by dropping triangle_user@: +# MaxScale authenticates a client against the backend user table +# using the CLIENT's own source address, so that account is what +# lets Delta log in THROUGH MaxScale. Dropping it does not close +# a bypass, it just breaks the site. See deploy/mariadb/README.md. +# Divergence -> gtid_strict_mode=ON on both nodes, so a rejoining old primary +# that diverged is REFUSED by auto_rejoin instead of silently +# corrupting the dataset. +# +# Residual, accepted: while DB2 is down, semi-sync degrades to async +# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-loss guarantee lapses; +# and MaxScale itself remains a single point of failure. Closing either needs a +# third node. See deploy/mariadb/README.md. +auto_failover=true +# Consecutive failed monitor passes before the master is declared down. +# 5 x 2000ms = ~10s of confirmed unreachability, which rides out a container +# migration or a brief network blip without promoting. +failcount=5 +failover_timeout=90000ms +# Rejoin a returning old primary as a replica automatically; gtid_strict_mode +# makes this refuse rather than corrupt if it diverged. auto_rejoin=true -auto_failover=false +# Force every non-primary node read_only, so the demoted server cannot take +# writes even if something reaches it directly. +enforce_read_only_slaves=true +# Two-node primary/replica and nothing else — reject any topology MaxScale did +# not expect rather than trying to reason about it. +enforce_simple_topology=true # --- Service: read/write split router ---------------------------------------- [Read-Write-Split] diff --git a/observability/blackbox/blackbox.yml b/observability/blackbox/blackbox.yml new file mode 100644 index 0000000..3c098d6 --- /dev/null +++ b/observability/blackbox/blackbox.yml @@ -0,0 +1,33 @@ +# blackbox_exporter modules for deploy/compose.observability.yml. +# +# Purpose: probe MaxScale's client listener from OUTSIDE the database tier. +# Everything else in this stack watches the CMS itself, which cannot report the +# one failure that matters most here — MaxScale being dead. Since the write path +# is fenced so that MaxScale is the ONLY route to the databases (see +# deploy/mariadb/README.md), MaxScale down is a total outage, and MaxScale's own +# alert script cannot report it: the process that would send the alert is the +# process that died. +# +# A TCP connect against :4006 is deliberately chosen over the MaxScale REST API: +# - 4006 is the port the CMS actually uses, so this tests the real dependency +# rather than a management interface that could be healthy while routing is +# not. +# - It needs no new exposure. The admin API (8989) would have to be opened to +# Delta, and it can reconfigure MaxScale, so it is a far worse thing to +# expose than a port the CMS already talks to. +# - MaxScale 24.02 serves no Prometheus endpoint (verified: /metrics and +# /v1/metrics both 404), so scraping it directly is not an option anyway. + +modules: + # Plain TCP handshake. We do NOT speak the MySQL protocol here: MaxScale sends + # a server greeting on connect, and completing a real login would need + # credentials in this file. Accepting the connection is sufficient evidence + # that MaxScale is alive and listening. + tcp_connect: + prober: tcp + # Comfortably under Prometheus's scrape_timeout so a slow probe surfaces as + # a failed probe rather than a scrape error, which would alert differently. + timeout: 5s + tcp: + ip_protocol_fallback: false + preferred_ip_protocol: ip4 diff --git a/observability/grafana/provisioning/alerting/maxscale.yml b/observability/grafana/provisioning/alerting/maxscale.yml new file mode 100644 index 0000000..9341a32 --- /dev/null +++ b/observability/grafana/provisioning/alerting/maxscale.yml @@ -0,0 +1,102 @@ +# Grafana unified-alerting provisioning: the database tier's outside-in watchdog. +# +# This exists to cover the one gap MaxScale's own alert script cannot: MaxScale +# being dead. The write path is fenced so MaxScale is the only route to the +# databases, so that is a total outage — and the script that would report it +# runs on the machine that stopped. See deploy/mariadb/README.md. +# +# $SLACK_WEBHOOK_URL is interpolated by Grafana from the container environment, +# supplied by compose.observability.yml. Provisioning rejects an empty URL, so +# the compose file defaults it to a placeholder; delivery then fails loudly in +# Grafana's log rather than silently doing nothing. +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: slack-triangle + receivers: + - uid: slack-triangle-receiver + type: slack + settings: + url: $SLACK_WEBHOOK_URL + title: '{{ template "slack.default.title" . }}' + text: |- + {{ range .Alerts }}*{{ .Status | toUpper }}* {{ .Labels.alertname }} + {{ .Annotations.summary }} + {{ .Annotations.description }} + {{ end }} + # Recovery matters as much as onset here: a database outage that ended + # is information someone is actively waiting for. + disableResolveMessage: false + +policies: + - orgId: 1 + receiver: slack-triangle + group_by: [alertname] + # Short waits: this is a total-outage alert, not a capacity warning. + group_wait: 10s + group_interval: 1m + # Re-notify hourly while still firing, so an unresolved outage does not fall + # off the radar after a single message. + repeat_interval: 1h + +groups: + - orgId: 1 + name: database-tier + folder: Alerts + # Evaluate every 30s. Prometheus scrapes every 15s, so this cannot miss a + # sustained failure while staying cheap. + interval: 30s + rules: + - uid: maxscale-unreachable + title: MaxScale unreachable + # Deliberately NOT "no data = alerting" for the probe itself: see + # noDataState below. + condition: threshold + for: 1m + data: + - refId: probe + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: prometheus + model: + refId: probe + editorMode: code + expr: probe_success{service="maxscale"} + instant: true + range: false + - refId: threshold + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: __expr__ + model: + refId: threshold + type: threshold + expression: probe + conditions: + - evaluator: + type: lt + params: [1] + # If the probe series vanishes entirely, something is wrong with + # Prometheus or blackbox rather than with MaxScale — but the practical + # consequence is the same: nobody is watching the database tier. Alert + # on it rather than failing open. + noDataState: Alerting + execErrState: Alerting + labels: + severity: critical + component: database + annotations: + summary: "MaxScale is not accepting connections on 10.248.40.183:4006" + description: >- + A TCP probe from Delta to MaxScale's client listener has failed for + 1 minute. The CMS reaches the databases ONLY through MaxScale (3306 + on DB1/DB2 is firewalled to the MaxScale host and the DB peer), so + this is a full database outage, not a degraded read path. + MaxScale's own Slack alerts cannot fire for this — the process that + sends them is the one that is down. + Check: systemctl status maxscale on 10.248.40.183, then + maxctrl list servers. If MaxScale is healthy and this still fires, + suspect the network path from Delta rather than the database tier. diff --git a/observability/prometheus/prometheus.delta.yml b/observability/prometheus/prometheus.delta.yml index e46df74..aee676c 100644 --- a/observability/prometheus/prometheus.delta.yml +++ b/observability/prometheus/prometheus.delta.yml @@ -33,3 +33,32 @@ scrape_configs: - targets: ["triangle-cms-backend-green-1:8080"] labels: slot: green + + # Watchdog for the database tier's single point of failure. + # + # The write path is fenced so MaxScale is the ONLY route to the databases, so + # MaxScale being down is a total outage — and it is precisely the outage + # MaxScale's own alert script cannot report, because the process that would + # send the alert is the one that died. This probes it from Delta instead. + # + # DB1/DB2 are deliberately NOT probed here: their 3306 is firewalled to the + # MaxScale host and the DB peer, so Delta cannot reach them by design, and a + # target that can never succeed is a permanently firing alert. + - job_name: blackbox-tcp + metrics_path: /probe + params: + module: [tcp_connect] + static_configs: + - targets: ["10.248.40.183:4006"] + labels: + service: maxscale + # The standard blackbox indirection: Prometheus scrapes the EXPORTER, and + # the real target travels as a query parameter. Without this rewrite + # Prometheus would try to scrape 10.248.40.183:4006 as if it served metrics. + relabel_configs: + - source_labels: [__address__] + target_label: __param_target + - source_labels: [__param_target] + target_label: instance + - target_label: __address__ + replacement: blackbox:9115