From 6eba1c6b317da534a112a90240e25c73d0f69de8 Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 4 Aug 2026 21:00:33 -0400 Subject: [PATCH 1/5] Make DB2 a safe automated failover target, not just a read replica DB2 exists now (THETRIANGLE-DB2-LXC, CT 111), renumbered to 10.248.40.155 after briefly sharing DB1's address. Wire it in as a failover target and turn on MaxScale's auto_failover, which was off because promoting a replica on a 2-node async pair loses acknowledged writes and can split-brain. Neither objection is answered by the failover flag itself, so: - Semi-sync replication with wait_point=AFTER_SYNC. DB1 does not commit to the storage engine until DB2 holds the event, so a promotion cannot drop a write the CMS was told succeeded. AFTER_COMMIT (the default) would not give this -- it makes the write visible before the ack, which is exactly the losing window. - Fence the write path instead of trying to reach quorum, which two nodes cannot do. MaxScale becomes the only route to the databases, so "MaxScale cannot see DB1" implies "the CMS cannot see DB1" and promoting DB2 cannot produce two servers taking writes. - gtid_strict_mode was already ON, so a diverged old primary is refused by auto_rejoin rather than corrupting the dataset. Two bugs found while doing this: - replica.cnf used relaxed durability (innodb_flush_log_at_trx_commit=2, sync_binlog=0). Right for a read cache, wrong for a failover target twice over: a promoted DB2 would run production without ACID, and semi-sync's ack would only mean "in DB2's page cache". Now 1/1, matching the primary. - setup-replica.sh passed --gtid alone, which records NO replication position -- in MariaDB it only changes the format of what --master-data emits. A replica seeded that way starts from the beginning of binlogs that expire after 7 days. Now --master-data=1 (=2 comments the line out), with a hard check that the dump actually contains the position. Adds provision-db2.sh to install MariaDB 11.8 and the replica config in one step; it refuses to run anywhere but DB2, since ssh gives no warning when an ARP change puts you on a different host. This changes source-of-truth config only. Nothing is applied to the DB hosts by merging it -- see "Bringing up DB2" in deploy/mariadb/README.md, and do not enable auto_failover before semi-sync is confirmed engaged and the write path is fenced. Co-Authored-By: Claude Opus 5 --- deploy/mariadb/README.md | 296 +++++++++++++++++++++++++++----- deploy/mariadb/primary.cnf | 22 +++ deploy/mariadb/provision-db2.sh | 108 ++++++++++++ deploy/mariadb/replica.cnf | 39 +++-- deploy/mariadb/setup-replica.sh | 70 ++++++-- deploy/maxscale/maxscale.cnf | 44 ++++- 6 files changed, 507 insertions(+), 72 deletions(-) create mode 100755 deploy/mariadb/provision-db2.sh diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index e1e728b..050efe0 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -6,25 +6,32 @@ The CMS opens a single connection to the endpoint configured as which splits writes to the primary and reads to the replica. ``` - CMS / app host (Delta) DB primary host DB replica host - ┌───────────────────────┐ ┌────────────────┐ ┌────────────────┐ - │ cms-blue cms-green │ writes │ DB1 │ GTID│ DB2 (planned) │ - │ │ │ │────────▶│ (server_id 1) │────▶│ (server_id 2) │ - │ └────┬───┘ │ │ binlog+ACID │async│ read_only │ - └───────────┼───────────┘ reads └────────────────┘ └────────────────┘ - └──────────▶ MaxScale (rwsplit :4006) ──────────────▶ + CMS / app host (Delta) DB primary host DB replica host + ┌───────────────────────┐ ┌────────────────┐ ┌────────────────┐ + │ cms-blue cms-green │ writes │ DB1 │ GTID │ DB2 │ + │ │ │ │────────▶│ (server_id 1) │──────▶│ (server_id 2) │ + │ └────┬───┘ │ │ binlog+ACID │◀──ack─│ read_only │ + └───────────┼───────────┘ reads └────────────────┘ semi- └────────────────┘ + └──────────▶ MaxScale (rwsplit :4006) ─────sync──────▶ + mariadbmon: auto_failover ``` -| Host | Address | Role | Software | -| --- | --- | --- | --- | -| `THETRIANGLE-DB1-LXC` | `10.248.40.154` | primary | MariaDB 11.8 LTS | -| `THETRIANGLE-MAXSCALE` | `10.248.40.183` | proxy | MaxScale 24.02 | -| DB2 | not provisioned | replica | — | +| Host | Address | CT | Role | Software | +| --- | --- | --- | --- | --- | +| `THETRIANGLE-DB1-LXC` | `10.248.40.154` | 108 | primary | MariaDB 11.8 LTS | +| `THETRIANGLE-DB2-LXC` | `10.248.40.155` | 111 | replica / failover target | MariaDB 11.8 LTS | +| `THETRIANGLE-MAXSCALE` | `10.248.40.183` | 109 | proxy | MaxScale 24.02 | -Both are unprivileged **LXC containers**, 4 vCPU / 4 GB / 63 GB, installed from -apt. Docker was deliberately not used: it needs Proxmox-side nesting on an +All three are unprivileged **LXC containers**, 4 vCPU / 4 GB / 63 GB, installed +from apt. Docker was deliberately not used: it needs Proxmox-side nesting on an unprivileged container and costs ~300 MB of a 4 GB budget. +> **DB2 was created ~2026-08-03 with DB1's address**, `10.248.40.154`, and both +> were live on the bridge — which host you reached depended on which ARP entry +> won on your path. It has since been renumbered to `10.248.40.155` +> (MAC `bc:24:11:71:f9:f8`). Confirm `hostname` before trusting the output of +> anything you send to a DB host. + ## What this directory is These `.cnf` files are the **source of truth for what is installed on those @@ -33,9 +40,10 @@ hosts** — edit here, copy up, restart the service. | File | Installed as | Host | | --- | --- | --- | | [primary.cnf](primary.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-primary.cnf` | DB1 | -| [replica.cnf](replica.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf` | DB2, when it exists | +| [replica.cnf](replica.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf` | DB2 | | [../maxscale/maxscale.cnf](../maxscale/maxscale.cnf) | `/etc/maxscale.cnf` | MaxScale | -| [setup-replica.sh](setup-replica.sh) | run once on DB2 | DB2, when it exists | +| [provision-db2.sh](provision-db2.sh) | run once on DB2 | DB2 | +| [setup-replica.sh](setup-replica.sh) | run once on DB2, after the above | DB2 | **The `70-` prefix is load-bearing.** MariaDB reads `mariadb.conf.d/` in lexical order and Ubuntu's stock `50-server.cnf` sets `bind-address = 127.0.0.1`. A file @@ -73,7 +81,7 @@ never left the boxes. ``` PRIMARY_HOST=10.248.40.154 - REPLICA_HOST=192.0.2.2 # RFC 5737 placeholder until DB2 exists + REPLICA_HOST=10.248.40.155 MARIADB_PORT=3306 MAXSCALE_USER=maxscale MAXSCALE_PASSWORD=... @@ -84,26 +92,140 @@ never left the boxes. | Account | Grants | Why | | --- | --- | --- | | `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | the app, via MaxScale | -| `triangle_user@10.248.40.168` | `ALL PRIVILEGES ON triangle.*` | Delta direct; cutover verification only, droppable | -| `maxscale@10.248.40.183` | monitor + account-table reads | no failover privileges | -| `repl@'%'` | `REPLICATION SLAVE` | pending DB2 | +| `maxscale@10.248.40.183` | monitor, account reads, **+ failover admin** | promotes/demotes on failover | +| `repl@10.248.40.155` | `REPLICATION SLAVE` | DB2's replication link | `ALL PRIVILEGES` rather than DML-only because the CMS runs additive DDL (`ADD COLUMN IF NOT EXISTS`) at startup. -## Adding the replica (DB2) +**`triangle_user@10.248.40.168` (Delta direct) must be dropped**, and is listed +here only so it is not recreated. It was a cutover-verification account that let +Delta bypass MaxScale. Leaving it in place defeats the fencing that makes +automated failover safe: it is a second write path to DB1 that survives MaxScale +deciding DB1 is dead. See *Why automated failover is safe here* below. + +The same account set must exist on **DB2**, because on promotion DB2 serves the +application. `repl@10.248.40.154` is needed there too so that the old primary +can replicate back after `auto_rejoin`. Grants replicate automatically once +replication is running (they are DDL on `mysql.*`), so creating them on DB1 +after DB2 is attached is sufficient — but verify rather than assume. + +## Bringing up DB2 + +Ordered so the cluster is never in a state where automated failover could fire +at a replica that is not ready. **Do not enable `auto_failover` before step 6.** + +**Prerequisite — `tadmin` has no passwordless sudo on DB2.** It is in the `sudo` +group but no `NOPASSWD` rule exists, unlike DB1 and MaxScale. Either add one to +match the other two hosts, or run steps 1–3 from the Proxmox console. + +0. **Take a backup of DB1 first.** DB1 is still the only copy of the data (see + Notes). Automated failover is a mechanism for *promoting* a server, not a + substitute for being able to restore one. + ✅ *Done 2026-08-05: `/var/backups/triangle/triangle-predb2-20260805-005529.sql.gz` + on DB1 — 36.7 MB gzipped, 97.6 MB raw, 16 tables, `gzip -t` clean.* + +1–2. **Install MariaDB 11.8 and the replica config** — run + [provision-db2.sh](provision-db2.sh) on DB2 as root. It adds the + `deb.mariadb.org` 11.8 repo (same series as DB1, so a promotion is not also a + version change), installs `replica.cnf` as + `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf`, restarts, and verifies + `server_id=2`, the bind address, and durability. It refuses to run unless the + hostname is `THETRIANGLE-DB2-LXC`. + + *Verified reachable from DB2: DB1:3306, MaxScale:4006, deb.mariadb.org:443.* + +3. **Create the replication accounts on DB1.** + ✅ *Done 2026-08-05: `repl@10.248.40.155` and `repl@10.248.40.154` both + created with `REPLICATION SLAVE`, password from + `/root/triangle-db-credentials.env`.* The `.154` one is so the **old primary + can replicate back** from DB2 after `auto_rejoin`; without it, a rejoin fails + on access denied. + + A pre-existing **`repl@'%'` is still present** and should be dropped — a + wildcard replication account defeats the point of host-scoping, and DB1's + 3306 is reachable from the whole subnet until the firewall step below. + +4. **Run [setup-replica.sh](setup-replica.sh) on DB2** (not on DB1). It takes a + GTID-consistent `--single-transaction --master-data=1` dump from DB1, restores + it, and starts replication. DB1 keeps serving throughout; the only lock is a + brief global read lock at the start, held just long enough to read the binlog + position: + + ``` + PRIMARY_HOST=10.248.40.154 \ + DUMP_USER=root DUMP_PASSWORD=... \ + REPL_USER=repl REPL_PASSWORD=... \ + sudo -E sh setup-replica.sh + ``` + + `DUMP_USER` must be a privileged account, **not** `repl`: `REPLICATION SLAVE` + grants the binlog stream but no table reads, so dumping as `repl` fails with + "SELECT command denied". The script is idempotent — it exits early if + replication is already running. + + Confirm `Slave_IO_Running: Yes`, `Slave_SQL_Running: Yes`, and + `Seconds_Behind_Master: 0` before continuing. + +5. **Turn on semi-sync and confirm it engages.** The setting is persisted in the + `.cnf` files, but it is dynamic, so it can be applied without a restart: + + ```sql + -- DB1 + SET GLOBAL rpl_semi_sync_master_enabled = ON; + SET GLOBAL rpl_semi_sync_master_wait_point = AFTER_SYNC; + SET GLOBAL rpl_semi_sync_master_timeout = 1000; + SET GLOBAL rpl_semi_sync_master_wait_no_slave = OFF; + -- DB2 + SET GLOBAL rpl_semi_sync_slave_enabled = ON; + STOP SLAVE IO_THREAD; START SLAVE IO_THREAD; -- required: the slave only + -- registers as semi-sync when + -- the IO thread reconnects + ``` + + Then on DB1, **verify rather than assume**: + + ```sql + SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- must be ON + SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1 + SHOW STATUS LIKE 'Rpl_semi_sync_master_no_tx'; -- should stop climbing + ``` + + `Rpl_semi_sync_master_status = OFF` with replication otherwise healthy means + it silently degraded to async — the zero-loss guarantee is not in force and + failover is not yet safe to enable. + +6. **Fence the write path.** This is what removes the split-brain risk, and it + is a prerequisite for step 8, not an optional hardening pass: + + ```sql + DROP USER 'triangle_user'@'10.248.40.168'; -- Delta's direct bypass + ``` + + and restrict 3306 on both DB hosts to MaxScale and the DB peer. **Do this + with Proxmox console access open** — DB1's firewall is currently wide open + (ufw inactive, iptables ACCEPT) and it was left that way deliberately to + avoid an SSH lockout on a remote host. -1. Provision the host, install MariaDB 11.8, install `replica.cnf` as - `70-triangle-replica.cnf` **with `bind-address` set** to DB2's NIC. -2. Run [setup-replica.sh](setup-replica.sh) on DB2 — it takes a GTID-consistent - `--single-transaction` dump from DB1 (non-blocking; DB1 keeps serving), - restores it, and starts replication. Pass `DUMP_USER`/`DUMP_PASSWORD` for a - privileged account: dumping as `repl` fails, because `REPLICATION SLAVE` - grants the binlog stream but no table reads. -3. Point `REPLICA_HOST` in MaxScale's `backend.env` at DB2 and restart MaxScale. - `maxscale.cnf` already carries the two-server topology, so no config edit is - needed. -4. Confirm `maxctrl list servers` shows `Master, Running` and `Slave, Running`. +7. **Grant MaxScale the failover privileges** it does not currently hold, on + DB1: + + ```sql + GRANT REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD + ON *.* TO 'maxscale'@'10.248.40.183'; + ``` + +8. **Point MaxScale at DB2 and enable failover.** Set + `REPLICA_HOST=10.248.40.155` in `/etc/maxscale.secrets.d/backend.env` + (it is the RFC 5737 placeholder `192.0.2.2` until this is done), copy up the + new `maxscale.cnf`, and `systemctl restart maxscale`. + + Confirm `maxctrl list servers` shows `Master, Running` **and** + `Slave, Running` — a replica showing `Down` here means MaxScale still has the + placeholder address. + +9. **Test the failover before trusting it**, at a quiet hour, with the backup + from step 0 in hand. See below. ## Verifying the split @@ -118,18 +240,84 @@ session — on the primary. `causal_reads=local` makes MaxScale wait for the replica to reach the write's GTID, so a session always sees its own writes despite lag and **no app-level split is needed**. -## Enabling automated failover (optional) +## Why automated failover is safe here + +`auto_failover` is **on**. The two standard objections to automating promotion +on a 2-node pair are real, and neither is answered by the failover setting +itself — they are answered by the replication mode and the network topology. + +**"Async replication loses the tail on promotion."** True, and it is why this +was left off originally. Replication is now **semi-synchronous with +`wait_point=AFTER_SYNC`**: DB1 does not commit to the storage engine, and +therefore never acknowledges to the CMS, until DB2 has the binlog event +durably. A promoted DB2 cannot be missing a write that the application was told +succeeded. `AFTER_COMMIT` — the MariaDB default — would *not* give this: it +makes the write visible to other sessions before the ack, which is exactly the +window that loses data. + +**"Two nodes can't tell 'primary is dead' from 'I can't see the primary'."** +Also true, and unfixable at that layer: a 2-node cluster has no quorum, so it +cannot vote. The risk is removed structurally instead, by making MaxScale the +**only** path to the databases — Delta's direct `triangle_user` grant is +dropped and 3306 is firewalled to MaxScale and the DB peer. Once that holds, +"MaxScale cannot reach DB1" implies "the CMS cannot reach DB1", so promoting +DB2 cannot result in two servers taking application writes. The partition that +would split-brain a quorum-less cluster instead just moves all traffic to the +promoted node, which is the desired outcome. + +**"The old primary comes back and clobbers things."** `gtid_strict_mode=ON` on +both nodes. A returning DB1 that diverged is *refused* by `auto_rejoin` and +sits there needing a human, rather than replicating conflicting history. + +### What is still not covered + +Honest limits, all of which need a third node to close: + +- **While DB2 is down, semi-sync degrades to async** and the zero-loss guarantee + lapses. This is deliberate — `rpl_semi_sync_master_wait_no_slave=OFF` — because + the alternative is DB1 stalling every commit for `rpl_semi_sync_master_timeout` + whenever DB2 is offline, turning a replica outage into a site outage. So + DB2-down-then-DB1-dies can still lose writes. **Alert on + `Rpl_semi_sync_master_status`** rather than assuming the guarantee holds. +- **MaxScale is a single point of failure** and the sole arbiter. If it dies, + the CMS is down regardless of how healthy both databases are. Fencing the + write path to MaxScale deepens this dependency — that is the price of removing + split-brain without a quorum. +- **Failover is not backup.** Promotion protects against a host dying, not + against a bad migration or a `DROP TABLE`, both of which replicate to DB2 in + milliseconds. + +### Planned maintenance + +Use **switchover**, not failover — it demotes the old primary cleanly instead of +assuming it is dead: + +``` +maxctrl call command mariadbmon switchover MariaDB-Monitor replica primary +``` + +### Testing it + +Do this once, at a quiet hour, with a fresh DB1 backup, **before** relying on it: + +``` +maxctrl list servers # baseline: Master/Slave, Running +``` + +Stop MariaDB on DB1 (`systemctl stop mariadb`), then watch MaxScale's log at +`/var/log/maxscale/maxscale.log`. Within ~10s (`failcount` 5 × `monitor_interval` +2000ms) it should log `master_down`, promote DB2, and `maxctrl list servers` +should show DB2 as `Master, Running`. Confirm the CMS still serves and can write. -`auto_failover` is **off**. Promoting a replica on a 2-node async pair risks -split-brain and data loss, and should be a deliberate ops action. To enable: +Then start DB1 again and confirm `auto_rejoin` brings it back as +`Slave, Running`. **If it does not rejoin, that is the safety net working, not a +bug** — check for divergence before forcing anything. Finish by switching back +with the `switchover` command above so DB1 is primary again. -1. Grant `REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD` to - `MAXSCALE_USER` on the primary. -2. Set `auto_failover=true` in [maxscale.cnf](../maxscale/maxscale.cnf). -3. Prefer `maxctrl call command mariadbmon switchover` for planned maintenance. +### Manual failover, without MaxScale -Manual failover without MaxScale: on the replica `STOP SLAVE; RESET SLAVE ALL;`, -set `read_only=OFF`, repoint `PRIMARY_HOST`, rebuild the old primary as a replica. +On DB2: `STOP SLAVE; RESET SLAVE ALL;` then `SET GLOBAL read_only=OFF`. Repoint +`PRIMARY_HOST` and rebuild the old primary as a replica. ## Notes / gotchas @@ -150,10 +338,30 @@ set `read_only=OFF`, repoint `PRIMARY_HOST`, rebuild the old primary as a replic reachable from the whole subnet. It should be restricted to the MaxScale host (and DB2 later); this was left alone deliberately to avoid an SSH lockout on a remote host, so do it with console access available. -- **DB1 is the only copy of the data.** The old dev container and its volume are - gone. The pre-cutover dump on Delta at +- **`mariadb-dump --gtid` on its own records NOTHING.** It only changes the + *format* of the position emitted by `--master-data`/`--dump-slave`, so without + one of those the dump carries no replication start position at all. A replica + seeded from such a dump begins at its own empty `gtid_slave_pos` — i.e. the + start of the primary's binlogs, which expire after 7 days + (`binlog_expire_logs_seconds`) — so replication either dies with error 1236 or + replays history on top of the restored data. `setup-replica.sh` carried this + bug and now passes `--master-data=1`, and hard-fails if the dump comes out + without an active `SET GLOBAL gtid_slave_pos=` line. It must be `=1`: `=2` + emits the same line **commented out**. Verified against 11.8.8. +- **DB1 is the only copy of the data until DB2 is replicating.** The old dev + container and its volume are gone. The pre-cutover dump on Delta at `~/triangle-deploy/backups/triangle-precutover-20260729-2137.sql.gz` is a - point-in-time artifact, not a backup rotation — real backups are still owed. + point-in-time artifact, not a backup rotation — real backups are still owed, + and a replica is not one: a `DROP TABLE` reaches DB2 in milliseconds. +- **`tadmin` has no `NOPASSWD` sudo on DB2**, unlike DB1 and MaxScale. It is in + the `sudo` group, so an interactive password works, but every scripted step in + the bring-up runbook fails without a rule matching the other two hosts. Note + the failure mode is quiet: `ssh ... 'sudo -n ...' ` prints "sudo: a password is + required" to stderr while the pipeline reports success, because + `cmd | ssh ...` returns the *local* command's exit status. +- **DB2 has no `10-eth0-static.network` override.** Its address is pinned in the + CT config (`pct config 111`) but not inside the container, so it relies on a + single layer where DB1 and MaxScale have two. Add the override to match. - `server_id` must be unique per node (1 primary / 2 replica); `gtid_domain_id` must match (1 here). - Schema changes: the CMS runs additive, idempotent migrations at startup that diff --git a/deploy/mariadb/primary.cnf b/deploy/mariadb/primary.cnf index 987b5b8..2893f80 100644 --- a/deploy/mariadb/primary.cnf +++ b/deploy/mariadb/primary.cnf @@ -74,3 +74,25 @@ binlog_expire_logs_seconds = 604800 # keep 7 days of binlogs for replica gtid_domain_id = 1 gtid_strict_mode = ON log_slave_updates = ON # lets the replica be chained / used for backups + +# --- Semi-synchronous replication: MASTER side -------------------------------- +# This is the setting that makes MaxScale's automated failover safe to turn on. +# Without it, replication is async: DB1 acknowledges a commit to the CMS before +# DB2 has seen it, so promoting DB2 silently discards the tail of the write +# stream. With it, a commit is not acknowledged until DB2 has the event. +rpl_semi_sync_master_enabled = ON +# AFTER_SYNC (not the AFTER_COMMIT default): wait for the replica's ack BEFORE +# committing to the storage engine, so a write is never visible to other +# sessions on DB1 until it is safe on DB2. AFTER_COMMIT makes the write visible +# first, which is precisely the window that loses data on promotion. +rpl_semi_sync_master_wait_point = AFTER_SYNC +# How long a commit waits for an ack before degrading to async, in ms. +rpl_semi_sync_master_timeout = 1000 +# OFF is load-bearing for availability. With the ON default, DB1 pays the full +# timeout above on EVERY commit while no replica is connected — so planned DB2 +# maintenance would add a second of latency to every write. OFF drops straight +# to async the moment there is no semi-sync replica, and back to semi-sync when +# DB2 reconnects. The tradeoff is real and must be understood: while DB2 is +# down, the zero-loss guarantee is NOT in force. Alert on +# Rpl_semi_sync_master_status = OFF rather than assuming it holds. +rpl_semi_sync_master_wait_no_slave = OFF diff --git a/deploy/mariadb/provision-db2.sh b/deploy/mariadb/provision-db2.sh new file mode 100755 index 0000000..6da526f --- /dev/null +++ b/deploy/mariadb/provision-db2.sh @@ -0,0 +1,108 @@ +#!/bin/sh +# One-time: install MariaDB 11.8 on DB2 and put the replica config in place. +# +# Run this ON DB2 (THETRIANGLE-DB2-LXC, 10.248.40.155) as root: +# +# sudo sh provision-db2.sh +# +# It does NOT start replication and does NOT touch MaxScale — it only gets a +# correctly-configured, correctly-bound MariaDB running. Run setup-replica.sh +# afterwards, then follow "Bringing up DB2" in README.md from step 5. +# +# Idempotent: safe to re-run. Existing repo/key/config are refreshed in place +# and apt skips packages already at the right version. +# +# NOTE ON ACCESS: tadmin has no NOPASSWD sudo on DB2 (unlike DB1 and MaxScale), +# so this cannot be driven over ssh non-interactively until such a rule exists: +# echo 'tadmin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/90-tadmin +# chmod 440 /etc/sudoers.d/90-tadmin +set -eu + +EXPECT_HOST=THETRIANGLE-DB2-LXC +EXPECT_ADDR=10.248.40.155 +SERIES=11.8 # LTS. deb.mariadb.org carries ONLY LTS lines. +CNF_SRC="$(dirname "$0")/replica.cnf" +CNF_DST=/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf + +# --- Guards ------------------------------------------------------------------ +# DB2 was briefly live on DB1's address (10.248.40.154) and ssh gives no +# host-key warning when the ARP winner changes underneath you. Never let this +# script run against the primary: it would overwrite the primary's config with +# a read_only replica config. +[ "$(id -u)" = 0 ] || { echo "must run as root" >&2; exit 1; } +if [ "$(hostname)" != "$EXPECT_HOST" ]; then + echo "REFUSING: hostname is '$(hostname)', expected '$EXPECT_HOST'." >&2 + echo "You are not on DB2. Check which host you actually reached." >&2 + exit 1 +fi +if ! ip -4 addr show | grep -q "inet ${EXPECT_ADDR}/"; then + echo "REFUSING: ${EXPECT_ADDR} is not configured on this host." >&2 + exit 1 +fi +[ -f "$CNF_SRC" ] || { echo "cannot find replica.cnf next to this script" >&2; exit 1; } + +echo "==> Host verified: $(hostname) / ${EXPECT_ADDR}" + +# --- MariaDB apt repo --------------------------------------------------------- +# Note this is the SERVER key. MaxScale uses a different key entirely and is not +# installed here — see README.md. +echo "==> Adding MariaDB ${SERIES} repository" +apt-get update -qq +apt-get install -y -qq curl gpg apt-transport-https ca-certificates + +install -d -m 0755 /etc/apt/keyrings +curl -fsSL https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY \ + | gpg --dearmor --yes -o /etc/apt/keyrings/mariadb.gpg +chmod 0644 /etc/apt/keyrings/mariadb.gpg + +. /etc/os-release +cat > /etc/apt/sources.list.d/mariadb.list < Installing mariadb-server" +DEBIAN_FRONTEND=noninteractive apt-get install -y -qq mariadb-server mariadb-client +mariadbd --version + +# --- Config ------------------------------------------------------------------- +# The 70- prefix is load-bearing: Ubuntu's stock 50-server.cnf sets +# bind-address = 127.0.0.1 and mariadb.conf.d is read in lexical order, so a +# file sorting before it cannot override the bind and the replica would be +# unreachable from both MaxScale and the primary. +echo "==> Installing ${CNF_DST}" +install -o root -g root -m 0644 "$CNF_SRC" "$CNF_DST" + +echo "==> Restarting mariadb" +systemctl enable --now mariadb +systemctl restart mariadb + +# --- Verify ------------------------------------------------------------------- +echo "==> Verifying" +mariadb -N -B -e "SELECT @@hostname, @@server_id, @@read_only, @@gtid_domain_id, @@gtid_strict_mode" + +# server_id must differ from DB1's (1) or replication refuses to start. +SID=$(mariadb -N -B -e "SELECT @@server_id") +[ "$SID" = "2" ] || { echo "FAIL: server_id is ${SID}, expected 2" >&2; exit 1; } + +# The whole point of the 70- prefix. If this shows 127.0.0.1, the config did not +# take and nothing downstream will work. +echo "--- listening sockets ---" +ss -ltnp 2>/dev/null | grep 3306 || echo "WARNING: nothing listening on 3306" +if ! ss -ltn 2>/dev/null | grep -q "${EXPECT_ADDR}:3306\|0.0.0.0:3306\|\*:3306"; then + echo "FAIL: not bound to ${EXPECT_ADDR}:3306 — check ${CNF_DST} ordering" >&2 + exit 1 +fi + +# Durability must match the primary: DB2 is a failover target, not a read cache. +echo "--- durability (must be 1 / 1) ---" +mariadb -N -B -e "SELECT @@innodb_flush_log_at_trx_commit, @@sync_binlog" + +echo +echo "OK. MariaDB ${SERIES} is installed, bound to ${EXPECT_ADDR}, and read_only." +echo "NEXT: run setup-replica.sh on this host to seed from DB1 and start" +echo "replication, then continue at README.md 'Bringing up DB2' step 5." diff --git a/deploy/mariadb/replica.cnf b/deploy/mariadb/replica.cnf index ec12a31..6f64ca3 100644 --- a/deploy/mariadb/replica.cnf +++ b/deploy/mariadb/replica.cnf @@ -1,19 +1,19 @@ [mysqld] # ============================================================================= -# Triangle CMS — MariaDB READ REPLICA (DB2) config. NOT YET DEPLOYED: DB2 does -# not exist as of 2026-07-30, so this is the config to install when it lands. -# Runs on its OWN host (separate from DB1), replicating asynchronously from the -# primary via GTID. Serves READ traffic only. +# Triangle CMS — MariaDB REPLICA + FAILOVER TARGET (DB2) config. +# Host: THETRIANGLE-DB2-LXC, 10.248.40.155, CT 111 (4 vCPU / 4 GB / 63 GB). +# Runs on its OWN host (separate from DB1), replicating from the primary via +# GTID with semi-synchronous acknowledgement. Serves READ traffic, and is +# promoted to primary by MaxScale's mariadbmon on failover. # Installed natively at /etc/mysql/mariadb.conf.d/70-triangle-replica.cnf — the # 70- prefix is load-bearing, see bind-address below. # ============================================================================= # --- Network ------------------------------------------------------------------ -# UNCOMMENT AND SET to DB2's internal NIC address before starting MariaDB. # Ubuntu's stock 50-server.cnf binds 127.0.0.1, which would leave the replica # unreachable from both MaxScale and the primary; this file must sort after it. # Firewall 3306 to the MaxScale and primary hosts only. -# bind-address = 10.248.40.xxx +bind-address = 10.248.40.155 # --- InnoDB memory ----------------------------------------------------------- # Sized to match the primary; see the rationale in primary.cnf. Adjust the two @@ -24,13 +24,21 @@ innodb_buffer_pool_size = 1G innodb_log_file_size = 256M innodb_log_buffer_size = 32M -# --- Durability: relaxed on the replica --------------------------------------- -# A read replica can re-fetch anything it loses on crash from the primary via -# GTID, so we trade per-commit fsyncs for throughput on apply. Never do this on -# the primary. -innodb_flush_log_at_trx_commit = 2 -sync_binlog = 0 +# --- Durability: FULL, same as the primary ------------------------------------ +# Deliberately NOT the relaxed (2 / 0) setting usually given to a read replica. +# Two reasons, both consequences of DB2 being a failover target rather than a +# pure read cache: +# 1. On promotion DB2 *becomes* the primary. Relaxed settings would silently +# leave production running without per-commit fsyncs until someone noticed. +# 2. Semi-sync's guarantee is only as strong as the ack. With sync_binlog=0 +# the replica acks once the event is in the OS page cache, so a power loss +# on DB2 discards writes the primary already told the client were durable. +# The CMS write volume is trivial (a newsroom, not a transaction processor), so +# the throughput this costs is not measurable here. +innodb_flush_log_at_trx_commit = 1 # fsync redo on every commit +sync_binlog = 1 # fsync binlog on every commit innodb_flush_method = O_DIRECT +innodb_doublewrite = ON # --- Concurrency / caches ---------------------------------------------------- max_connections = 300 # replica typically fields more read conns @@ -70,3 +78,10 @@ binlog_expire_logs_seconds = 604800 # Parallel apply keeps replication lag low under write bursts from the primary. slave_parallel_threads = 4 slave_parallel_mode = optimistic + +# --- Semi-synchronous replication: SLAVE side --------------------------------- +# The replica acknowledges each binlog event back to the primary, which is what +# lets automated failover promote DB2 without losing acknowledged commits. Must +# be ON here for the primary's AFTER_SYNC wait to ever be satisfied — if this is +# OFF, the primary just times out and degrades to async on every commit. +rpl_semi_sync_slave_enabled = ON diff --git a/deploy/mariadb/setup-replica.sh b/deploy/mariadb/setup-replica.sh index 19c4852..f9de510 100755 --- a/deploy/mariadb/setup-replica.sh +++ b/deploy/mariadb/setup-replica.sh @@ -14,9 +14,9 @@ # (unix_socket auth is the default for root on a native apt install). # # Idempotency: it exits early if replication is already running, so re-running -# after a hiccup is safe. A GTID-consistent dump (--gtid) records the exact -# primary position, so START SLAVE ... MASTER_USE_GTID=slave_pos resumes with no -# gaps or duplicates. +# after a hiccup is safe. The dump records the exact primary position via +# --gtid --master-data=1 (BOTH are needed — see the dump step below), so +# START SLAVE ... MASTER_USE_GTID=slave_pos resumes with no gaps or duplicates. set -eu : "${PRIMARY_HOST:?PRIMARY_HOST is required (DB1's address)}" @@ -50,12 +50,34 @@ until mariadb -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP done echo "dumping ${MARIADB_DATABASE} from primary (GTID-consistent) ..." -# --gtid emits SET GLOBAL gtid_slave_pos=...; --single-transaction takes no lock -# on InnoDB, so the primary keeps serving throughout. +# --master-data=1 is REQUIRED and is not optional decoration. `--gtid` on its own +# emits NOTHING: in MariaDB it only changes the FORMAT of the position recorded +# by --master-data/--dump-slave, so `--gtid` without one of those produces a dump +# carrying no replication start position at all. The replica would then begin +# from its own empty gtid_slave_pos, i.e. from the very start of the primary's +# binlogs — which are expired after binlog_expire_logs_seconds (7 days), so +# replication either dies with error 1236 or replays history on top of the +# restored data. Verified against 11.8.8: --gtid alone emits no gtid line; +# --master-data=1 emits an ACTIVE `SET GLOBAL gtid_slave_pos='1-1-...';`, which +# is what the CHANGE MASTER ... MASTER_USE_GTID=slave_pos below consumes. +# It must be =1, not =2 — =2 comments that same line out. +# +# --single-transaction keeps the dump consistent without locking InnoDB tables, +# but note that combining it with --master-data does take a brief global read +# lock at the very start, just long enough to read the binlog position. It is +# milliseconds, not the length of the dump, and the primary serves throughout. mariadb-dump -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP_PASSWORD}" \ - --single-transaction --gtid --routines --triggers --events \ + --single-transaction --gtid --master-data=1 --routines --triggers --events \ --databases "${MARIADB_DATABASE}" > "${DUMP_FILE}" +# Fail loudly here rather than starting replication from a bogus position. +if ! grep -q "^SET GLOBAL gtid_slave_pos=" "${DUMP_FILE}"; then + echo "ERROR: dump contains no active 'SET GLOBAL gtid_slave_pos=' line." >&2 + echo "Replication would start from the wrong position. Check that" >&2 + echo "${DUMP_USER} holds RELOAD/BINLOG MONITOR on the primary." >&2 + exit 1 +fi + echo "loading dump into replica ..." # read_only=ON is set in replica.cnf; root is exempt (it holds SUPER), so the # restore lands without having to relax it. @@ -77,8 +99,34 @@ sleep 2 local_sql -e "SHOW SLAVE STATUS\G" | grep -E "Slave_IO_Running|Slave_SQL_Running|Seconds_Behind_Master|Last_.*Error" || true echo echo "done. Verify 'Slave_IO_Running: Yes' and 'Slave_SQL_Running: Yes' above." -echo "maxscale.cnf already carries a [replica] server, so on the MaxScale host" -echo "just point REPLICA_HOST in /etc/maxscale.secrets.d/backend.env at DB2" -echo "(it is currently the RFC 5737 placeholder 192.0.2.2), then" -echo "'systemctl restart maxscale' and confirm 'maxctrl list servers' shows" -echo "the replica as Slave, Running." +echo +echo "NEXT, in order — see 'Bringing up DB2' in README.md. Do NOT enable" +echo "auto_failover until semi-sync is confirmed engaged and the write path is" +echo "fenced; promoting an async replica loses acknowledged writes." +echo +echo " 1. Enable semi-sync on the PRIMARY (dynamic, no restart needed):" +echo " SET GLOBAL rpl_semi_sync_master_enabled = ON;" +echo " SET GLOBAL rpl_semi_sync_master_wait_point = AFTER_SYNC;" +echo " SET GLOBAL rpl_semi_sync_master_timeout = 1000;" +echo " SET GLOBAL rpl_semi_sync_master_wait_no_slave = OFF;" +echo " replica.cnf already sets rpl_semi_sync_slave_enabled=ON here, and the" +echo " CHANGE MASTER above reconnected the IO thread, so this node registers" +echo " as a semi-sync client as soon as the primary is enabled." +echo +echo " 2. CONFIRM it engaged, on the primary — do not assume:" +echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- must be ON" +echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1" +echo " Healthy replication with status OFF means it degraded to async." +echo +echo " 3. Fence the write path: DROP USER 'triangle_user'@'10.248.40.168' and" +echo " firewall 3306 to the MaxScale host and the DB peer. Have console" +echo " access open; the DB firewalls are currently wide open." +echo +echo " 4. On the MaxScale host, set REPLICA_HOST in" +echo " /etc/maxscale.secrets.d/backend.env to this node (it is the RFC 5737" +echo " placeholder 192.0.2.2 until then), grant the maxscale user" +echo " REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD," +echo " copy up maxscale.cnf, and 'systemctl restart maxscale'." +echo +echo " 5. Confirm 'maxctrl list servers' shows Master, Running and Slave," +echo " Running — then test a real failover before relying on it." diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf index b5528a6..a80fff8 100644 --- a/deploy/maxscale/maxscale.cnf +++ b/deploy/maxscale/maxscale.cnf @@ -45,12 +45,46 @@ servers=primary,replica user=$MAXSCALE_USER password=$MAXSCALE_PASSWORD monitor_interval=2000ms -# Keep replicas read-only automatically. Automated failover is OFF by default: -# promoting a replica on a 2-node async pair risks split-brain/data loss and -# should be a deliberate ops action. See deploy/mariadb/README.md to enable. -enforce_read_only_slaves=true + +# --- Automated failover ------------------------------------------------------ +# ON. This was previously off because promoting a replica on a 2-node ASYNC pair +# loses the un-replicated tail of the write stream and can split-brain. Both of +# those are addressed elsewhere rather than by leaving failover off: +# +# Data loss -> replication is now SEMI-SYNCHRONOUS with wait_point=AFTER_SYNC +# (see deploy/mariadb/primary.cnf). DB1 does not acknowledge a +# commit until DB2 holds the event durably, so a promotion +# cannot discard an acknowledged write. +# Split-brain -> a 2-node cluster has no quorum and therefore cannot VOTE on +# who is alive. Instead the write path is fenced: MaxScale is +# the ONLY route to the databases (Delta's direct +# triangle_user@10.248.40.168 grant is dropped, 3306 is +# firewalled to this host and the DB peer). So "MaxScale cannot +# see DB1" also means "the CMS cannot see DB1", and promoting +# DB2 cannot produce two servers taking application writes. +# Divergence -> gtid_strict_mode=ON on both nodes, so a rejoining old primary +# that diverged is REFUSED by auto_rejoin instead of silently +# corrupting the dataset. +# +# Residual, accepted: while DB2 is down, semi-sync degrades to async +# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-loss guarantee lapses; +# and MaxScale itself remains a single point of failure. Closing either needs a +# third node. See deploy/mariadb/README.md. +auto_failover=true +# Consecutive failed monitor passes before the master is declared down. +# 5 x 2000ms = ~10s of confirmed unreachability, which rides out a container +# migration or a brief network blip without promoting. +failcount=5 +failover_timeout=90000ms +# Rejoin a returning old primary as a replica automatically; gtid_strict_mode +# makes this refuse rather than corrupt if it diverged. auto_rejoin=true -auto_failover=false +# Force every non-primary node read_only, so the demoted server cannot take +# writes even if something reaches it directly. +enforce_read_only_slaves=true +# Two-node primary/replica and nothing else — reject any topology MaxScale did +# not expect rather than trying to reason about it. +enforce_simple_topology=true # --- Service: read/write split router ---------------------------------------- [Read-Write-Split] From d3d9be983609015643b9e14aba1dbac7755a5969 Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 4 Aug 2026 21:35:22 -0400 Subject: [PATCH 2/5] Fence at the network only; do not drop the app's Delta-scoped grant Correcting guidance this branch introduced and I acted on against production. It told you to drop triangle_user@ as "Delta's direct bypass" before enabling failover. That account is not a bypass: MaxScale authenticates a client against the backend user table using the CLIENT's own source address, so triangle_user@ is exactly how the CMS logs in through MaxScale, while triangle_user@ is what lets MaxScale then open the backend connection. Both are required. Dropping it took the site down with Error 1045 on every DB-backed route while /v1/health kept returning 200, so it looked healthy until a page was loaded. Restored by copying the password hash from the surviving maxscale-scoped row. The fencing that makes auto_failover safe is purely the firewall: 3306 on both DB hosts now accepts only MaxScale and the DB peer, so Delta cannot reach the databases directly regardless of what accounts exist. That property is what the split-brain argument rests on. Also documents a second trap found on the live bring-up: setup-replica.sh dumps --databases triangle, which excludes mysql.*, so DB2 came up with none of the application accounts. A failover then promotes a server that nothing can authenticate to. Backfill under SET SESSION sql_log_bin=0 so the accounts stay out of the replication stream, and verify both hosts rather than assuming grants replicated. Co-Authored-By: Claude Opus 5 --- deploy/mariadb/README.md | 81 ++++++++++++++++++++++----------- deploy/mariadb/setup-replica.sh | 19 ++++++-- deploy/maxscale/maxscale.cnf | 17 ++++--- 3 files changed, 80 insertions(+), 37 deletions(-) diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index 050efe0..81d1b5e 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -91,24 +91,29 @@ never left the boxes. | Account | Grants | Why | | --- | --- | --- | -| `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | the app, via MaxScale | +| `triangle_user@10.248.40.168` | `ALL PRIVILEGES ON triangle.*` | **client-side auth**: lets Delta log in *through* MaxScale | +| `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | **backend-side auth**: lets MaxScale open the backend connection | | `maxscale@10.248.40.183` | monitor, account reads, **+ failover admin** | promotes/demotes on failover | | `repl@10.248.40.155` | `REPLICATION SLAVE` | DB2's replication link | +| `repl@10.248.40.154` | `REPLICATION SLAVE` | reverse link, for `auto_rejoin` after failover | `ALL PRIVILEGES` rather than DML-only because the CMS runs additive DDL (`ADD COLUMN IF NOT EXISTS`) at startup. -**`triangle_user@10.248.40.168` (Delta direct) must be dropped**, and is listed -here only so it is not recreated. It was a cutover-verification account that let -Delta bypass MaxScale. Leaving it in place defeats the fencing that makes -automated failover safe: it is a second write path to DB1 that survives MaxScale -deciding DB1 is dead. See *Why automated failover is safe here* below. - -The same account set must exist on **DB2**, because on promotion DB2 serves the -application. `repl@10.248.40.154` is needed there too so that the old primary -can replicate back after `auto_rejoin`. Grants replicate automatically once -replication is running (they are DDL on `mysql.*`), so creating them on DB1 -after DB2 is attached is sufficient — but verify rather than assume. +**Both `triangle_user` rows are required — do not "clean up" the Delta-scoped +one.** MaxScale authenticates a client against the backend user table by the +**client's own source address**, then connects to the backend from its own. Drop +either and the CMS gets `Error 1045`. See the warning in step 6. + +**Every one of these must exist on DB2 as well**, because on promotion DB2 serves +the application and MaxScale re-authenticates everything against *its* user +table. **`setup-replica.sh` does NOT copy them** — it dumps `--databases +triangle`, which excludes `mysql.*` — so accounts created *before* replication +started are absent on DB2, and only those created *after* replicate. Verify with +`SELECT CONCAT(user,'@',host) FROM mysql.user` on both hosts; a mismatch here +means a "successful" failover promotes a server nothing can log in to. To backfill +without polluting the replication stream, apply them on DB2 under +`SET SESSION sql_log_bin=0`. ## Bringing up DB2 @@ -195,17 +200,36 @@ match the other two hosts, or run steps 1–3 from the Proxmox console. it silently degraded to async — the zero-loss guarantee is not in force and failover is not yet safe to enable. -6. **Fence the write path.** This is what removes the split-brain risk, and it - is a prerequisite for step 8, not an optional hardening pass: +6. **Fence the write path — at the network, and ONLY at the network.** This is + what removes the split-brain risk, and it is a prerequisite for step 8, not + an optional hardening pass. Restrict 3306 on both DB hosts to MaxScale and + the DB peer: - ```sql - DROP USER 'triangle_user'@'10.248.40.168'; -- Delta's direct bypass + ```sh + ufw allow 22/tcp # BEFORE enabling + ufw allow from 10.248.40.183 to any port 3306 proto tcp # MaxScale + ufw allow from to any port 3306 proto tcp # replication + ufw --force enable ``` - and restrict 3306 on both DB hosts to MaxScale and the DB peer. **Do this - with Proxmox console access open** — DB1's firewall is currently wide open - (ufw inactive, iptables ACCEPT) and it was left that way deliberately to - avoid an SSH lockout on a remote host. + Add the `allow` rules **before** `enable`, and do it **with Proxmox console + access open** — `pct enter ` gets you back in if you cut yourself off. + Do DB2 first: a mistake there costs replication, a mistake on DB1 costs the + site. + + > ⚠️ **Do NOT "fence" this by dropping `triangle_user@10.248.40.168`.** + > It looks like a direct-bypass account and it is not. **MaxScale + > authenticates a client against the backend's user table using the + > CLIENT's own source address**, so `triangle_user@` is precisely + > what lets the CMS log in *through* MaxScale; `triangle_user@` + > is what lets MaxScale then open the backend connection. **Both are + > required.** Dropping the Delta-scoped one closes no bypass and takes the + > site down with `Error 1045 Access denied for user + > 'triangle_user'@'10.248.40.168'` on any DB-backed route, while + > `/v1/health` keeps returning 200 — so it looks fine until someone loads a + > page. This was done and reverted on 2026-08-05. The firewall above is what + > actually removes the bypass, because Delta is no longer permitted to reach + > 3306 at all. 7. **Grant MaxScale the failover privileges** it does not currently hold, on DB1: @@ -258,12 +282,17 @@ window that loses data. **"Two nodes can't tell 'primary is dead' from 'I can't see the primary'."** Also true, and unfixable at that layer: a 2-node cluster has no quorum, so it cannot vote. The risk is removed structurally instead, by making MaxScale the -**only** path to the databases — Delta's direct `triangle_user` grant is -dropped and 3306 is firewalled to MaxScale and the DB peer. Once that holds, -"MaxScale cannot reach DB1" implies "the CMS cannot reach DB1", so promoting -DB2 cannot result in two servers taking application writes. The partition that -would split-brain a quorum-less cluster instead just moves all traffic to the -promoted node, which is the desired outcome. +**only** path to the databases — 3306 on both DB hosts is firewalled to +MaxScale and the DB peer, so nothing else can open a connection at all. Once +that holds, "MaxScale cannot reach DB1" implies "the CMS cannot reach DB1", so +promoting DB2 cannot result in two servers taking application writes. The +partition that would split-brain a quorum-less cluster instead just moves all +traffic to the promoted node, which is the desired outcome. + +Note this fencing is **purely a network property**. It is tempting to also +revoke the app's Delta-scoped grant as "a second write path", but that account +is not a bypass — it is how MaxScale authenticates the client — and removing it +only breaks the site. See step 6. **"The old primary comes back and clobbers things."** `gtid_strict_mode=ON` on both nodes. A returning DB1 that diverged is *refused* by `auto_rejoin` and diff --git a/deploy/mariadb/setup-replica.sh b/deploy/mariadb/setup-replica.sh index f9de510..cb868a2 100755 --- a/deploy/mariadb/setup-replica.sh +++ b/deploy/mariadb/setup-replica.sh @@ -118,15 +118,24 @@ echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- must be ON" echo " SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1" echo " Healthy replication with status OFF means it degraded to async." echo -echo " 3. Fence the write path: DROP USER 'triangle_user'@'10.248.40.168' and" -echo " firewall 3306 to the MaxScale host and the DB peer. Have console" -echo " access open; the DB firewalls are currently wide open." +echo " 3. Copy the app/monitor/repl accounts to THIS host. The dump above is" +echo " --databases triangle, which EXCLUDES mysql.*, so any account created" +echo " before replication started is missing here and a promotion would" +echo " yield a server nothing can log in to. Apply them under" +echo " SET SESSION sql_log_bin=0 so they stay out of the binlog." echo -echo " 4. On the MaxScale host, set REPLICA_HOST in" +echo " 4. Fence the write path AT THE NETWORK: firewall 3306 to the MaxScale" +echo " host and the DB peer, adding 'ufw allow 22/tcp' BEFORE enabling." +echo " Do NOT drop triangle_user@ to 'close a bypass' — MaxScale" +echo " authenticates clients by their own source address, so that account" +echo " is how the CMS logs in through MaxScale. Dropping it just breaks" +echo " the site. Have console access open." +echo +echo " 5. On the MaxScale host, set REPLICA_HOST in" echo " /etc/maxscale.secrets.d/backend.env to this node (it is the RFC 5737" echo " placeholder 192.0.2.2 until then), grant the maxscale user" echo " REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD," echo " copy up maxscale.cnf, and 'systemctl restart maxscale'." echo -echo " 5. Confirm 'maxctrl list servers' shows Master, Running and Slave," +echo " 6. Confirm 'maxctrl list servers' shows Master, Running and Slave," echo " Running — then test a real failover before relying on it." diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf index a80fff8..32398ab 100644 --- a/deploy/maxscale/maxscale.cnf +++ b/deploy/maxscale/maxscale.cnf @@ -56,12 +56,17 @@ monitor_interval=2000ms # commit until DB2 holds the event durably, so a promotion # cannot discard an acknowledged write. # Split-brain -> a 2-node cluster has no quorum and therefore cannot VOTE on -# who is alive. Instead the write path is fenced: MaxScale is -# the ONLY route to the databases (Delta's direct -# triangle_user@10.248.40.168 grant is dropped, 3306 is -# firewalled to this host and the DB peer). So "MaxScale cannot -# see DB1" also means "the CMS cannot see DB1", and promoting -# DB2 cannot produce two servers taking application writes. +# who is alive. Instead the write path is fenced AT THE NETWORK: +# 3306 on both DB hosts is firewalled to this host and the DB +# peer only, so MaxScale is the ONLY route to the databases. +# "MaxScale cannot see DB1" therefore also means "the CMS cannot +# see DB1", and promoting DB2 cannot produce two servers taking +# application writes. +# Do NOT try to fence this by dropping triangle_user@: +# MaxScale authenticates a client against the backend user table +# using the CLIENT's own source address, so that account is what +# lets Delta log in THROUGH MaxScale. Dropping it does not close +# a bypass, it just breaks the site. See deploy/mariadb/README.md. # Divergence -> gtid_strict_mode=ON on both nodes, so a rejoining old primary # that diverged is REFUSED by auto_rejoin instead of silently # corrupting the dataset. From 4e3aeba174ae2d9f14a0e99f30838e33de4d897e Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 4 Aug 2026 21:46:27 -0400 Subject: [PATCH 3/5] Fix the two failures a real failover test exposed Ran an actual failover against production: stopped DB1, MaxScale promoted DB2 in under 10s, the CMS kept serving and writing against the promoted node. Coming back was where it broke, twice. auto_rejoin looped forever on "Failed to prepare (demote) standalone server for rejoin". The monitor user was missing BINLOG ADMIN, which MariaDB 10.5+ split out of SUPER, so mariadbmon could not run SET @@session.sql_log_bin=0 while demoting the returning primary. The grant list in this README predated that split. With that fixed the rejoin succeeded and then reverted ~2s later (new_slave -> lost_slave). replication_user/replication_password default to the MONITOR user, so mariadbmon built the link as maxscale@, which does not exist -- the monitor account is host-scoped to the MaxScale host. That is what the repl accounts are for, and nothing was pointing MaxScale at them. Now set explicitly. MaxScale surfaces this only as lost_slave; the real 1045 is on the rejoining node. Also enables semi-sync on BOTH sides on BOTH nodes. DB2 previously had only the slave side, so a promotion silently dropped to asynchronous replication -- losing the zero-data-loss guarantee at precisely the moment it had just been needed. Caught before the test, and confirmed during it: the promoted DB2 held Rpl_semi_sync_master_status=ON with one client. Verified end state: DB1 Master, DB2 Slave, equal GTID, 0 lag, semi-sync ON with 1 client, CMS health and homepage both 200. Co-Authored-By: Claude Opus 5 --- deploy/mariadb/README.md | 56 ++++++++++++++++++++++++++++++++---- deploy/mariadb/primary.cnf | 5 ++++ deploy/mariadb/replica.cnf | 16 +++++++++-- deploy/maxscale/maxscale.cnf | 11 +++++++ 4 files changed, 80 insertions(+), 8 deletions(-) diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index 81d1b5e..56bd39b 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -85,8 +85,14 @@ never left the boxes. MARIADB_PORT=3306 MAXSCALE_USER=maxscale MAXSCALE_PASSWORD=... + REPL_USER=repl + REPL_PASSWORD=... # what mariadbmon writes into CHANGE MASTER ``` + All six are required. Removing any one leaves an unsubstituted `$VAR` in the + config and MaxScale will not start — worth remembering when editing this file + with `sed`, where a range delete can silently take an adjacent line with it. + ## Accounts on DB1 | Account | Grants | Why | @@ -231,14 +237,30 @@ match the other two hosts, or run steps 1–3 from the Proxmox console. > actually removes the bypass, because Delta is no longer permitted to reach > 3306 at all. -7. **Grant MaxScale the failover privileges** it does not currently hold, on - DB1: +7. **Grant MaxScale the failover privileges**, on **both** hosts: ```sql - GRANT REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD + GRANT REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD, + BINLOG ADMIN, CONNECTION ADMIN, REPLICATION MASTER ADMIN, + READ_ONLY ADMIN, SHOW DATABASES ON *.* TO 'maxscale'@'10.248.40.183'; ``` + **`BINLOG ADMIN` is the one that is easy to miss and it breaks `auto_rejoin` + outright.** MariaDB 10.5+ split the old catch-all `SUPER` into discrete + privileges, so holding `SUPER` no longer implies it. Without it mariadbmon + cannot run `SET @@session.sql_log_bin=0` while demoting a returning primary + and loops forever on: + + ``` + Failed to prepare (demote) standalone server 'primary' for rejoin. + ``` + + Apply it on the current replica under `SET SESSION sql_log_bin=0` as well — + a grant made only on the primary reaches the replica by replication, but the + node that needs it during a rejoin is the one that is *not* currently + replicating. + 8. **Point MaxScale at DB2 and enable failover.** Set `REPLICA_HOST=10.248.40.155` in `/etc/maxscale.secrets.d/backend.env` (it is the RFC 5737 placeholder `192.0.2.2` until this is done), copy up the @@ -339,9 +361,31 @@ Stop MariaDB on DB1 (`systemctl stop mariadb`), then watch MaxScale's log at should show DB2 as `Master, Running`. Confirm the CMS still serves and can write. Then start DB1 again and confirm `auto_rejoin` brings it back as -`Slave, Running`. **If it does not rejoin, that is the safety net working, not a -bug** — check for divergence before forcing anything. Finish by switching back -with the `switchover` command above so DB1 is primary again. +`Slave, Running`. **If it does not rejoin, that is often the safety net working +rather than a bug** — compare `@@gtid_current_pos` on both before forcing +anything. A returning node that is merely *behind* (its GTID is a prefix of the +new primary's) is cleanly rejoinable; one that is genuinely diverged is refused +by `gtid_strict_mode`, and that refusal is correct. + +Two non-divergence failures seen on the first real test, both worth recognising: + +- **`Failed to prepare (demote) standalone server for rejoin`, repeating every + monitor tick** — the monitor user is missing `BINLOG ADMIN`. See step 7. +- **Rejoin appears to succeed then instantly reverts** (`new_slave` followed by + `lost_slave` about two seconds later). The monitor built the replication link + with the wrong credentials; `replication_user`/`replication_password` default + to the *monitor* user, which is host-scoped to the MaxScale host and so does + not exist from the rejoining node. `maxscale.cnf` now sets them to `repl` + explicitly. MaxScale reports this only as `lost_slave` — the real error is on + the rejoining node, in `SHOW SLAVE STATUS` `Last_IO_Error` (1045). + **After fixing it, clear the stale connection** with + `STOP SLAVE; RESET SLAVE ALL;` on the rejoining node: while a replica + connection exists the node is no longer "standalone", so the monitor will not + rebuild it and simply leaves it broken. + +Finish by switching back with the `switchover` command above so DB1 is primary +again, and confirm semi-sync re-engages (`Rpl_semi_sync_master_status = ON`, +`clients = 1`) on whichever node ends up primary. ### Manual failover, without MaxScale diff --git a/deploy/mariadb/primary.cnf b/deploy/mariadb/primary.cnf index 2893f80..854b42e 100644 --- a/deploy/mariadb/primary.cnf +++ b/deploy/mariadb/primary.cnf @@ -96,3 +96,8 @@ rpl_semi_sync_master_timeout = 1000 # down, the zero-loss guarantee is NOT in force. Alert on # Rpl_semi_sync_master_status = OFF rather than assuming it holds. rpl_semi_sync_master_wait_no_slave = OFF +# Slave side, inert while this node is the primary. Present so that after a +# failover and auto_rejoin — when this node comes back as a REPLICA of DB2 — it +# registers as a semi-sync client instead of silently leaving the new primary +# running asynchronously. Mirrors replica.cnf; both nodes carry both roles. +rpl_semi_sync_slave_enabled = ON diff --git a/deploy/mariadb/replica.cnf b/deploy/mariadb/replica.cnf index 6f64ca3..377df7a 100644 --- a/deploy/mariadb/replica.cnf +++ b/deploy/mariadb/replica.cnf @@ -79,9 +79,21 @@ binlog_expire_logs_seconds = 604800 slave_parallel_threads = 4 slave_parallel_mode = optimistic -# --- Semi-synchronous replication: SLAVE side --------------------------------- -# The replica acknowledges each binlog event back to the primary, which is what +# --- Semi-synchronous replication --------------------------------------------- +# BOTH sides are enabled on BOTH nodes, because either node can hold either role +# after a failover. MariaDB only acts on the side matching its current role, so +# the master settings sit inert here until this node is promoted. +# +# Slave side: acknowledges each binlog event back to the primary, which is what # lets automated failover promote DB2 without losing acknowledged commits. Must # be ON here for the primary's AFTER_SYNC wait to ever be satisfied — if this is # OFF, the primary just times out and degrades to async on every commit. rpl_semi_sync_slave_enabled = ON +# Master side: inert while this node is a replica, load-bearing the moment it is +# promoted. Without it a failover silently drops to asynchronous replication — +# losing the zero-data-loss guarantee at exactly the moment you have just proven +# you need it. Values must match primary.cnf; see the rationale there. +rpl_semi_sync_master_enabled = ON +rpl_semi_sync_master_wait_point = AFTER_SYNC +rpl_semi_sync_master_timeout = 1000 +rpl_semi_sync_master_wait_no_slave = OFF diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf index 32398ab..4bfb638 100644 --- a/deploy/maxscale/maxscale.cnf +++ b/deploy/maxscale/maxscale.cnf @@ -44,6 +44,17 @@ module=mariadbmon servers=primary,replica user=$MAXSCALE_USER password=$MAXSCALE_PASSWORD +# Credentials mariadbmon writes into CHANGE MASTER when it promotes, demotes or +# rejoins a node. MUST be set explicitly: when omitted these DEFAULT TO THE +# MONITOR USER above, and mariadbmon then builds the replication link as +# maxscale@ — an account that does not exist, because +# the monitor user is host-scoped to the MaxScale host. The symptom is a rejoin +# that appears to succeed and then immediately flips back +# ("new_slave" followed by "lost_slave" ~2s later), with the real cause only +# visible on the rejoining node as SHOW SLAVE STATUS Last_IO_Error 1045. +# `repl` is host-scoped to BOTH DB hosts precisely so either direction works. +replication_user=$REPL_USER +replication_password=$REPL_PASSWORD monitor_interval=2000ms # --- Automated failover ------------------------------------------------------ From 2a610fc9302805ab6717812b51460df931ac6066 Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 4 Aug 2026 22:01:24 -0400 Subject: [PATCH 4/5] Alert to Slack on failover, via the monitor's own script hook Adds deploy/maxscale/maxscale-alert.sh, wired in as mariadbmon's script=. It fires within one monitor tick rather than waiting for a scrape, and runs on the MaxScale host, so it does not depend on Delta or the observability stack being up. It always appends to /var/log/maxscale/failover-events.log and posts to Slack only if /etc/maxscale.secrets.d/alert.env supplies a webhook, so it is safe to install before the webhook exists -- which is how it is deployed right now, log-only pending the URL. The webhook stays out of maxscale.cnf, which is world-readable 0644. Two things found by testing it rather than reading the docs: slave_up was missing from events=. It is NOT the same transition as new_slave -- new_slave is [Running]->[Slave,Running], slave_up is [Down]->[Slave,Running] -- so a replica outage alerted on the way down and went silent on recovery. This fails silently in the worst way: the only symptom is an alert that never arrives. Rpl_semi_sync_master_status is NOT a usable health signal, and the guidance here previously said to alert on it. Stopping DB2 showed status stay ON through seven unacknowledged commits while clients sat at 0 and no_tx climbed 0->7. With wait_no_slave=OFF the master never enters the state that would flip it. Alert on Rpl_semi_sync_master_clients == 0. Verified end to end by stopping and starting DB2: both slave_down and slave_up reached the log, semi-sync re-engaged with clients=1, and the CMS stayed 200 throughout. Not covered, and stated in the README: the script cannot report that MaxScale itself died, which -- given the write path is fenced to MaxScale -- is a total outage. That needs an external check. Co-Authored-By: Claude Opus 5 --- deploy/mariadb/README.md | 62 ++++++++++++++++--- deploy/mariadb/primary.cnf | 12 +++- deploy/maxscale/maxscale-alert.sh | 99 +++++++++++++++++++++++++++++++ deploy/maxscale/maxscale.cnf | 22 +++++++ 4 files changed, 184 insertions(+), 11 deletions(-) create mode 100755 deploy/maxscale/maxscale-alert.sh diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index 56bd39b..427161b 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -197,14 +197,18 @@ match the other two hosts, or run steps 1–3 from the Proxmox console. Then on DB1, **verify rather than assume**: ```sql - SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- must be ON - SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1 - SHOW STATUS LIKE 'Rpl_semi_sync_master_no_tx'; -- should stop climbing + SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1 <- the real check + SHOW STATUS LIKE 'Rpl_semi_sync_master_yes_tx'; -- rises on acknowledged commits + SHOW STATUS LIKE 'Rpl_semi_sync_master_no_tx'; -- must NOT keep climbing + SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- ON, but see below ``` - `Rpl_semi_sync_master_status = OFF` with replication otherwise healthy means - it silently degraded to async — the zero-loss guarantee is not in force and - failover is not yet safe to enable. + **`clients` is the signal, not `status`.** With + `rpl_semi_sync_master_wait_no_slave=OFF`, `status` stays `ON` even while + nothing is acknowledging — verified by stopping DB2 on 2026-08-05: seven + commits completed unacknowledged (`no_tx` 0 → 7) with `status` still `ON` + and `clients` at 0. So `clients = 1` plus `yes_tx` rising is what shows the + guarantee is actually in force; `status = ON` on its own proves nothing. 6. **Fence the write path — at the network, and ONLY at the network.** This is what removes the split-brain risk, and it is a prerequisite for step 8, not @@ -329,7 +333,10 @@ Honest limits, all of which need a third node to close: the alternative is DB1 stalling every commit for `rpl_semi_sync_master_timeout` whenever DB2 is offline, turning a replica outage into a site outage. So DB2-down-then-DB1-dies can still lose writes. **Alert on - `Rpl_semi_sync_master_status`** rather than assuming the guarantee holds. + `Rpl_semi_sync_master_clients == 0`, NOT on `Rpl_semi_sync_master_status`** — + with `wait_no_slave=OFF` the status stays `ON` while commits go + unacknowledged, so it is not evidence the guarantee holds. A `slave_down` / + `lost_slave` Slack alert (see Alerting) covers the same condition. - **MaxScale is a single point of failure** and the sole arbiter. If it dies, the CMS is down regardless of how healthy both databases are. Fencing the write path to MaxScale deepens this dependency — that is the price of removing @@ -384,8 +391,45 @@ Two non-divergence failures seen on the first real test, both worth recognising: rebuild it and simply leaves it broken. Finish by switching back with the `switchover` command above so DB1 is primary -again, and confirm semi-sync re-engages (`Rpl_semi_sync_master_status = ON`, -`clients = 1`) on whichever node ends up primary. +again, and confirm semi-sync re-engages (`Rpl_semi_sync_master_clients = 1`) +on whichever node ends up primary. + +## Alerting + +`maxscale.cnf` sets `script=` on the monitor, so mariadbmon runs +[../maxscale/maxscale-alert.sh](../maxscale/maxscale-alert.sh) (installed as +`/usr/local/bin/maxscale-alert.sh`, 0755) as the `maxscale` user on each event +in `events=`. It fires within one monitor tick, rather than waiting for a +scrape, and it does not depend on Delta or the observability stack being up. + +It **always** appends to `/var/log/maxscale/failover-events.log` and posts to +Slack only if `/etc/maxscale.secrets.d/alert.env` (0640 `root:maxscale`) +supplies a webhook: + +``` +SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... +``` + +With that empty it degrades to log-only, so it is safe to install first. The +webhook must not go in `maxscale.cnf`, which is world-readable 0644. + +**Event names are not intuitive, and getting them wrong fails silently — you +only notice by the alert that never arrives.** In particular `new_slave` +(`[Running]→[Slave,Running]`, a standalone node joining) and `slave_up` +(`[Down]→[Slave,Running]`, a node returning from an outage) are different +transitions; listing only the former alerts on the way down and stays silent on +recovery. Confirm what actually fired with: + +```sh +grep "changed state" /var/log/maxscale/maxscale.log +``` + +### What it does not cover + +The script can only report events MaxScale is alive to observe, so **it cannot +tell you MaxScale itself has died** — and because the write path is fenced to +MaxScale, that is a total outage. Closing this needs an external check +(Delta's Prometheus can reach the host); it is not built yet. ### Manual failover, without MaxScale diff --git a/deploy/mariadb/primary.cnf b/deploy/mariadb/primary.cnf index 854b42e..1b2f620 100644 --- a/deploy/mariadb/primary.cnf +++ b/deploy/mariadb/primary.cnf @@ -93,8 +93,16 @@ rpl_semi_sync_master_timeout = 1000 # maintenance would add a second of latency to every write. OFF drops straight # to async the moment there is no semi-sync replica, and back to semi-sync when # DB2 reconnects. The tradeoff is real and must be understood: while DB2 is -# down, the zero-loss guarantee is NOT in force. Alert on -# Rpl_semi_sync_master_status = OFF rather than assuming it holds. +# down, the zero-loss guarantee is NOT in force. +# +# MONITOR THIS WITH Rpl_semi_sync_master_clients, NOT ..._status. Verified by +# stopping DB2 on 2026-08-05: status stayed **ON** through seven unacknowledged +# commits while clients sat at 0 and no_tx climbed 0 -> 7. With +# wait_no_slave=OFF the master never enters the "off" state it would otherwise +# fall into, so status is not evidence the guarantee is holding. +# Rpl_semi_sync_master_clients == 0 -> nothing is acknowledging: ALERT +# Rpl_semi_sync_master_no_tx rising -> commits completing unacknowledged +# Rpl_semi_sync_master_yes_tx rising -> the guarantee is actually in force rpl_semi_sync_master_wait_no_slave = OFF # Slave side, inert while this node is the primary. Present so that after a # failover and auto_rejoin — when this node comes back as a REPLICA of DB2 — it diff --git a/deploy/maxscale/maxscale-alert.sh b/deploy/maxscale/maxscale-alert.sh new file mode 100755 index 0000000..4bed0e4 --- /dev/null +++ b/deploy/maxscale/maxscale-alert.sh @@ -0,0 +1,99 @@ +#!/bin/sh +# Alert hook for MaxScale's mariadbmon, invoked via `script=` in maxscale.cnf. +# Installed on THETRIANGLE-MAXSCALE as /usr/local/bin/maxscale-alert.sh (0755). +# +# MaxScale runs this as the `maxscale` user on every event listed in `events=`, +# substituting $EVENT/$INITIATOR/$NODELIST/$PARENT before exec. It is bounded by +# script_timeout (90s) — if it hangs, the monitor blocks, so every outbound call +# here MUST have its own timeout. +# +# Design notes: +# - It ALWAYS writes the local log first and posts to Slack second. A failover +# that happened is a fact worth keeping even if Slack is unreachable, and +# the log is what you correlate against maxscale.log afterwards. +# - It exits 0 unconditionally. A non-zero exit here is noise in maxscale.log +# and there is nothing MaxScale can usefully do about a failed notification. +# - Until SLACK_WEBHOOK_URL is configured it degrades to log-only rather than +# failing, so it is safe to install before the webhook exists. +# +# The webhook lives in /etc/maxscale.secrets.d/alert.env (0640 root:maxscale), +# NOT here and NOT in maxscale.cnf, which is world-readable 0644: +# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... +set -u + +EVENT="${1:-unknown}" +INITIATOR="${2:-unknown}" +NODELIST="${3:-}" +PARENT="${4:-}" + +TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LOG=/var/log/maxscale/failover-events.log + +# --- Always record locally, before anything that can fail -------------------- +printf '%s event=%s initiator=%s nodes=%s parent=%s\n' \ + "$TS" "$EVENT" "$INITIATOR" "$NODELIST" "$PARENT" >> "$LOG" 2>/dev/null + +# --- Classify ----------------------------------------------------------------- +# new_master is the one that means an automated failover actually promoted a +# node. lost_slave/slave_down matter more than they look: with only two nodes, +# losing the replica means semi-sync degrades to async +# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-data-loss guarantee +# stops holding until it returns. +case "$EVENT" in + new_master) + ICON=":rotating_light:"; SEV="FAILOVER" + NOTE="A node was promoted. The old primary needs checking before it is trusted again." ;; + master_down|lost_master) + ICON=":rotating_light:"; SEV="PRIMARY DOWN" + NOTE="Writes are failing until a promotion completes." ;; + slave_down|lost_slave) + ICON=":warning:"; SEV="REPLICA DOWN" + NOTE="Semi-sync has degraded to async: writes are no longer guaranteed durable on two nodes." ;; + master_up|slave_up|new_slave|server_up) + # slave_up ([Down]->[Slave,Running]) is the recovery counterpart of + # slave_down; new_slave ([Running]->[Slave,Running]) is a standalone node + # joining. Both belong here or an outage never reports that it ended. + ICON=":white_check_mark:"; SEV="RECOVERED" + NOTE="Confirm with 'maxctrl list servers' and check Rpl_semi_sync_master_clients=1 on the primary." ;; + server_down) + ICON=":warning:"; SEV="SERVER DOWN" + NOTE="A backend stopped responding to the monitor." ;; + *) + ICON=":information_source:"; SEV="EVENT" + NOTE="" ;; +esac + +# --- Post to Slack, if configured -------------------------------------------- +[ -r /etc/maxscale.secrets.d/alert.env ] && . /etc/maxscale.secrets.d/alert.env +[ -n "${SLACK_WEBHOOK_URL:-}" ] || exit 0 + +# Build the message with RAW values, then escape exactly once when it becomes +# JSON. Escaping the fields individually and then escaping the whole string +# again would double every backslash. +TEXT="${ICON} *MaxScale ${SEV}* — \`${EVENT}\` +*initiator:* ${INITIATOR} +*nodes:* ${NODELIST:-n/a} +*parent:* ${PARENT:-n/a} +*time:* ${TS} +${NOTE}" + +# JSON string escaping: backslash and quote, then fold the real newlines into +# \n. Literal newlines inside a JSON string are invalid and Slack rejects the +# payload outright, so the multi-line message above must be collapsed here. +# The ':a;N;$!ba' idiom slurps the whole input before substituting. +json_esc() { + printf '%s' "$1" \ + | sed 's/\\/\\\\/g; s/"/\\"/g' \ + | sed ':a;N;$!ba;s/\n/\\n/g' +} + +# --max-time well under script_timeout so a slow Slack cannot stall the monitor. +# Output is discarded: an error response body can echo the URL back. +curl -sS -X POST \ + --max-time 10 \ + -H 'Content-Type: application/json' \ + --data "{\"text\":\"$(json_esc "$TEXT")\"}" \ + "$SLACK_WEBHOOK_URL" >/dev/null 2>&1 \ + || printf '%s event=%s slack_post_failed\n' "$TS" "$EVENT" >> "$LOG" 2>/dev/null + +exit 0 diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf index 4bfb638..f638d84 100644 --- a/deploy/maxscale/maxscale.cnf +++ b/deploy/maxscale/maxscale.cnf @@ -57,6 +57,28 @@ replication_user=$REPL_USER replication_password=$REPL_PASSWORD monitor_interval=2000ms +# --- Alerting ---------------------------------------------------------------- +# Fired by the monitor on each event below, as the `maxscale` user. See +# deploy/maxscale/maxscale-alert.sh — it always writes +# /var/log/maxscale/failover-events.log and posts to Slack only if +# /etc/maxscale.secrets.d/alert.env supplies a webhook, so it is safe to install +# before that exists. The script must stay well inside script_timeout or it +# blocks the monitor; the curl inside it is capped at 10s. +# +# NOTE this can only report what MaxScale is alive to observe — it cannot tell +# you MaxScale itself has died. That needs an external check. +script=/usr/local/bin/maxscale-alert.sh $EVENT $INITIATOR $NODELIST $PARENT +script_timeout=90000ms +# Explicit rather than the default "all": these are the transitions worth waking +# someone for. slave_down/lost_slave are included deliberately — on a two-node +# pair, losing the replica silently degrades semi-sync to async, so it is the +# moment the zero-data-loss guarantee stops holding. +# `slave_up` and `new_slave` are BOTH needed and are not the same transition: +# new_slave is [Running]->[Slave,Running] (a standalone node joining), slave_up +# is [Down]->[Slave,Running] (a node coming back from an outage). Listing only +# new_slave means an outage alerts on the way down and stays silent on recovery. +events=master_down,master_up,lost_master,new_master,slave_down,slave_up,lost_slave,new_slave,server_down,server_up + # --- Automated failover ------------------------------------------------------ # ON. This was previously off because promoting a replica on a 2-node ASYNC pair # loses the un-replicated tail of the write stream and can split-brain. Both of From 6659ecd35e6ae2ab50b819cba0120af7355849e2 Mon Sep 17 00:00:00 2001 From: ssavutu Date: Tue, 4 Aug 2026 22:18:31 -0400 Subject: [PATCH 5/5] Watch MaxScale from outside, since it cannot report its own death The failover alert script runs on the MaxScale host, so the one outage it can never report is MaxScale itself stopping -- and because the write path is fenced so MaxScale is the only route to the databases, that is a total outage, not a degraded read path. Covers it from Delta: blackbox_exporter TCP-connects to 10.248.40.183:4006, Prometheus scrapes the probe, and a provisioned Grafana rule alerts to Slack when probe_success == 0 for 1m. Probing :4006 rather than the admin API is deliberate. 4006 is the port the CMS actually uses, so this tests the real dependency instead of a management interface that could be healthy while routing is not; and 8989 can reconfigure MaxScale, so opening it to Delta would be a far worse thing to expose. MaxScale 24.02 serves no Prometheus endpoint in any case (/metrics and /v1/metrics both 404), so scraping it directly was never available. DB1/DB2 are not probed: their 3306 is firewalled to the MaxScale host and the DB peer, so Delta cannot reach them by design and the target would alert forever. noDataState is Alerting on purpose -- a missing probe series means nobody is watching the database tier, which is worth waking someone for even though the cause is Prometheus rather than MaxScale. The Slack webhook comes from SLACK_WEBHOOK_URL and defaults to a non-functional placeholder, because Grafana provisioning rejects an empty URL and a missing webhook must not stop the stack from starting. Until it is set the rule still fires in Grafana and delivery fails in the log, which is visible rather than silent. Verified end to end by pointing the probe at a closed port: the rule went pending -> firing and Grafana routed it to receiver=slack-triangle, failing delivery only on the placeholder URL. No CMS outage was needed to test it. Co-Authored-By: Claude Opus 5 --- deploy/compose.observability.yml | 31 ++++++ deploy/mariadb/README.md | 63 ++++++++++- observability/blackbox/blackbox.yml | 33 ++++++ .../provisioning/alerting/maxscale.yml | 102 ++++++++++++++++++ observability/prometheus/prometheus.delta.yml | 29 +++++ 5 files changed, 253 insertions(+), 5 deletions(-) create mode 100644 observability/blackbox/blackbox.yml create mode 100644 observability/grafana/provisioning/alerting/maxscale.yml diff --git a/deploy/compose.observability.yml b/deploy/compose.observability.yml index ce57b79..ce07169 100644 --- a/deploy/compose.observability.yml +++ b/deploy/compose.observability.yml @@ -76,6 +76,14 @@ services: environment: GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER is required} GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD is required} + # Slack destination for the provisioned MaxScale-down alert. Deliberately + # NOT `:?required` -- the alert rule and its Grafana UI state are useful + # on their own, and a missing webhook must not stop the whole stack from + # starting. The default is a syntactically valid but non-functional URL: + # Grafana's provisioning rejects an empty one, so a placeholder is what + # lets this deploy before the webhook exists. Delivery then fails and is + # logged until it is replaced, which is visible rather than silent. + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL:-https://hooks.slack.com/services/REPLACE-ME} ports: - "127.0.0.1:3000:3000" volumes: @@ -84,6 +92,29 @@ services: # datasource or dashboard is a file drop rather than a Compose edit. - ../observability/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro,z - ../observability/grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro,z + - ../observability/grafana/provisioning/alerting:/etc/grafana/provisioning/alerting:ro,z + networks: + - observability_net + + # Probes MaxScale's client listener from Delta. This is the only thing in the + # stack that can report the database tier being unreachable: MaxScale's own + # alert script cannot, because a dead MaxScale sends nothing. See + # ../observability/blackbox/blackbox.yml for why it is a TCP connect on :4006 + # rather than the admin API. + blackbox: + image: prom/blackbox-exporter:v0.25.0 + restart: unless-stopped + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + command: ["--config.file=/etc/blackbox/blackbox.yml"] + volumes: + - ../observability/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro,z + # No ports published: only Prometheus talks to it, over the internal + # network. It probes OUTWARD to the MaxScale host, which needs no inbound + # exposure here. networks: - observability_net diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md index 427161b..70f6870 100644 --- a/deploy/mariadb/README.md +++ b/deploy/mariadb/README.md @@ -424,12 +424,65 @@ recovery. Confirm what actually fired with: grep "changed state" /var/log/maxscale/maxscale.log ``` -### What it does not cover +### The MaxScale-down watchdog -The script can only report events MaxScale is alive to observe, so **it cannot -tell you MaxScale itself has died** — and because the write path is fenced to -MaxScale, that is a total outage. Closing this needs an external check -(Delta's Prometheus can reach the host); it is not built yet. +The script can only report events MaxScale is alive to observe, so it cannot +tell you MaxScale *itself* died — and because the write path is fenced to +MaxScale, that is a total outage. That gap is covered from **outside** the +database tier, by the observability stack on Delta: + +``` +blackbox_exporter --TCP connect--> 10.248.40.183:4006 + ^ | + | scrape v + Prometheus ----> Grafana alert ----> Slack + probe_success == 0 for 1m +``` + +- [../../observability/blackbox/blackbox.yml](../../observability/blackbox/blackbox.yml) + — a plain TCP connect, no MySQL login, so no credentials are needed. +- [../../observability/prometheus/prometheus.delta.yml](../../observability/prometheus/prometheus.delta.yml) + — job `blackbox-tcp`, with the usual exporter relabel indirection. +- [../../observability/grafana/provisioning/alerting/maxscale.yml](../../observability/grafana/provisioning/alerting/maxscale.yml) + — rule `maxscale-unreachable` plus the `slack-triangle` contact point. + +**Why `:4006` and not the admin API:** 4006 is the port the CMS actually uses, +so it tests the real dependency; the admin API (8989) would have to be opened to +Delta and it can reconfigure MaxScale, which is a much worse thing to expose. +MaxScale 24.02 serves no Prometheus endpoint anyway (`/metrics` and +`/v1/metrics` both 404). + +**DB1/DB2 are deliberately not probed** — their 3306 is firewalled to the +MaxScale host and the DB peer, so Delta cannot reach them by design and such a +target would alert forever. + +`noDataState: Alerting` is intentional: if the probe series disappears, nobody +is watching the database tier, which is worth waking someone for even though +the cause is Prometheus rather than MaxScale. + +> **The Slack webhook is supplied by `SLACK_WEBHOOK_URL` in +> `observability.env`**, and defaults to a non-functional placeholder so the +> stack still starts without it (Grafana's provisioning rejects an empty URL). +> Until it is set, the alert fires correctly in Grafana and delivery fails in +> the Grafana log — visible, not silent. Note this is a **second** place the +> webhook is needed: the MaxScale host has its own copy in +> `/etc/maxscale.secrets.d/alert.env`, because the two alert paths run on +> different machines by design. + +**Testing it by changing the probe target leaves a stale series behind.** The +old `instance` keeps its last value inside Prometheus's 5-minute instant-query +lookback, so the rule goes on firing for several minutes after you revert. That +is an artifact of the test, not of the alert: in normal operation the target +never changes, `probe_success` moves 1→0→1 on one series, and recovery is +immediate. + +**The observability stack does not live in the runner's checkout.** It runs from +`/home/tadmin/triangle-observability` on Delta — a hand-copied tree, not a git +clone — with `--env-file ../observability.env` (not `cms.env`, despite what +deploy/README.md says elsewhere). Changing any file above means copying it there +and restarting the affected service; note `docker compose up -d` will NOT +restart Prometheus for a config-file-only change, so `restart prometheus` +explicitly. ### Manual failover, without MaxScale diff --git a/observability/blackbox/blackbox.yml b/observability/blackbox/blackbox.yml new file mode 100644 index 0000000..3c098d6 --- /dev/null +++ b/observability/blackbox/blackbox.yml @@ -0,0 +1,33 @@ +# blackbox_exporter modules for deploy/compose.observability.yml. +# +# Purpose: probe MaxScale's client listener from OUTSIDE the database tier. +# Everything else in this stack watches the CMS itself, which cannot report the +# one failure that matters most here — MaxScale being dead. Since the write path +# is fenced so that MaxScale is the ONLY route to the databases (see +# deploy/mariadb/README.md), MaxScale down is a total outage, and MaxScale's own +# alert script cannot report it: the process that would send the alert is the +# process that died. +# +# A TCP connect against :4006 is deliberately chosen over the MaxScale REST API: +# - 4006 is the port the CMS actually uses, so this tests the real dependency +# rather than a management interface that could be healthy while routing is +# not. +# - It needs no new exposure. The admin API (8989) would have to be opened to +# Delta, and it can reconfigure MaxScale, so it is a far worse thing to +# expose than a port the CMS already talks to. +# - MaxScale 24.02 serves no Prometheus endpoint (verified: /metrics and +# /v1/metrics both 404), so scraping it directly is not an option anyway. + +modules: + # Plain TCP handshake. We do NOT speak the MySQL protocol here: MaxScale sends + # a server greeting on connect, and completing a real login would need + # credentials in this file. Accepting the connection is sufficient evidence + # that MaxScale is alive and listening. + tcp_connect: + prober: tcp + # Comfortably under Prometheus's scrape_timeout so a slow probe surfaces as + # a failed probe rather than a scrape error, which would alert differently. + timeout: 5s + tcp: + ip_protocol_fallback: false + preferred_ip_protocol: ip4 diff --git a/observability/grafana/provisioning/alerting/maxscale.yml b/observability/grafana/provisioning/alerting/maxscale.yml new file mode 100644 index 0000000..9341a32 --- /dev/null +++ b/observability/grafana/provisioning/alerting/maxscale.yml @@ -0,0 +1,102 @@ +# Grafana unified-alerting provisioning: the database tier's outside-in watchdog. +# +# This exists to cover the one gap MaxScale's own alert script cannot: MaxScale +# being dead. The write path is fenced so MaxScale is the only route to the +# databases, so that is a total outage — and the script that would report it +# runs on the machine that stopped. See deploy/mariadb/README.md. +# +# $SLACK_WEBHOOK_URL is interpolated by Grafana from the container environment, +# supplied by compose.observability.yml. Provisioning rejects an empty URL, so +# the compose file defaults it to a placeholder; delivery then fails loudly in +# Grafana's log rather than silently doing nothing. +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: slack-triangle + receivers: + - uid: slack-triangle-receiver + type: slack + settings: + url: $SLACK_WEBHOOK_URL + title: '{{ template "slack.default.title" . }}' + text: |- + {{ range .Alerts }}*{{ .Status | toUpper }}* {{ .Labels.alertname }} + {{ .Annotations.summary }} + {{ .Annotations.description }} + {{ end }} + # Recovery matters as much as onset here: a database outage that ended + # is information someone is actively waiting for. + disableResolveMessage: false + +policies: + - orgId: 1 + receiver: slack-triangle + group_by: [alertname] + # Short waits: this is a total-outage alert, not a capacity warning. + group_wait: 10s + group_interval: 1m + # Re-notify hourly while still firing, so an unresolved outage does not fall + # off the radar after a single message. + repeat_interval: 1h + +groups: + - orgId: 1 + name: database-tier + folder: Alerts + # Evaluate every 30s. Prometheus scrapes every 15s, so this cannot miss a + # sustained failure while staying cheap. + interval: 30s + rules: + - uid: maxscale-unreachable + title: MaxScale unreachable + # Deliberately NOT "no data = alerting" for the probe itself: see + # noDataState below. + condition: threshold + for: 1m + data: + - refId: probe + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: prometheus + model: + refId: probe + editorMode: code + expr: probe_success{service="maxscale"} + instant: true + range: false + - refId: threshold + relativeTimeRange: + from: 300 + to: 0 + datasourceUid: __expr__ + model: + refId: threshold + type: threshold + expression: probe + conditions: + - evaluator: + type: lt + params: [1] + # If the probe series vanishes entirely, something is wrong with + # Prometheus or blackbox rather than with MaxScale — but the practical + # consequence is the same: nobody is watching the database tier. Alert + # on it rather than failing open. + noDataState: Alerting + execErrState: Alerting + labels: + severity: critical + component: database + annotations: + summary: "MaxScale is not accepting connections on 10.248.40.183:4006" + description: >- + A TCP probe from Delta to MaxScale's client listener has failed for + 1 minute. The CMS reaches the databases ONLY through MaxScale (3306 + on DB1/DB2 is firewalled to the MaxScale host and the DB peer), so + this is a full database outage, not a degraded read path. + MaxScale's own Slack alerts cannot fire for this — the process that + sends them is the one that is down. + Check: systemctl status maxscale on 10.248.40.183, then + maxctrl list servers. If MaxScale is healthy and this still fires, + suspect the network path from Delta rather than the database tier. diff --git a/observability/prometheus/prometheus.delta.yml b/observability/prometheus/prometheus.delta.yml index e46df74..aee676c 100644 --- a/observability/prometheus/prometheus.delta.yml +++ b/observability/prometheus/prometheus.delta.yml @@ -33,3 +33,32 @@ scrape_configs: - targets: ["triangle-cms-backend-green-1:8080"] labels: slot: green + + # Watchdog for the database tier's single point of failure. + # + # The write path is fenced so MaxScale is the ONLY route to the databases, so + # MaxScale being down is a total outage — and it is precisely the outage + # MaxScale's own alert script cannot report, because the process that would + # send the alert is the one that died. This probes it from Delta instead. + # + # DB1/DB2 are deliberately NOT probed here: their 3306 is firewalled to the + # MaxScale host and the DB peer, so Delta cannot reach them by design, and a + # target that can never succeed is a permanently firing alert. + - job_name: blackbox-tcp + metrics_path: /probe + params: + module: [tcp_connect] + static_configs: + - targets: ["10.248.40.183:4006"] + labels: + service: maxscale + # The standard blackbox indirection: Prometheus scrapes the EXPORTER, and + # the real target travels as a query parameter. Without this rewrite + # Prometheus would try to scrape 10.248.40.183:4006 as if it served metrics. + relabel_configs: + - source_labels: [__address__] + target_label: __param_target + - source_labels: [__param_target] + target_label: instance + - target_label: __address__ + replacement: blackbox:9115