Make DB2 a safe automated failover target

[ssavutu]

DB2 (THETRIANGLE-DB2-LXC, 10.248.40.155) is now a semi-synchronous replica of DB1 and MaxScale's auto_failover is on. This has already been applied and failover-tested against production — merging changes nothing on the DB hosts, it only lands the source-of-truth config and the runbook.

Why auto_failover was off, and what changed

Promoting a replica on a 2-node async pair loses acknowledged writes and can split-brain. Neither is fixed by the failover flag, so:

• Semi-sync, wait_point=AFTER_SYNC — DB1 doesn't commit to the storage engine until DB2 holds the event, so a promotion can't drop a write the CMS was told succeeded. AFTER_COMMIT (the default) would not do this; it makes the write visible before the ack, which is exactly the losing window.
• Network fencing — two nodes have no quorum and can't vote, so instead 3306 on both hosts is firewalled to MaxScale and the DB peer. "MaxScale can't see DB1" then implies "the CMS can't see DB1", and promoting DB2 cannot produce two servers taking writes.
• gtid_strict_mode=ON — a diverged old primary is refused by auto_rejoin rather than corrupting.

Accepted limits, documented in the README: semi-sync degrades to async while DB2 is down (wait_no_slave=OFF, deliberate — the alternative turns a replica outage into a site outage), and MaxScale remains a SPOF. Both need a third node. Failover is also not backup; real backups are still owed.

Failover test

Stopped DB1 → promoted in <10s → CMS served and wrote against DB2 → restarted DB1 → rejoined as replica → switched back. End state verified: equal GTID, 0 lag, semi-sync ON with 1 client, CMS 200.

Bugs fixed

• mariadb-dump --gtid alone records no position. It only changes the format of what --master-data emits. A replica seeded that way starts from the beginning of binlogs that expire in 7 days. setup-replica.sh had this; now --master-data=1 (=2 comments the line out) plus a hard check that aborts if the position is missing.
• replica.cnf used relaxed durability — right for a read cache, wrong for a failover target twice over: a promoted DB2 would run production without ACID, and semi-sync's ack would only mean "in DB2's page cache".
• Semi-sync was configured one-sided. Both roles are now enabled on both nodes; otherwise a promotion silently drops to async.
• maxscale was missing BINLOG ADMIN (MariaDB 10.5+ split it out of SUPER), which made auto_rejoin loop forever on "Failed to prepare (demote) standalone server".
• replication_user defaults to the monitor user, so rejoin built CHANGE MASTER as maxscale@<rejoining node> — nonexistent. Surfaced only as new_slave → lost_slave ~2s later. Now pinned to repl.

Please read the warning in step 6

The second commit reverts guidance the first one introduced, because I followed it and took production down. It said to drop triangle_user@<delta-ip> as "Delta's direct bypass". It is not a bypass: MaxScale authenticates a client against the backend user table using the client's own source address, so that account is how the CMS logs in through MaxScale. Dropping it closed nothing and returned Error 1045 on every DB-backed route — while /v1/health kept returning 200, so it looked healthy until a page was loaded.

Related: setup-replica.sh dumps --databases triangle, which excludes mysql.*, so DB2 initially had none of the application accounts. A failover would have promoted a server nothing could authenticate to. Both traps are now documented with the symptoms to recognise them by.

Reviewer notes

• deploy/mariadb/provision-db2.sh is new; it refuses to run anywhere but DB2, since ssh gives no warning when an ARP change puts you on a different host.
• maxscale.cnf was verified with maxscale --config-check on the live 24.02.9 binary.
• backend.env gained REPL_USER/REPL_PASSWORD; all six keys are required or MaxScale won't start.

🤖 Generated with Claude Code

[ssavutu]

Added failover alerting (2a610fc).

How it works: maxscale-alert.sh is wired in as mariadbmon's script=, so the monitor invokes it directly on each event in events=. It fires within one monitor tick instead of waiting for a scrape, and it runs on the MaxScale host — so it still works when Delta or the observability stack is down.

It always appends to /var/log/maxscale/failover-events.log and posts to Slack only if /etc/maxscale.secrets.d/alert.env supplies a webhook. It is deployed and live right now in log-only mode, pending a webhook URL. The webhook deliberately does not live in maxscale.cnf, which is world-readable 0644.

Two bugs the test caught that reading the docs would not have:

• slave_up was missing from events=. It is not the same transition as new_slave — new_slave is [Running]→[Slave,Running] (a standalone node joining), slave_up is [Down]→[Slave,Running] (a node returning). A replica outage therefore alerted on the way down and went silent on recovery. The only symptom of this class of mistake is an alert that never arrives.
• Rpl_semi_sync_master_status is not a usable health signal, and this branch previously told you to alert on it. Stopping DB2 showed status stay ON through seven unacknowledged commits while clients sat at 0 and no_tx climbed 0→7 — with wait_no_slave=OFF the master never enters the state that would flip it. Alert on Rpl_semi_sync_master_clients == 0.

Verified end to end by stopping and restarting DB2: both slave_down and slave_up reached the log, semi-sync re-engaged at clients=1, CMS stayed 200 throughout.

Known gap, documented rather than papered over: the script cannot report that MaxScale itself died — and since the write path is now fenced to MaxScale, that is a total outage. That needs an external check from Delta's Prometheus and is not built here.

[ssavutu]

Added the MaxScale-down watchdog (6659ecd) — the gap the previous commit documented but did not close.

blackbox_exporter --TCP connect--> 10.248.40.183:4006
        ^                                  |
        | scrape                           v
   Prometheus  ---->  Grafana alert  ---->  Slack
                      probe_success == 0 for 1m

The failover script runs on MaxScale, so the one outage it can never report is MaxScale itself stopping — which, now that the write path is fenced, is a total outage rather than a degraded read path. This watches it from Delta instead.

Design choices worth a reviewer's attention:

• Probes :4006, not the admin API. 4006 is the port the CMS actually uses, so it tests the real dependency rather than a management interface that could be healthy while routing is not — and 8989 can reconfigure MaxScale, so opening it to Delta is a far worse thing to expose. MaxScale 24.02 serves no Prometheus endpoint anyway (/metrics and /v1/metrics both 404), so scraping it directly was never on the table.
• DB1/DB2 are deliberately not probed. Their 3306 is firewalled to the MaxScale host and the DB peer, so Delta cannot reach them by design; such a target would alert forever.
• noDataState: Alerting — a missing probe series means nobody is watching the database tier, which is worth waking someone for even though the cause is Prometheus rather than MaxScale.

Verified without any CMS outage by pointing the probe at a closed port on the same host: rule went pending → firing, and Grafana routed it to receiver=slack-triangle, failing delivery only on the placeholder webhook. Reverted, alert returned to inactive.

Two operational notes now in the README:

• The observability stack does not live in the runner's checkout. It runs from /home/tadmin/triangle-observability — a hand-copied tree, not a git clone — with --env-file ../observability.env, not cms.env as deploy/README.md states elsewhere. And compose up -d will not restart Prometheus for a config-file-only change; it needs restart prometheus explicitly.
• Testing by changing the probe target leaves a stale series inside Prometheus's 5-minute instant-query lookback, so the rule keeps firing for several minutes after you revert. That is an artifact of the test method, not the alert — in normal operation the target never changes and recovery is immediate.

[ssavutu]

Wired up the real webhook (9eac385) — with one correction.

The webhook is a Discord one, not Slack. Both alert paths were built against the wrong provider and neither would have delivered a single message. Now converted to Discord's native format rather than its Slack-compatibility shim — Discord accepts Slack-shaped payloads on a /slack URL suffix, but silently discards formatting it doesn't understand, so the native field renders correctly and fails honestly:

• maxscale-alert.sh posts {"content": ...}, with Discord markdown (**bold** — single asterisks are italics there)
• The Grafana contact point is type discord, not slack aimed at a Discord URL
• SLACK_WEBHOOK_URL → DISCORD_WEBHOOK_URL throughout

Two things worth flagging beyond the rename:

• Added --fail to the script's curl. Without it curl exits 0 on a 4xx/5xx, so a rejected payload or a revoked webhook would have looked like success and never reached the failure log. For an alerting path that is the worst possible failure mode — you'd believe you were covered.
• Added deleteContactPoints for the orphaned slack-triangle receiver. Removing a contact point from a provisioning file does not delete it from Grafana; it would have sat there indefinitely looking like a live destination.

Verified live on both paths: the MaxScale script posted a test event with no discord_post_failed line (which --fail now makes meaningful), and the Grafana rule fired and delivered with no notifier error — where the placeholder URL had previously logged a loud Failed to send.

Also gitignored .webhook, which was sitting untracked-but-committable in the working tree. Confirmed it appears in no commit on any branch.

[ssavutu]

Moved the watchdog off Grafana, and redesigned the messages (cfbb593).

Watchdog now lives in Prometheus + Alertmanager

blackbox --TCP--> 10.248.40.183:4006
   | scrape
   v
Prometheus ---> Alertmanager ---> Discord

Delta's Prometheus is a datasource for the Triangle Grafana, not part of it — so a Grafana-owned rule would have vanished the moment the local Grafana was retired, while blackbox kept probing, Prometheus kept scraping, and every dashboard kept looking healthy. The only symptom would have been an alert that never arrived, during exactly the window nobody was watching for one. Grafana's noDataState: Alerting becomes an explicit MaxScaleProbeMissing rule on absent().

Messages are embeds now

The first version dumped the whole annotation as prose, and a teammate replied "do I ignore this?" — which is the only review that matters for an alert. Each message now has a coloured bar (red act now / yellow degraded / green over), a title saying what happened rather than naming the event constant, one line of consequence, and addresses as fields. MaxScale's [10.248.40.155]:3306 bracket formatting is stripped; it's IPv6-safe quoting that means nothing for an IPv4 pair.

One trap worth knowing

Grafana provisioning never deletes — removing a definition from disk leaves the object live in its database, so alerting/maxscale.yml is kept as a deletion-only file. Ordering there is load-bearing and I got it wrong first: deleting a contact point that a notification policy still references makes Grafana exit and crash-loop with

ProvisioningServiceImpl run error: contact points:
[alerting.notifications.contact-points.referenced]

and dropping the policies: block doesn't remove the policy either. resetPolicies has to hand routing back to the default first, in a separate start. Grafana was down for about two minutes while I sorted that; it's documented in the file itself so the next person doesn't repeat it.

Verified: rule fired → Alertmanager → Discord with no delivery error, and the MaxScale script posted an embed with no discord_post_failed. Grafana is back to zero alert rules and one built-in contact point.

[ssavutu]

Dropped Grafana from the observability stack (2437694).

Everything routes to the central Triangle Grafana, which reaches this stack through the Nginx endpoints in nginx/triangle-{prometheus,loki}.conf, so a second Grafana on Delta was pure duplication.

This is only safe because the alerting moved out first. The database-tier rules went to Prometheus + Alertmanager in the previous commit precisely so they'd survive this — pulling Grafana costs no alerting at all, only dashboards, which are the central instance's job anyway.

Nothing was lost. I diffed the live dashboard against the repo copy before removing anything: gisbxcj was byte-identical, no un-pulled UI edits. Its datasources pointed at compose-internal DNS (http://prometheus:9090), so they were local-only and never portable to central.

Remaining stack: prometheus · loki · promtail · blackbox · alertmanager.

The migration trap, now documented

The dashboard hard-binds to datasource UIDs — 16 panel references to prometheus, 3 to loki — and Grafana assigns a random UID to any datasource created through the UI. Get it wrong and the dashboard imports cleanly and renders completely empty, with no error outside the individual panels. Provision them, or set the UID explicitly.

observability/grafana/ is now just dashboards/gisbxcj.json, the source of truth to import centrally. pull-dashboards.sh points at whichever Grafana holds the dashboards rather than a local one that no longer exists.

Verified after removal: Prometheus healthy with both rules loaded and Alertmanager attached, blackbox probe returning 1, Loki still serving queries, DB tier and CMS untouched.

One loose end I deliberately did not clean up: the triangle-observability_grafana_data volume is still on Delta. Deleting it is irreversible and it's confirmed redundant, so it's yours to drop when you're happy:

docker volume rm triangle-observability_grafana_data

[ssavutu]

The observability stack now deploys from CI (bdf949f).

It was the last thing on Delta that existed only because someone had run docker compose by hand, out of a directory none of the tooling knew about. Bad property for anything; worse for the thing that pages you when the database dies — it drifts from the repo silently and the first symptom is an alert that never arrives.

deploy-observability.sh runs as a Deploy Delta step after the CMS deploy, syncing observability/ and compose.observability.yml from the runner's checkout into ~triangle-runner/triangle-observability.

It copies rather than running in place on purpose — actions/checkout resets the checkout on every deploy, which would yank the bind-mount sources out from under a long-lived stack. That's exactly why the old README told people to install it somewhere else by hand; this automates that instead of ignoring it.

Two traps it has to handle

• docker compose up -d does NOT restart a container when only the contents of a mounted config file changed. The spec is identical, so it reports Running and the new config never takes effect. I hit this earlier in this PR with Prometheus. The script fingerprints the synced tree and restarts explicitly when it differs — and skips restarting when it doesn't, so an ordinary CMS deploy costs nothing.
• The sync is --inplace. Plain rsync writes a temp file and renames, giving every file a new inode while a running container's bind mount still holds the old one. That serves stale config that looks correctly deployed.

Safety

Separate Compose project, so up -d cannot recreate or stop the CMS slots — a failure here leaves the CMS deployed and serving. It runs after deploy.sh because Prometheus joins the CMS network, declared external. It fails the job if Prometheus comes up with zero alerting rules or no attached Alertmanager, because a stack that runs but silently doesn't alert is worse than one that's plainly down.

Needing no env file is what made this clean, and that fell out of dropping Grafana: DISCORD_WEBHOOK_FILE is the only variable left and it has a default, so there are no secrets to plumb through CI.

Migrated and verified on Delta

Moved off /home/tadmin (unwritable by the runner) to the runner-owned path, with all four data volumes and Prometheus history intact. Re-running is a confirmed no-op; a changed rule file does trigger the restart. Stale copy removed — but first I moved out its one irreplaceable file, the plaintext datasource password, since Nginx only stores it hashed and deleting it would have meant a password reset rather than a lookup. It now lives at /etc/triangle-observability/loki-datasource-password.

The Nginx sites are still installed by hand; only the Compose stack behind them deploys automatically.

[ssavutu]

The Nginx sites auto-deploy now too (5c74568) — that was the last hand-installed piece.

Done without widening the runner's privileges

The files install into /etc/nginx/triangle-observability/, a directory the runner owns, which a root-owned conf.d file pulls in with a wildcard include. The runner's sudo rights stay exactly what the CMS deploy already needed — nginx -t and nginx -s reload — instead of gaining write access to /etc/nginx or an "install this as root" rule. It's the same shape as /etc/nginx/triangle-cms/, which it already owns for blue/green.

Needs a one-time root bootstrap (already done on Delta, documented in the README).

Transactional, because this Nginx also serves the CMS

Live files are snapshotted, new ones installed, and nginx -t runs before any reload. A config that fails validation is reverted and the step fails. Nginx keeps serving the old config throughout — and the part that actually matters, no broken file is left on disk for the next reload to trip over, which could be the CMS deploy's.

triangle-cms.conf is deliberately excluded. It's the live site's own server block — a much larger blast radius than two loopback-proxying endpoints, and it changes about never. Automating it would trade real risk for no meaningful gain.

Verified on Delta, all three paths

Case	Result
Unchanged sites	skips the reload entirely
Valid change	installs and reloads
Broken config	fails nginx -t, reverts, exits 1, CMS + both endpoints still 200

Live files confirmed identical to the repo afterwards, and the old sites-available/sites-enabled copies removed — leaving both would have collided on the same listen ports.

One latent bug fixed on the way

The compose helper is renamed obs_compose. It was shadowing the compose() that common.sh binds to compose.cms.yml — which worked only by definition order. Anyone moving the source line would have silently pointed every call in this script at the CMS stack.

Final state: CMS 200, both authenticated datasource endpoints 200, five observability services up, DB1 Master / DB2 Slave at equal GTID.