Fix alerting delivery, and deploy observability from CI - #183
Merged
Conversation
The webhook turned out to be a Discord one (https://discord.com/api/webhooks/...), so both alert paths were built against the wrong provider and neither would have delivered. Uses Discord's native format rather than its Slack-compatibility shim. Discord will accept Slack-shaped payloads on a /slack URL suffix, but it discards formatting it does not understand, so the native field renders correctly and fails honestly: - maxscale-alert.sh posts {"content": ...} instead of {"text": ...}, and uses Discord markdown (**bold**; single asterisks are italics). - The Grafana contact point is type "discord", not "slack" pointed at a Discord URL. - SLACK_WEBHOOK_URL is now DISCORD_WEBHOOK_URL throughout. Adds --fail to the script's curl. Without it curl exits 0 on a 4xx/5xx, so a rejected payload or a revoked webhook would have looked like success and never reached the failure log -- which for an alerting path is the worst possible failure mode. Adds deleteContactPoints for the orphaned slack-triangle receiver. Removing a contact point from a provisioning file does NOT delete it from Grafana; it would otherwise have sat there indefinitely looking like a live destination. Also gitignores .webhook, which was sitting untracked-but-committable in the working tree. Verified live on both paths: the MaxScale script posted a test event with no discord_post_failed line (curl --fail makes that meaningful), and the Grafana rule fired and delivered with no notifier error, where the placeholder URL had previously logged a loud failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes, both prompted by seeing the alerts land in #alerts. The MaxScale-down watchdog now lives in Prometheus and Alertmanager instead of Grafana's provisioned alerting. Delta's Prometheus is a DATASOURCE for the Triangle Grafana, not part of it, so a Grafana-owned rule would have disappeared the moment the local Grafana was retired -- while blackbox kept probing, Prometheus kept scraping, and every dashboard kept looking healthy. The only symptom would have been an alert that never arrived, during exactly the window nobody was watching for one. The rule now sits next to the data it reads and does not care which Grafana is in front. Grafana's noDataState becomes an explicit MaxScaleProbeMissing rule on absent(). The messages are now Discord embeds rather than walls of bold labels. The first version dumped the whole annotation as prose and a teammate replied "do I ignore this?", which is the only review that matters for an alert. Each one now has a coloured bar (red act now, yellow degraded, green over), a title that says what happened rather than naming the event constant, one line of consequence, and the addresses as fields. MaxScale's [10.248.40.155]:3306 bracket formatting is stripped -- it is IPv6-safe quoting that means nothing for an IPv4 pair. Grafana's alerting provisioning is kept as a deletion-only file, because provisioning never deletes: removing a definition from disk leaves the object live. Ordering there is load-bearing and I got it wrong first -- deleting a contact point a notification policy still references makes Grafana exit with "contact-points.referenced" and crash-loop, and dropping the policies block does not remove the policy. resetPolicies has to hand routing back to the default first, in a separate pass. Verified: rule fired -> Alertmanager -> Discord with no delivery error, and the MaxScale script posted an embed with no discord_post_failed. Grafana is back to zero alert rules and one built-in contact point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything routes to the central Triangle Grafana, which reaches this
stack through the Nginx endpoints in nginx/triangle-{prometheus,loki}
.conf. A second Grafana on Delta only duplicated that.
Safe to remove because the alerting no longer lives there. The
database-tier rules moved to Prometheus + Alertmanager in the previous
commit precisely so they would survive this; pulling Grafana now costs
no alerting at all, only dashboards, and those are the central
instance's job anyway.
Nothing was lost with it. The one dashboard it held, gisbxcj, was
verified byte-identical to the repo copy before removal -- no un-pulled
UI edits. Its datasources pointed at compose-internal DNS
(http://prometheus:9090), so they were local-only and never portable to
the central instance.
The dashboard JSON moves to observability/grafana/dashboards/ and stays
the source of truth to import centrally. The provisioning tree is gone:
datasources were local-only, and the alerting file existed solely to
delete what earlier commits had created, which it has now done.
Documents the trap that migration turns on: the dashboard hard-binds to
datasource UIDs "prometheus" and "loki" -- 16 panel references and 3 --
and Grafana assigns a random UID to anything created through the UI. Get
it wrong and the dashboard imports cleanly and renders empty, with no
error outside the panels themselves.
pull-dashboards.sh now points at whichever Grafana holds the dashboards
rather than a local one that no longer exists.
Verified after removal: Prometheus healthy with both rules loaded and
Alertmanager attached, blackbox probe returning 1, Loki still serving.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stack was the last thing on Delta that only existed because someone had run docker compose by hand, out of a directory nobody's tooling knew about. That is a bad property for anything, and a worse one for the thing that pages you when the database dies: it drifts from the repo silently, and the first symptom is an alert that never arrives. deploy-observability.sh now runs as a Deploy Delta step, after the CMS deploy. It syncs observability/ and compose.observability.yml from the runner's checkout into ~triangle-runner/triangle-observability and runs Compose from there. It copies rather than running in place because actions/checkout resets the checkout on every deploy, which would yank the bind-mount sources out from under a long-lived stack -- the reason the old README told people to install it somewhere else by hand. The destination is owned by the runner and outside _work/, so Actions never touches it. Two things this has to handle that are easy to miss: `docker compose up -d` does NOT restart a container when only the CONTENTS of a mounted config file changed. The spec is identical, so it reports "Running" and the new config never takes effect. The script fingerprints the synced tree and restarts explicitly when it differs, and skips restarting entirely when it does not, so an ordinary CMS deploy costs nothing. The sync is --inplace. Plain rsync writes a temp file and renames, so every file gets a new inode while a running container's bind mount still holds the old one -- stale config that looks correctly deployed. It cannot disturb the CMS: separate Compose project, so up -d cannot recreate or stop the slots. It runs after deploy.sh because Prometheus joins the CMS network, declared external. It fails the job if Prometheus comes up with no alerting rules or no attached Alertmanager, since a stack that runs but silently does not alert is worse than one that is plainly down. Needing no env file is what made this simple, and that fell out of removing Grafana: DISCORD_WEBHOOK_FILE is the only variable left and it has a default, so there are no secrets to plumb through CI. Migrated and verified on Delta: stack moved off /home/tadmin to the runner-owned path with all four data volumes and Prometheus history intact, re-running is a no-op, and a changed rule file does trigger the restart. The stale /home/tadmin copy is gone; its one irreplaceable file, the plaintext datasource password Nginx only stores hashed, was moved to /etc/triangle-observability/ first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last hand-installed piece. The Loki and Prometheus datasource endpoints the central Grafana connects to were host config that only matched the repo because someone had copied it there. Done without widening the runner's privileges. The files install into /etc/nginx/triangle-observability/, a directory the runner OWNS, which a root-owned conf.d file pulls in with a wildcard include. The runner's sudo rights stay exactly what the CMS deploy already needs -- nginx -t and nginx -s reload -- rather than gaining write access to /etc/nginx or an "install this as root" rule. Same shape as /etc/nginx/triangle-cms/, which it already owns for blue/green. The install is transactional because this Nginx also serves the CMS: the live files are snapshotted, the new ones installed, and nginx -t runs BEFORE any reload. A config that fails validation is reverted and the step fails. Nginx keeps serving the old config throughout, and -- the part that actually matters -- no broken file is left on disk for the next reload to trip over, which could be the CMS deploy's. triangle-cms.conf is deliberately excluded. It is the live site's own server block, a much larger blast radius than two loopback-proxying endpoints, and it changes about never. The compose helper is renamed obs_compose. It was shadowing the compose() that common.sh binds to compose.cms.yml, which worked only by definition order -- moving the source line would have silently pointed this script at the CMS stack. Verified on Delta by exercising all three paths: unchanged sites skip the reload, a valid change installs and reloads, and a deliberately broken config fails nginx -t, reverts, exits non-zero, and leaves Nginx, the CMS and both authenticated endpoints healthy. Live files confirmed identical to the repo afterwards, and the old sites-available/sites-enabled copies removed so the two cannot collide on the same listen ports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #182, which merged while this work was still in progress — these five commits landed on that branch after its merge point, so they never made it in.
This matters more than a normal follow-up:
maincurrently documents and configures alerting for Slack, and production has been on Discord since. Everything here is already live on Delta and verified; merging reconciles the repo with the hosts.What's in it
9eac385cfbb5932437694bdf949f5c74568The through-line
The alerting built in #182 pointed at the wrong provider and wouldn't have delivered a single message — the webhook is Discord, not Slack. Fixing that exposed the rest: the watchdog lived in Grafana's provisioned alerting, which meant it would silently vanish when the local Grafana was retired in favour of the central Triangle Grafana. So it moved to Prometheus + Alertmanager, next to the data it reads, and then Grafana came out of the stack entirely.
At which point the remaining problem was that none of it deployed. The observability stack only matched the repo because someone had run
docker composeby hand in a directory none of the tooling knew about — a bad property generally, and a worse one for the thing that pages you when the database dies.Notable decisions, all documented in-tree
curl --failin the alert script — without it curl exits 0 on 4xx/5xx, so a revoked webhook looks like success and never reaches the failure log.actions/checkoutresets the runner workspace every deploy and would yank bind-mount sources from under a long-lived stack.up -ddoes not restart a container when only a mounted file's contents changed — the script fingerprints the tree and restarts explicitly, and skips when nothing changed.--inplace— plain rsync renames, giving new inodes a running container's bind mount never sees.include, so the runner's sudo stays at exactlynginx -tandnginx -s reload. Transactional: validate before reload, revert and fail otherwise, so no broken file is left for the CMS deploy's next reload.triangle-cms.confis deliberately excluded from auto-deploy — live site server block, much larger blast radius, changes about never.Verification
All of it was exercised against production rather than reasoned about: a real failover test (promotion in <10s, CMS served and wrote against the promoted node, rejoin, switch back), Discord delivery confirmed on both paths, and the Nginx deploy tested through all three cases including a deliberately broken config that reverted and exited non-zero with the CMS still serving.
Current live state: DB1
Master/ DB2Slaveat equal GTID, semi-syncclients=1, five observability services up, CMS 200 on both slots, both authenticated datasource endpoints 200.Deploying this
Merging triggers Deploy Delta, which will now run
deploy-observability.shfor the first time from CI. It should be a near no-op — the stack is already at these versions from the same script run by hand — but it's the first automated run, so worth watching.Still open, unchanged: central Grafana wiring (mind the
prometheus/lokidatasource UIDs) and real backups.🤖 Generated with Claude Code