diff --git a/.env.example b/.env.example index c3aa994..9076281 100644 --- a/.env.example +++ b/.env.example @@ -1,7 +1,5 @@ MARIADB_ROOT_PASSWORD=change-this-root-password MARIADB_PASSWORD=change-this-db-password -GRAFANA_ADMIN_USER=change-this-admin-user -GRAFANA_ADMIN_PASSWORD=change-this-admin-password # Authentik OIDC - required for auth on write endpoints. OIDC_ISSUER_URL= diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 978f86e..391b5ad 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -78,17 +78,3 @@ jobs: NGINX_ACTIVE_INCLUDE: ${{ vars.DELTA_NGINX_ACTIVE_INCLUDE }} PUBLIC_BASE_URL: ${{ vars.DELTA_PUBLIC_BASE_URL }} run: deploy/scripts/deploy.sh "${{ steps.sha.outputs.value }}" - - # Runs AFTER the CMS deploy, deliberately. Prometheus joins the CMS - # Compose network, which is declared external, so that network has to - # exist before this stack will start. - # - # It is a separate Compose project, so nothing here can recreate or stop - # the CMS slots -- a failure below leaves the CMS deployed and serving. It - # still fails the job, because an observability stack that quietly stopped - # matching the repo is exactly the state that lets a database outage go - # unannounced. - - name: Deploy observability stack - env: - OBSERVABILITY_DIR: ${{ vars.DELTA_OBSERVABILITY_DIR }} - run: deploy/scripts/deploy-observability.sh diff --git a/.gitignore b/.gitignore index b3e5d12..a8a1e69 100644 --- a/.gitignore +++ b/.gitignore @@ -16,8 +16,8 @@ frontend/.env server/data/ deploy/*.env !deploy/*.env.example -deploy/runner/ ..env.un~ -# Local secret: Discord webhook for failover alerts (see deploy/mariadb/README.md) -.webhook +# Local DB dumps (contain cms_users password hashes and cms_sessions). +# Kept here so Borg backs them up -- /var/lib/docker volumes are NOT backed up. +db-backups/ diff --git a/README.md b/README.md index 0321f03..93c1933 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ Headless CMS replacement for The Triangle, with: - `server/` Go API - `frontend/` React frontend -- `observability/` Loki + Promtail + Prometheus, plus a Grafana in the local dev - stack only — on Delta the dashboards live in the central Triangle Grafana - (see `deploy/README.md`) +- The observability stack — Prometheus, Loki, Promtail, Alertmanager, blackbox + and the Grafana dashboards — lives in + [`triangle-infrastructure`](https://github.com/DrexelTriangle/triangle-infrastructure). + The backend still exposes `GET /metrics`; scrape it directly in local dev. - `scripts/` local setup helpers API docs and data models: https://github.com/DrexelTriangle/triangle-cms/wiki @@ -92,13 +93,17 @@ python ./scripts/generate_wordpress_sql.py 3. Start services (Path A is recommended): -Path A (recommended): full Docker stack (CMS + observability): +Path A (recommended): full Docker stack (CMS + MariaDB): ```bash python ./scripts/setup_containers.py ``` -This stack uses a shared Docker bridge network scoped to the Compose project, so services can resolve each other by container/service name (for example `mariadb`, `loki`, `grafana`). +This stack uses a shared Docker bridge network scoped to the Compose project, so services can resolve each other by container/service name (for example `mariadb`). + +The dev stack no longer runs Loki, Promtail or Grafana: their config files were +shared with Delta's stack and moved to `triangle-infrastructure` so there is one +source of truth. Use `docker compose logs` locally. Path B: Docker MariaDB + local Go backend: diff --git a/deploy/README.md b/deploy/README.md index a5d9274..d4042f3 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -20,215 +20,42 @@ deployment Compose project. The backend connects to the external database/proxy endpoint supplied by `DB_HOST` and `DB_PORT` in the host-only `cms.env`. Loki and Promtail do run on Delta, but in the separate `triangle-observability` -Compose project (`compose.observability.yml`). Nothing in CI/CD starts it: the -deploy scripts pin `COMPOSE_FILE` to `compose.cms.yml` so that a CMS deploy can -never tear the log stack down. It is brought up by hand, once, and `restart: -unless-stopped` keeps it up across deploys. - -## Exposing Loki to the Triangle Grafana - -The Triangle Grafana queries Delta's Loki as a datasource. Loki has no -authentication of its own in any configuration, so it is published only to -`127.0.0.1:13100` and fronted by the `nginx/triangle-loki.conf` site on port -3100, which adds basic auth, blocks the write path, and 404s everything that is -not a query. - -1. **The stack deploys itself.** `deploy/scripts/deploy-observability.sh` runs - as a step of the Deploy Delta workflow, immediately after the CMS deploy, so - an observability change reaches Delta the same way application code does — - merge to main and wait. Nothing here needs starting by hand. - - It syncs `observability/` and `compose.observability.yml` from the runner's - checkout into `~triangle-runner/triangle-observability` and runs Compose from - there. **It copies rather than running in place on purpose:** the checkout is - the only tree on Delta that contains `observability/`, but `actions/checkout` - resets it on every deploy, which would yank the bind-mount sources out from - under a long-lived stack. The destination is owned by the runner and sits - outside `_work/`, so Actions never touches it. Override with the - `DELTA_OBSERVABILITY_DIR` repository variable if the host is laid out - differently. - - It **cannot** disturb the CMS: this is a separate Compose project, so - `up -d` cannot recreate or stop the slots. The one real coupling is that - Prometheus joins the CMS network, declared `external`, which is why the step - runs after the CMS deploy rather than beside it — and why a from-scratch - bring-up needs the CMS stack up first. - - Two behaviours worth knowing before changing anything here: - - - **A config-only change does not restart anything by itself.** `up -d` sees - an identical Compose spec and reports `Running` even though a mounted file's - contents changed, so the new config never takes effect. The script - fingerprints the synced tree and issues an explicit `restart` when it - differs — and skips it entirely when it does not, so an ordinary CMS deploy - costs nothing. - - **The sync is `--inplace`.** Ordinary `rsync` writes a temp file and - renames, giving every file a new inode, while a running container's bind - mount still holds the old one. That combination serves stale config that - looks correctly deployed. - - The step fails the job if Prometheus comes up with zero alerting rules or no - attached Alertmanager, because a stack that is running but silently not - alerting is worse than one that is plainly down. - - To run it by hand (from a repo checkout on Delta): +Compose project, which now lives in [`triangle-infrastructure`](https://github.com/DrexelTriangle/triangle-infrastructure). +Nothing in this repo starts or stops it: the deploy scripts pin `COMPOSE_FILE` +to `compose.cms.yml` so that a CMS deploy can never tear the log stack down, and +`restart: unless-stopped` keeps it up across deploys. + +## Observability + +Prometheus, Loki, Promtail, Alertmanager, blackbox and the two read-only nginx +endpoints the central Grafana queries **moved to +[`triangle-infrastructure`](https://github.com/DrexelTriangle/triangle-infrastructure)** and are applied with +`ansible-playbook playbooks/observability.yml`. They are no longer deployed by +this repo's workflow. + +Two things about it still constrain the CMS, so they are recorded here rather +than only there: + +- **Prometheus joins this project's Compose network**, declared `external` on + its side. The CMS stack must be up before the observability stack will start, + and **renaming a service in `compose.cms.yml` breaks scraping in a different + repository.** `observability/prometheus/prometheus.delta.yml` over there + scrapes both slots by container name. +- **The backend exposes unauthenticated Prometheus metrics on `GET /metrics`.** + That is safe only because the host nginx proxies just `/v1` and `/swagger`, so + nothing routes it in from outside and Prometheus reaches it on the slot's + loopback port. If a vhost ever forwards `/metrics`, put it behind auth first: + it exposes route names, traffic volumes and error rates. + +`up{job="cms"}` returns **two** series, one per slot, labelled `slot="blue"` / +`slot="green"`. The idle slot being up is normal and says nothing about which +one serves traffic. + +The local development stack in the repo-root `docker-compose.yml` keeps its own +Prometheus (`observability/prometheus/prometheus.dev.yml`). Its Loki, Promtail +and Grafana services were removed when their shared config files moved, so +`docker compose logs` is the local log story now. - ``` - deploy/scripts/deploy-observability.sh - ``` - -2. **The Nginx sites deploy automatically too**, in the same step. They are - installed into `/etc/nginx/triangle-observability/`, a directory the runner - OWNS, which a root-owned `/etc/nginx/conf.d/triangle-observability.conf` - pulls in with a wildcard `include`. - - That indirection is deliberate: it keeps the runner's sudo rights at exactly - the two commands the CMS deploy already needs — `nginx -t` and - `nginx -s reload` — rather than granting write access to `/etc/nginx` or a - general "install this file as root" rule. It is the same shape as - `/etc/nginx/triangle-cms/`, which the runner already owns for blue/green. - - The install is transactional, because this Nginx also serves the CMS: the - live files are snapshotted, the new ones installed, and `nginx -t` run - **before** any reload. A config that fails validation is reverted and the step - fails — Nginx keeps serving the old config throughout, and no broken file is - left on disk for the *next* reload (which could be the CMS deploy's) to trip - over. - - `nginx/triangle-cms.conf` is deliberately NOT deployed this way: it is the - live site's own server block, a materially larger blast radius than two - loopback-proxying endpoints, and it changes about never. - - The **one-time root bootstrap**, already done on Delta. Note the sites no - longer live in `sites-available`/`sites-enabled` — a copy in both places - would collide on the same `listen` ports: - - ``` - sudo install -d -o triangle-runner -g triangle-runner -m 0750 \ - /etc/nginx/triangle-observability - printf 'include /etc/nginx/triangle-observability/*.conf;\n' \ - | sudo tee /etc/nginx/conf.d/triangle-observability.conf >/dev/null - sudo nginx -t && sudo nginx -s reload - ``` - -3. Create the datasource credentials (still a manual, one-time step — it is a - secret, not config): - - ``` - sudo htpasswd -B -c /etc/nginx/triangle-observability.htpasswd triangle-grafana - sudo chown root:www-data /etc/nginx/triangle-observability.htpasswd - sudo chmod 0640 /etc/nginx/triangle-observability.htpasswd - sudo nginx -t && sudo nginx -s reload - ``` - - Both endpoints share one htpasswd file, so the Triangle Grafana uses the same - credentials for its Loki and Prometheus datasources. The plaintext is kept at - `/etc/triangle-observability/loki-datasource-password` (0600 root) — Nginx - stores only a bcrypt hash, so losing that file means resetting the password - rather than looking it up. - - Note that `htpasswd` is not installed on Delta by default; it ships in - `apache2-utils` on Debian/Ubuntu and `httpd-tools` on RHEL-family hosts. - -4. Verify from Delta before handing the details over. Expect `401` then `200`: - - ``` - curl -s -o /dev/null -w '%{http_code}\n' localhost:3100/loki/api/v1/labels - curl -s -o /dev/null -w '%{http_code}\n' -u triangle-grafana \ - localhost:3100/loki/api/v1/labels - ``` - - `curl -s localhost:3100/ready` reports `Pattern Ingester not ready: waiting - for 15s after being ready` indefinitely, and that is expected rather than a - fault: `loki-config.yml` enables `pattern_ingester`, whose readiness never - latches in a single-binary deployment. Loki serves queries normally. Use the - authenticated `/loki/api/v1/labels` call above as the real health signal. - - Then confirm logs are actually arriving, which is the check that catches a - log-driver mismatch. This must return a non-empty list of container names: - - ``` - curl -s -u triangle-grafana \ - 'localhost:3100/loki/api/v1/label/container/values' - ``` - -5. In the Triangle Grafana, add two datasources, both with Basic auth enabled - and the same credentials. No path prefix or extra headers are needed: - - | Type | URL | - | --- | --- | - | Loki | `http://:3100` | - | Prometheus | `http://:9090` | - - **Name them so they end in `-delta`** — in practice `prometheus-delta` and - `loki-delta`. The UIDs no longer matter: the CMS dashboard resolves its - datasources through two template variables (`prometheus_ds`, `loki_ds`) - whose `regex` is `/-delta$/`, so the dropdowns match only these two and bind - to them on import. Name one of them something else and it drops out of the - dropdown, and the panels go blank again. - - This replaced hardcoded UIDs, which could not work here: the central Grafana - already had its own Prometheus and Loki holding the UIDs `prometheus` and - `loki`, and Grafana assigns a random UID to anything created through the UI. - Panels kept querying the *central* pair and rendered empty — the display name - was never what the dashboard looked at. The failure is silent: the dashboard - imports cleanly and every panel just shows no data. - - The dashboard needs both datasources: 16 panel references query Prometheus - and only 3 query Loki, so a Loki-only setup renders a mostly empty dashboard. - -6. Import `observability/grafana/dashboards/gisbxcj.json`. - -**Delta no longer runs its own Grafana** (removed 2026-08-05). Every dashboard -lives in the central Triangle Grafana, which reaches this stack through the two -Nginx endpoints above, so a second local Grafana only duplicated it. Nothing was -lost with it — the one dashboard it held was byte-identical to the repo copy. - -Removing it costs no alerting: the database-tier alerts live in Prometheus and -Alertmanager, not Grafana, precisely so they survive this (see -`deploy/mariadb/README.md`). Basic auth on these endpoints travels in cleartext; -fold them into TLS when TLS lands on Delta. - -### Metrics and the CMS dashboard - -The backend exposes Prometheus metrics on `GET /metrics`, and the stack runs a -Prometheus that scrapes both slots by container name -(`observability/prometheus/prometheus.delta.yml`). To do that it attaches to the -CMS project's network, declared `external` so this stack never owns it. The -consequence is an ordering dependency: the CMS stack must be up first, or -Compose refuses to start this one with a missing-network error. - -Scraping via the host gateway does not work, and it is worth knowing why before -"simplifying" it back: the slots publish to `127.0.0.1:8081/8082`, loopback -only, so a container dialling `172.17.0.1` gets connection refused. - -The "CMS Dashboard" JSON lives at `observability/grafana/dashboards/gisbxcj.json` -and is imported into the central Grafana by hand — there is no local Grafana to -provision it into any more. - -`/metrics` is unauthenticated. That is safe only because Nginx proxies just -`/v1` and `/swagger`, so nothing routes it in from outside and Prometheus -reaches it on the slot's loopback port. If a vhost ever forwards `/metrics`, put -it behind auth first: it exposes route names, traffic volumes, and error rates. - -`up{job="cms"}` returns **two** series, one per slot, each labelled -`slot="blue"` / `slot="green"`. The idle slot being up is normal and says -nothing about which one serves traffic. - -Dashboards can be edited in the Grafana UI, and those edits live only in that -Grafana's database until pulled back into the repo. Point the script at whichever -Grafana holds them — now the central one, not Delta: - -**Check the diff before committing a pull.** Grafana serializes the *resolved* -datasource of each panel, so a round-trip through the UI can write concrete UIDs -back over the `${prometheus_ds}` / `${loki_ds}` references and re-create exactly -the breakage the template variables exist to prevent. If `git diff` after a pull -shows `"uid"` values that are not `${...}`, restore them before committing. - -``` -GF_URL=https:// \ - GRAFANA_ADMIN_USER=... GRAFANA_ADMIN_PASSWORD=... scripts/pull-dashboards.sh -``` ### Reading blue/green logs in Grafana @@ -270,12 +97,10 @@ behind Nginx. - `compose.cms.yml` - Delta-only blue/green frontend/backend slots. - `cms.env.example` - sanitized variable-name-only production env template. -- `nginx/triangle-cms.conf` - host Nginx site template. +- `nginx/triangle-cms.conf` - **moved** to [`triangle-infrastructure`](https://github.com/DrexelTriangle/triangle-infrastructure) + (`roles/delta_cms_host/`). Install it with `playbooks/delta-host.yml`. - `nginx/triangle-cms-active-upstreams.conf.example` - generated include seed. -- `nginx/triangle-loki.conf` - read-only Loki endpoint for the Triangle Grafana. - `scripts/deploy.sh` - deploy exact SHA to inactive slot, switch, smoke test. -- `scripts/deploy-observability.sh` - sync and apply the observability stack and - its Nginx sites; runs as a Deploy Delta step after `deploy.sh`. - `scripts/rollback.sh` - explicit rollback to the other slot or named slot. ## One-Time Server Bootstrap @@ -292,7 +117,8 @@ a manual server task: `/usr/sbin/nginx -s reload`. 5. Place the host-only production env file at the path configured by `DELTA_CMS_ENV_FILE`. Do not put it in git. -6. Install `nginx/triangle-cms.conf` as an enabled Nginx site. +6. Install the host Nginx site with `triangle-infrastructure`'s + `playbooks/delta-host.yml` (see "Installing the Nginx site" below). 7. Create the narrow runtime-state directory and seed the active upstream include: @@ -323,29 +149,30 @@ such as `/etc/nginx/sites-available/triangle-cms.conf` remains root-owned. ### Installing the Nginx site -Steps 6-8 above, concretely. The repo is not checked out on Delta, so copy the -two files over first (from a workstation, at the repo root): +**This is Ansible's job now.** The site and the runner-owned state directory are +`roles/delta_cms_host` in [`triangle-infrastructure`](https://github.com/DrexelTriangle/triangle-infrastructure): ```bash -scp deploy/nginx/triangle-cms.conf \ - deploy/nginx/triangle-cms-active-upstreams.conf.example \ - @:/tmp/ +ansible-playbook playbooks/delta-host.yml --limit thetriangle-delta --check --diff +ansible-playbook playbooks/delta-host.yml --limit thetriangle-delta ``` -Then on Delta: +The role creates `/etc/nginx/triangle-cms` (`0750`, +`triangle-runner:triangle-runner`) and **deliberately does not manage +`active-upstreams.conf` inside it** — that file is written by `deploy.sh` on +every release and read back to determine the live slot, so anything that +templates it silently reverts production to the other slot. -```bash -sudo cp /tmp/triangle-cms.conf /etc/nginx/sites-available/triangle-cms.conf -sudo ln -sf /etc/nginx/sites-available/triangle-cms.conf /etc/nginx/sites-enabled/ +Seed the include once, by hand, from this repo's +`nginx/triangle-cms-active-upstreams.conf.example`: -sudo install -d -o triangle-runner -g triangle-runner -m 0750 /etc/nginx/triangle-cms +```bash sudo install -o triangle-runner -g triangle-runner -m 0644 \ - /tmp/triangle-cms-active-upstreams.conf.example \ + triangle-cms-active-upstreams.conf.example \ /etc/nginx/triangle-cms/active-upstreams.conf # The stock default site also matches `server_name _` and can win the vhost pick. sudo rm -f /etc/nginx/sites-enabled/default - sudo nginx -t && sudo systemctl reload nginx ``` diff --git a/deploy/compose.cms.yml b/deploy/compose.cms.yml index 0e80767..048cbee 100644 --- a/deploy/compose.cms.yml +++ b/deploy/compose.cms.yml @@ -53,7 +53,7 @@ x-backend-base: &backend-base EMBEDDINGS_URL: ${EMBEDDINGS_URL-http://embeddings:8000} volumes: # CephFS media tree (host). rw so the upload endpoint can store new files; - # host Nginx serves the same tree read-only (see deploy/nginx/triangle-cms.conf). + # host Nginx serves the same tree read-only (that site is in triangle-infrastructure). - ${MEDIA_HOST_PATH:-/mnt/cephfs/media}:/mnt/cephfs/media:rw healthcheck: test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/v1/health/db || exit 1"] diff --git a/deploy/compose.observability.yml b/deploy/compose.observability.yml deleted file mode 100644 index 1e11977..0000000 --- a/deploy/compose.observability.yml +++ /dev/null @@ -1,190 +0,0 @@ -# Observability stack (Loki + Promtail + Prometheus + Alertmanager + blackbox), -# deployed and -# upgraded independently of the CMS. This is intentionally a SEPARATE Compose -# project from compose.cms.yml: a CMS deploy must never `docker compose down` -# this stack, and an observability change must never touch the CMS slots or -# MariaDB. -# -# Promtail collects container logs through the Docker daemon API, and Prometheus -# scrapes each CMS slot's /metrics over the host gateway. Both therefore work -# regardless of the CMS living in a different Compose project on its own network. -# -# Usage (from this directory): -# docker compose -f compose.observability.yml --env-file up -d -# -# There is NO Grafana here. It was removed 2026-08-05: every dashboard now lives -# in the central Triangle Grafana, which consumes this stack's Prometheus and -# Loki as datasources over the Nginx endpoints in nginx/triangle-{prometheus, -# loki}.conf. Running a second Grafana on Delta only duplicated that, and the -# alerting it used to own now lives in Prometheus + Alertmanager, which is why -# removing it costs no alerting -- see deploy/mariadb/README.md. -# -# The dashboard JSON is still kept in observability/grafana/dashboards/ as the -# source of truth to import into the central Grafana. Its panels do NOT hard-bind -# to datasource UIDs any more: they resolve through the "prometheus_ds" and -# "loki_ds" template variables, because the central Grafana already had its own -# Prometheus and Loki and the Delta ones necessarily got different UIDs there. -# Both variables carry regex /-delta$/, so they bind to the Delta datasources on -# import rather than defaulting to the central pair and coming up empty. That -# regex is the one coupling left: a Delta datasource renamed out of the *-delta -# convention drops out of the dropdown and the panels go blank again. - -name: triangle-observability - -services: - loki: - image: grafana/loki:3.5.6 - restart: unless-stopped - command: ["-config.file=/etc/loki/config.yaml"] - # Published on loopback ONLY, and on 13100 rather than 3100, so that host - # Nginx can bind the natural :3100 for the outside world (see - # nginx/triangle-loki.conf) without a port clash. Loki runs with - # auth_enabled: false and has no authentication of its own, so nothing here - # may ever be published to a routable address directly -- Nginx is what adds - # basic auth and blocks the write path. - ports: - - "127.0.0.1:13100:3100" - healthcheck: - test: ["CMD", "loki", "-verify-config=true", "-config.file=/etc/loki/config.yaml"] - interval: 20s - timeout: 5s - retries: 5 - start_period: 10s - volumes: - - ../observability/loki-config.yml:/etc/loki/config.yaml:ro,z - - loki_data:/loki - networks: - - observability_net - - promtail: - image: grafana/promtail:3.5.6 - restart: unless-stopped - user: "0:0" - read_only: true - security_opt: - - no-new-privileges:true - cap_drop: - - ALL - command: ["-config.file=/etc/promtail/config.yml"] - depends_on: - loki: - condition: service_healthy - volumes: - - ../observability/promtail-config.yml:/etc/promtail/config.yml:ro,z - - promtail_positions:/tmp - # Read-only Docker socket: Promtail discovers containers and streams their - # logs through the daemon API rather than tailing files, because Delta's - # `local` log driver writes binary logs that file-tailing cannot parse. - # See the comment in promtail-config.yml. Note that :ro on the socket - # limits nothing the daemon does -- the API is fully privileged either - # way -- so this container is effectively root on the host, same as any - # socket-mounted agent. - - /var/run/docker.sock:/var/run/docker.sock:ro - networks: - - observability_net - - # Probes MaxScale's client listener from Delta. This is the only thing in the - # stack that can report the database tier being unreachable: MaxScale's own - # alert script cannot, because a dead MaxScale sends nothing. See - # ../observability/blackbox/blackbox.yml for why it is a TCP connect on :4006 - # rather than the admin API. - blackbox: - image: prom/blackbox-exporter:v0.25.0 - restart: unless-stopped - read_only: true - security_opt: - - no-new-privileges:true - cap_drop: - - ALL - command: ["--config.file=/etc/blackbox/blackbox.yml"] - volumes: - - ../observability/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro,z - # No ports published: only Prometheus talks to it, over the internal - # network. It probes OUTWARD to the MaxScale host, which needs no inbound - # exposure here. - networks: - - observability_net - - # Delivers Prometheus's database-tier alerts to Discord. Deliberately NOT - # Grafana's built-in alerting: Delta's Prometheus is only a DATASOURCE for the - # Triangle Grafana, so a Grafana-owned rule would disappear when the local - # instance is retired -- while the metric kept being scraped and the dashboards - # kept looking healthy. Keeping rule and delivery next to the data means - # neither depends on which Grafana is in front. - alertmanager: - image: prom/alertmanager:v0.28.1 - restart: unless-stopped - read_only: true - security_opt: - - no-new-privileges:true - cap_drop: - - ALL - command: - - "--config.file=/etc/alertmanager/alertmanager.yml" - - "--storage.path=/alertmanager" - volumes: - - ../observability/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro,z - # The Discord webhook, host-side and never in git. Alertmanager does no - # environment substitution in its config, so the URL is read from this - # path rather than injected -- same reasoning as MaxScale's alert.env. - - ${DISCORD_WEBHOOK_FILE:-/etc/triangle-observability/discord-webhook}:/etc/alertmanager/discord-webhook:ro - - alertmanager_data:/alertmanager - # Loopback only, and on 19093 to match the 1-prefixed convention the other - # loopback services here use. Alertmanager has no authentication of its own - # and its API can silence alerts, so it must never reach a routable address. - ports: - - "127.0.0.1:19093:9093" - networks: - - observability_net - - prometheus: - image: prom/prometheus:v3.6.0 - restart: unless-stopped - command: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - # Match Loki's two-week retention so the two halves of the dashboard - # cover the same window and neither outgrows the disk it shares with the - # CMS. - - "--storage.tsdb.retention.time=14d" - # Loopback only, on 19090 so host Nginx can bind the natural 9090 for the - # Triangle Grafana (see nginx/triangle-prometheus.conf). Prometheus has no - # authentication of its own, so this must never be published to a routable - # address directly. Note that --web.enable-admin-api and - # --web.enable-lifecycle are deliberately absent above: without them the - # delete-series, reload and quit endpoints do not exist at all. - ports: - - "127.0.0.1:19090:9090" - volumes: - - ../observability/prometheus/prometheus.delta.yml:/etc/prometheus/prometheus.yml:ro,z - - ../observability/prometheus/rules:/etc/prometheus/rules:ro,z - - prometheus_data:/prometheus - # Prometheus joins the CMS project's network read-only so it can scrape the - # slots by container name. - # - # The host gateway does NOT work here, which is easy to get wrong: the CMS - # slots publish to 127.0.0.1:8081/8082, loopback only, so a container - # reaching for the gateway address (172.17.0.1) gets connection refused. - # Joining the network reaches the containers on their own port 8080 instead, - # and needs no change to compose.cms.yml and no new host exposure. - networks: - - observability_net - - cms_net - -volumes: - alertmanager_data: - loki_data: - promtail_positions: - prometheus_data: - -networks: - observability_net: - driver: bridge - # The CMS project's network, created by compose.cms.yml. Declared external so - # this stack attaches to it rather than owning it -- nothing here can create, - # modify, or remove it. The cost of this is a real ordering dependency: the - # CMS stack must be up before this one starts, or Compose fails with a missing - # network instead of silently scraping nothing. - cms_net: - external: true - name: triangle-cms_triangle_net diff --git a/deploy/mariadb/README.md b/deploy/mariadb/README.md deleted file mode 100644 index 0814eb3..0000000 --- a/deploy/mariadb/README.md +++ /dev/null @@ -1,566 +0,0 @@ -# Production MariaDB — primary + MaxScale (native installs, off Delta) - -The database tier runs on its own hosts, **not** on Delta and **not** in Docker. -The CMS opens a single connection to the endpoint configured as -`DB_HOST`/`DB_PORT` in Delta's host-only `cms.env`; that endpoint is MaxScale, -which splits writes to the primary and reads to the replica. - -``` - CMS / app host (Delta) DB primary host DB replica host - ┌───────────────────────┐ ┌────────────────┐ ┌────────────────┐ - │ cms-blue cms-green │ writes │ DB1 │ GTID │ DB2 │ - │ │ │ │────────▶│ (server_id 1) │──────▶│ (server_id 2) │ - │ └────┬───┘ │ │ binlog+ACID │◀──ack─│ read_only │ - └───────────┼───────────┘ reads └────────────────┘ semi- └────────────────┘ - └──────────▶ MaxScale (rwsplit :4006) ─────sync──────▶ - mariadbmon: auto_failover -``` - -| Host | Address | CT | Role | Software | -| --- | --- | --- | --- | --- | -| `THETRIANGLE-DB1-LXC` | `10.248.40.154` | 108 | primary | MariaDB 11.8 LTS | -| `THETRIANGLE-DB2-LXC` | `10.248.40.155` | 111 | replica / failover target | MariaDB 11.8 LTS | -| `THETRIANGLE-MAXSCALE` | `10.248.40.183` | 109 | proxy | MaxScale 24.02 | - -All three are unprivileged **LXC containers**, 4 vCPU / 4 GB / 63 GB, installed -from apt. Docker was deliberately not used: it needs Proxmox-side nesting on an -unprivileged container and costs ~300 MB of a 4 GB budget. - -> **DB2 was created ~2026-08-03 with DB1's address**, `10.248.40.154`, and both -> were live on the bridge — which host you reached depended on which ARP entry -> won on your path. It has since been renumbered to `10.248.40.155` -> (MAC `bc:24:11:71:f9:f8`). Confirm `hostname` before trusting the output of -> anything you send to a DB host. - -## What this directory is - -These `.cnf` files are the **source of truth for what is installed on those -hosts** — edit here, copy up, restart the service. - -| File | Installed as | Host | -| --- | --- | --- | -| [primary.cnf](primary.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-primary.cnf` | DB1 | -| [replica.cnf](replica.cnf) | `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf` | DB2 | -| [../maxscale/maxscale.cnf](../maxscale/maxscale.cnf) | `/etc/maxscale.cnf` | MaxScale | -| [provision-db2.sh](provision-db2.sh) | run once on DB2 | DB2 | -| [setup-replica.sh](setup-replica.sh) | run once on DB2, after the above | DB2 | - -**The `70-` prefix is load-bearing.** MariaDB reads `mariadb.conf.d/` in lexical -order and Ubuntu's stock `50-server.cnf` sets `bind-address = 127.0.0.1`. A file -sorting before it cannot override that, and the primary would be unreachable -from MaxScale. - -## Installing - -MariaDB (DB1) is the standard `deb.mariadb.org` repo. Note that it carries -**only LTS lines** — 10.6, 10.11, 11.4, 11.8. 11.7 was a short-term release and -has been withdrawn, so 11.8 is the floor here; the configs were re-verified -against 11.8 before install. - -MaxScale needs its **own GPG key**, rotated 2025-12-10 — neither the MariaDB -Server key nor the Enterprise key will validate it: - -``` -https://supplychain.mariadb.com/MariaDB-MaxScale-GPG-KEY # BB2A36F3…5D87FACA8C27D14E -https://dlm.mariadb.com/repo/maxscale/latest/apt jammy main -``` - -## Secrets - -Nothing here is committed, and the passwords were generated on-host and have -never left the boxes. - -- **DB1** — `/root/triangle-db-credentials.env` (0600) holds the app, - replication, and MaxScale passwords. `APP_PASSWORD` is what Delta's `cms.env` - needs as `DB_PASSWORD`. -- **MaxScale** — `/etc/maxscale.cnf` is world-readable 0644, so backend - addresses and the service password live in - `/etc/maxscale.secrets.d/backend.env` (0640 `root:maxscale`) and reach the - process through the systemd drop-in - `/etc/systemd/system/maxscale.service.d/10-backend-env.conf`: - - ``` - PRIMARY_HOST=10.248.40.154 - REPLICA_HOST=10.248.40.155 - MARIADB_PORT=3306 - MAXSCALE_USER=maxscale - MAXSCALE_PASSWORD=... - REPL_USER=repl - REPL_PASSWORD=... # what mariadbmon writes into CHANGE MASTER - ``` - - All six are required. Removing any one leaves an unsubstituted `$VAR` in the - config and MaxScale will not start — worth remembering when editing this file - with `sed`, where a range delete can silently take an adjacent line with it. - -## Accounts on DB1 - -| Account | Grants | Why | -| --- | --- | --- | -| `triangle_user@10.248.40.168` | `ALL PRIVILEGES ON triangle.*` | **client-side auth**: lets Delta log in *through* MaxScale | -| `triangle_user@10.248.40.183` | `ALL PRIVILEGES ON triangle.*` | **backend-side auth**: lets MaxScale open the backend connection | -| `maxscale@10.248.40.183` | monitor, account reads, **+ failover admin** | promotes/demotes on failover | -| `repl@10.248.40.155` | `REPLICATION SLAVE` | DB2's replication link | -| `repl@10.248.40.154` | `REPLICATION SLAVE` | reverse link, for `auto_rejoin` after failover | - -`ALL PRIVILEGES` rather than DML-only because the CMS runs additive DDL -(`ADD COLUMN IF NOT EXISTS`) at startup. - -**Both `triangle_user` rows are required — do not "clean up" the Delta-scoped -one.** MaxScale authenticates a client against the backend user table by the -**client's own source address**, then connects to the backend from its own. Drop -either and the CMS gets `Error 1045`. See the warning in step 6. - -**Every one of these must exist on DB2 as well**, because on promotion DB2 serves -the application and MaxScale re-authenticates everything against *its* user -table. **`setup-replica.sh` does NOT copy them** — it dumps `--databases -triangle`, which excludes `mysql.*` — so accounts created *before* replication -started are absent on DB2, and only those created *after* replicate. Verify with -`SELECT CONCAT(user,'@',host) FROM mysql.user` on both hosts; a mismatch here -means a "successful" failover promotes a server nothing can log in to. To backfill -without polluting the replication stream, apply them on DB2 under -`SET SESSION sql_log_bin=0`. - -## Bringing up DB2 - -Ordered so the cluster is never in a state where automated failover could fire -at a replica that is not ready. **Do not enable `auto_failover` before step 6.** - -**Prerequisite — `tadmin` has no passwordless sudo on DB2.** It is in the `sudo` -group but no `NOPASSWD` rule exists, unlike DB1 and MaxScale. Either add one to -match the other two hosts, or run steps 1–3 from the Proxmox console. - -0. **Take a backup of DB1 first.** DB1 is still the only copy of the data (see - Notes). Automated failover is a mechanism for *promoting* a server, not a - substitute for being able to restore one. - ✅ *Done 2026-08-05: `/var/backups/triangle/triangle-predb2-20260805-005529.sql.gz` - on DB1 — 36.7 MB gzipped, 97.6 MB raw, 16 tables, `gzip -t` clean.* - -1–2. **Install MariaDB 11.8 and the replica config** — run - [provision-db2.sh](provision-db2.sh) on DB2 as root. It adds the - `deb.mariadb.org` 11.8 repo (same series as DB1, so a promotion is not also a - version change), installs `replica.cnf` as - `/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf`, restarts, and verifies - `server_id=2`, the bind address, and durability. It refuses to run unless the - hostname is `THETRIANGLE-DB2-LXC`. - - *Verified reachable from DB2: DB1:3306, MaxScale:4006, deb.mariadb.org:443.* - -3. **Create the replication accounts on DB1.** - ✅ *Done 2026-08-05: `repl@10.248.40.155` and `repl@10.248.40.154` both - created with `REPLICATION SLAVE`, password from - `/root/triangle-db-credentials.env`.* The `.154` one is so the **old primary - can replicate back** from DB2 after `auto_rejoin`; without it, a rejoin fails - on access denied. - - A pre-existing **`repl@'%'` is still present** and should be dropped — a - wildcard replication account defeats the point of host-scoping, and DB1's - 3306 is reachable from the whole subnet until the firewall step below. - -4. **Run [setup-replica.sh](setup-replica.sh) on DB2** (not on DB1). It takes a - GTID-consistent `--single-transaction --master-data=1` dump from DB1, restores - it, and starts replication. DB1 keeps serving throughout; the only lock is a - brief global read lock at the start, held just long enough to read the binlog - position: - - ``` - PRIMARY_HOST=10.248.40.154 \ - DUMP_USER=root DUMP_PASSWORD=... \ - REPL_USER=repl REPL_PASSWORD=... \ - sudo -E sh setup-replica.sh - ``` - - `DUMP_USER` must be a privileged account, **not** `repl`: `REPLICATION SLAVE` - grants the binlog stream but no table reads, so dumping as `repl` fails with - "SELECT command denied". The script is idempotent — it exits early if - replication is already running. - - Confirm `Slave_IO_Running: Yes`, `Slave_SQL_Running: Yes`, and - `Seconds_Behind_Master: 0` before continuing. - -5. **Turn on semi-sync and confirm it engages.** The setting is persisted in the - `.cnf` files, but it is dynamic, so it can be applied without a restart: - - ```sql - -- DB1 - SET GLOBAL rpl_semi_sync_master_enabled = ON; - SET GLOBAL rpl_semi_sync_master_wait_point = AFTER_SYNC; - SET GLOBAL rpl_semi_sync_master_timeout = 1000; - SET GLOBAL rpl_semi_sync_master_wait_no_slave = OFF; - -- DB2 - SET GLOBAL rpl_semi_sync_slave_enabled = ON; - STOP SLAVE IO_THREAD; START SLAVE IO_THREAD; -- required: the slave only - -- registers as semi-sync when - -- the IO thread reconnects - ``` - - Then on DB1, **verify rather than assume**: - - ```sql - SHOW STATUS LIKE 'Rpl_semi_sync_master_clients'; -- must be 1 <- the real check - SHOW STATUS LIKE 'Rpl_semi_sync_master_yes_tx'; -- rises on acknowledged commits - SHOW STATUS LIKE 'Rpl_semi_sync_master_no_tx'; -- must NOT keep climbing - SHOW STATUS LIKE 'Rpl_semi_sync_master_status'; -- ON, but see below - ``` - - **`clients` is the signal, not `status`.** With - `rpl_semi_sync_master_wait_no_slave=OFF`, `status` stays `ON` even while - nothing is acknowledging — verified by stopping DB2 on 2026-08-05: seven - commits completed unacknowledged (`no_tx` 0 → 7) with `status` still `ON` - and `clients` at 0. So `clients = 1` plus `yes_tx` rising is what shows the - guarantee is actually in force; `status = ON` on its own proves nothing. - -6. **Fence the write path — at the network, and ONLY at the network.** This is - what removes the split-brain risk, and it is a prerequisite for step 8, not - an optional hardening pass. Restrict 3306 on both DB hosts to MaxScale and - the DB peer: - - ```sh - ufw allow 22/tcp # BEFORE enabling - ufw allow from 10.248.40.183 to any port 3306 proto tcp # MaxScale - ufw allow from to any port 3306 proto tcp # replication - ufw --force enable - ``` - - Add the `allow` rules **before** `enable`, and do it **with Proxmox console - access open** — `pct enter ` gets you back in if you cut yourself off. - Do DB2 first: a mistake there costs replication, a mistake on DB1 costs the - site. - - > ⚠️ **Do NOT "fence" this by dropping `triangle_user@10.248.40.168`.** - > It looks like a direct-bypass account and it is not. **MaxScale - > authenticates a client against the backend's user table using the - > CLIENT's own source address**, so `triangle_user@` is precisely - > what lets the CMS log in *through* MaxScale; `triangle_user@` - > is what lets MaxScale then open the backend connection. **Both are - > required.** Dropping the Delta-scoped one closes no bypass and takes the - > site down with `Error 1045 Access denied for user - > 'triangle_user'@'10.248.40.168'` on any DB-backed route, while - > `/v1/health` keeps returning 200 — so it looks fine until someone loads a - > page. This was done and reverted on 2026-08-05. The firewall above is what - > actually removes the bypass, because Delta is no longer permitted to reach - > 3306 at all. - -7. **Grant MaxScale the failover privileges**, on **both** hosts: - - ```sql - GRANT REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD, - BINLOG ADMIN, CONNECTION ADMIN, REPLICATION MASTER ADMIN, - READ_ONLY ADMIN, SHOW DATABASES - ON *.* TO 'maxscale'@'10.248.40.183'; - ``` - - **`BINLOG ADMIN` is the one that is easy to miss and it breaks `auto_rejoin` - outright.** MariaDB 10.5+ split the old catch-all `SUPER` into discrete - privileges, so holding `SUPER` no longer implies it. Without it mariadbmon - cannot run `SET @@session.sql_log_bin=0` while demoting a returning primary - and loops forever on: - - ``` - Failed to prepare (demote) standalone server 'primary' for rejoin. - ``` - - Apply it on the current replica under `SET SESSION sql_log_bin=0` as well — - a grant made only on the primary reaches the replica by replication, but the - node that needs it during a rejoin is the one that is *not* currently - replicating. - -8. **Point MaxScale at DB2 and enable failover.** Set - `REPLICA_HOST=10.248.40.155` in `/etc/maxscale.secrets.d/backend.env` - (it is the RFC 5737 placeholder `192.0.2.2` until this is done), copy up the - new `maxscale.cnf`, and `systemctl restart maxscale`. - - Confirm `maxctrl list servers` shows `Master, Running` **and** - `Slave, Running` — a replica showing `Down` here means MaxScale still has the - placeholder address. - -9. **Test the failover before trusting it**, at a quiet hour, with the backup - from step 0 in hand. See below. - -## Verifying the split - -``` -maxctrl list servers # roles + replication lag -maxctrl list services # connections/routing -maxctrl list sessions # which backend a live session is on -``` - -Reads should land on the replica, writes — and reads-after-writes within a -session — on the primary. `causal_reads=local` makes MaxScale wait for the -replica to reach the write's GTID, so a session always sees its own writes -despite lag and **no app-level split is needed**. - -## Why automated failover is safe here - -`auto_failover` is **on**. The two standard objections to automating promotion -on a 2-node pair are real, and neither is answered by the failover setting -itself — they are answered by the replication mode and the network topology. - -**"Async replication loses the tail on promotion."** True, and it is why this -was left off originally. Replication is now **semi-synchronous with -`wait_point=AFTER_SYNC`**: DB1 does not commit to the storage engine, and -therefore never acknowledges to the CMS, until DB2 has the binlog event -durably. A promoted DB2 cannot be missing a write that the application was told -succeeded. `AFTER_COMMIT` — the MariaDB default — would *not* give this: it -makes the write visible to other sessions before the ack, which is exactly the -window that loses data. - -**"Two nodes can't tell 'primary is dead' from 'I can't see the primary'."** -Also true, and unfixable at that layer: a 2-node cluster has no quorum, so it -cannot vote. The risk is removed structurally instead, by making MaxScale the -**only** path to the databases — 3306 on both DB hosts is firewalled to -MaxScale and the DB peer, so nothing else can open a connection at all. Once -that holds, "MaxScale cannot reach DB1" implies "the CMS cannot reach DB1", so -promoting DB2 cannot result in two servers taking application writes. The -partition that would split-brain a quorum-less cluster instead just moves all -traffic to the promoted node, which is the desired outcome. - -Note this fencing is **purely a network property**. It is tempting to also -revoke the app's Delta-scoped grant as "a second write path", but that account -is not a bypass — it is how MaxScale authenticates the client — and removing it -only breaks the site. See step 6. - -**"The old primary comes back and clobbers things."** `gtid_strict_mode=ON` on -both nodes. A returning DB1 that diverged is *refused* by `auto_rejoin` and -sits there needing a human, rather than replicating conflicting history. - -### What is still not covered - -Honest limits, all of which need a third node to close: - -- **While DB2 is down, semi-sync degrades to async** and the zero-loss guarantee - lapses. This is deliberate — `rpl_semi_sync_master_wait_no_slave=OFF` — because - the alternative is DB1 stalling every commit for `rpl_semi_sync_master_timeout` - whenever DB2 is offline, turning a replica outage into a site outage. So - DB2-down-then-DB1-dies can still lose writes. **Alert on - `Rpl_semi_sync_master_clients == 0`, NOT on `Rpl_semi_sync_master_status`** — - with `wait_no_slave=OFF` the status stays `ON` while commits go - unacknowledged, so it is not evidence the guarantee holds. A `slave_down` / - `lost_slave` Discord alert (see Alerting) covers the same condition. -- **MaxScale is a single point of failure** and the sole arbiter. If it dies, - the CMS is down regardless of how healthy both databases are. Fencing the - write path to MaxScale deepens this dependency — that is the price of removing - split-brain without a quorum. -- **Failover is not backup.** Promotion protects against a host dying, not - against a bad migration or a `DROP TABLE`, both of which replicate to DB2 in - milliseconds. - -### Planned maintenance - -Use **switchover**, not failover — it demotes the old primary cleanly instead of -assuming it is dead: - -``` -maxctrl call command mariadbmon switchover MariaDB-Monitor replica primary -``` - -### Testing it - -Do this once, at a quiet hour, with a fresh DB1 backup, **before** relying on it: - -``` -maxctrl list servers # baseline: Master/Slave, Running -``` - -Stop MariaDB on DB1 (`systemctl stop mariadb`), then watch MaxScale's log at -`/var/log/maxscale/maxscale.log`. Within ~10s (`failcount` 5 × `monitor_interval` -2000ms) it should log `master_down`, promote DB2, and `maxctrl list servers` -should show DB2 as `Master, Running`. Confirm the CMS still serves and can write. - -Then start DB1 again and confirm `auto_rejoin` brings it back as -`Slave, Running`. **If it does not rejoin, that is often the safety net working -rather than a bug** — compare `@@gtid_current_pos` on both before forcing -anything. A returning node that is merely *behind* (its GTID is a prefix of the -new primary's) is cleanly rejoinable; one that is genuinely diverged is refused -by `gtid_strict_mode`, and that refusal is correct. - -Two non-divergence failures seen on the first real test, both worth recognising: - -- **`Failed to prepare (demote) standalone server for rejoin`, repeating every - monitor tick** — the monitor user is missing `BINLOG ADMIN`. See step 7. -- **Rejoin appears to succeed then instantly reverts** (`new_slave` followed by - `lost_slave` about two seconds later). The monitor built the replication link - with the wrong credentials; `replication_user`/`replication_password` default - to the *monitor* user, which is host-scoped to the MaxScale host and so does - not exist from the rejoining node. `maxscale.cnf` now sets them to `repl` - explicitly. MaxScale reports this only as `lost_slave` — the real error is on - the rejoining node, in `SHOW SLAVE STATUS` `Last_IO_Error` (1045). - **After fixing it, clear the stale connection** with - `STOP SLAVE; RESET SLAVE ALL;` on the rejoining node: while a replica - connection exists the node is no longer "standalone", so the monitor will not - rebuild it and simply leaves it broken. - -Finish by switching back with the `switchover` command above so DB1 is primary -again, and confirm semi-sync re-engages (`Rpl_semi_sync_master_clients = 1`) -on whichever node ends up primary. - -## Alerting - -`maxscale.cnf` sets `script=` on the monitor, so mariadbmon runs -[../maxscale/maxscale-alert.sh](../maxscale/maxscale-alert.sh) (installed as -`/usr/local/bin/maxscale-alert.sh`, 0755) as the `maxscale` user on each event -in `events=`. It fires within one monitor tick, rather than waiting for a -scrape, and it does not depend on Delta or the observability stack being up. - -It **always** appends to `/var/log/maxscale/failover-events.log` and posts to -Discord only if `/etc/maxscale.secrets.d/alert.env` (0640 `root:maxscale`) -supplies a webhook: - -``` -DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... -``` - -With that empty it degrades to log-only, so it is safe to install first. The -webhook must not go in `maxscale.cnf`, which is world-readable 0644. - -**Event names are not intuitive, and getting them wrong fails silently — you -only notice by the alert that never arrives.** In particular `new_slave` -(`[Running]→[Slave,Running]`, a standalone node joining) and `slave_up` -(`[Down]→[Slave,Running]`, a node returning from an outage) are different -transitions; listing only the former alerts on the way down and stays silent on -recovery. Confirm what actually fired with: - -```sh -grep "changed state" /var/log/maxscale/maxscale.log -``` - -### The MaxScale-down watchdog - -The script can only report events MaxScale is alive to observe, so it cannot -tell you MaxScale *itself* died — and because the write path is fenced to -MaxScale, that is a total outage. That gap is covered from **outside** the -database tier: - -``` -blackbox_exporter --TCP connect--> 10.248.40.183:4006 - | ^ - v scrape | (probe fails when MaxScale is down) - Prometheus ---> Alertmanager ---> Discord - rules/database.yml discord_configs -``` - -- [../../observability/blackbox/blackbox.yml](../../observability/blackbox/blackbox.yml) - — a plain TCP connect, no MySQL login, so no credentials are needed. -- [../../observability/prometheus/rules/database.yml](../../observability/prometheus/rules/database.yml) - — `MaxScaleUnreachable` and `MaxScaleProbeMissing`. -- [../../observability/alertmanager/alertmanager.yml](../../observability/alertmanager/alertmanager.yml) - — routing and the Discord receiver. - -**This deliberately does NOT live in Grafana.** Delta's Prometheus is a -*datasource* for the Triangle Grafana, not part of it, so a Grafana-owned alert -rule would vanish the moment the local Grafana is retired — while blackbox kept -probing, Prometheus kept scraping, and every dashboard kept looking healthy. The -only symptom would be an alert that never arrives. Keeping the rule next to the -data means it does not care which Grafana is in front. - -**Why `:4006` and not the admin API:** 4006 is the port the CMS actually uses, -so it tests the real dependency; the admin API (8989) would have to be opened to -Delta and it can reconfigure MaxScale, which is a much worse thing to expose. -MaxScale 24.02 serves no Prometheus endpoint anyway (`/metrics` and -`/v1/metrics` both 404). - -**DB1/DB2 are deliberately not probed** — their 3306 is firewalled to the -MaxScale host and the DB peer, so Delta cannot reach them by design and such a -target would alert forever. - -`MaxScaleProbeMissing` replaces Grafana's `noDataState: Alerting`: if the probe -series stops existing, nobody is watching the database tier, and that is not -allowed to fail open. - -> **The Discord webhook is read from a file, not an environment variable** — -> Alertmanager does no env substitution in its config. It lives at -> `/etc/triangle-observability/discord-webhook` on Delta (0644 inside a 0700 -> directory; the container reads the bind mount directly, so the tight directory -> costs nothing). Override the path with `DISCORD_WEBHOOK_FILE`. -> This is the **second** place the webhook is needed: the MaxScale host has its -> own copy in `/etc/maxscale.secrets.d/alert.env`, because the two alert paths -> run on different machines by design. - -**Grafana provisioning only adds and updates — it never deletes.** Removing an -alert rule or contact point from a provisioning file leaves it live in Grafana's -database. `alerting/maxscale.yml` is therefore kept as a deletion-only file. -**Order matters and getting it wrong crash-loops Grafana**: a contact point that -a notification policy still references cannot be deleted, and Grafana exits with - -``` -ProvisioningServiceImpl run error: contact points: -[alerting.notifications.contact-points.referenced] -``` - -rather than starting without its provisioning. Dropping the `policies:` block -does not remove the policy either, so `resetPolicies` must hand routing back to -the default before the contact points can go — and it takes a separate start to -apply, so this is a two-pass change. - -**Testing by changing the probe target leaves a stale series behind.** The old -`instance` keeps its last value inside Prometheus's 5-minute lookback, so the -rule goes on firing for several minutes after you revert. That is an artifact of -the test, not the alert: in normal operation the target never changes, -`probe_success` moves 1→0→1 on one series, and recovery is immediate. - -**The observability stack deploys automatically**, as a step of the Deploy Delta -workflow (`deploy/scripts/deploy-observability.sh`), so changing any file above -means merging to main — not copying anything to Delta by hand. It runs from -`~triangle-runner/triangle-observability`, synced from the runner's checkout, -because `actions/checkout` resets the checkout itself on every deploy and would -yank the bind-mount sources. The script restarts services only when the synced -config actually changed, which matters because `docker compose up -d` will NOT -restart a container when only a mounted file's contents differ. - -### Manual failover, without MaxScale - -On DB2: `STOP SLAVE; RESET SLAVE ALL;` then `SET GLOBAL read_only=OFF`. Repoint -`PRIMARY_HOST` and rebuild the old primary as a replica. - -## Notes / gotchas - -- **MaxScale mis-sizes its cache on LXC.** The container's cgroup `memory.max` - reads `max`, so MaxScale ignores lxcfs's 4 GB `/proc/meminfo` and takes 15% of - the *Proxmox host's* ~62 GB — it sized the query classifier cache at 9.38 GiB - on a 4 GB box. `query_classifier_cache_size=64M` is now pinned explicitly. - Any other memory-autotuning service on these containers has the same trap. -- **Both hosts need a DHCP reservation.** PVE wrote `eth0.network` with - `DHCP = no` and no `Address=`, so the addresses originally came from one-off - `dhclient` runs; DB1 silently lost its IPv4 when the lease expired. Static - overrides now live at `/etc/systemd/network/10-eth0-static.network` on both - (systemd-networkd applies the lexically first match, so `10-` beats PVE's - file and survives regeneration) — but the addresses came out of the DHCP pool - and can still be reassigned to someone else. MACs: DB1 - `bc:24:11:e5:cc:58`, MaxScale `bc:24:11:83:05:ea`. -- **DB1's firewall is open.** ufw is inactive and iptables is ACCEPT, so 3306 is - reachable from the whole subnet. It should be restricted to the MaxScale host - (and DB2 later); this was left alone deliberately to avoid an SSH lockout on a - remote host, so do it with console access available. -- **`mariadb-dump --gtid` on its own records NOTHING.** It only changes the - *format* of the position emitted by `--master-data`/`--dump-slave`, so without - one of those the dump carries no replication start position at all. A replica - seeded from such a dump begins at its own empty `gtid_slave_pos` — i.e. the - start of the primary's binlogs, which expire after 7 days - (`binlog_expire_logs_seconds`) — so replication either dies with error 1236 or - replays history on top of the restored data. `setup-replica.sh` carried this - bug and now passes `--master-data=1`, and hard-fails if the dump comes out - without an active `SET GLOBAL gtid_slave_pos=` line. It must be `=1`: `=2` - emits the same line **commented out**. Verified against 11.8.8. -- **DB1 is the only copy of the data until DB2 is replicating.** The old dev - container and its volume are gone. The pre-cutover dump on Delta at - `~/triangle-deploy/backups/triangle-precutover-20260729-2137.sql.gz` is a - point-in-time artifact, not a backup rotation — real backups are still owed, - and a replica is not one: a `DROP TABLE` reaches DB2 in milliseconds. -- **`tadmin` has no `NOPASSWD` sudo on DB2**, unlike DB1 and MaxScale. It is in - the `sudo` group, so an interactive password works, but every scripted step in - the bring-up runbook fails without a rule matching the other two hosts. Note - the failure mode is quiet: `ssh ... 'sudo -n ...' ` prints "sudo: a password is - required" to stderr while the pipeline reports success, because - `cmd | ssh ...` returns the *local* command's exit status. -- **DB2 has no `10-eth0-static.network` override.** Its address is pinned in the - CT config (`pct config 111`) but not inside the container, so it relies on a - single layer where DB1 and MaxScale have two. Add the override to match. -- `server_id` must be unique per node (1 primary / 2 replica); `gtid_domain_id` - must match (1 here). -- Schema changes: the CMS runs additive, idempotent migrations at startup that - replicate cleanly. Keep DDL expand-only so a rollback never drops a column the - old version needs. Large `ALTER`s run on the primary and replicate — watch - replica lag during them. -- `mariadb-dump` emits `SQL_MODE='NO_AUTO_VALUE_ON_ZERO'` in its preamble, which - is what preserves `articles.id = 0`. A hand-written `INSERT` script must set - it manually. diff --git a/deploy/mariadb/primary.cnf b/deploy/mariadb/primary.cnf deleted file mode 100644 index 1b2f620..0000000 --- a/deploy/mariadb/primary.cnf +++ /dev/null @@ -1,111 +0,0 @@ -[mysqld] -# --- Network ------------------------------------------------------------------ -# Listen on this host's internal NIC only. Ubuntu's stock 50-server.cnf binds to -# 127.0.0.1, which would make the primary unreachable from MaxScale, so this file -# MUST sort after it (installed as 70-triangle-primary.cnf). Local admin still -# works over the unix socket. Firewall 3306 to the MaxScale and replica hosts -# only — nothing else should reach the database directly. -bind-address = 10.248.40.154 - -# ============================================================================= -# Triangle CMS — MariaDB PRIMARY (writes) tuning + replication source config. -# Target host: THETRIANGLE-DB1-LXC, a dedicated 4 vCPU / 4 GB unprivileged LXC. -# Installed natively at /etc/mysql/mariadb.conf.d/70-triangle-primary.cnf — the -# 70- prefix is load-bearing, see the bind-address note above. -# ============================================================================= - -# --- InnoDB memory: the single most important knob --------------------------- -# The buffer pool caches data+indexes in RAM. The entire `triangle` dataset is -# ~92 MB, so 1G holds all of it many times over with room for growth; the rest of -# the 4 GB stays available for the OS, connection buffers, and per-thread -# sort/join memory. Raise toward 2G only if the dataset grows past ~1 GB — going -# bigger on a 4 GB host buys nothing and risks the OOM killer. -innodb_buffer_pool_size = 1G -# NOTE: innodb_buffer_pool_instances was REMOVED in MariaDB 10.6 and is not a -# system variable in 11.x. mariadbd silently accepts and ignores it as a startup -# option (verified against 11.7), so it is simply omitted here. -# -# innodb_redo_log_capacity is MySQL 8.0.30+, NOT MariaDB. Setting it makes -# mariadbd refuse to start ("unknown variable"). MariaDB sizes the redo log with -# innodb_log_file_size. 256M is far more than this write volume needs and keeps -# crash recovery fast. -innodb_log_file_size = 256M -innodb_log_buffer_size = 32M - -# --- Durability: full ACID on the primary (do not weaken) -------------------- -innodb_flush_log_at_trx_commit = 1 # fsync redo on every commit -sync_binlog = 1 # fsync binlog on every commit -innodb_flush_method = O_DIRECT -innodb_doublewrite = ON - -# --- Concurrency / caches ---------------------------------------------------- -max_connections = 200 # 2 CMS slots pool DB conns; leave headroom -thread_cache_size = 64 -table_open_cache = 4000 -table_definition_cache = 2000 -innodb_read_io_threads = 8 -innodb_write_io_threads = 8 -innodb_io_capacity = 1000 # assumes SSD; raise (e.g. 4000) on NVMe -innodb_io_capacity_max = 2000 -innodb_purge_threads = 4 - -# --- Protocol / timeouts ----------------------------------------------------- -max_allowed_packet = 64M -wait_timeout = 600 -interactive_timeout = 600 - -# --- Character set (matches the schema: utf8mb4) ----------------------------- -character-set-server = utf8mb4 -collation-server = utf8mb4_unicode_ci - -# --- Slow query log (kept inside the data volume; no extra mount needed) ----- -slow_query_log = 1 -slow_query_log_file = /var/lib/mysql/slow-query.log -long_query_time = 1 -log_queries_not_using_indexes = 0 - -# --- Replication: PRIMARY (binary log + GTID) -------------------------------- -server_id = 1 -log_bin = /var/lib/mysql/mysql-bin -log_bin_index = /var/lib/mysql/mysql-bin.index -binlog_format = ROW -binlog_row_image = MINIMAL -binlog_expire_logs_seconds = 604800 # keep 7 days of binlogs for replica catch-up -gtid_domain_id = 1 -gtid_strict_mode = ON -log_slave_updates = ON # lets the replica be chained / used for backups - -# --- Semi-synchronous replication: MASTER side -------------------------------- -# This is the setting that makes MaxScale's automated failover safe to turn on. -# Without it, replication is async: DB1 acknowledges a commit to the CMS before -# DB2 has seen it, so promoting DB2 silently discards the tail of the write -# stream. With it, a commit is not acknowledged until DB2 has the event. -rpl_semi_sync_master_enabled = ON -# AFTER_SYNC (not the AFTER_COMMIT default): wait for the replica's ack BEFORE -# committing to the storage engine, so a write is never visible to other -# sessions on DB1 until it is safe on DB2. AFTER_COMMIT makes the write visible -# first, which is precisely the window that loses data on promotion. -rpl_semi_sync_master_wait_point = AFTER_SYNC -# How long a commit waits for an ack before degrading to async, in ms. -rpl_semi_sync_master_timeout = 1000 -# OFF is load-bearing for availability. With the ON default, DB1 pays the full -# timeout above on EVERY commit while no replica is connected — so planned DB2 -# maintenance would add a second of latency to every write. OFF drops straight -# to async the moment there is no semi-sync replica, and back to semi-sync when -# DB2 reconnects. The tradeoff is real and must be understood: while DB2 is -# down, the zero-loss guarantee is NOT in force. -# -# MONITOR THIS WITH Rpl_semi_sync_master_clients, NOT ..._status. Verified by -# stopping DB2 on 2026-08-05: status stayed **ON** through seven unacknowledged -# commits while clients sat at 0 and no_tx climbed 0 -> 7. With -# wait_no_slave=OFF the master never enters the "off" state it would otherwise -# fall into, so status is not evidence the guarantee is holding. -# Rpl_semi_sync_master_clients == 0 -> nothing is acknowledging: ALERT -# Rpl_semi_sync_master_no_tx rising -> commits completing unacknowledged -# Rpl_semi_sync_master_yes_tx rising -> the guarantee is actually in force -rpl_semi_sync_master_wait_no_slave = OFF -# Slave side, inert while this node is the primary. Present so that after a -# failover and auto_rejoin — when this node comes back as a REPLICA of DB2 — it -# registers as a semi-sync client instead of silently leaving the new primary -# running asynchronously. Mirrors replica.cnf; both nodes carry both roles. -rpl_semi_sync_slave_enabled = ON diff --git a/deploy/mariadb/provision-db2.sh b/deploy/mariadb/provision-db2.sh deleted file mode 100755 index 6da526f..0000000 --- a/deploy/mariadb/provision-db2.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/sh -# One-time: install MariaDB 11.8 on DB2 and put the replica config in place. -# -# Run this ON DB2 (THETRIANGLE-DB2-LXC, 10.248.40.155) as root: -# -# sudo sh provision-db2.sh -# -# It does NOT start replication and does NOT touch MaxScale — it only gets a -# correctly-configured, correctly-bound MariaDB running. Run setup-replica.sh -# afterwards, then follow "Bringing up DB2" in README.md from step 5. -# -# Idempotent: safe to re-run. Existing repo/key/config are refreshed in place -# and apt skips packages already at the right version. -# -# NOTE ON ACCESS: tadmin has no NOPASSWD sudo on DB2 (unlike DB1 and MaxScale), -# so this cannot be driven over ssh non-interactively until such a rule exists: -# echo 'tadmin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/90-tadmin -# chmod 440 /etc/sudoers.d/90-tadmin -set -eu - -EXPECT_HOST=THETRIANGLE-DB2-LXC -EXPECT_ADDR=10.248.40.155 -SERIES=11.8 # LTS. deb.mariadb.org carries ONLY LTS lines. -CNF_SRC="$(dirname "$0")/replica.cnf" -CNF_DST=/etc/mysql/mariadb.conf.d/70-triangle-replica.cnf - -# --- Guards ------------------------------------------------------------------ -# DB2 was briefly live on DB1's address (10.248.40.154) and ssh gives no -# host-key warning when the ARP winner changes underneath you. Never let this -# script run against the primary: it would overwrite the primary's config with -# a read_only replica config. -[ "$(id -u)" = 0 ] || { echo "must run as root" >&2; exit 1; } -if [ "$(hostname)" != "$EXPECT_HOST" ]; then - echo "REFUSING: hostname is '$(hostname)', expected '$EXPECT_HOST'." >&2 - echo "You are not on DB2. Check which host you actually reached." >&2 - exit 1 -fi -if ! ip -4 addr show | grep -q "inet ${EXPECT_ADDR}/"; then - echo "REFUSING: ${EXPECT_ADDR} is not configured on this host." >&2 - exit 1 -fi -[ -f "$CNF_SRC" ] || { echo "cannot find replica.cnf next to this script" >&2; exit 1; } - -echo "==> Host verified: $(hostname) / ${EXPECT_ADDR}" - -# --- MariaDB apt repo --------------------------------------------------------- -# Note this is the SERVER key. MaxScale uses a different key entirely and is not -# installed here — see README.md. -echo "==> Adding MariaDB ${SERIES} repository" -apt-get update -qq -apt-get install -y -qq curl gpg apt-transport-https ca-certificates - -install -d -m 0755 /etc/apt/keyrings -curl -fsSL https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY \ - | gpg --dearmor --yes -o /etc/apt/keyrings/mariadb.gpg -chmod 0644 /etc/apt/keyrings/mariadb.gpg - -. /etc/os-release -cat > /etc/apt/sources.list.d/mariadb.list < Installing mariadb-server" -DEBIAN_FRONTEND=noninteractive apt-get install -y -qq mariadb-server mariadb-client -mariadbd --version - -# --- Config ------------------------------------------------------------------- -# The 70- prefix is load-bearing: Ubuntu's stock 50-server.cnf sets -# bind-address = 127.0.0.1 and mariadb.conf.d is read in lexical order, so a -# file sorting before it cannot override the bind and the replica would be -# unreachable from both MaxScale and the primary. -echo "==> Installing ${CNF_DST}" -install -o root -g root -m 0644 "$CNF_SRC" "$CNF_DST" - -echo "==> Restarting mariadb" -systemctl enable --now mariadb -systemctl restart mariadb - -# --- Verify ------------------------------------------------------------------- -echo "==> Verifying" -mariadb -N -B -e "SELECT @@hostname, @@server_id, @@read_only, @@gtid_domain_id, @@gtid_strict_mode" - -# server_id must differ from DB1's (1) or replication refuses to start. -SID=$(mariadb -N -B -e "SELECT @@server_id") -[ "$SID" = "2" ] || { echo "FAIL: server_id is ${SID}, expected 2" >&2; exit 1; } - -# The whole point of the 70- prefix. If this shows 127.0.0.1, the config did not -# take and nothing downstream will work. -echo "--- listening sockets ---" -ss -ltnp 2>/dev/null | grep 3306 || echo "WARNING: nothing listening on 3306" -if ! ss -ltn 2>/dev/null | grep -q "${EXPECT_ADDR}:3306\|0.0.0.0:3306\|\*:3306"; then - echo "FAIL: not bound to ${EXPECT_ADDR}:3306 — check ${CNF_DST} ordering" >&2 - exit 1 -fi - -# Durability must match the primary: DB2 is a failover target, not a read cache. -echo "--- durability (must be 1 / 1) ---" -mariadb -N -B -e "SELECT @@innodb_flush_log_at_trx_commit, @@sync_binlog" - -echo -echo "OK. MariaDB ${SERIES} is installed, bound to ${EXPECT_ADDR}, and read_only." -echo "NEXT: run setup-replica.sh on this host to seed from DB1 and start" -echo "replication, then continue at README.md 'Bringing up DB2' step 5." diff --git a/deploy/mariadb/replica.cnf b/deploy/mariadb/replica.cnf deleted file mode 100644 index 377df7a..0000000 --- a/deploy/mariadb/replica.cnf +++ /dev/null @@ -1,99 +0,0 @@ -[mysqld] -# ============================================================================= -# Triangle CMS — MariaDB REPLICA + FAILOVER TARGET (DB2) config. -# Host: THETRIANGLE-DB2-LXC, 10.248.40.155, CT 111 (4 vCPU / 4 GB / 63 GB). -# Runs on its OWN host (separate from DB1), replicating from the primary via -# GTID with semi-synchronous acknowledgement. Serves READ traffic, and is -# promoted to primary by MaxScale's mariadbmon on failover. -# Installed natively at /etc/mysql/mariadb.conf.d/70-triangle-replica.cnf — the -# 70- prefix is load-bearing, see bind-address below. -# ============================================================================= - -# --- Network ------------------------------------------------------------------ -# Ubuntu's stock 50-server.cnf binds 127.0.0.1, which would leave the replica -# unreachable from both MaxScale and the primary; this file must sort after it. -# Firewall 3306 to the MaxScale and primary hosts only. -bind-address = 10.248.40.155 - -# --- InnoDB memory ----------------------------------------------------------- -# Sized to match the primary; see the rationale in primary.cnf. Adjust the two -# together if the DB hosts are ever resized. -innodb_buffer_pool_size = 1G -# innodb_buffer_pool_instances was removed in MariaDB 10.6; see primary.cnf. -# innodb_redo_log_capacity is MySQL-only and DOES block startup. See primary.cnf. -innodb_log_file_size = 256M -innodb_log_buffer_size = 32M - -# --- Durability: FULL, same as the primary ------------------------------------ -# Deliberately NOT the relaxed (2 / 0) setting usually given to a read replica. -# Two reasons, both consequences of DB2 being a failover target rather than a -# pure read cache: -# 1. On promotion DB2 *becomes* the primary. Relaxed settings would silently -# leave production running without per-commit fsyncs until someone noticed. -# 2. Semi-sync's guarantee is only as strong as the ack. With sync_binlog=0 -# the replica acks once the event is in the OS page cache, so a power loss -# on DB2 discards writes the primary already told the client were durable. -# The CMS write volume is trivial (a newsroom, not a transaction processor), so -# the throughput this costs is not measurable here. -innodb_flush_log_at_trx_commit = 1 # fsync redo on every commit -sync_binlog = 1 # fsync binlog on every commit -innodb_flush_method = O_DIRECT -innodb_doublewrite = ON - -# --- Concurrency / caches ---------------------------------------------------- -max_connections = 300 # replica typically fields more read conns -thread_cache_size = 64 -table_open_cache = 4000 -table_definition_cache = 2000 -innodb_read_io_threads = 8 -innodb_write_io_threads = 8 -innodb_io_capacity = 1000 # raise on NVMe -innodb_io_capacity_max = 2000 - -# --- Protocol / timeouts ----------------------------------------------------- -max_allowed_packet = 64M -wait_timeout = 600 -interactive_timeout = 600 - -# --- Character set ----------------------------------------------------------- -character-set-server = utf8mb4 -collation-server = utf8mb4_unicode_ci - -# --- Slow query log ---------------------------------------------------------- -slow_query_log = 1 -slow_query_log_file = /var/lib/mysql/slow-query.log -long_query_time = 1 - -# --- Replication: REPLICA ---------------------------------------------------- -server_id = 2 # MUST be unique per node -read_only = ON # app read user has no SUPER, so this blocks stray writes -relay_log = /var/lib/mysql/relay-bin -relay_log_index = /var/lib/mysql/relay-bin.index -gtid_domain_id = 1 # must match the primary -gtid_strict_mode = ON -log_slave_updates = ON # own binlog → usable as a backup source / for chaining -log_bin = /var/lib/mysql/mysql-bin -binlog_format = ROW -binlog_expire_logs_seconds = 604800 -# Parallel apply keeps replication lag low under write bursts from the primary. -slave_parallel_threads = 4 -slave_parallel_mode = optimistic - -# --- Semi-synchronous replication --------------------------------------------- -# BOTH sides are enabled on BOTH nodes, because either node can hold either role -# after a failover. MariaDB only acts on the side matching its current role, so -# the master settings sit inert here until this node is promoted. -# -# Slave side: acknowledges each binlog event back to the primary, which is what -# lets automated failover promote DB2 without losing acknowledged commits. Must -# be ON here for the primary's AFTER_SYNC wait to ever be satisfied — if this is -# OFF, the primary just times out and degrades to async on every commit. -rpl_semi_sync_slave_enabled = ON -# Master side: inert while this node is a replica, load-bearing the moment it is -# promoted. Without it a failover silently drops to asynchronous replication — -# losing the zero-data-loss guarantee at exactly the moment you have just proven -# you need it. Values must match primary.cnf; see the rationale there. -rpl_semi_sync_master_enabled = ON -rpl_semi_sync_master_wait_point = AFTER_SYNC -rpl_semi_sync_master_timeout = 1000 -rpl_semi_sync_master_wait_no_slave = OFF diff --git a/deploy/mariadb/setup-replica.sh b/deploy/mariadb/setup-replica.sh deleted file mode 100755 index cb868a2..0000000 --- a/deploy/mariadb/setup-replica.sh +++ /dev/null @@ -1,141 +0,0 @@ -#!/bin/sh -# One-time: provision DB2 as a read replica of DB1 and start GTID replication. -# -# Run this ON THE REPLICA HOST (as root, or under sudo) after MariaDB is -# installed there natively and replica.cnf is in place as -# /etc/mysql/mariadb.conf.d/70-triangle-replica.cnf: -# -# PRIMARY_HOST=10.248.40.154 \ -# DUMP_USER=root DUMP_PASSWORD=... \ -# REPL_USER=repl REPL_PASSWORD=... \ -# sh setup-replica.sh -# -# Local admin goes over the unix socket as root, so no local password is needed -# (unix_socket auth is the default for root on a native apt install). -# -# Idempotency: it exits early if replication is already running, so re-running -# after a hiccup is safe. The dump records the exact primary position via -# --gtid --master-data=1 (BOTH are needed — see the dump step below), so -# START SLAVE ... MASTER_USE_GTID=slave_pos resumes with no gaps or duplicates. -set -eu - -: "${PRIMARY_HOST:?PRIMARY_HOST is required (DB1's address)}" -PRIMARY_PORT="${PRIMARY_PORT:-3306}" -: "${REPL_USER:?REPL_USER is required}" -: "${REPL_PASSWORD:?REPL_PASSWORD is required}" -MARIADB_DATABASE="${MARIADB_DATABASE:-triangle}" - -# The dump is taken as a SEPARATE, privileged account — NOT as REPL_USER. -# `repl` holds only REPLICATION SLAVE, which grants the binlog stream but no -# table reads, so dumping as `repl` fails with "SELECT command denied". Use -# root, or any account with SELECT/SHOW VIEW/TRIGGER/EVENT on the database. -: "${DUMP_USER:?DUMP_USER is required (an account that can SELECT the data, e.g. root)}" -: "${DUMP_PASSWORD:?DUMP_PASSWORD is required}" - -DUMP_FILE="$(mktemp /var/tmp/primary-dump.XXXXXX.sql)" -chmod 600 "${DUMP_FILE}" -trap 'rm -f "${DUMP_FILE}"' EXIT INT TERM - -local_sql() { mariadb "$@"; } - -if local_sql -N -e "SHOW SLAVE STATUS\G" 2>/dev/null | grep -q "Slave_IO_Running: Yes"; then - echo "replica already running; nothing to do" - exit 0 -fi - -echo "waiting for primary ${PRIMARY_HOST}:${PRIMARY_PORT} ..." -until mariadb -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP_PASSWORD}" \ - -e "SELECT 1" >/dev/null 2>&1; do - sleep 3 -done - -echo "dumping ${MARIADB_DATABASE} from primary (GTID-consistent) ..." -# --master-data=1 is REQUIRED and is not optional decoration. `--gtid` on its own -# emits NOTHING: in MariaDB it only changes the FORMAT of the position recorded -# by --master-data/--dump-slave, so `--gtid` without one of those produces a dump -# carrying no replication start position at all. The replica would then begin -# from its own empty gtid_slave_pos, i.e. from the very start of the primary's -# binlogs — which are expired after binlog_expire_logs_seconds (7 days), so -# replication either dies with error 1236 or replays history on top of the -# restored data. Verified against 11.8.8: --gtid alone emits no gtid line; -# --master-data=1 emits an ACTIVE `SET GLOBAL gtid_slave_pos='1-1-...';`, which -# is what the CHANGE MASTER ... MASTER_USE_GTID=slave_pos below consumes. -# It must be =1, not =2 — =2 comments that same line out. -# -# --single-transaction keeps the dump consistent without locking InnoDB tables, -# but note that combining it with --master-data does take a brief global read -# lock at the very start, just long enough to read the binlog position. It is -# milliseconds, not the length of the dump, and the primary serves throughout. -mariadb-dump -h"${PRIMARY_HOST}" -P"${PRIMARY_PORT}" -u"${DUMP_USER}" -p"${DUMP_PASSWORD}" \ - --single-transaction --gtid --master-data=1 --routines --triggers --events \ - --databases "${MARIADB_DATABASE}" > "${DUMP_FILE}" - -# Fail loudly here rather than starting replication from a bogus position. -if ! grep -q "^SET GLOBAL gtid_slave_pos=" "${DUMP_FILE}"; then - echo "ERROR: dump contains no active 'SET GLOBAL gtid_slave_pos=' line." >&2 - echo "Replication would start from the wrong position. Check that" >&2 - echo "${DUMP_USER} holds RELOAD/BINLOG MONITOR on the primary." >&2 - exit 1 -fi - -echo "loading dump into replica ..." -# read_only=ON is set in replica.cnf; root is exempt (it holds SUPER), so the -# restore lands without having to relax it. -local_sql < "${DUMP_FILE}" - -echo "starting replication ..." -local_sql < to 'close a bypass' — MaxScale" -echo " authenticates clients by their own source address, so that account" -echo " is how the CMS logs in through MaxScale. Dropping it just breaks" -echo " the site. Have console access open." -echo -echo " 5. On the MaxScale host, set REPLICA_HOST in" -echo " /etc/maxscale.secrets.d/backend.env to this node (it is the RFC 5737" -echo " placeholder 192.0.2.2 until then), grant the maxscale user" -echo " REPLICATION SLAVE ADMIN, SUPER, PROCESS, EVENT, SET USER, RELOAD," -echo " copy up maxscale.cnf, and 'systemctl restart maxscale'." -echo -echo " 6. Confirm 'maxctrl list servers' shows Master, Running and Slave," -echo " Running — then test a real failover before relying on it." diff --git a/deploy/maxscale/maxscale-alert.sh b/deploy/maxscale/maxscale-alert.sh deleted file mode 100755 index 03e069e..0000000 --- a/deploy/maxscale/maxscale-alert.sh +++ /dev/null @@ -1,132 +0,0 @@ -#!/bin/sh -# Alert hook for MaxScale's mariadbmon, invoked via `script=` in maxscale.cnf. -# Installed on THETRIANGLE-MAXSCALE as /usr/local/bin/maxscale-alert.sh (0755). -# -# MaxScale runs this as the `maxscale` user on every event listed in `events=`, -# substituting $EVENT/$INITIATOR/$NODELIST/$PARENT before exec. It is bounded by -# script_timeout (90s) — if it hangs, the monitor blocks, so every outbound call -# here MUST have its own timeout. -# -# Design notes: -# - It ALWAYS writes the local log first and posts to Discord second. A failover -# that happened is a fact worth keeping even if Discord is unreachable, and -# the log is what you correlate against maxscale.log afterwards. -# - It exits 0 unconditionally. A non-zero exit here is noise in maxscale.log -# and there is nothing MaxScale can usefully do about a failed notification. -# - Until DISCORD_WEBHOOK_URL is configured it degrades to log-only rather than -# failing, so it is safe to install before the webhook exists. -# -# The webhook lives in /etc/maxscale.secrets.d/alert.env (0640 root:maxscale), -# NOT here and NOT in maxscale.cnf, which is world-readable 0644: -# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... -# -# Discord's NATIVE payload is used ({"content": ...}), not Slack's -# ({"text": ...}). Discord will accept Slack-shaped payloads if you append -# /slack to the webhook URL, but that path silently drops formatting it does not -# understand, so the native field is the honest choice. Note the markup differs -# from Slack: bold is **text**, not *text*. -set -u - -EVENT="${1:-unknown}" -INITIATOR="${2:-unknown}" -NODELIST="${3:-}" -PARENT="${4:-}" - -TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -LOG=/var/log/maxscale/failover-events.log - -# --- Always record locally, before anything that can fail -------------------- -printf '%s event=%s initiator=%s nodes=%s parent=%s\n' \ - "$TS" "$EVENT" "$INITIATOR" "$NODELIST" "$PARENT" >> "$LOG" 2>/dev/null - -# --- Classify ----------------------------------------------------------------- -# new_master is the one that means an automated failover actually promoted a -# node. lost_slave/slave_down matter more than they look: with only two nodes, -# losing the replica means semi-sync degrades to async -# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-data-loss guarantee -# stops holding until it returns. -# Colours are Discord's own palette, as decimal, so the embed's left bar reads -# at a glance without anyone parsing the text: red = act now, yellow = degraded, -# green = over, blurple = informational. -RED=15548997 # ED4245 -YELLOW=16705372 # FEE75C -GREEN=5763719 # 57F287 -BLURPLE=5793266 # 5865F2 - -case "$EVENT" in - new_master) - TITLE="Failover — replica promoted to primary"; COLOR=$RED - NOTE="Writes are going to the new primary. Check the old one before it rejoins." ;; - master_down|lost_master) - TITLE="Primary database is down"; COLOR=$RED - NOTE="Writes are failing until a promotion completes." ;; - slave_down|lost_slave) - TITLE="Replica is down"; COLOR=$YELLOW - NOTE="Semi-sync dropped to async — writes are no longer guaranteed on two nodes." ;; - master_up|slave_up|new_slave|server_up) - # slave_up ([Down]->[Slave,Running]) is the recovery counterpart of - # slave_down; new_slave ([Running]->[Slave,Running]) is a standalone node - # joining. Both belong here or an outage never reports that it ended. - TITLE="Database tier back to normal"; COLOR=$GREEN - NOTE="Replication is healthy again." ;; - server_down) - TITLE="Database backend is down"; COLOR=$YELLOW - NOTE="A backend stopped answering the monitor." ;; - *) - TITLE="MaxScale event"; COLOR=$BLURPLE - NOTE="" ;; -esac - -# MaxScale renders addresses as [10.248.40.155]:3306. The brackets are its -# IPv6-safe formatting and carry no meaning for an IPv4 pair, so strip them — -# they are pure noise in a notification someone reads on a phone. -INITIATOR_D=$(printf '%s' "$INITIATOR" | tr -d '[]') -NODELIST_D=$(printf '%s' "${NODELIST:-}" | tr -d '[]' | sed 's/,/, /g') - -# --- Post to Discord, if configured ------------------------------------------ -[ -r /etc/maxscale.secrets.d/alert.env ] && . /etc/maxscale.secrets.d/alert.env -[ -n "${DISCORD_WEBHOOK_URL:-}" ] || exit 0 - -# JSON string escaping: backslash and quote, then fold real newlines into \n. -# Literal newlines inside a JSON string are invalid and Discord rejects the -# payload outright. The ':a;N;$!ba' idiom slurps the whole input first. -json_esc() { - printf '%s' "$1" \ - | sed 's/\\/\\\\/g; s/"/\\"/g' \ - | sed ':a;N;$!ba;s/\n/\\n/g' -} - -# An embed rather than a plain message. Discord renders it with a coloured bar, -# a real title and aligned fields, which is both easier to triage at a glance -# and unmistakably a machine notice rather than someone talking. The `parent` -# value is dropped from the display -- it is usually "n/a" and never actionable -# -- but it is still written to the log above, where it costs nothing. -PAYLOAD=$(cat </dev/null 2>&1 \ - || printf '%s event=%s discord_post_failed\n' "$TS" "$EVENT" >> "$LOG" 2>/dev/null - -exit 0 diff --git a/deploy/maxscale/maxscale.cnf b/deploy/maxscale/maxscale.cnf deleted file mode 100644 index 4691295..0000000 --- a/deploy/maxscale/maxscale.cnf +++ /dev/null @@ -1,151 +0,0 @@ -# MaxScale — read/write splitting proxy for Triangle CMS. -# -# Deployed to THETRIANGLE-MAXSCALE (10.248.40.183) as /etc/maxscale.cnf, from a -# native apt install of MaxScale 24.02 — see deploy/mariadb/README.md. Backend -# addresses and credentials come from environment variables -# (substitute_variables=true): PRIMARY_HOST, REPLICA_HOST, MARIADB_PORT, -# MAXSCALE_USER, MAXSCALE_PASSWORD, supplied by the systemd drop-in -# /etc/systemd/system/maxscale.service.d/10-backend-env.conf, which reads -# /etc/maxscale.secrets.d/backend.env (0640 root:maxscale). This file itself is -# world-readable 0644, so no secret may be written into it directly. - -[maxscale] -threads=auto -# Allow $ENV references below to be replaced with environment variables. On the -# native install these come from a root-only systemd EnvironmentFile -# (/etc/maxscale.secrets.d/backend.env) so the service password is never written -# into this world-readable file. -substitute_variables=true - -# MUST be set explicitly on a container host. MaxScale defaults this to ~15% of -# system memory, but inside an LXC whose cgroup limit reads "max" it sees the -# Proxmox HOST's RAM, not the container's — on a 4 GB container it sized the -# cache at 9.38 GiB, which would invite the OOM killer. The CMS issues a small, -# highly repetitive query set, so a small cache is all it needs. -query_classifier_cache_size=64M - -# --- Backend servers (on their own physical hosts) --------------------------- -[primary] -type=server -address=$PRIMARY_HOST -port=$MARIADB_PORT -protocol=MariaDBBackend - -[replica] -type=server -address=$REPLICA_HOST -port=$MARIADB_PORT -protocol=MariaDBBackend - -# --- Monitor: discovers which node is primary vs replica --------------------- -[MariaDB-Monitor] -type=monitor -module=mariadbmon -servers=primary,replica -user=$MAXSCALE_USER -password=$MAXSCALE_PASSWORD -# Credentials mariadbmon writes into CHANGE MASTER when it promotes, demotes or -# rejoins a node. MUST be set explicitly: when omitted these DEFAULT TO THE -# MONITOR USER above, and mariadbmon then builds the replication link as -# maxscale@ — an account that does not exist, because -# the monitor user is host-scoped to the MaxScale host. The symptom is a rejoin -# that appears to succeed and then immediately flips back -# ("new_slave" followed by "lost_slave" ~2s later), with the real cause only -# visible on the rejoining node as SHOW SLAVE STATUS Last_IO_Error 1045. -# `repl` is host-scoped to BOTH DB hosts precisely so either direction works. -replication_user=$REPL_USER -replication_password=$REPL_PASSWORD -monitor_interval=2000ms - -# --- Alerting ---------------------------------------------------------------- -# Fired by the monitor on each event below, as the `maxscale` user. See -# deploy/maxscale/maxscale-alert.sh — it always writes -# /var/log/maxscale/failover-events.log and posts to Discord only if -# /etc/maxscale.secrets.d/alert.env supplies a webhook, so it is safe to install -# before that exists. The script must stay well inside script_timeout or it -# blocks the monitor; the curl inside it is capped at 10s. -# -# NOTE this can only report what MaxScale is alive to observe — it cannot tell -# you MaxScale itself has died. That needs an external check. -script=/usr/local/bin/maxscale-alert.sh $EVENT $INITIATOR $NODELIST $PARENT -script_timeout=90000ms -# Explicit rather than the default "all": these are the transitions worth waking -# someone for. slave_down/lost_slave are included deliberately — on a two-node -# pair, losing the replica silently degrades semi-sync to async, so it is the -# moment the zero-data-loss guarantee stops holding. -# `slave_up` and `new_slave` are BOTH needed and are not the same transition: -# new_slave is [Running]->[Slave,Running] (a standalone node joining), slave_up -# is [Down]->[Slave,Running] (a node coming back from an outage). Listing only -# new_slave means an outage alerts on the way down and stays silent on recovery. -events=master_down,master_up,lost_master,new_master,slave_down,slave_up,lost_slave,new_slave,server_down,server_up - -# --- Automated failover ------------------------------------------------------ -# ON. This was previously off because promoting a replica on a 2-node ASYNC pair -# loses the un-replicated tail of the write stream and can split-brain. Both of -# those are addressed elsewhere rather than by leaving failover off: -# -# Data loss -> replication is now SEMI-SYNCHRONOUS with wait_point=AFTER_SYNC -# (see deploy/mariadb/primary.cnf). DB1 does not acknowledge a -# commit until DB2 holds the event durably, so a promotion -# cannot discard an acknowledged write. -# Split-brain -> a 2-node cluster has no quorum and therefore cannot VOTE on -# who is alive. Instead the write path is fenced AT THE NETWORK: -# 3306 on both DB hosts is firewalled to this host and the DB -# peer only, so MaxScale is the ONLY route to the databases. -# "MaxScale cannot see DB1" therefore also means "the CMS cannot -# see DB1", and promoting DB2 cannot produce two servers taking -# application writes. -# Do NOT try to fence this by dropping triangle_user@: -# MaxScale authenticates a client against the backend user table -# using the CLIENT's own source address, so that account is what -# lets Delta log in THROUGH MaxScale. Dropping it does not close -# a bypass, it just breaks the site. See deploy/mariadb/README.md. -# Divergence -> gtid_strict_mode=ON on both nodes, so a rejoining old primary -# that diverged is REFUSED by auto_rejoin instead of silently -# corrupting the dataset. -# -# Residual, accepted: while DB2 is down, semi-sync degrades to async -# (rpl_semi_sync_master_wait_no_slave=OFF) and the zero-loss guarantee lapses; -# and MaxScale itself remains a single point of failure. Closing either needs a -# third node. See deploy/mariadb/README.md. -auto_failover=true -# Consecutive failed monitor passes before the master is declared down. -# 5 x 2000ms = ~10s of confirmed unreachability, which rides out a container -# migration or a brief network blip without promoting. -failcount=5 -failover_timeout=90000ms -# Rejoin a returning old primary as a replica automatically; gtid_strict_mode -# makes this refuse rather than corrupt if it diverged. -auto_rejoin=true -# Force every non-primary node read_only, so the demoted server cannot take -# writes even if something reaches it directly. -enforce_read_only_slaves=true -# Two-node primary/replica and nothing else — reject any topology MaxScale did -# not expect rather than trying to reason about it. -enforce_simple_topology=true - -# --- Service: read/write split router ---------------------------------------- -[Read-Write-Split] -type=service -router=readwritesplit -servers=primary,replica -user=$MAXSCALE_USER -password=$MAXSCALE_PASSWORD -# Route reads to the replica but guarantee a session sees its own prior writes -# (MaxScale waits for the replica to reach the write's GTID). Fixes replica-lag -# read-after-write without any app change. -causal_reads=local -causal_reads_timeout=10s -# Replay an in-flight transaction on the new primary if a backend drops, so a -# failover doesn't surface transient errors to the CMS. -transaction_replay=true -master_reconnection=true -# If the primary is unreachable, fail writes but keep serving reads. -master_failure_mode=fail_on_write - -# --- Listener the CMS connects to -------------------------------------------- -[Read-Write-Listener] -type=listener -service=Read-Write-Split -protocol=MariaDBClient -port=4006 diff --git a/deploy/nginx/triangle-cms.conf b/deploy/nginx/triangle-cms.conf deleted file mode 100644 index 9273794..0000000 --- a/deploy/nginx/triangle-cms.conf +++ /dev/null @@ -1,96 +0,0 @@ -# Install as a host Nginx site, for example: -# /etc/nginx/sites-available/triangle-cms.conf -# and symlink into sites-enabled. The active slot include is generated by the -# deploy/rollback scripts. -# -# Initial deployment intentionally uses HTTP and server_name _. It will work via -# Delta's VPN IP or hostname. When cms.thetriangle.org is ready, replace -# server_name and add HTTPS/TLS termination here; the backend should remain -# internal HTTP behind Nginx. - -server { - listen 80; - server_name _; - - include /etc/nginx/triangle-cms/active-upstreams.conf; - - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - - location = /healthz { - access_log off; - default_type text/plain; - return 200 "ok\n"; - } - - location = /swagger { - return 301 /swagger/; - } - - location /swagger/ { - proxy_pass $triangle_cms_backend; - } - - location = /v1 { - proxy_pass $triangle_cms_backend; - } - - location /v1/ { - # POST /v1/media accepts up to MEDIA_MAX_UPLOAD_BYTES (90 MiB by - # default -- the legacy corpus has full-res camera originals near 77 - # MiB). Nginx's own default is 1m, so without this it rejects any - # ordinary phone photo with a 413 before the request reaches the CMS, - # and the app-side limit never gets a say. Keep this at or above the - # backend's limit so the backend is the one that decides. - client_max_body_size 91m; - # A 90 MiB body over a slow uplink takes minutes; the stock 60s applies - # per read, but raise the ceilings so a stalled-but-alive upload is not - # killed mid-flight. - client_body_timeout 300s; - proxy_read_timeout 300s; - proxy_send_timeout 300s; - proxy_pass $triangle_cms_backend; - } - - # Legacy WordPress media, migrated to CephFS. Static, read-only, images only. - # The CMS upload endpoint (backend container) is the only writer; Nginx just - # reads. There is no PHP/FastCGI handler here, so nothing can execute even if - # a script file slipped into the corpus. - location /wp-content/ { - root /mnt/cephfs/media; # /wp-content/x -> /mnt/cephfs/media/wp-content/x - try_files $uri =404; # never fall through to the app - autoindex off; - limit_except GET HEAD { deny all; } - - # Refuse to serve executables/scripts outright (there were .php/.exe - # web-shells on the WP origin; the rsync excluded them, this is defense - # in depth). - location ~* \.(php\d?|phtml|phar|cgi|pl|py|sh|exe|asp|aspx|jsp)$ { return 403; } - - add_header X-Content-Type-Options "nosniff" always; - add_header Content-Security-Policy "default-src 'none'; img-src 'self'" always; - # Filenames encode the exact size, so files are immutable. Set this only - # via add_header -- `expires` would emit a second, weaker Cache-Control - # and CDNs disagree about which duplicate wins. - # - # Deliberately NOT `always`: that flag also stamps 30-day-immutable onto - # 404s, and Cloudflare then pins "this file does not exist" at the edge - # for a month. Any file added to the corpus after something first - # requested it stays invisible until the TTL expires or someone purges -- - # a migrated image that is provably on disk and served fine by this nginx - # still 404s publicly, which reads as a failed copy rather than a cache - # hit. Without the flag add_header applies only to 2xx/3xx, so misses - # fall back to Cloudflare's short default 404 TTL and self-heal. - add_header Cache-Control "public, max-age=2592000, immutable"; - } - - # Never serve dotfiles (keep ACME http-01 working if it is ever added here). - location ~ /\.(?!well-known) { deny all; } - - location / { - proxy_pass $triangle_cms_frontend; - } -} diff --git a/deploy/nginx/triangle-loki.conf b/deploy/nginx/triangle-loki.conf deleted file mode 100644 index bc56451..0000000 --- a/deploy/nginx/triangle-loki.conf +++ /dev/null @@ -1,89 +0,0 @@ -# Read-only, password-protected Loki endpoint for the Triangle Grafana. -# -# Install as a host Nginx site, for example: -# /etc/nginx/sites-available/triangle-loki.conf -# and symlink into sites-enabled. -# -# Why this exists: Loki in compose.observability.yml runs with -# `auth_enabled: false`. That switch turns off Loki's *multi-tenancy*, not -# authentication -- Loki has no authentication at all, in any configuration. -# Publishing container port 3100 to a routable address would therefore hand -# anyone on the VPN unauthenticated read AND write access to every log line the -# CMS has ever emitted. So the container is published to 127.0.0.1:13100 and -# this site is the only way in. -# -# One credential pair covers this and the Prometheus endpoint (triangle-prometheus.conf). -# -# Grafana datasource config on the Triangle Grafana side: -# Type: Loki -# URL: http://:3100 -# Auth: Basic auth, with the user/password from the htpasswd file below -# No path prefix is needed. Loki's own API paths already start with /loki/ -# (/loki/api/v1/query_range and friends), which is what `location /loki/` -# below matches. -# -# Setup: -# 1. Create the credentials (htpasswd ships in apache2-utils / httpd-tools): -# sudo htpasswd -B -c /etc/nginx/triangle-observability.htpasswd triangle-grafana -# Store the password in the team's secret manager alongside the Grafana -# admin credentials; it is the only thing protecting this endpoint. -# 2. sudo chown root:www-data /etc/nginx/triangle-observability.htpasswd -# sudo chmod 0640 /etc/nginx/triangle-observability.htpasswd -# 3. sudo nginx -t && sudo nginx -s reload -# -# This listens on a dedicated port rather than adding a /loki/ location to the -# CMS site, so that a mistake here can never affect the CMS vhost, and so the -# datasource URL is the port a Grafana admin already expects Loki to be on. -# -# When TLS lands on Delta, terminate it here too -- basic auth over plain HTTP -# sends the password in cleartext on every query, and it is only tolerable -# while this endpoint is VPN-internal. - -server { - listen 3100; - server_name _; - - # Loki's query responses are large and slow relative to a normal API call; - # a wide dashboard range can legitimately take minutes. - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_read_timeout 300s; - proxy_send_timeout 300s; - proxy_buffering off; - - # Promtail pushes to Loki over the Compose network, not through Nginx, so - # the ingestion path never needs to be reachable from outside. Refuse it - # explicitly: a Grafana datasource only ever reads, and leaving push open - # would let any holder of the read password forge or flood log lines. - location = /loki/api/v1/push { - return 403; - } - - location /loki/ { - auth_basic "Triangle Loki"; - auth_basic_user_file /etc/nginx/triangle-observability.htpasswd; - proxy_pass http://127.0.0.1:13100; - } - - # Grafana's "Save & test" probes /loki/api/v1/status/buildinfo for version - # detection. That is under the /loki/ prefix above, so it needs no rule of - # its own -- verified against Loki 3.5.6, which 404s the unprefixed - # /api/v1/status/buildinfo that some docs suggest. - - # Unauthenticated liveness probe for our own use. /ready reports only - # whether the ingester is ready; it leaks no log content. - location = /ready { - access_log off; - proxy_pass http://127.0.0.1:13100; - } - - # Everything else on this port -- /metrics, /config, the admin and delete - # APIs -- stays unreachable. /config in particular echoes the running - # configuration back to the caller. - location / { - return 404; - } -} diff --git a/deploy/nginx/triangle-prometheus.conf b/deploy/nginx/triangle-prometheus.conf deleted file mode 100644 index 12d4c1a..0000000 --- a/deploy/nginx/triangle-prometheus.conf +++ /dev/null @@ -1,60 +0,0 @@ -# Read-only, password-protected Prometheus endpoint for the Triangle Grafana. -# -# Install as a host Nginx site, for example: -# /etc/nginx/sites-available/triangle-prometheus.conf -# and symlink into sites-enabled. -# -# Same shape and same reasoning as triangle-loki.conf: Prometheus has no -# authentication of its own, so the container publishes to 127.0.0.1:19090 and -# this site is the only way in. It shares triangle-observability.htpasswd with -# the Loki endpoint, so one credential pair covers both datasources. -# -# Grafana datasource config on the Triangle Grafana side: -# Type: Prometheus -# URL: http://:9090 -# Auth: Basic auth, same credentials as the Loki datasource -# -# The container is published on 19090 so Nginx can bind the natural 9090 that a -# Grafana admin expects, without a port clash. - -server { - listen 9090; - server_name _; - - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_read_timeout 300s; - proxy_send_timeout 300s; - - # The admin API can delete series and wipe the TSDB. It is already off -- - # neither --web.enable-admin-api nor --web.enable-lifecycle is passed in - # compose.observability.yml -- but deny it here too so that turning either - # flag on later for local debugging does not silently expose destructive - # endpoints to everything that can reach this port. - location /api/v1/admin/ { - return 403; - } - - # Query API. This is everything a Grafana datasource needs: query, - # query_range, series, labels, label values, metadata, buildinfo. - location /api/ { - auth_basic "Triangle Prometheus"; - auth_basic_user_file /etc/nginx/triangle-observability.htpasswd; - proxy_pass http://127.0.0.1:19090; - } - - # Unauthenticated liveness probe for our own use. Reports readiness only. - location = /-/ready { - access_log off; - proxy_pass http://127.0.0.1:19090; - } - - # Everything else stays unreachable: the web UI, /metrics, /config (which - # echoes the running scrape configuration), /-/reload and /-/quit. - location / { - return 404; - } -} diff --git a/deploy/proxmox/README.md b/deploy/proxmox/README.md deleted file mode 100644 index 59cc1dc..0000000 --- a/deploy/proxmox/README.md +++ /dev/null @@ -1,111 +0,0 @@ -# Proxmox host tooling - -Runs on the **Proxmox host**, not on Delta and not in a container. - -## Why the IP-conflict watch exists - -`10.248.40.154` (DB1) and `10.248.40.183` (MaxScale) were taken from Drexel's -DHCP pool by one-off `dhclient` runs. They are now static in **both** the -container config (`pct config ` → `net0 ... ip=`) and inside the container -(`/etc/systemd/network/10-eth0-static.network`), so our hosts keep their -addresses — that part is solved, and it is what fixed DB1's ~20-minute outage -when its lease lapsed on 2026-07-29. - -What is **not** solved: those addresses were never excluded from the pool. -DHCP for this subnet is central Drexel (`dhcp-server-identifier 10.254.5.41`, -off-subnet via a relay), so **nothing on the Proxmox host can reserve or exclude -an address** — that needs a request to Drexel IT, and as of 2026-07-30 none has -been made. The old leases expire **2026-08-05**, after which that server may -hand `.154` or `.183` to another device. - -Two hosts answering for one address does not fail cleanly. It looks like -intermittent, unexplainable connection errors against the database. This check -turns that into an immediate, named alert. - -**It detects; it does not prevent.** The fix is still a pool exclusion from -Drexel IT. - -## Install - -```bash -install -m 0755 ip-conflict-watch.sh /usr/local/sbin/ip-conflict-watch.sh -install -m 0644 ip-conflict-watch.service /etc/systemd/system/ -install -m 0644 ip-conflict-watch.timer /etc/systemd/system/ -systemctl daemon-reload -systemctl enable --now ip-conflict-watch.timer -``` - -Verify, and confirm it reports the expected MACs: - -```bash -systemctl start ip-conflict-watch.service -journalctl -t triangle-ip-watch -n 10 --no-pager -systemctl list-timers ip-conflict-watch.timer -``` - -Expected output: - -``` -[daemon.info] 10.248.40.154 OK (bc:24:11:e5:cc:58) -[daemon.info] 10.248.40.183 OK (bc:24:11:83:05:ea) -``` - -## Exit codes - -| code | meaning | -| --- | --- | -| 0 | both addresses answered, only from the expected MAC | -| 1 | **conflict** — a foreign MAC also answered | -| 2 | a host did not answer ARP at all (container down, or wrong `IFACE`) | - -A non-zero exit fails the unit, so `systemctl status ip-conflict-watch` and -`systemctl list-units --failed` both surface it. - -## Configuration - -Defaults are in the script. To change the interface or watch more hosts without -editing it, drop `/etc/triangle-ip-watch.conf`: - -```bash -IFACE=vmbr0 -WATCH=( - "10.248.40.154=bc:24:11:e5:cc:58" - "10.248.40.183=bc:24:11:83:05:ea" -) -``` - -Keep the MACs in step with `pct config | grep net0`. A stale expected MAC -produces a false conflict alert. - -## Getting alerted - -The check logs to the journal under tag `triangle-ip-watch`, at `daemon.err` -for a conflict. That is deliberately the whole of it — how alerts leave this -box is a local decision. Two options: - -- **systemd**, mail on failure — add `OnFailure=status-email@%n.service` to the - service unit with a mail-sending template unit. -- **Promtail**, if the observability stack is ever pointed at this host — scrape - the journal and alert on the `triangle-ip-watch` tag at priority 3. - -## After 2026-08-05 - -Re-check by hand once the old leases have lapsed, since that is the window when -a reassignment can first happen: - -```bash -arping -I vmbr0 -c 2 10.248.40.154 # expect bc:24:11:e5:cc:58 -arping -I vmbr0 -c 2 10.248.40.183 # expect bc:24:11:83:05:ea -``` - -Note `arping -D` from the Proxmox host **always** gets a reply — our own -containers answer their own probes. That is not a conflict. Identity needs plain -`arping` (which prints the responder MAC) or `ip neigh show `. - -## Still to do - -- Request a **pool exclusion** (not a reservation — these hosts never request a - lease, so a reservation would never be exercised) for `.154` and `.183` from - Drexel IT, and ask them to confirm the dynamic range so it can be verified. -- CT 113 `THETRIANGLE-REACT` is still `ip=dhcp`. It *does* request a lease, so - for that one a **reservation** is the right ask. diff --git a/deploy/proxmox/ip-conflict-watch.service b/deploy/proxmox/ip-conflict-watch.service deleted file mode 100644 index adea907..0000000 --- a/deploy/proxmox/ip-conflict-watch.service +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=Triangle DB tier IP-conflict check -Documentation=https://github.com/DrexelTriangle/triangle-cms/blob/main/deploy/proxmox/README.md - -[Service] -Type=oneshot -ExecStart=/usr/local/sbin/ip-conflict-watch.sh -# arping needs raw sockets; everything else is dropped. -CapabilityBoundingSet=CAP_NET_RAW -AmbientCapabilities=CAP_NET_RAW -NoNewPrivileges=true -ProtectSystem=strict -ProtectHome=true -PrivateTmp=true diff --git a/deploy/proxmox/ip-conflict-watch.sh b/deploy/proxmox/ip-conflict-watch.sh deleted file mode 100755 index d1c436c..0000000 --- a/deploy/proxmox/ip-conflict-watch.sh +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/env bash -# Detect an IP conflict on the Triangle DB tier. -# -# The DB hosts' addresses are static in Proxmox and inside the containers, but -# they were taken from Drexel's DHCP pool and were never excluded from it -# (central DHCP lives on 10.254.5.41; Proxmox is not the DHCP server, so nothing -# on this host can reserve them). Our hosts therefore keep their addresses, but -# nothing stops that server leasing the same address to some other device once -# the old leases lapse. This cannot prevent that -- it detects it quickly. -# -# Runs on the PROXMOX HOST, which shares a bridge with the containers. ARP is -# link-local, so this does not work from a routed workstation. -# -# Exit: 0 all good, 1 conflict (a foreign MAC answered), 2 a host did not answer. -set -uo pipefail - -IFACE="${IFACE:-vmbr0}" -TAG="triangle-ip-watch" - -# ip=expected-mac. Keep in sync with `pct config | grep net0`. -WATCH=( - "10.248.40.154=bc:24:11:e5:cc:58" # THETRIANGLE-DB1-LXC (CT 108, MariaDB) - "10.248.40.183=bc:24:11:83:05:ea" # THETRIANGLE-MAXSCALE (CT 109) -) - -# Optional overrides, e.g. to add a host without editing this file. -# shellcheck source=/dev/null -[[ -r /etc/triangle-ip-watch.conf ]] && source /etc/triangle-ip-watch.conf - -log() { logger -t "$TAG" -p "$1" -- "$2"; echo "[$1] $2"; } - -status=0 - -for entry in "${WATCH[@]}"; do - ip="${entry%%=*}" - expected="$(tr '[:upper:]' '[:lower:]' <<<"${entry#*=}")" - - # -c 3 -w 3: three probes, give up after three seconds either way. - out="$(arping -c 3 -w 3 -I "$IFACE" "$ip" 2>/dev/null)" - macs="$(grep -oiE '([0-9a-f]{2}:){5}[0-9a-f]{2}' <<<"$out" | tr '[:upper:]' '[:lower:]' | sort -u)" - - if [[ -z "$macs" ]]; then - # Not a conflict: the container is probably down, or the bridge is wrong. - # Worth surfacing, but it must not read as "someone stole the address". - log daemon.warning "$ip did not answer ARP on $IFACE (host down, or wrong interface?)" - [[ $status -eq 0 ]] && status=2 - continue - fi - - foreign="$(grep -v "^${expected}$" <<<"$macs" || true)" - if [[ -n "$foreign" ]]; then - # Two devices answering for one address. On a database host this shows up - # as intermittent, inexplicable connection failures rather than a clean - # outage, so it is worth an alarm rather than a warning. - log daemon.err "IP CONFLICT on $ip: expected $expected, also answered by: $(tr '\n' ' ' <<<"$foreign")" - status=1 - else - log daemon.info "$ip OK ($expected)" - fi -done - -exit "$status" diff --git a/deploy/proxmox/ip-conflict-watch.timer b/deploy/proxmox/ip-conflict-watch.timer deleted file mode 100644 index 196a895..0000000 --- a/deploy/proxmox/ip-conflict-watch.timer +++ /dev/null @@ -1,12 +0,0 @@ -[Unit] -Description=Run the Triangle DB tier IP-conflict check every 15 minutes - -[Timer] -OnBootSec=5min -OnUnitActiveSec=15min -# Keeps every PVE node from probing on the same second if this is ever cloned. -RandomizedDelaySec=60s -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/deploy/scripts/deploy-observability.sh b/deploy/scripts/deploy-observability.sh deleted file mode 100755 index 3911bdf..0000000 --- a/deploy/scripts/deploy-observability.sh +++ /dev/null @@ -1,295 +0,0 @@ -#!/usr/bin/env bash -# Deploy the observability stack (Prometheus, Alertmanager, blackbox, Loki, -# Promtail) as part of the normal Delta deployment, so it stops depending on -# someone remembering to run it by hand. -# -# Run from the repo root, after the CMS deploy has completed: -# deploy/scripts/deploy-observability.sh -# -# WHY THIS COPIES INSTEAD OF RUNNING IN PLACE -# The runner's checkout is the only tree on Delta that contains observability/, -# but `actions/checkout` resets it on every deploy. A long-lived stack bind- -# mounting config out of it would have its mount sources yanked mid-run. So the -# files are synced to a stable directory the runner owns and Actions never -# touches, and Compose runs from there. That directory used to be maintained by -# hand; this script is what replaces the hand. -# -# IT MUST NEVER TOUCH THE CMS. compose.observability.yml is a separate Compose -# project, so `up -d` here cannot recreate or stop the CMS slots. The one real -# coupling is that Prometheus joins the CMS network, declared external, so the -# CMS stack has to exist first -- which is why this runs after deploy.sh rather -# than beside it. -set -Eeuo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" -REPO_DIR="$(cd "${DEPLOY_DIR}/.." && pwd)" - -# nginx_test / nginx_reload / atomic_install_file, shared with the CMS deploy so -# both paths validate and reload Nginx exactly the same way. -# shellcheck source=deploy/scripts/common.sh -source "${SCRIPT_DIR}/common.sh" - -# Default under the runner's own home: writable by it, and outside _work/ so -# actions/checkout never resets it. Overridable for a differently-laid-out host. -OBSERVABILITY_DIR="${OBSERVABILITY_DIR:-${HOME}/triangle-observability}" -COMPOSE_PROJECT="triangle-observability" -HEALTH_TIMEOUT="${OBSERVABILITY_HEALTH_TIMEOUT:-120}" - -SRC_TREE="${REPO_DIR}/observability" -SRC_COMPOSE="${DEPLOY_DIR}/compose.observability.yml" -DST_COMPOSE_DIR="${OBSERVABILITY_DIR}/deploy" -DST_COMPOSE="${DST_COMPOSE_DIR}/compose.observability.yml" - -for f in "${SRC_COMPOSE}"; do - [[ -f "$f" ]] || { echo "required file not found: $f" >&2; exit 1; } -done -[[ -d "${SRC_TREE}" ]] || { echo "required directory not found: ${SRC_TREE}" >&2; exit 1; } - -# Deliberately NOT named `compose`: common.sh defines a compose() bound to -# compose.cms.yml and the CMS env file. Shadowing it would work only because of -# definition order, and anyone moving the `source` line above would silently -# point every call in this script at the CMS stack. -obs_compose() { - # No --env-file: since Grafana was removed the only variable left is - # DISCORD_WEBHOOK_FILE, which carries a default, so this stack needs no - # secrets at deploy time. The webhook itself is a root-owned file on the host - # that only Docker reads. - docker compose -p "${COMPOSE_PROJECT}" -f "${DST_COMPOSE}" "$@" -} - -# Fingerprint of everything that gets mounted into a container. Used to decide -# whether a restart is needed at all, so an ordinary CMS deploy that changed no -# observability file costs nothing. -tree_fingerprint() { - local dir="$1" - [[ -d "$dir" ]] || { echo "absent"; return; } - find "$dir" -type f -exec sha256sum {} + 2>/dev/null \ - | sed "s#${dir}/##" | sort -k2 | sha256sum | cut -d' ' -f1 -} - -before="$(tree_fingerprint "${OBSERVABILITY_DIR}/observability")" -before_compose="$( [[ -f "${DST_COMPOSE}" ]] && sha256sum "${DST_COMPOSE}" | cut -d' ' -f1 || echo absent )" - -echo "syncing observability config -> ${OBSERVABILITY_DIR}" -mkdir -p "${DST_COMPOSE_DIR}" - -# --inplace is load-bearing. Without it rsync writes a temp file and renames, -# which gives every synced file a NEW INODE -- and a running container's bind -# mount holds the old one, so it would keep serving stale config even after a -# restart picked up nothing. --delete keeps removed files from lingering, which -# is how a deleted alert rule actually stops being evaluated. -if command -v rsync >/dev/null 2>&1; then - rsync -a --delete --inplace "${SRC_TREE}/" "${OBSERVABILITY_DIR}/observability/" - rsync -a --inplace "${SRC_COMPOSE}" "${DST_COMPOSE}" -else - # cp -f truncates and rewrites in place, preserving the inode, so it is safe - # here for the same reason --inplace is. The rm -rf first is what stands in - # for --delete. - rm -rf "${OBSERVABILITY_DIR}/observability" - mkdir -p "${OBSERVABILITY_DIR}/observability" - cp -R "${SRC_TREE}/." "${OBSERVABILITY_DIR}/observability/" - cp -f "${SRC_COMPOSE}" "${DST_COMPOSE}" -fi - -after="$(tree_fingerprint "${OBSERVABILITY_DIR}/observability")" -after_compose="$(sha256sum "${DST_COMPOSE}" | cut -d' ' -f1)" - -changed=0 -[[ "${before}" != "${after}" ]] && changed=1 -[[ "${before_compose}" != "${after_compose}" ]] && changed=1 - -echo "applying compose (project ${COMPOSE_PROJECT})" -# --remove-orphans is what retires a service deleted from the compose file -- -# Grafana left this way. It is scoped to this project, so it cannot reach the -# CMS slots. -obs_compose up -d --remove-orphans - -if (( changed )); then - # `up -d` does NOT restart a container when only the CONTENTS of a mounted - # config file changed -- the Compose spec is identical, so it sees nothing to - # do and reports "Running". Every config-only change therefore needs an - # explicit restart or it silently does not take effect. This is the single - # easiest thing to get wrong here. - echo "config changed; restarting services to pick it up" - obs_compose restart -else - echo "no observability config change; skipping restart" -fi - -echo "verifying" -deadline=$(( SECONDS + HEALTH_TIMEOUT )) - -wait_for() { - local label="$1" url="$2" - until curl -fsS -o /dev/null --max-time 5 "${url}"; do - if (( SECONDS >= deadline )); then - echo "timed out waiting for ${label} (${url})" >&2 - obs_compose ps >&2 || true - return 1 - fi - sleep 3 - done - echo " ok ${label}" -} - -wait_for "prometheus" "http://127.0.0.1:19090/-/healthy" -wait_for "alertmanager" "http://127.0.0.1:19093/-/healthy" - -# A stack that is up but has silently dropped its alert rules is worse than one -# that is down, because it looks fine. Assert the rules actually loaded and that -# Prometheus is talking to Alertmanager, rather than trusting "container is -# running". -# -# These have to RETRY, not check once. /-/healthy goes green as soon as the web -# server is listening, which is well before the rule manager has evaluated the -# rule files and before the notifier has resolved alertmanager:9093 through -# Docker DNS. Asserting immediately after a restart is a race, and it is the -# race that made this step fail with "no active alertmanager" on a stack that -# was in fact fine seconds later. -retry_until() { - local label="$1"; shift - until "$@"; do - if (( SECONDS >= deadline )); then - echo "timed out: ${label}" >&2 - obs_compose ps >&2 || true - return 1 - fi - sleep 3 - done - echo " ok ${label}" -} - -has_alert_rules() { - local n - n="$(curl -fsS --max-time 5 http://127.0.0.1:19090/api/v1/rules \ - | grep -o '"name":"[^"]*"' | wc -l)" || return 1 - (( n > 0 )) -} - -has_active_alertmanager() { - # Match the discovered URL, not the JSON keys: "activeAlertmanagers" and - # "droppedAlertmanagers" are always present, but capitalised, so a - # case-sensitive match on lowercase "alertmanager:9093" only hits a real - # entry. Checking activeAlertmanagers is non-empty would otherwise need a - # JSON parser this host is not guaranteed to have. - curl -fsS --max-time 5 http://127.0.0.1:19090/api/v1/alertmanagers \ - | grep -q 'alertmanager:9093' -} - -retry_until "alert rules loaded" has_alert_rules \ - || { echo "prometheus loaded no alerting rules -- check observability/prometheus/rules/" >&2; exit 1; } -retry_until "alertmanager attached" has_active_alertmanager \ - || { echo "prometheus has no active alertmanager; alerts would fire into nothing" >&2; exit 1; } - -obs_compose ps --format ' {{.Service}}\t{{.Status}}' - -# --- Nginx sites ------------------------------------------------------------- -# The read-only Loki and Prometheus endpoints the central Grafana connects to. -# These are HOST config, not containers, so they deploy differently: the files -# are installed into a directory the runner OWNS, which a root-owned -# /etc/nginx/conf.d/triangle-observability.conf pulls in with a wildcard include. -# -# That indirection is the whole point. It keeps the runner's sudo rights at -# exactly the two commands the CMS deploy already needs -- `nginx -t` and -# `nginx -s reload` -- instead of granting it write access to /etc/nginx or a -# general "install this file as root" rule. It is the same shape as -# /etc/nginx/triangle-cms/, which the runner already owns for blue/green. -NGINX_SITES_DIR="${NGINX_SITES_DIR:-/etc/nginx/triangle-observability}" -NGINX_SITES=(triangle-loki.conf triangle-prometheus.conf) - -deploy_nginx_sites() { - local staging prior_dir changed=0 s src dst - if [[ ! -d "${NGINX_SITES_DIR}" ]]; then - cat >&2 </dev/null - sudo nginx -t && sudo nginx -s reload - -Failing rather than skipping: an Nginx site that silently stopped tracking the -repo is how the datasource endpoints drift out from under the Grafana using them. -EOF - return 1 - fi - [[ -w "${NGINX_SITES_DIR}" ]] || { - echo "${NGINX_SITES_DIR} is not writable by $(id -un); it must be owned by the runner" >&2 - return 1 - } - - for s in "${NGINX_SITES[@]}"; do - [[ -f "${DEPLOY_DIR}/nginx/${s}" ]] || { - echo "missing source: ${DEPLOY_DIR}/nginx/${s}" >&2; return 1; } - cmp -s "${DEPLOY_DIR}/nginx/${s}" "${NGINX_SITES_DIR}/${s}" || changed=1 - done - - if (( ! changed )); then - echo " no nginx site change; skipping reload" - return 0 - fi - - # Snapshot whatever is live so a config that fails validation can be undone - # without a human. Nginx keeps serving the OLD config until a successful - # reload, so a failed `nginx -t` here is harmless as long as we put the files - # back -- the danger is leaving broken files on disk for the NEXT reload, - # which could be the CMS deploy's. - prior_dir="$(mktemp -d)" - staging="$(mktemp -d "${NGINX_SITES_DIR}/.stage.XXXXXX")" - # shellcheck disable=SC2064 - trap "rm -rf '${prior_dir}' '${staging}'" RETURN - - for s in "${NGINX_SITES[@]}"; do - [[ -f "${NGINX_SITES_DIR}/${s}" ]] && cp -p "${NGINX_SITES_DIR}/${s}" "${prior_dir}/${s}" - done - - echo " installing nginx sites: ${NGINX_SITES[*]}" - for s in "${NGINX_SITES[@]}"; do - src="${DEPLOY_DIR}/nginx/${s}" - dst="${NGINX_SITES_DIR}/${s}" - cp "${src}" "${staging}/${s}" - chmod 0644 "${staging}/${s}" - mv -f "${staging}/${s}" "${dst}" - done - - restore_prior_sites() { - local t - for t in "${NGINX_SITES[@]}"; do - if [[ -f "${prior_dir}/${t}" ]]; then - cp -p "${prior_dir}/${t}" "${NGINX_SITES_DIR}/${t}" - else - rm -f "${NGINX_SITES_DIR}/${t}" - fi - done - } - - if ! nginx_test; then - echo "nginx validation failed with the new observability sites; reverting" >&2 - restore_prior_sites - if ! nginx_test; then - echo "CRITICAL: reverted observability sites still fail nginx validation; operator intervention required" >&2 - return 10 - fi - return 1 - fi - - if ! nginx_reload; then - echo "nginx reload failed; reverting observability sites" >&2 - restore_prior_sites - nginx_test && nginx_reload || { - echo "CRITICAL: reload failed and restoration could not be reloaded; operator intervention required" >&2 - return 11 - } - return 1 - fi - - echo " ok nginx sites installed and reloaded" -} - -echo "deploying nginx sites" -deploy_nginx_sites - -echo "observability deployment complete" diff --git a/deploy/scripts/deploy_scripts_test.sh b/deploy/scripts/deploy_scripts_test.sh index 1ab3c2c..83d57df 100755 --- a/deploy/scripts/deploy_scripts_test.sh +++ b/deploy/scripts/deploy_scripts_test.sh @@ -282,11 +282,11 @@ test_lock_contention() { exec 8>&- } -test_host_nginx_health_uses_default_type() { - local nginx_conf="${SCRIPT_DIR}/../nginx/triangle-cms.conf" - assert_file_contains "${nginx_conf}" 'default_type text/plain' - assert_file_not_contains "${nginx_conf}" 'add_header Content-Type text/plain' -} +# test_host_nginx_health_uses_default_type moved with the file it guards. +# nginx/triangle-cms.conf now lives in triangle-infrastructure +# (roles/delta_cms_host/files/), and the assertion is tests/site-contracts.sh +# over there. Asserting on a file this repo no longer ships would only test +# whichever stale copy happened to be on the runner. test_no_old_production_include_path_references() { local old_path @@ -312,7 +312,6 @@ test_failed_readiness_leaves_active_slot test_failed_public_smoke_rolls_back test_invalid_and_malicious_sha test_lock_contention -test_host_nginx_health_uses_default_type test_no_old_production_include_path_references echo "deploy script tests passed" diff --git a/docker-compose.yml b/docker-compose.yml index 189ae94..8e0bfd0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -93,91 +93,9 @@ services: networks: - triangle_net - loki: - image: grafana/loki:3.5.6 - restart: unless-stopped - command: ["-config.file=/etc/loki/config.yaml"] - healthcheck: - test: ["CMD", "loki", "-verify-config=true", "-config.file=/etc/loki/config.yaml"] - interval: 20s - timeout: 5s - retries: 5 - start_period: 10s - volumes: - - ./observability/loki-config.yml:/etc/loki/config.yaml:ro,z - - loki_data:/loki - networks: - - triangle_net - - promtail: - image: grafana/promtail:3.5.6 - restart: unless-stopped - user: "0:0" - read_only: true - security_opt: - - no-new-privileges:true - # Local-dev only. On SELinux hosts (Fedora) the daemon socket is labelled - # container_var_run_t, which a confined container may not connect to: the - # mount succeeds, Promtail starts, and every discovery refresh fails with - # "permission denied while trying to connect to the Docker daemon socket". - # Delta is Ubuntu 24.04 with no SELinux, so compose.observability.yml - # deliberately does NOT carry this relaxation. - - label:disable - cap_drop: - - ALL - command: ["-config.file=/etc/promtail/config.yml"] - depends_on: - loki: - condition: service_healthy - volumes: - - ./observability/promtail-config.yml:/etc/promtail/config.yml:ro,z - - promtail_positions:/tmp - # Must match deploy/compose.observability.yml: the shared promtail config - # discovers containers over the Docker API rather than tailing files. - - /var/run/docker.sock:/var/run/docker.sock:ro - networks: - - triangle_net - - grafana: - image: grafana/grafana:12.2.0 - restart: unless-stopped - depends_on: - loki: - condition: service_healthy - environment: - GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER is required} - GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD is required} - ports: - - "127.0.0.1:3000:3000" - volumes: - - grafana_data:/var/lib/grafana - - ./observability/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro,z - - ./observability/grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro,z - networks: - - triangle_net - - prometheus: - image: prom/prometheus:v3.6.0 - restart: unless-stopped - command: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - volumes: - # Dev scrapes the single TLS `cms` service; Delta scrapes two plain-HTTP - # slots. Different files rather than one conditional config. - - ./observability/prometheus/prometheus.dev.yml:/etc/prometheus/prometheus.yml:ro,z - - prometheus_data:/prometheus - ports: - - "127.0.0.1:9090:9090" - networks: - - triangle_net volumes: mariadb_data: - loki_data: - promtail_positions: - grafana_data: - prometheus_data: networks: triangle_net: diff --git a/frontend/src/lib/clipboard.ts b/frontend/src/lib/clipboard.ts index 7cb87f0..a4d7e91 100644 --- a/frontend/src/lib/clipboard.ts +++ b/frontend/src/lib/clipboard.ts @@ -2,8 +2,9 @@ * Copy text to the clipboard, returning whether it worked. * * navigator.clipboard exists only on secure origins. The CMS is still served - * over plain HTTP on Delta (see deploy/nginx/triangle-cms.conf), where the whole - * API is simply undefined -- so the modern path alone would fail on exactly the + * over plain HTTP on Delta (the host Nginx site lives in the + * triangle-infrastructure repo), where the whole API is undefined -- so the + * modern path alone would fail on exactly the * deployment editors use today. Fall back to the deprecated execCommand copy, * which has no such restriction, and only report failure if both fail. */ diff --git a/observability/alertmanager/alertmanager.yml b/observability/alertmanager/alertmanager.yml deleted file mode 100644 index d16e9ea..0000000 --- a/observability/alertmanager/alertmanager.yml +++ /dev/null @@ -1,43 +0,0 @@ -# Alertmanager: routes Prometheus's database-tier alerts to Discord. -# -# Exists so the MaxScale watchdog does not live inside Grafana. Delta's -# Prometheus is a datasource for the Triangle Grafana rather than part of it, so -# keeping the rule and its delivery here means neither depends on which Grafana -# is in front. See observability/prometheus/rules/database.yml. -# -# The webhook is NOT in this file. It is read from a path (0640, root:root on -# the host) so the config can stay in git, matching how MaxScale keeps its own -# secrets out of the world-readable maxscale.cnf. - -global: - # How long after the last firing evaluation an alert is considered resolved. - # Prometheus re-sends every evaluation while a rule fires, so this only has to - # outlast a couple of scrape intervals. - resolve_timeout: 5m - -route: - receiver: discord - # One notification per alert name rather than per instance. With two rules and - # one probe there is nothing to fan out, and grouping keeps a flapping backend - # from producing a message per transition. - group_by: [alertname] - # Short waits: these are total-outage alerts, not capacity warnings. - group_wait: 10s - group_interval: 1m - # Re-notify hourly while still firing, so an unresolved outage does not fall - # off the channel after a single message. - repeat_interval: 1h - -receivers: - - name: discord - discord_configs: - - webhook_url_file: /etc/alertmanager/discord-webhook - # Alertmanager colours the embed by status on its own — red firing, - # green resolved — so the text does not need to carry the severity. - # Keep both fields SHORT: this is a phone notification first and a - # document never. The detail belongs in the runbook, not the alert. - title: '{{ .CommonAnnotations.summary }}' - message: |- - {{ .CommonAnnotations.detail }} - {{ range .Alerts }} - `{{ .Labels.instance }}`{{ end }} diff --git a/observability/blackbox/blackbox.yml b/observability/blackbox/blackbox.yml deleted file mode 100644 index 3c098d6..0000000 --- a/observability/blackbox/blackbox.yml +++ /dev/null @@ -1,33 +0,0 @@ -# blackbox_exporter modules for deploy/compose.observability.yml. -# -# Purpose: probe MaxScale's client listener from OUTSIDE the database tier. -# Everything else in this stack watches the CMS itself, which cannot report the -# one failure that matters most here — MaxScale being dead. Since the write path -# is fenced so that MaxScale is the ONLY route to the databases (see -# deploy/mariadb/README.md), MaxScale down is a total outage, and MaxScale's own -# alert script cannot report it: the process that would send the alert is the -# process that died. -# -# A TCP connect against :4006 is deliberately chosen over the MaxScale REST API: -# - 4006 is the port the CMS actually uses, so this tests the real dependency -# rather than a management interface that could be healthy while routing is -# not. -# - It needs no new exposure. The admin API (8989) would have to be opened to -# Delta, and it can reconfigure MaxScale, so it is a far worse thing to -# expose than a port the CMS already talks to. -# - MaxScale 24.02 serves no Prometheus endpoint (verified: /metrics and -# /v1/metrics both 404), so scraping it directly is not an option anyway. - -modules: - # Plain TCP handshake. We do NOT speak the MySQL protocol here: MaxScale sends - # a server greeting on connect, and completing a real login would need - # credentials in this file. Accepting the connection is sufficient evidence - # that MaxScale is alive and listening. - tcp_connect: - prober: tcp - # Comfortably under Prometheus's scrape_timeout so a slow probe surfaces as - # a failed probe rather than a scrape error, which would alert differently. - timeout: 5s - tcp: - ip_protocol_fallback: false - preferred_ip_protocol: ip4 diff --git a/observability/grafana/dashboards/gisbxcj.json b/observability/grafana/dashboards/gisbxcj.json deleted file mode 100644 index 78ca773..0000000 --- a/observability/grafana/dashboards/gisbxcj.json +++ /dev/null @@ -1,962 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": { - "type": "grafana", - "uid": "-- Grafana --" - }, - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "fiscalYearStartMonth": 0, - "graphTooltip": 0, - "id": null, - "links": [], - "panels": [ - { - "datasource": { - "type": "loki", - "uid": "${loki_ds}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "logs/min" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 0 - }, - "id": 10, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } - }, - "pluginVersion": "12.1.1", - "targets": [ - { - "direction": "backward", - "editorMode": "code", - "expr": "sum by (level) (count_over_time({service_name=\"cms\"} | json [1m]))", - "legendFormat": "{{level}}", - "queryType": "range", - "refId": "A" - } - ], - "title": "Log Volume by level", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "opacity", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "max": 0, - "min": 100, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percent" - }, - "overrides": [ - { - "__systemRef": "hideSeriesFrom", - "matcher": { - "id": "byNames", - "options": { - "mode": "exclude", - "names": [ - "GET /v1/health" - ], - "prefix": "All except:", - "readOnly": true - } - }, - "properties": [ - { - "id": "custom.hideFrom", - "value": { - "legend": false, - "tooltip": false, - "viz": false - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 0 - }, - "id": 1, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "exemplar": false, - "expr": "rate(http_requests_total[5m]) \n", - "format": "time_series", - "legendFormat": "{{route}}", - "range": true, - "refId": "A" - } - ], - "title": "Request rate (req/s)", - "type": "timeseries" - }, - { - "datasource": { - "uid": "${prometheus_ds}", - "type": "prometheus" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "mappings": [], - "max": 100, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percent" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 8 - }, - "id": 5, - "options": { - "minVizHeight": 75, - "minVizWidth": 75, - "orientation": "auto", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showThresholdLabels": false, - "showThresholdMarkers": false, - "sizing": "auto" - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "editorMode": "code", - "expr": "up{job=\"cms\"} * 100", - "legendFormat": "__auto", - "range": true, - "refId": "A" - } - ], - "title": "Service Up>Down", - "type": "gauge" - }, - { - "datasource": { - "uid": "${prometheus_ds}", - "type": "prometheus" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "mappings": [], - "max": 100, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percent" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 8 - }, - "id": 3, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "editorMode": "code", - "expr": "(\n sum(rate(http_requests_total{status=~\"4..\"}[5m])) or vector(0)\n)\n/\nsum(rate(http_requests_total[5m])) * 100", - "legendFormat": "__auto", - "range": true, - "refId": "A" - } - ], - "title": "Error rate ", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "decbytes" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 16 - }, - "id": 7, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "process_resident_memory_bytes{job=\"cms\"} ", - "legendFormat": "Memory", - "range": true, - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "go_goroutines{job=\"cms\"}", - "hide": false, - "instant": false, - "legendFormat": "Goroutine count", - "range": true, - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "rate(process_cpu_seconds_total{job=\"cms\"}[5ms])", - "hide": false, - "instant": false, - "legendFormat": "Cpu core", - "range": true, - "refId": "C" - } - ], - "title": "Go runtime", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "hue", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineStyle": { - "fill": "solid" - }, - "lineWidth": 1, - "pointSize": 6, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - } - }, - "overrides": [ - { - "__systemRef": "hideSeriesFrom", - "matcher": { - "id": "byNames", - "options": { - "mode": "exclude", - "names": [ - "403" - ], - "prefix": "All except:", - "readOnly": true - } - }, - "properties": [ - { - "id": "custom.hideFrom", - "value": { - "legend": false, - "tooltip": true, - "viz": true - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 16 - }, - "id": 2, - "options": { - "legend": { - "calcs": [ - "last" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "single", - "sort": "none" - } - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "sum by (status) (rate(http_requests_total[5m]))", - "format": "time_series", - "legendFormat": "{{status}}", - "range": true, - "refId": "A" - } - ], - "title": "Request rate by status", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "description": "How long requests take. p50 = typical, p95/p99 = the slowest requests. Rising p99 = slow tail.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "hue", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 24 - }, - "id": 4, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "none" - } - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))", - "legendFormat": "p95", - "range": true, - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))", - "hide": false, - "instant": false, - "legendFormat": "p50", - "range": true, - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))", - "hide": false, - "instant": false, - "legendFormat": "p99", - "range": true, - "refId": "C" - } - ], - "title": "Request latency", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "description": "Where traffic concentrates\n", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "custom": { - "align": "auto", - "cellOptions": { - "mode": "lcd", - "type": "gauge", - "valueDisplayMode": "text" - }, - "footer": { - "reducers": [] - }, - "inspect": false, - "tooltip": { - "field": "GET /v1/articles/{slug}" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percent" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 32 - }, - "id": 6, - "options": { - "cellHeight": "sm", - "frameIndex": 1, - "showHeader": true - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${prometheus_ds}" - }, - "editorMode": "code", - "expr": "topk(5, sum by(route) (rate(http_requests_total[5m]))) * 1000", - "legendFormat": "__auto", - "range": true, - "refId": "A" - } - ], - "title": "Top routes", - "type": "table" - }, - { - "datasource": { - "type": "loki", - "uid": "${loki_ds}" - }, - "description": "Main explorer, without health and metrics", - "fieldConfig": { - "defaults": {}, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 40 - }, - "id": 8, - "options": { - "dedupStrategy": "none", - "enableInfiniteScrolling": false, - "enableLogDetails": true, - "prettifyLogMessage": true, - "showCommonLabels": false, - "showLabels": false, - "showTime": true, - "sortOrder": "Descending", - "wrapLogMessage": false - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "direction": "backward", - "editorMode": "code", - "expr": "{service=\"cms\"} != \"/metrics\" ", - "queryType": "range", - "refId": "A" - } - ], - "title": "CMS logs", - "type": "logs" - }, - { - "datasource": { - "type": "loki", - "uid": "${loki_ds}" - }, - "description": "All errors from 400 and above are displayed here", - "fieldConfig": { - "defaults": {}, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 40 - }, - "id": 9, - "options": { - "dedupStrategy": "none", - "enableInfiniteScrolling": false, - "enableLogDetails": true, - "prettifyLogMessage": true, - "showCommonLabels": false, - "showLabels": false, - "showTime": false, - "sortOrder": "Descending", - "wrapLogMessage": false - }, - "pluginVersion": "12.2.0", - "targets": [ - { - "direction": "backward", - "editorMode": "code", - "expr": "{service=\"cms\"} | json | status >= 400", - "queryType": "range", - "refId": "A" - } - ], - "title": "Error logs", - "type": "logs" - } - ], - "preload": false, - "refresh": "5s", - "schemaVersion": 41, - "tags": [], - "templating": { - "list": [ - { - "current": {}, - "hide": 0, - "includeAll": false, - "label": "Prometheus", - "multi": false, - "name": "prometheus_ds", - "options": [], - "query": "prometheus", - "queryValue": "", - "refresh": 1, - "regex": "/-delta$/", - "skipUrlSync": false, - "type": "datasource" - }, - { - "current": {}, - "hide": 0, - "includeAll": false, - "label": "Loki", - "multi": false, - "name": "loki_ds", - "options": [], - "query": "loki", - "queryValue": "", - "refresh": 1, - "regex": "/-delta$/", - "skipUrlSync": false, - "type": "datasource" - } - ] - }, - "time": { - "from": "now-24h", - "to": "now" - }, - "timepicker": {}, - "timezone": "browser", - "title": "CMS Dashboard", - "uid": "gisbxcj", - "version": 4 -} diff --git a/observability/loki-config.yml b/observability/loki-config.yml deleted file mode 100644 index d39f8f0..0000000 --- a/observability/loki-config.yml +++ /dev/null @@ -1,62 +0,0 @@ -auth_enabled: false - -server: - http_listen_port: 3100 - grpc_listen_port: 9096 - -common: - instance_addr: 127.0.0.1 - path_prefix: /loki - storage: - filesystem: - chunks_directory: /loki/chunks - rules_directory: /loki/rules - replication_factor: 1 - ring: - kvstore: - store: inmemory - -schema_config: - configs: - - from: 2024-01-01 - store: tsdb - object_store: filesystem - schema: v13 - index: - prefix: index_ - period: 24h - -ruler: - alertmanager_url: http://localhost:9093 - -limits_config: - allow_structured_metadata: true - volume_enabled: true - # Two weeks. Without this Loki keeps every line forever: the chunks live on a - # local filesystem volume on Delta, so unbounded growth is a disk-full - # incident on the host that also runs the CMS, not just a large bill. - retention_period: 336h - -# Retention is enforced by the compactor, and ONLY by the compactor -- setting -# retention_period alone changes nothing at all. `retention_enabled: true` is -# what actually deletes expired chunks; leaving it false is the usual reason a -# configured retention silently never takes effect. -compactor: - working_directory: /loki/compactor - compaction_interval: 10m - retention_enabled: true - # Grace period between a chunk being marked for deletion and the delete - # happening, so a mistaken retention change can be reverted before data is - # actually gone. - retention_delete_delay: 2h - delete_request_store: filesystem - -pattern_ingester: - enabled: true - -query_range: - results_cache: - cache: - embedded_cache: - enabled: true - max_size_mb: 100 diff --git a/observability/prometheus/prometheus.delta.yml b/observability/prometheus/prometheus.delta.yml deleted file mode 100644 index 6bd6501..0000000 --- a/observability/prometheus/prometheus.delta.yml +++ /dev/null @@ -1,77 +0,0 @@ -# Delta scrape config, for deploy/compose.observability.yml. -# -# Two differences from the dev config, both structural rather than cosmetic: -# -# 1. Plain HTTP. Delta runs the backend with CMS_SERVER_MODE=internal-http -# behind Nginx, so the slots speak HTTP on their loopback ports. Scraping -# https:// here fails the handshake and the dashboard shows no data. -# -# 2. Both slots, always. Blue and green both run permanently and only one is -# wired up in Nginx, so scraping "the backend" is not a single target. Each -# carries a `slot` label; `up{job="cms"}` will legitimately show two series. -# Which one is live is decided by Nginx, not by anything Prometheus can -# see -- see the blue/green section in deploy/README.md. -# -# Targets are container names on the CMS project's network, which Prometheus -# joins (see compose.observability.yml). Going via the host gateway does not -# work: the slots publish to 127.0.0.1 only, so the gateway address refuses the -# connection. Port 8080 is the in-container port, not the published one. - -global: - scrape_interval: 15s - -# Alerting lives here rather than in Grafana so it survives the switch to the -# Triangle Grafana, for which this Prometheus is only a datasource. A -# Grafana-owned rule would vanish with the local instance while the metric kept -# being scraped — dashboards fine, alerts silently gone. -rule_files: - - /etc/prometheus/rules/*.yml - -alerting: - alertmanagers: - - static_configs: - - targets: ["alertmanager:9093"] - -scrape_configs: - # The job name must stay "cms": the CMS dashboard - # (observability/grafana/dashboards/gisbxcj.json) filters on job="cms" in - # queries like up{job="cms"} and go_goroutines{job="cms"}. - - job_name: cms - metrics_path: /metrics - scheme: http - static_configs: - - targets: ["triangle-cms-backend-blue-1:8080"] - labels: - slot: blue - - targets: ["triangle-cms-backend-green-1:8080"] - labels: - slot: green - - # Watchdog for the database tier's single point of failure. - # - # The write path is fenced so MaxScale is the ONLY route to the databases, so - # MaxScale being down is a total outage — and it is precisely the outage - # MaxScale's own alert script cannot report, because the process that would - # send the alert is the one that died. This probes it from Delta instead. - # - # DB1/DB2 are deliberately NOT probed here: their 3306 is firewalled to the - # MaxScale host and the DB peer, so Delta cannot reach them by design, and a - # target that can never succeed is a permanently firing alert. - - job_name: blackbox-tcp - metrics_path: /probe - params: - module: [tcp_connect] - static_configs: - - targets: ["10.248.40.183:4006"] - labels: - service: maxscale - # The standard blackbox indirection: Prometheus scrapes the EXPORTER, and - # the real target travels as a query parameter. Without this rewrite - # Prometheus would try to scrape 10.248.40.183:4006 as if it served metrics. - relabel_configs: - - source_labels: [__address__] - target_label: __param_target - - source_labels: [__param_target] - target_label: instance - - target_label: __address__ - replacement: blackbox:9115 diff --git a/observability/prometheus/prometheus.dev.yml b/observability/prometheus/prometheus.dev.yml deleted file mode 100644 index d7df5ff..0000000 --- a/observability/prometheus/prometheus.dev.yml +++ /dev/null @@ -1,23 +0,0 @@ -# Local development scrape config, for the root docker-compose.yml stack. -# -# The dev `cms` service serves TLS on 8080 with the self-signed cert in -# server/certs, hence scheme: https and insecure_skip_verify. Delta runs the -# backend in CMS_SERVER_MODE=internal-http behind Nginx and has two slots -# rather than one, so it uses prometheus.delta.yml instead. - -global: - scrape_interval: 15s - -scrape_configs: - # The job name must stay "cms": the CMS dashboard - # (observability/grafana/dashboards/gisbxcj.json) filters on job="cms" in - # queries like up{job="cms"} and go_goroutines{job="cms"}. - - job_name: cms - metrics_path: /metrics - scheme: https - tls_config: - insecure_skip_verify: true - static_configs: - - targets: ["cms:8080"] - labels: - slot: dev diff --git a/observability/prometheus/rules/database.yml b/observability/prometheus/rules/database.yml deleted file mode 100644 index 6544b11..0000000 --- a/observability/prometheus/rules/database.yml +++ /dev/null @@ -1,41 +0,0 @@ -# Database-tier alerting rules, evaluated by Prometheus and routed to Discord by -# Alertmanager. -# -# This lived in Grafana's provisioned alerting until 2026-08-05. It moved here so -# it survives the switch to the Triangle Grafana: Delta's Prometheus is a -# DATASOURCE for that Grafana, not part of it, so a rule that lives in Prometheus -# keeps evaluating no matter which Grafana is in front — while a Grafana-owned -# rule would have disappeared with the local instance, silently. The metric would -# have kept being scraped and the dashboards would have looked fine; only the -# alert would have gone missing, which is the failure mode you never notice. - -groups: - - name: database-tier - interval: 30s - rules: - # The one outage MaxScale's own alert script can never report: itself. - # The write path is fenced so MaxScale is the only route to the databases, - # so this is a total outage, not a degraded read path. - - alert: MaxScaleUnreachable - expr: probe_success{service="maxscale"} == 0 - for: 1m - labels: - severity: critical - component: database - annotations: - summary: "MaxScale is not accepting connections" - detail: "The CMS cannot reach the database at all. Check `systemctl status maxscale`, then `maxctrl list servers`." - - # Replaces Grafana's noDataState: Alerting. If the probe series stops - # existing, nobody is watching the database tier — the cause is Prometheus - # or blackbox rather than MaxScale, but the consequence is the same, so it - # is not allowed to fail open. - - alert: MaxScaleProbeMissing - expr: absent(probe_success{service="maxscale"}) - for: 5m - labels: - severity: warning - component: observability - annotations: - summary: "MaxScale probe has stopped reporting" - detail: "Nothing is watching the database tier. Check the blackbox exporter on Delta." diff --git a/observability/promtail-config.yml b/observability/promtail-config.yml deleted file mode 100644 index 8a55592..0000000 --- a/observability/promtail-config.yml +++ /dev/null @@ -1,51 +0,0 @@ -server: - http_listen_port: 9080 - grpc_listen_port: 0 - -positions: - filename: /tmp/positions.yaml - -clients: - - url: http://loki:3100/loki/api/v1/push - -scrape_configs: - # Logs are collected through the Docker API, not by tailing files. - # - # This is not a stylistic choice. Delta's /etc/docker/daemon.json sets - # `"log-driver": "local"`, so container logs are written as a binary protobuf - # stream to /var/lib/docker/containers//local-logs/container.log. The - # obvious file-tailing setup globs for `*-json.log`, which the `local` driver - # never produces: Promtail starts cleanly, reports itself healthy, matches - # zero targets, and ships nothing. Nothing in Loki or Grafana surfaces that as - # an error -- it looks exactly like an application that never logs. - # - # docker_sd_configs reads through the daemon's log API instead, which is - # driver-agnostic, so this keeps working whether the host is on `local` or - # `json-file`. It also discovers containers as they appear, which matters - # here: blue/green deploys replace containers on every release. - - job_name: docker - docker_sd_configs: - - host: unix:///var/run/docker.sock - # Blue/green cutovers create containers mid-deploy; a short refresh - # keeps the window where a new slot's logs go uncollected small. - refresh_interval: 15s - relabel_configs: - # Docker reports names with a leading slash ("/triangle-cms-backend-blue-1"). - - source_labels: ["__meta_docker_container_name"] - regex: "/?(.*)" - target_label: container - - source_labels: ["__meta_docker_container_log_stream"] - target_label: stream - - source_labels: ["__meta_docker_container_label_com_docker_compose_project"] - target_label: compose_project - - source_labels: ["__meta_docker_container_label_com_docker_compose_service"] - target_label: compose_service - - target_label: job - replacement: docker - pipeline_stages: - # The backend tags its structured lines with a service field; promote it - # to a label so Grafana can filter on it. - - regex: - expression: ".*(?:service=|\\\"service\\\":\\\")(?P[a-zA-Z0-9_-]+).*" - - labels: - service: diff --git a/scripts/pull-dashboards.sh b/scripts/pull-dashboards.sh deleted file mode 100755 index 8507678..0000000 --- a/scripts/pull-dashboards.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail -cd "$(dirname "$0")/.." -[ -f .env ] && { set -a; . ./.env; set +a; } -# Pulls dashboards out of a Grafana into the repo, so UI edits survive the -# instance they were made in. -# -# Delta no longer runs a Grafana (removed 2026-08-05), so GF_URL now points at -# whichever Grafana holds the dashboards -- in practice the central Triangle -# Grafana. Point it there and supply that instance's credentials: -# -# GF_URL=https:// GRAFANA_ADMIN_USER=... \ -# GRAFANA_ADMIN_PASSWORD=... scripts/pull-dashboards.sh -# -# The default below is the old local address and will simply fail to connect -# now, which is the intended loud failure rather than a silent no-op. -# -# ALWAYS review `git diff` on the pulled JSON. The CMS dashboard resolves its -# datasources through the "prometheus_ds" and "loki_ds" template variables, and -# Grafana serializes whatever each panel resolved to -- so a pull can replace -# every "${prometheus_ds}" with a concrete UID and silently re-introduce the -# breakage those variables exist to prevent. Concrete UIDs are per-Grafana, so -# the committed result would render empty anywhere else. -GF_URL="${GF_URL:-http://127.0.0.1:3000}" -# GRAFANA_ADMIN_* are what the Compose files use; GF_USER/GF_PASS stay as -# fallbacks so an existing local .env keeps working. -GF_USER="${GRAFANA_ADMIN_USER:-${GF_USER:?set GRAFANA_ADMIN_USER (in .env or env)}}" -GF_PASS="${GRAFANA_ADMIN_PASSWORD:-${GF_PASS:?set GRAFANA_ADMIN_PASSWORD (in .env or env)}}" -OUT_DIR="observability/grafana/dashboards" -mkdir -p "$OUT_DIR" -curl -s -u "$GF_USER:$GF_PASS" "$GF_URL/api/search?type=dash-db" \ - | jq -r '.[].uid' \ - | while read -r uid; do - curl -s -u "$GF_USER:$GF_PASS" "$GF_URL/api/dashboards/uid/$uid" \ - | jq '.dashboard | .id = null' \ - > "$OUT_DIR/$uid.json" - echo "pulled $uid -> $OUT_DIR/$uid.json" - done diff --git a/scripts/setup_containers.py b/scripts/setup_containers.py index e80d429..1b5d27b 100755 --- a/scripts/setup_containers.py +++ b/scripts/setup_containers.py @@ -58,16 +58,15 @@ def main() -> int: print("Resetting compose services and volumes...") run_command([*compose_cmd, "down", "-v", "--remove-orphans"], cwd=root_dir) - print("Starting mariadb, cms, loki, and promtail...") + print("Starting mariadb and cms...") run_command([*compose_cmd, "up", "-d", "--build", "--remove-orphans"], cwd=root_dir) printable = " ".join(compose_cmd) print("\nStack is up. Useful commands:") print(f" {printable} ps") print(f" {printable} logs -f cms") - print(f" {printable} logs -f promtail") print(f" {printable} down") - print(f" {printable} down -v # remove volumes (DB/Loki data)") + print(f" {printable} down -v # remove volumes (DB data)") return 0 diff --git a/server/internal/database/homepage_carousel_settings.go b/server/internal/database/homepage_carousel_settings.go index 8d487e6..f69a21c 100644 --- a/server/internal/database/homepage_carousel_settings.go +++ b/server/internal/database/homepage_carousel_settings.go @@ -18,7 +18,8 @@ const homepageCarouselSettingKey = "homepage_carousel" // on the Scalene side silently broke the slide. // // The prefix sits under wp-content/uploads because that is the only tree Nginx -// serves (see deploy/nginx/triangle-cms.conf); it is outside the YYYY/MM layout +// serves (the host Nginx site is in the triangle-infrastructure repo); it is +// outside the YYYY/MM layout // so the media reindex and the legacy WP corpus stay visibly separate. // // That location is served with 30-day immutable caching, which assumes a