firedrill 0.5.0: pgBackRest and WAL-G backups, webhook alerts

README.md

@@ -85,6 +85,7 @@ firedrill run path/to/dump.dump            # restore and report
 firedrill run s3://backups/pg/daily/       # the newest object under a prefix
 firedrill run --live DATABASE_URL --keep backups/   # dump a running database now, prove it restores
 firedrill run dump.dump --image 'pgvector/pgvector:pg{major}'  # extensions the stock image lacks
+firedrill physical pgbackrest --repo DIR --target "..."      # pgBackRest / WAL-G, with PITR -- see below
 firedrill run dump.dump --json report.json # machine-readable
 firedrill run dump.dump --rto 45m          # exceeding the budget is a finding
 firedrill run dump.dump --tier fast        # schema only, for every commit
@@ -200,6 +201,38 @@ supplies.
 A plain dump can only be restored whole, so the `fast` and `sample` tiers
 refuse it (`TIER_UNSUPPORTED`) instead of quietly running a full restore.
 
+## MySQL and MariaDB
+
+`firedrill run backup.sql` recognises a `mysqldump` or `mariadb-dump` file
+(plain or `.gz`) by its own header and takes the same path Postgres does:
+the server version comes from the dump, the restore runs in `mysql:<version>`
+or `mariadb:<version>`, and then it runs smoke, your `semantics:` checks, and
+`CHECK TABLE` on every restored table.
+
+- **Truncation.** Both tools end a finished dump with `-- Dump completed on
+  ...`. The `mysql` client restores a dump cut at a statement boundary with
+  exit 0, so a missing trailer is `ARCHIVE_TRUNCATED` (critical) before
+  anything is restored.
+- **Errors.** The client stops at the first failing statement; its
+  `ERROR nnnn (state) at line N` becomes a finding naming the line, since
+  everything after it was never restored.
+- **MariaDB 10.5.25+/11.x** dumps open with a `/*M!999999\- enable the
+  sandbox mode */` line before the header -- the reason they won't load into
+  MySQL. firedrill reads past it and restores them into MariaDB.
+- The password reaches the client through `MYSQL_PWD` inside the container,
+  never argv.
+
+Tested on [datacharmer/test_db](https://github.com/datacharmer/test_db)'s
+employees database (300,024 employees, 2,844,047 salary rows), dumped by
+each engine's own tool:
+
+| dump | result |
+|---|---|
+| MySQL 8.4, 168 MB | PASS: 8 tables, both exact counts, `CHECK TABLE` clean (restore 63 s) |
+| MariaDB 11.4, 172 MB, with the sandbox-mode line | PASS (restore 25 s) |
+| the MySQL dump gzipped, 36 MB | PASS |
+| the MySQL dump cut off at 80 MB | FAIL: `ARCHIVE_TRUNCATED`, the client's `ERROR 1064 at line 265`, 7 of 8 tables, both counts wrong |
+
 ## Which PostgreSQL versions
 
 Tested end to end against **13, 14, 15, 16, 17 and 18** — dump on any of them,
@@ -404,6 +437,97 @@ catalog facts. It has no field that could hold a row, which is pinned by a
 test, because it is the artefact of this tool most likely to be committed to
 a repo by accident.
 
+## pgBackRest and WAL-G
+
+Most production Postgres isn't backed up with `pg_dump` at all. It's
+backed up with [pgBackRest](https://pgbackrest.org) or
+[WAL-G](https://github.com/wal-g/wal-g): a base backup plus a WAL archive.
+firedrill restores those **with the tool itself**, replays the WAL, and puts
+the recovered server through the same checks:
+
+```
+$ firedrill physical pgbackrest --repo /backups/pgbackrest --target "2026-09-26 20:28:59+00" --config boundary.yml
+  backup    pgBackRest 20260926-202837F  stanza demo  finished 2026-09-26T20:28:56+00:00
+  source    PostgreSQL 16  -> recovered to 2026-09-26 20:28:59+00 on postgres:16
+
+  [ok  ] recover    23.81s  20260926-202837F -> 2026-09-26 20:28:59+00
+  [ok  ] smoke       0.84s  shop: 2 table(s)
+  [ok  ] semantics   0.72s  2 check(s) in shop
+  [ok  ] integrity   2.39s  1 sequence(s)
+
+  PASS -- recovered to the target, and the boundary is where it should be.
+```
+
+```bash
+firedrill physical pgbackrest --repo DIR                 # newest backup, all archived WAL
+firedrill physical walg --repo DIR --target "2026-09-26 20:30:23+00"
+firedrill physical pgbackrest --stanza main --postgres 16  # a remote (S3/GCS/Azure) repository
+firedrill physical walg --repo DIR --max-age 26h          # and the newest backup must be recent
+```
+
+- The major version comes from the repository (`backup.info`, the WAL-G stop
+  sentinel), and the restore runs in `postgres:<major>` with the tool added:
+  pgBackRest from the image's own PGDG repository, WAL-G's release binary
+  pinned by sha256. `--image` takes a `{major}` template for extension images.
+- **A remote repository** is configured the way the tools are always
+  configured -- `PGBACKREST_REPO1_TYPE=s3`, `WALG_S3_PREFIX`, `AWS_*`, and so
+  on. Those variables are passed to the container **by name**, never value.
+- **It never archives into the repository it checks.** A restored data
+  directory carries production's own `archive_command`; started as it is, the
+  drill would push WAL into production's archive. The server is started with
+  `-c archive_mode=off` (which outranks every file), and then *asked*: if it
+  reports anything but `off`, the drill stops with `ARCHIVING_ENABLED`.
+  Tested with a backup that carries `archive_mode = 'on'` from `ALTER SYSTEM`:
+  the repository held the same WAL files before and after. A local
+  repository is also mounted read-only.
+
+Measured against repositories made by pgBackRest 2.59.1 and WAL-G v3.0.9
+themselves (built by `tests/make_physical.py`, in CI):
+
+| case | result |
+|---|---|
+| replay all archived WAL | PASS: all 3 tables back, including the 2 written after the backup |
+| recover to the timestamp between `before_target` and `after_target` | PASS: exactly one of them |
+| a target later than the archive | FAIL `PITR_TARGET_UNREACHED` |
+| a target before every backup (retention already pruned what you'd need) | FAIL `PITR_TARGET_BEFORE_BACKUP` |
+| 64 bytes zeroed in one pgBackRest file | FAIL `RESTORE_FAILED`: *zlib threw error: [-3] data error* |
+| newest backup older than `--max-age` | FAIL `BACKUP_STALE` |
+
+A restore that ran and broke says **FAIL**: the backup was tested, and it is
+broken. COULD NOT VERIFY stays reserved for a check that could not happen.
+
+## Alerts, and running it on a schedule
+
+```bash
+export BACKUP_ALERTS=https://hooks.slack.com/services/...
+firedrill physical pgbackrest --repo /backups --max-age 26h --notify BACKUP_ALERTS
+```
+
+`--notify ENV_VAR` POSTs one JSON message when a drill fails **or could not
+run**. It carries `text` (Slack, Mattermost) and `content` (Discord) with a
+one-line verdict, plus `ok`, `verified`, `worst` and the finding rules for
+anything else. `--notify-on always` reports passes too. The argument is the
+variable's *name*: a webhook URL is a credential. A webhook that refuses the
+message fails the run rather than being skipped quietly, and its URL never
+appears in the error.
+
+Without GitHub Actions, a systemd timer on any host with Docker does it:
+
+```ini
+# /etc/systemd/system/firedrill.service
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/firedrill.env          # BACKUP_ALERTS=..., PGBACKREST_REPO1_* ...
+ExecStart=/usr/local/bin/firedrill physical pgbackrest --stanza main --postgres 16 --max-age 26h --notify BACKUP_ALERTS
+
+# /etc/systemd/system/firedrill.timer
+[Timer]
+OnCalendar=*-*-* 04:17:00
+Persistent=true
+[Install]
+WantedBy=timers.target
+```
+
 ## Point-in-time recovery
 
 A dump proves you can get *a* database back. PITR proves you can get it back to

action.yml

@@ -35,6 +35,40 @@ inputs:
       for databases using extensions the stock postgres image lacks.
     required: false
     default: ""
+  physical:
+    description: >-
+      pgbackrest or walg: restore a physical backup with that tool instead of a
+      dump. Configure a remote repository with PGBACKREST_* / WALG_* / AWS_*
+      variables in the step's `env:`, or give a local one in `repo`.
+    required: false
+    default: ""
+  repo:
+    description: With `physical`, a local repository directory.
+    required: false
+    default: ""
+  stanza:
+    description: "With `physical: pgbackrest`, the stanza."
+    required: false
+    default: ""
+  target:
+    description: With `physical`, recover to this timestamp instead of the end of the archive.
+    required: false
+    default: ""
+  max-age:
+    description: With `physical`, fail when the newest backup is older than this, e.g. 26h.
+    required: false
+    default: ""
+  postgres:
+    description: With `physical` and a remote repository, the Postgres major version.
+    required: false
+    default: ""
+  notify:
+    description: >-
+      Name of an environment variable holding a Slack/Discord/any webhook URL
+      (set it from a secret with `env:`). Posts the verdict when the drill fails
+      or could not run.
+    required: false
+    default: ""
   config:
     description: Path to firedrill.yml. Defaults to one beside the checkout.
     required: false
@@ -118,9 +152,25 @@ runs:
       # becomes an unactionable one.
       run: |
         set -uo pipefail
+        if [ -n "${{ inputs.physical }}" ]; then
+          args=(physical "${{ inputs.physical }}")
+          [ -n "${{ inputs.repo }}" ]     && args+=(--repo "${{ inputs.repo }}")
+          [ -n "${{ inputs.stanza }}" ]   && args+=(--stanza "${{ inputs.stanza }}")
+          [ -n "${{ inputs.target }}" ]   && args+=(--target "${{ inputs.target }}")
+          [ -n "${{ inputs.max-age }}" ]  && args+=(--max-age "${{ inputs.max-age }}")
+          [ -n "${{ inputs.postgres }}" ] && args+=(--postgres "${{ inputs.postgres }}")
+          [ -n "${{ inputs.notify }}" ]   && args+=(--notify "${{ inputs.notify }}")
+          [ -n "${{ inputs.image }}" ]    && args+=(--image "${{ inputs.image }}")
+          [ -n "${{ inputs.config }}" ]   && args+=(--config "${{ inputs.config }}")
+          [ -n "${{ inputs.junit }}" ]    && args+=(--junit "${{ inputs.junit }}")
+          [ -n "${{ inputs.json }}" ]     && args+=(--json "${{ inputs.json }}")
+          args+=(--fail-on "${{ inputs.fail-on }}")
+          firedrill "${args[@]}" | tee firedrill-output.txt || true
+        else
         args=(run)
         [ -n "${{ inputs.dump }}" ]     && args+=("${{ inputs.dump }}")
         [ -n "${{ inputs.live }}" ]     && args+=(--live "${{ inputs.live }}")
+        [ -n "${{ inputs.notify }}" ]   && args+=(--notify "${{ inputs.notify }}")
         [ -n "${{ inputs.keep }}" ]     && args+=(--keep "${{ inputs.keep }}")
         [ -n "${{ inputs.image }}" ]    && args+=(--image "${{ inputs.image }}")
         [ -n "${{ inputs.config }}" ]   && args+=(--config "${{ inputs.config }}")
@@ -132,6 +182,7 @@ runs:
         args+=(--fail-on "${{ inputs.fail-on }}")
 
         firedrill "${args[@]}" | tee firedrill-output.txt || true
+        fi
 
         report="${{ inputs.json }}"
         if [ -n "$report" ] && [ -f "$report" ]; then

firedrill/cli.py

@@ -102,6 +102,43 @@ def build_parser() -> argparse.ArgumentParser:
     drill_pitr.add_argument("--junit", metavar="PATH")
     drill_pitr.add_argument("--quiet", action="store_true")
 
+    phys = sub.add_parser(
+        "physical", help="restore a pgBackRest or WAL-G backup with the tool itself, and report")
+    phys.add_argument("tool", choices=("pgbackrest", "walg"))
+    phys.add_argument("--repo", metavar="DIR",
+                      help="a local repository (mounted read-only). Omit it for a remote "
+                           "one configured through PGBACKREST_* / WALG_* / AWS_* variables")
+    phys.add_argument("--stanza", help="pgBackRest stanza (found automatically if there is one)")
+    phys.add_argument("--backup", metavar="NAME",
+                      help="a specific backup set or base backup (default: the newest)")
+    phys.add_argument("--target", metavar="TIMESTAMP",
+                      help="recover to this moment instead of replaying all archived WAL")
+    phys.add_argument("--database", metavar="NAME",
+                      help="where semantics checks run (default: the only database there is)")
+    phys.add_argument("--postgres", metavar="MAJOR",
+                      help="the major version; needed for a remote repository")
+    phys.add_argument("--max-age", metavar="DURATION",
+                      help="the newest backup must be younger than this, e.g. 26h")
+    phys.add_argument("--image", metavar="TEMPLATE",
+                      help="base image with a {major} slot, e.g. 'pgvector/pgvector:pg{major}'")
+    phys.add_argument("--config", metavar="PATH")
+    phys.add_argument("--no-config", action="store_true")
+    phys.add_argument("--fail-on", choices=SEVERITIES, default="high")
+    phys.add_argument("--ready-timeout", type=int, default=600,
+                      help="seconds to wait for restore and WAL replay (default: %(default)s)")
+    phys.add_argument("--json", metavar="PATH")
+    phys.add_argument("--junit", metavar="PATH")
+    phys.add_argument("--quiet", action="store_true")
+
+    for parser_ in (run, drill_pitr, phys):
+        parser_.add_argument(
+            "--notify", metavar="ENV_VAR",
+            help="POST the verdict to the webhook URL held in this variable (Slack, "
+                 "Discord, Mattermost, or any endpoint). The name, never the URL: a "
+                 "webhook URL is a credential")
+        parser_.add_argument("--notify-on", choices=("fail", "always"), default="fail",
+                             help="when to notify (default: fail -- including COULD NOT VERIFY)")
+
     sub.add_parser("clean", help="remove containers left behind by a crash")
     sub.add_parser("doctor", help="check the environment before trusting a run")
     return parser
@@ -143,6 +180,16 @@ def main(argv: list[str] | None = None) -> int:
     # for and still print a pass.
     try:
         cfg = _load_config(args)
+        if args.command == "physical":
+            if args.image and "{major}" not in args.image:
+                raise config.ConfigError("--image needs a {major} slot")
+            result = drill.run_physical(
+                args.tool, repo=args.repo, stanza=args.stanza, backup=args.backup,
+                target=args.target, database=args.database, cfg=cfg,
+                flavour=args.image or cfg.image or "", pin_major=args.postgres,
+                max_age=_duration(args.max_age) if args.max_age else None,
+                fail_on=args.fail_on, ready_timeout=args.ready_timeout)
+            return _emit(result, args)
         if args.command == "pitr":
             result = drill.run_pitr(
                 args.base, args.wal, args.target, cfg=cfg,
@@ -228,6 +275,15 @@ def _emit(result, args) -> int:
         else:
             with open(args.json, "w", encoding="utf-8") as handle:
                 handle.write(blob + "\n")
+    if getattr(args, "notify", None):
+        from . import notify
+        try:
+            notify.send(result, args.notify, args.notify_on)
+        except notify.NotifyError as exc:
+            # The drill's verdict stands; an alert that did not go out is said
+            # loudly rather than swallowed, and fails the run.
+            print(f"firedrill: could not notify: {exc}", file=sys.stderr)
+            return result.exit_code or 1
     return result.exit_code