Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/backup-destination-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wpelevator/tug': minor
---

Add `tug backup destination check`, which opens the control repository and the repository of every project with a backup target from the control host, creates missing ones and exits nonzero when any fails. `tug backup destination set` runs the check after saving (skip it with `--no-check`). Backup destinations may now use any built-in restic backend: `s3:`, `rest:`, `b2:`, `azure:`, `gs:` and `swift:`. `tug backup destination set` keeps exit code 0 when the check fails, since the destination is saved. Adds `POST /api/v1/backup/destination/check` and `deployed` in the repository password response.
5 changes: 5 additions & 0 deletions .changeset/rest-repository-probe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@wpelevator/tug': patch
---

Decide whether a backup repository is missing from restic's exit code 10 instead of its message, in control and on nodes. Control's own backup now creates the `_control` repository on a REST server, and a mistyped S3 bucket fails instead of being created. The protection against creating a mistyped bucket applies once each node's agent is upgraded, because nodes run the check themselves. Automatic repository creation now requires restic 0.17 or newer, which matters only with a custom `TUG_RESTIC_IMAGE`; the error names a restic older than 0.17. `backup.check` no longer initializes a repository, and a wrong repository password is named in the error. `sftp:` and `rclone:` destinations were accepted but never worked, so rejecting them with the reason is a fix. `tug backup password` and the repository password endpoint work before a project is deployed and say whether it has been, and `tug backup run` explains that a project that was never deployed has no backup targets yet.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Managed ingress balances HTTP requests across serving replicas with round robin.
| --- | --- | --- |
| Get images to servers | Builds your images on control (or on your machine with `tug deploy --build-local`) and sends them over connections the agents already make. Or push to any HTTPS registry and declare `image:`, and control pins the tag to a digest before deploying. A failed build leaves the nodes untouched. | [Control plane](docs/control-plane.md#image-delivery-without-a-registry) |
| Serve HTTPS | Managed Caddy serves a hostname with `tls: auto`, gets the certificate and renews it. DNS must point at the server and ports 80 and 443 must be reachable. | [Ingress](docs/ingress.md) |
| Back up and restore | Scheduled [restic](https://restic.net) backups of volumes and databases to S3-compatible storage. `tug restore run db` brings one back. Control backs itself up every night. | [Backups](docs/backups.md) |
| Back up and restore | Scheduled [restic](https://restic.net) backups of volumes and databases to S3-compatible storage, a restic REST server and other restic backends. `tug restore run db` brings one back. Control backs itself up every night. | [Backups](docs/backups.md) |
| Keep secrets out of git | `tug secret set` keeps them encrypted on control and delivers each only to the nodes whose services use it. Every `${NAME}` in your Compose file comes from control, never from the shell or `.env` of whoever runs the deploy. | [Where values come from](docs/compose.md#where-values-come-from) |
| Roll back | `tug rollback` returns to a previous deployment, and a top-level `x-tug: { rollback: auto }` does it when a node reports the new version unhealthy. It cannot undo database writes or schema changes. | [Health checks](docs/healthchecks.md) |
| Update without a restart | `x-tug: { rollout: { strategy: rolling } }` starts the new revision beside the old one and switches ingress once it is healthy. It needs ingress, a tug health check and one published port. The default `recreate` restarts the container and can interrupt traffic. | [Rolling rollouts](docs/compose.md#rolling-rollouts) |
Expand Down
1 change: 1 addition & 0 deletions docs/access-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@ The project column says how a scoped credential's allowed projects are checked:
| `GET /api/v1/backup/destination` | `admin` | Control-wide; URL and variable names, never values |
| `POST /api/v1/backup/destination` | `admin` | Control-wide |
| `DELETE /api/v1/backup/destination` | `admin` | Control-wide |
| `POST /api/v1/backup/destination/check` | `admin` | Control-wide; opens the destination's repositories and creates missing ones |
| `GET /api/v1/backup/repository-password` | `admin` | Query `project` prints that project's repository password; `control` prints the control repository password |
| `POST /api/v1/secrets` | `admin` | Control-wide |
| `DELETE /api/v1/secrets/:name` | `admin` | Control-wide |
Expand Down
61 changes: 56 additions & 5 deletions docs/backups.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# Backups and recovery

Mark a service or a volume with `x-tug.backup`, give control one S3-compatible destination, and agents run [restic](https://restic.net) on the node that holds the data on a schedule. There is no backup container to build or keep alive.
Mark a service or a volume with `x-tug.backup`, give control one restic repository location (S3-compatible storage, a REST server and other [destinations](#destinations)), and agents run [restic](https://restic.net) on the node that holds the data on a schedule. There is no backup container to build or keep alive.

## Set up your first backup

You need S3-compatible storage reachable from the workload node, and credentials allowed to use the bucket. Create the bucket first. Mark each target in your Compose file as shown in [Compose extensions](compose.md#backups), and see the [WordPress example](https://github.com/wpelevator/tug/blob/main/examples/wordpress/README.md) for a database and files together. A top-level schedule only supplies defaults and enables nothing.
You need storage restic can use, reachable from the control host and from the workload node, and credentials allowed to use it. The example uses S3-compatible storage, see [Destinations](#destinations) for the others. Create the bucket first. Mark each target in your Compose file as shown in [Compose extensions](compose.md#backups), and see the [WordPress example](https://github.com/wpelevator/tug/blob/main/examples/wordpress/README.md) for a database and files together. A top-level schedule only supplies defaults and enables nothing.

```sh
# Supply the values through your shell or secret manager. Do not commit them.
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
tug backup destination set s3:https://s3.example.com/bucket/tug-backups --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY
tug backup destination check
tug deploy
tug backup run
tug backup list --check
Expand All @@ -18,6 +19,8 @@ tug backup password

| Command | What it tells you |
| --- | --- |
| `backup destination set` | Saves the destination, then checks it from the control host. A failed check is a warning and the exit code stays 0 (`--no-check` skips the check) |
| `backup destination check` | Opens the control repository and each project's repository, creates missing ones and exits nonzero if any fails. Skips project repositories when `_control` fails |
| `backup run` | Waits for the configured targets and reports failure |
| `backup list --check` | Whether each target succeeded, and exits nonzero if not |
| `backup password` | The repository and its password. Store it, with the storage credentials, off the server |
Expand All @@ -27,6 +30,51 @@ The password is derived from control's master key, so losing both the password a

Control also backs itself up to the same destination every night, see [Control plane backup](#control-plane-backup). Configure [signed webhooks](observability.md#alerting) for backup failures, missed protection and recovery notifications. Delivery is best effort; keep an independent scheduled `tug backup list --check`.

## Destinations

A destination is a restic repository location. `tug backup destination set <url>` accepts these forms, and each project and control get their own repository below it. Credentials are never part of the URL: pass them as `--env` variables, which control stores encrypted. `RESTIC_PASSWORD`, `RESTIC_REPOSITORY` and `RESTIC_PASSWORD_FILE` are managed by tug and refused.

| URL form | Credentials (`--env`) | Status |
| --- | --- | --- |
| `s3:https://host/bucket[/prefix]` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (required) | Exercised on real hosts |
| `rest:http://host:8000/path` | `RESTIC_REST_USERNAME`, `RESTIC_REST_PASSWORD`, or none for a server started with `--no-auth` | Verified against restic's `rest-server` |
| `b2:bucket:path` | `B2_ACCOUNT_ID`, `B2_ACCOUNT_KEY` | Accepted, not yet exercised in a real-host run |
| `azure:container:/path` | `AZURE_ACCOUNT_NAME` with `AZURE_ACCOUNT_KEY` or `AZURE_ACCOUNT_SAS` | Accepted, not yet exercised in a real-host run |
| `gs:bucket:/path` | `GOOGLE_PROJECT_ID` with `GOOGLE_ACCESS_TOKEN`, or `GOOGLE_APPLICATION_CREDENTIALS` | Accepted, not yet exercised in a real-host run. A credentials file path must exist inside the helper container, which tug cannot provide, so prefer `GOOGLE_ACCESS_TOKEN` |
| `swift:container:/path` | The `OS_*` variables restic documents for your Keystone setup | Accepted, not yet exercised in a real-host run |
| `/absolute/path` | None | Development only: the path is read on each node and is not shared between nodes |

Each backend's variables are listed in restic's [Preparing a new repository](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html).

### REST server

A [rest-server](https://github.com/restic/rest-server) is the simplest destination you host yourself:

```sh
export RESTIC_REST_USERNAME RESTIC_REST_PASSWORD
tug backup destination set rest:http://backup.internal:8000/tug --env RESTIC_REST_USERNAME --env RESTIC_REST_PASSWORD
```

Plain HTTP is acceptable only on a private network or VPN that the control host and every node share. Use `rest:https://` anywhere else. A server started with `--no-auth` needs no `--env`.

### Not supported

`sftp:` is rejected because tug cannot give the restic helper container SSH keys or a `known_hosts` file. Run a rest-server and reach it over a private network or VPN, or use S3. `rclone:` is rejected because the helper image does not configure rclone.

### Check the destination

```sh
tug backup destination check
```

The check runs from the control host in a restic helper container. It opens `_control` and the repository of every project with an enabled backup target. An existing repository is `ok`. A missing one, which restic reports with exit code 10, is created with `restic init` and reported as `initialized`, which also proves the credentials can write. Anything else is `error` with restic's message: a wrong repository password (exit code 12), a bucket that does not exist, a refused login or an unreachable store. A bucket is never created by a check or a backup, so a typo in the bucket name fails instead. When `_control` fails, the project repositories are `skipped`, because they share the destination and would fail the same way, each after its own timeout. The command exits nonzero when any repository fails, and `tug backup destination show` returns the last result as `checkResult`. A check refuses to start while control's own backup runs or another check is running, and the whole check stops after four minutes.

`initialized` means the check created a new, empty repository. During a [migration](migration.md#coming-from-an-existing-restic-setup) an unexpected `created a new, empty repository` means the destination path is wrong, because the repository you expected would have been found.

`tug backup destination set` runs the same check right after saving. The destination stays saved and the exit code stays 0 when the check fails: stderr says `saved; the check failed` and names the failing repositories, and you fix the cause and run `tug backup destination check`. A control too old to check destinations prints a warning instead. Pass `--no-check` to skip it. Nodes connect to the destination at their first backup, so a passing check proves the control host's access, not that every node can reach the destination.

Repositories are created by the first check or backup only when restic is 0.17 or newer, the first version that reports a missing repository with exit code 10. A failure from an image older than 0.17 says so. The stock image is new enough, so this matters only with a custom `TUG_RESTIC_IMAGE`.

## Plan for recovery

Workload snapshots hold application data. Control archives hold tug state and keys. Store both away from the VPS, and keep their credentials and passwords somewhere you can reach if the server is lost.
Expand Down Expand Up @@ -107,6 +155,8 @@ backup:
| `backup.no_destination` | No destination is configured |
| `backup.cancelled` | Someone ran `tug backup cancel <run>`. A cancelled backup is a warning and is not retried |

Exit status is the signal to wire into monitoring: `tug backup list --check` exits nonzero when a target is stale, failed or missed, or when nothing is protected. Run it from cron or a monitor, and keep running it during a [migration](migration.md#coming-from-an-existing-restic-setup), when the old backup system may be failing without anyone noticing.

While control is down no backup starts, which is the price of one scheduler. Running jobs finish and report when control returns. `tug backup run` takes a backup now, and `tug backup list` shows the catalog, see [CLI](cli.md#backups).

## Restore
Expand Down Expand Up @@ -179,7 +229,7 @@ As soon as a backup destination exists, control backs itself up with nothing mor
| Who runs it | Control runs the restic helper on its own Docker socket, so the control host needs no node and no cron job |
| Stalls | restic retries a failing store (a missing bucket, a refused login, an unreachable host) for many minutes. When it prints only retry notices for two minutes (`TUG_CONTROL_BACKUP_STALL_MS`), the backup fails with restic's final message and last cause, which `tug backup list` shows |

Create the bucket first. A repository is initialized only when restic's final answer is that it does not exist, so a bucket created while a backup was retrying is picked up by the next attempt.
Create the bucket first. A repository is initialized only when restic exits with code 10 (the repository does not exist). A missing bucket makes restic retry and fail, so a bucket created while a backup was retrying is picked up by the next attempt.

`tug backup list` shows a `control` line with the last success in the destination now in use. Switching destination starts a first backup there and reports it as unprotected until that backup completes. A local-path destination (for development) is labelled `local path only: not off-host`. `tug backup list --check` fails when the last control backup failed, when none succeeded for twice the interval (two days by default), or when a destination has had no completed backup an hour after it was set. `tug events --project _control` shows `backup.completed` and `backup.failed`.

Expand Down Expand Up @@ -224,7 +274,8 @@ A node runs backup and restore jobs in lanes: a restore and its safety backup fi
| Control down | No backup starts. Running jobs finish and report when control returns, and a long gap yields one missed event and one catch-up run per target |
| Consistency | Live volumes are crash-consistent, not application-consistent. Use a command backup for anything with its own consistency |
| One node per volume | A volume is backed up from the node that holds it. A volume mounted from a file server with `driver_opts` is read through that node's mount, so back up the file server's own disk where you can |
| Not built | Repository maintenance (`restic check`, a read-verify of the data) and SFTP destinations |
| Not built | Repository maintenance (`restic check`, a read-verify of the data) |
| Rejected destinations | `sftp:` and `rclone:` URLs, see [Not supported](#not-supported) |
| Leftovers | A scratch restore leaves its volume (`tug_r<project-length>_<project>_<id>`) behind, and the safety snapshot of every in-place restore is kept for ever. No command lists or removes either. Remove scratch volumes with `docker volume rm` on the node, and safety snapshots with `restic forget` using the repository from `tug backup password` |
| Helper memory | The restic helper defaults to 512 MB, half of a 1 GB host. Lower it with `memory` under `x-tug.backup`. Larger repositories need more, so test the limit against your data |
| `tug exec` | Cancelling it does not stop the remote process |
Expand All @@ -244,6 +295,6 @@ Backup destination settings: the control plane uses one destination, and `tug ba
| Variable | Where | Default | Purpose |
| --- | --- | --- | --- |
| `TUG_BACKUP_TICK_MS` | Control | `30000` | Scheduler tick |
| `TUG_RESTIC_IMAGE` | Control | `restic/restic:0.19.1` | restic helper image |
| `TUG_RESTIC_IMAGE` | Control | `restic/restic:0.19.1` | restic helper image. Must be 0.17 or newer, the first release that exits with code 10 for a missing repository; older images cannot create repositories automatically |
| `TUG_BACKUP_CONCURRENCY` | Agent | `2` on 3 GiB or more, else `1` | Concurrent backup jobs per node |
| `TUG_CONTROL_BACKUP_INTERVAL_MS`, `TUG_CONTROL_BACKUP_RETRY_MS`, `TUG_CONTROL_BACKUP_STALL_MS`, `TUG_CONTROL_BACKUP_LIMIT_MS`, `TUG_CONTROL_BACKUP_DIR` | Control | See [Configuration](configuration.md#control-plane) | Control backup timing and location |
Loading
Loading