fix(backup): capture capstan.db in every run and rewrite the DR runbook (agent-os-36o)

README.md

@@ -89,9 +89,35 @@ Capstan includes a built-in backup engine powered by [restic](https://restic.net
 
 ### What gets backed up
 
-Each stack's compose directory is backed up as a restic snapshot tagged with the stack ID.
+Every backup run captures two kinds of thing:
+
+| Artifact | restic tag | Notes |
+| --- | --- | --- |
+| Each stack's compose directory | the stack ID | One snapshot per enabled stack |
+| `capstan.db` | `capstan-database` | Captured on **every** run, before the stacks |
+
+The database is what makes an instance *this* instance: user accounts, the
+encrypted git tokens and restic password, every setting, backup and auto-update
+policies, the stacks registry, and the audit log. Without it a restore gives you
+your compose files back and nothing else.
+
+It is snapshotted with SQLite's `VACUUM INTO` rather than copied. `capstan.db`
+runs in WAL mode, so a plain file copy taken while writes are in flight can be
+torn or missing recent commits.
+
+A run that saves every stack but fails to capture the database is reported as
+**partial**, never success.
+
 Backups can be triggered manually (Settings, Backup tab) or run on a schedule.
 
+> **`STORAGE_KEY` is not in the backup, and that is deliberate.** The secrets
+> inside `capstan.db` are encrypted with a key derived from it, so a stolen
+> backup does not yield your git tokens. The corollary is that restoring onto a
+> host with a *different* `STORAGE_KEY` gives you a database whose secrets
+> cannot be decrypted. Store `STORAGE_KEY` and `JWT_SECRET` in the same password
+> manager as the restic repository password. All three are needed for a complete
+> recovery.
+
 ### Configuration
 
 The quickest path is the **Settings UI** (Settings, Backup). All fields save to the
@@ -157,15 +183,146 @@ curl -X POST http://localhost:5001/api/v1/backups/restore \
 
 ### Disaster recovery
 
-If the host is lost, recover from an rclone remote:
+Follow this in order. **The database is restored before the stacks** — restoring
+stacks first gives you a Capstan that does not know they exist.
+
+#### Before you start, you need
+
+| Item | Why |
+| --- | --- |
+| The restic repository password | Without it the repository cannot be opened at all |
+| `STORAGE_KEY` | The old value, or every stored secret is unreadable |
+| `JWT_SECRET` | A different value invalidates existing sessions (survivable) |
+| Access to the rclone remote | The only copy that survives losing the data volume |
+
+If you have the repository but not `STORAGE_KEY`, you can still recover
+everything except the encrypted secrets — see step 6.
+
+#### 1. Deploy a fresh instance, stopped
+
+Bring up a new Capstan with the same `./data` bind mount path and the **same**
+`STORAGE_KEY` and `JWT_SECRET` as the lost instance, then stop the container:
+
+```bash
+docker compose -f docker-compose.prod.yaml up -d
+docker compose -f docker-compose.prod.yaml stop capstan
+```
+
+Stopping matters. Overwriting `capstan.db` underneath a running server corrupts
+it — the server holds the WAL open and will write over what you just restored.
+
+#### Run steps 2–4 inside the image, not on the host
+
+Every command below is run in a throwaway container built from the Capstan
+image, with the same volumes. Three reasons, all learned the hard way in a real
+drill:
+
+- `restic` and `rclone` ship **inside** the image at pinned versions. A freshly
+  provisioned recovery host usually has neither, and installing them mid-outage
+  is a poor use of the outage.
+- The rclone remote path stored in Capstan's settings is a path **as seen from
+  inside the container**. Running the same `rclone sync` on the host resolves it
+  against the host filesystem and fails with `directory not found`.
+- The paths inside the restic snapshot are container paths, so restoring in the
+  same namespace keeps them meaningful.
+
+Define this once, adjusting the paths to your deployment:
+
+```bash
+alias capstan-tools='docker run --rm \
+  -v ./data:/app/data \
+  -v ~/.config/rclone/rclone.conf:/home/appuser/.config/rclone/rclone.conf:ro \
+  --entrypoint sh ghcr.io/thinkbig1979/capstan:latest -c'
+```
+
+#### 2. Get the repository back from the remote
+
+```bash
+capstan-tools 'rclone sync <remote>:<path> /app/data/restic-repo --stats-one-line'
+```
+
+`<remote>:<path>` are exactly the values shown in Settings → Backup. This is
+what the UI's DR Restore does; doing it by hand keeps step 1's "container
+stopped" rule intact.
+
+#### 3. Find the database snapshot
+
+```bash
+capstan-tools '
+  printf "%s" "$RESTIC_PASSWORD" > /tmp/pw
+  export RESTIC_REPOSITORY=/app/data/restic-repo RESTIC_PASSWORD_FILE=/tmp/pw
+  restic snapshots --tag capstan-database'
+```
+
+The newest one is normally what you want. Note its short ID.
+
+#### 4. Restore the database
+
+The snapshot contains the file at `/app/data/backup-staging/capstan.db`, so a
+restore reproduces that path underneath `--target`:
+
+```bash
+capstan-tools '
+  set -e
+  printf "%s" "$RESTIC_PASSWORD" > /tmp/pw
+  export RESTIC_REPOSITORY=/app/data/restic-repo RESTIC_PASSWORD_FILE=/tmp/pw
+  restic restore <snapshot-id> --target /tmp/capstan-restore
+  cp /tmp/capstan-restore/app/data/backup-staging/capstan.db /app/data/capstan.db
+  rm -rf /tmp/capstan-restore
+  rm -f /app/data/capstan.db-wal /app/data/capstan.db-shm
+  chown appuser:appuser /app/data/capstan.db'
+```
+
+Two details that will bite if skipped:
+
+- **Delete `capstan.db-wal` and `capstan.db-shm`.** They belong to the database
+  you just replaced. Left in place, SQLite tries to apply them to the restored
+  file.
+- **`chown appuser:appuser`.** The restore runs as root; Capstan runs as
+  `appuser` (UID 1000 by default, remappable via `PUID`/`PGID`). A root-owned
+  `capstan.db` leaves the server unable to write to its own database.
+
+#### 5. Start Capstan and verify before going further
+
+```bash
+docker compose -f docker-compose.prod.yaml start capstan
+```
+
+Check all four, in this order. Each one fails differently:
+
+1. **Log in** with a pre-existing account. Failure here means the wrong
+   `capstan.db`, not a `STORAGE_KEY` problem.
+2. **Settings → Backup** shows your repository path, retention and schedule.
+   Empty defaults mean the database did not actually land.
+3. **A git-backed stack can pull.** This is the `STORAGE_KEY` test — the token
+   is decrypted to run it. An authentication failure here with the right
+   password means the key differs from the one that encrypted it.
+4. **Settings → Backup → History** lists runs from before the incident, and your
+   backup policies are still enabled.
+
+#### 6. If `STORAGE_KEY` was lost
+
+Everything except the encrypted secrets is recoverable. Accounts, settings,
+policies, history and the stacks registry all come back. What must be re-entered
+by hand: the restic repository password, git HTTPS tokens, and any other stored
+credential. Re-enter them in Settings; the new values are encrypted with the new
+key.
+
+#### 7. Restore the stacks
+
+Only now, with a working Capstan that knows about them, restore the compose
+directories from Settings → Backup → Snapshots, or via the API:
+
+```bash
+curl -X POST http://localhost:5001/api/v1/backups/restore \
+  -H 'Content-Type: application/json' \
+  -d '{"stackId":"<id>","snapshotId":"<short-id>","confirm":true}'
+```
 
-1. Deploy a fresh Capstan instance with the same `./data` bind mount path.
-2. Trigger a DR restore (Settings, Backup, DR Restore). This syncs the full
-   restic repository from the configured remote back to `/app/data/restic-repo`.
-3. Restore individual stacks via the Snapshots panel.
+#### Practise this
 
-The restic repository password is required for DR recovery. Store it separately
-from the server (e.g. a password manager).
+A runbook that has never been executed is a hypothesis. Run it end to end
+against a scratch instance at least once, before you need it.
 
 ## Volume Path Identity
 

backend/internal/database/database.go

@@ -2,7 +2,9 @@ package database
 
 import (
 	"database/sql"
+	"fmt"
 	"os"
+	"strings"
 	"time"
 
 	_ "modernc.org/sqlite"
@@ -92,3 +94,32 @@ func NewWithMigrationsAndEncryptor(dataDir string, encryptor TokenEncryptor) (*D
 func (d *DB) Close() error {
 	return d.db.Close()
 }
+
+// VacuumInto writes a consistent point-in-time copy of the database to dest
+// using SQLite's `VACUUM INTO`.
+//
+// This is not a convenience wrapper around a file copy — it is the only correct
+// way to snapshot this database. capstan.db runs in WAL mode (see the
+// journal_mode pragma above), so the .db file on disk is not self-contained:
+// recent commits live in the -wal sidecar until a checkpoint. Copying the file
+// while writes are in flight yields a torn or stale database. VACUUM INTO runs
+// through the SQL layer, so it observes a single consistent snapshot and emits
+// a standalone, already-compacted database with no sidecar files.
+//
+// It does not block writers.
+//
+// dest must not already exist — SQLite refuses to overwrite, and that refusal
+// is deliberate rather than something to work around: silently clobbering a
+// previous snapshot would destroy the artifact a restore depends on. Callers
+// remove a stale file explicitly if they intend to replace it.
+func (d *DB) VacuumInto(dest string) error {
+	// Parameter binding is not permitted for VACUUM INTO in SQLite, so the path
+	// is interpolated. Callers construct dest from configuration, never from
+	// user input; the quote-doubling keeps a path containing a single quote from
+	// terminating the literal.
+	quoted := "'" + strings.ReplaceAll(dest, "'", "''") + "'"
+	if _, err := d.db.Exec("VACUUM INTO " + quoted); err != nil {
+		return fmt.Errorf("vacuum into %s: %w", dest, err)
+	}
+	return nil
+}

backend/internal/database/database_vacuum_test.go

@@ -0,0 +1,72 @@
+package database
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestVacuumInto_ProducesAnOpenableStandaloneCopy pins the primitive the backup
+// engine relies on (agent-os-36o).
+//
+// The point is not that a file appears — it is that the file is a complete,
+// self-contained database containing data committed right up to the snapshot.
+// capstan.db runs in WAL mode, so a plain file copy can miss recent commits
+// that are still in the -wal sidecar. This asserts the copy round-trips through
+// a fresh connection and carries the data.
+func TestVacuumInto_ProducesAnOpenableStandaloneCopy(t *testing.T) {
+	t.Parallel()
+
+	src, err := NewWithMigrations(":memory:")
+	require.NoError(t, err)
+	t.Cleanup(func() { src.Close() })
+
+	require.NoError(t, src.SetSetting("canary", "vacuum-into-round-trip"))
+
+	// Stage the copy as <dir>/capstan.db so it can be reopened by passing <dir>
+	// to NewWithMigrations, which is exactly how a restore puts it back.
+	restoreDir := filepath.Join(t.TempDir(), "restored")
+	require.NoError(t, os.MkdirAll(restoreDir, 0o700))
+	dest := filepath.Join(restoreDir, "capstan.db")
+	require.NoError(t, src.VacuumInto(dest))
+
+	info, statErr := os.Stat(dest)
+	require.NoError(t, statErr, "the snapshot file must exist")
+	assert.Greater(t, info.Size(), int64(0), "the snapshot must not be empty")
+
+	// No sidecars: VACUUM INTO emits a standalone database, which is what makes
+	// it safe to hand to restic as a single file.
+	for _, sidecar := range []string{dest + "-wal", dest + "-shm"} {
+		_, sErr := os.Stat(sidecar)
+		assert.True(t, os.IsNotExist(sErr), "%s must not exist alongside the snapshot", sidecar)
+	}
+
+	// Reopen the copy independently and confirm the committed row is present.
+	restored, err := NewWithMigrations(restoreDir)
+	require.NoError(t, err, "the snapshot must be openable as a database in its own right")
+	t.Cleanup(func() { restored.Close() })
+
+	got, err := restored.GetSetting("canary")
+	require.NoError(t, err)
+	assert.Equal(t, "vacuum-into-round-trip", got,
+		"data committed before the snapshot must survive into the copy")
+}
+
+// TestVacuumInto_RefusesToOverwrite documents that SQLite will not clobber an
+// existing destination. Callers must remove a stale file deliberately —
+// silently overwriting would destroy the artifact a restore depends on.
+func TestVacuumInto_RefusesToOverwrite(t *testing.T) {
+	t.Parallel()
+
+	src, err := NewWithMigrations(":memory:")
+	require.NoError(t, err)
+	t.Cleanup(func() { src.Close() })
+
+	dest := filepath.Join(t.TempDir(), "snapshot.db")
+	require.NoError(t, os.WriteFile(dest, []byte("pre-existing"), 0o600))
+
+	assert.Error(t, src.VacuumInto(dest), "VACUUM INTO must refuse an existing destination")
+}