fix(backup): capture capstan.db in every run and rewrite the DR runbook (agent-os-36o)

[thinkbig1979]

capstan.db was in no snapshot at all. The backup engine's scope was each stack's compose directory; the database — accounts, the encrypted git tokens and restic password, every setting, backup and auto-update policies, the stacks registry, history and the audit log — was simply not backed up.

Two failure modes compounded: the restic repository defaults to <DataDir>/restic-repo, the same volume the database lives on, so losing that one volume lost both; and the runbook never mentioned it, so following it exactly gave you your compose files back and a Capstan with no users, no settings, no credentials, no policies and no history — discovered during an actual outage.

What changes

Every run, scheduled or manual, snapshots capstan.db before the per-stack loop, under its own tag capstan-database. Taking it first means a stack failure part-way through still leaves the database in the repository.

It uses SQLite's VACUUM INTO, not a file copy — capstan.db runs in WAL mode, so a raw copy taken while writes are in flight can be torn or missing recent commits. The staged plaintext copy is removed in a defer, whatever happens next.

A run that saves every stack but loses the database reports partial, never success. Green while the most important artifact is missing is the silent failure this issue exists to remove.

The staging path is fixed at <DataDir>/backup-staging/capstan.db rather than a random temp dir, because it becomes the path recorded inside the snapshot and therefore the path an operator types during a restore. Exported as DatabaseSnapshotPath() and pinned by a test, since README names it literally.

The drill

Full destroy-and-recover, following only the written runbook.

Setup — auth enabled, real user, a git-backed stack pointing at a private GitHub remote with an HTTPS token stored, deliberately non-default retention (keepDaily 3, keepWeekly 9, keepMonthly 11), a 1440-minute schedule, and an rclone remote on a separate bind mount from ./data.

Backup:

7c4512ff  capstan-database,capstan-backup,2026-07-31  /app/data/backup-staging/capstan.db  200.000 KiB
35ff99bf  stacks~drstack:default,capstan-backup,...   /opt/stacks/drstack                   26.731 KiB

Staging directory empty afterwards. Token confirmed encrypted at rest — 0 plaintext occurrences in capstan.db.

Destroy: rm -rf ./data — database and local repository, together. Offsite copy (10 files) untouched.

Recover: fresh container, different hostname, stopped; rclone sync from offsite; restic snapshots --tag capstan-database; restore; start.

Verify:

Check	Result
Log in with the pre-existing account	SUCCESS
Settings survived	keepDaily 3, keepWeekly 9, keepMonthly 11, schedule 1440, rcloneRemote offsite, syncAfterBackup true — scheduler restarted at 24h
Policies and history present	[('stacks~drstack:default', True, 'hot')], 1 run
Secrets decrypt	hasHttpsToken: true, hasPassword: true

The decrypt claim is proved rather than assumed, by re-running the identical restored database under a wrong STORAGE_KEY:

	correct key	wrong key
login	SUCCESS	SUCCESS (bcrypt hashes are not encrypted)
keepDaily / keepWeekly	3 / 9	3 / 9
hasPassword	true	false
hasHttpsToken	true	false

Those flags track successful decryption, not row presence — so the correct-key run demonstrates the restored secrets genuinely decrypt, and the wrong-key run is direct evidence for the runbook's STORAGE_KEY warning and its step 6.

Three runbook corrections that only executing it produced

• Steps 2–4 now run inside a throwaway container from the image. restic and rclone ship in the image at pinned versions and a fresh recovery host usually has neither. More importantly the rclone remote path in Capstan's settings is a path as seen from inside the container — running the documented rclone sync on the host fails with directory not found (observed).
• Delete capstan.db-wal / capstan.db-shm — they belong to the database being replaced.
• chown appuser:appuser the restored file. The restore runs as root, Capstan does not; a root-owned capstan.db leaves the server unable to write to its own database.

Still outstanding

The one acceptance item not yet demonstrated is an authenticated git pull with the restored token. The token decrypts (proved above) but the drill PAT currently carries Metadata: Read without Contents: Read, so GitHub returns 403 to git-upload-pack regardless of Capstan. Re-runnable in seconds once the token is rescoped; nothing in this PR depends on it.

Also filed from the drill: agent-os-pid — a database captured mid-run restores with that run stuck at running. Cosmetic, and the same shape appears when the process is killed mid-run, which nothing currently handles.

Gates

go build and go vet clean; 674 tests in 9 packages (667 baseline + 7 added).

Refs agent-os-36o.