Skip to content

SPA-92 Back up the stand as a pair: dump + MinIO volume, with a rehearsed restore - #85

Merged
Atmostone merged 1 commit into
masterfrom
alexeybalexeyb/spa-92-backup-and-restore
Sep 21, 2026
Merged

Atmostone merged 1 commit into
masterfrom
alexeybalexeyb/spa-92-backup-and-restore

Conversation

@Atmostone

Copy link
Copy Markdown
Owner

Closes SPA-92.

A Postgres dump is not a backup of this stand. quality_records.record_s3_path, tasks.log_archive_s3_path and knowledge_documents.s3_path hold keys into MinIO, not content — traces, execution snapshots and deliverables live in the volume. Dump the database alone and you restore rows that point into nothing, which is the loss this project already lived through once and the reason SPA-90 exists.

scripts/backup.sh writes one directory holding db.dump, minio.tar.gz (the whole volume, .minio.sys included), annotations.json, and a manifest.json that pairs them: sha256 per artifact, row counts, and the result of looking up every S3 key from the database in the volume being tarred. scripts/restore.sh --scratch rehearses into a throwaway container and volume, verifies artifacts against the manifest before touching anything, and re-asks the pairing question of the restored copy. --live is the disaster path: it refuses to run while api/scheduler/orchestrator/minio are up and requires the phrase to be typed.

The pairing check lives in scripts/_backup_lib.sh and is the same code both paths run — a rehearsal with its own re-implementation would prove the rehearsal works. Which columns hold S3 keys is derived from information_schema, so a migration adding a fourth is covered by construction.

A backup is insurance, so it is written even when it fails — deliberately the opposite of the reproduction bundle, which refuses to write an archive that does not verify. A bundle is a claim and a bad one should not exist; refusing to preserve an imperfect stand destroys the only copy of it. The verdict is carried instead: complete: false and a non-zero exit.

The first run of the pairing check passed having examined nothing. docker exec -i inside the collection loop drained the here-string feeding that loop, so the first query consumed the column list and the result was «0 keys, 0 missing, complete: true» — the same shape as SPA-89's fallback that could never be entered. Fixed structurally, and audited anyway: each column's collected count is compared against an independent SQL count, and a disagreement reports suspicious rather than a number.

Verified by making it fail

Four live runs on the stand, three of them negative:

Scenario Result
Happy path 16 keys, all resolving, verified; rehearsal reproduced all six row counts
A referenced blob stripped from the tarball rehearsal reports 1 missing + the manifest mismatch, exit 1
db.dump corrupted, manifest untouched refused on checksum before anything was restored, exit 1
Stand built with one dangling key backup written but incomplete, exit 1

The first attempt at the blob test deleted an orphan object no row references, and passed — the test was wrong, not the code.

Notes for the reviewer

  • The safety net the ticket described as «verified present» no longer exists on this machine — no .dump anywhere under $HOME. This backup is the only copy, not a second one.
  • The wipe half of the ticket was not performed: the live database is no longer the 4b/5b corpus it assumed, so there is nothing left to wipe.
  • docs/development.md gains a Backup and restore section; docs/CHANGELOG.md has the entry.

…with a manifest that checks the pairing and a restore rehearsed into throwaway containers
@Atmostone
Atmostone merged commit 241da37 into master Sep 21, 2026
1 of 2 checks passed
Atmostone added a commit that referenced this pull request Sep 21, 2026
…d-restore

SPA-92 Back up the stand as a pair: dump + MinIO volume, with a rehearsed restore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant