Repository navigation
SPA-92 Back up the stand as a pair: dump + MinIO volume, with a rehearsed restore - #85
Merged
Merged
Conversation
…with a manifest that checks the pairing and a restore rehearsed into throwaway containers
Atmostone
added a commit
that referenced
this pull request
Sep 21, 2026
…d-restore SPA-92 Back up the stand as a pair: dump + MinIO volume, with a rehearsed restore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes SPA-92.
A Postgres dump is not a backup of this stand.
quality_records.record_s3_path,tasks.log_archive_s3_pathandknowledge_documents.s3_pathhold keys into MinIO, not content — traces, execution snapshots and deliverables live in the volume. Dump the database alone and you restore rows that point into nothing, which is the loss this project already lived through once and the reason SPA-90 exists.scripts/backup.shwrites one directory holdingdb.dump,minio.tar.gz(the whole volume,.minio.sysincluded),annotations.json, and amanifest.jsonthat pairs them: sha256 per artifact, row counts, and the result of looking up every S3 key from the database in the volume being tarred.scripts/restore.sh --scratchrehearses into a throwaway container and volume, verifies artifacts against the manifest before touching anything, and re-asks the pairing question of the restored copy.--liveis the disaster path: it refuses to run while api/scheduler/orchestrator/minio are up and requires the phrase to be typed.The pairing check lives in
scripts/_backup_lib.shand is the same code both paths run — a rehearsal with its own re-implementation would prove the rehearsal works. Which columns hold S3 keys is derived frominformation_schema, so a migration adding a fourth is covered by construction.A backup is insurance, so it is written even when it fails — deliberately the opposite of the reproduction bundle, which refuses to write an archive that does not verify. A bundle is a claim and a bad one should not exist; refusing to preserve an imperfect stand destroys the only copy of it. The verdict is carried instead:
complete: falseand a non-zero exit.The first run of the pairing check passed having examined nothing.
docker exec -iinside the collection loop drained the here-string feeding that loop, so the first query consumed the column list and the result was «0 keys, 0 missing, complete: true» — the same shape as SPA-89's fallback that could never be entered. Fixed structurally, and audited anyway: each column's collected count is compared against an independent SQL count, and a disagreement reportssuspiciousrather than a number.Verified by making it fail
Four live runs on the stand, three of them negative:
verified; rehearsal reproduced all six row countsdb.dumpcorrupted, manifest untouchedincomplete, exit 1The first attempt at the blob test deleted an orphan object no row references, and passed — the test was wrong, not the code.
Notes for the reviewer
.dumpanywhere under$HOME. This backup is the only copy, not a second one.docs/development.mdgains a Backup and restore section;docs/CHANGELOG.mdhas the entry.