docs(dr): make the runbook's rclone step work as written - #36
Merged
Conversation
Step 2 of the disaster-recovery runbook failed for anyone following it
verbatim. The capstan-tools alias mounts rclone.conf at the appuser path, but
--entrypoint sh skips the entrypoint that drops privileges, so the throwaway
container runs as root with HOME=/root and rclone never opened the file:
NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
CRITICAL: Failed to create file system for "offsite:/offsite/capstan-repo":
didn't find section in config file ("offsite")
The message points at the remote, not at the config, so it sends you
debugging the wrong thing during an outage — and step 2 is the one that
retrieves the only copy of the repository that survived losing the volume.
Setting RCLONE_CONFIG explicitly fixes it regardless of which user runs, and
the reason is documented next to it so nobody "tidies" the variable away.
Also notes that a local-filesystem remote needs its own bind mount, since the
path is resolved inside the container.
Observed while re-running the destroy-and-recover drill for agent-os-36o.
agent-os-36o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of
agent-os-36o. Found by re-running the destroy-and-recover drill end to end against the fixedagent-os-qqwcode.The defect
Step 2 of the DR runbook — the one that retrieves the only copy of the restic repository that survived losing the data volume — fails for anyone following it verbatim.
The
capstan-toolsalias mountsrclone.confat/home/appuser/.config/rclone/rclone.conf, but--entrypoint shskips the entrypoint that would drop toappuser. The throwaway container therefore runs as root withHOME=/root, and rclone never opens the file that was just mounted:The runbook already knows the container runs as root — step 4 says so, and that is why it ends in
chown appuser:appuser. The alias just never accounted for it.What makes this worth a fix rather than a footnote: the error names the remote, not the config file. During an outage that sends you checking credentials and connectivity for a remote that is fine.
The fix
-e RCLONE_CONFIG=/home/appuser/.config/rclone/rclone.confon the alias — correct whichever user ends up running it — with the reason documented inline so it does not get tidied away later.Also notes that a local-filesystem remote needs its own bind mount on the alias, since the path resolves inside the container.
Verified: the same command, with the variable set, completes and lands
restic-repo/in the data volume. Steps 3, 4 and 5 then run exactly as written, and the drill reaches a working restored instance.Scope
README only. No code changes.