Recover a stateful service after losing a cloud region, from code and offsite backups.
Gitea and PostgreSQL run on a kubeadm-built Kubernetes cluster on Google Cloud VMs in Finland. When the primary region fails, one GitHub Actions workflow builds a new cluster in Belgium, restores the newest backup, and moves the public name git.sindrg.com to it. A scheduled watchdog can start that workflow on its own. The lab models organizations that must run Kubernetes on VMs they control. The project is complete and its environment removed; see Status.
Results · How recovery works · Stack · Status · Documentation
Only automated runs count. In the unattended drills of milestone 11, a workflow stops the primary, the watchdog starts the recovery, and nobody acts. The targets are a recovery time of at most 4 hours and a data loss of at most 2 hours (decision 0002). Two consecutive drills on 2026-10-19 met both, with a 2 GiB dataset on the primary:
| Drill | Detection | Recovery time | Data loss |
|---|---|---|---|
| 20261019T170003Z | 13 min 15 s | 34 min 21 s | 10 min 18 s |
| 20261019T185625Z | 18 min 15 s | 36 min 00 s | 5 min 09 s |
The recovery time runs from the first failed probe to the checks through the public name, so it includes detection. For drill 20261019T185625Z:
| Phase | Stages | Duration |
|---|---|---|
| Detection | From the first failed probe to the watchdog's Recover run |
18 min 15 s |
| Job setup | Before the first stage | 54 s |
| Preflight | preflight |
1 min 40 s |
| Build the cluster | provisioning, bootstrap, flux-reconcile |
11 min 01 s |
| Restore and check | restore, verification |
3 min 38 s |
| Fence and cut over | fence, dns-cutover, probe-recovery |
32 s |
| Total | 36 min 00 s |
Detection and building the cluster take most of the time. Detection needs five minutes of failed checks and then a watchdog run, which GitHub schedules every five minutes but started 12 and 13 minutes apart in these drills. The drill results list every drill, and the worklog compares them with the attended drills.
- Back up. Every 15 minutes the primary captures PostgreSQL and the Gitea repositories from one LVM snapshot while Gitea keeps running, encrypts them with age, and uploads them to a bucket in Belgium. The writer cannot delete or overwrite a backup.
- Detect. An uptime check probes
git.sindrg.comfrom three continents. A scheduled watchdog starts a recovery only when every checker has failed for five minutes, the runner cannot reach the primary either, and theAUTO_FAILOVERswitch is on. It started the recovery in every unattended drill of milestone 11. - Recover. The
Recoverworkflow signs in to Google Cloud without a stored key and runs these stages in order: preflight, provisioning, bootstrap, Flux reconcile, restore, verification, fence, DNS cutover, checks through the public name, and resuming backups. Each stage ends with a check. A failure stops the run before the cutover, and a rerun resumes at the failed stage. - Return. After a drill, the
Returnworkflow starts the primary, moves the name back, and removes the recovery environment. It discards the writes that the recovery cluster accepted.
The recovery region runs no VMs until a recovery starts. Terraform state, the deployment source, the backups, and the recovery credentials all stay reachable without the primary region. The architecture pages explain each part.
| Tool | Role |
|---|---|
| Terraform | VMs, network, load balancer, buckets, and the pipeline identity |
| Ansible and kubeadm | Node configuration and the Kubernetes bootstrap |
| Flux and Helm | The application layer, reconciled from this repository |
| Traefik and cert-manager | HTTPS through the Gateway API, with Let's Encrypt certificates |
| SOPS and age | Encrypted secrets in the public repository and encrypted backups |
| GitHub Actions | Tests and linters, pin and public-surface checks, the watchdog, and the recovery, return, and drill workflows |
| Cloud Monitoring, Healthchecks.io, Grafana Cloud | Uptime checks from outside, a backup heartbeat, and cluster metrics that outlive the primary region |
The project is complete: milestones 0 to 11, from the recovery contract to unattended drills, all passed their gates. The environment was removed after the close, so git.sindrg.com no longer answers; the results above link to the workflow runs. The worklogs record every milestone and its gate, and the close records the teardown.
Production readiness lists what a production deployment adds to this lab.
| To find | Read |
|---|---|
| How each part works, with diagrams | Architecture |
| Why it is built this way | Decision records |
| The recovery procedure | Regional recovery runbook |
| Automated runs, stage by stage | Worklog 10 |
| How to build it yourself | Reproduce the project |
| Everything else | Documentation index |
Run make to list the operator commands.
MIT.