Questions/Proposal on how to achieve high availability with docker swarms

[Musholic]

Hello,

I'm interested in contributing to this plugin to achieve true high availability with a Docker Swarm setup in case of node/disk failures or maintenance (with minimal data loss). I would like your input regarding the following points, in case I'm missing anything.

The assumption is that I have two hosts hA and hB that should hold the most recent data (duplicated in case one of them is not available). The goal is that a service running on hC will periodically replicate its data, and if it moves to hD, it will continue with the most recently available data on hA and hB.

• The current sync has a few issues: it's less efficient than a btrfs send, it does not check if the remote data is actually newer, and it does not delete local data (could be fixed with -u or --delete flags, I think).
• The replicate command does not send the same snapshot to multiple hosts at once, and will create a snapshot even if there is no change (so if I schedule a replicate every minute on two different hosts, it will create two snapshots every minute).
  • It's actually not a big deal as they can be purged.

To make my scenario work, I will have to implement some kind of snapshot_sync command that would execute automatically before the container starts, and will:

• Check the most recent snapshots on the given list of remote hosts.
• Check if one of them is more recent than its own snapshots.
• If yes, create a snapshot (as backup), fetch the remote snapshot, and restore it.
• Before the service exits, it should replicate a new snapshot to the given remote hosts (an additional replicate schedule, during the lifetime of the service, should probably be automated to avoid data loss).

For context, I'm really happy to have found your plugin as I tested a lot of replicated storage solutions (SeaweedFS, Garage, JuiceFS, rclone, ...), but all of them don't work well with async operations or with high latency between nodes; they divide the performance of my small nodes by a lot and use a lot of resources. Some don't work well with databases data. While this solution does not prevent the complete loss of data, I think it's currently the best compromise for me.

[ccomb]

Hi ! I'm glad to see some interest in this plugin! I've one in production for sensitive data for years, but it's an old version. I've started last year to improve it, but I'm still not using the latest version, and it possibly need some migration strategy because the name of the plugin is still with the old name.

And unfortunately I've got not so much time currently. :-/

Let me explain the original vision behind this plugin. I wanted to run several servers (say A, B, C) to host multiple applications, each on one server. This idea was not to run with an orchestrator (neither swarm or k8n) , but to run independant servers. The key idea was using the replicate schedule command to get data replication : each application has an active node, where the application runs, and a single replicate node, which hosts periodic incremental data of the active node.
In case the active node fails, or if we just want to move the application to another server, we freeze data on the active node, make a last send to the replicate node, which get the latest data and switch the replicate to be the new active node. the old active node becomes the replicate. This is some kind of peer to peer hosting replication.

I didn't want to go with multiple replicates because it leads to split-brain considerations and data starts to look like a git repository. Each replicate can be considered a branch, we never know which node contains the latest data. Unless maybe we replicate to two nodes instead of one, with the same frozen data? However in the end we need to know where the data is : I had implemented some kind of distributed repository with consul of what node hosts which data, but in the end it stard to look like a partial ochestrator.

The sync feature was added later, and the main initial purpose was to keep a volume of the same letsencrypt certificates on all the nodes, because we may have a reverse proxy in front of them.

Is that close or consistent with the way you want to use it?

[Musholic]

Thanks for your answer. I understand your worries about split-brain issues.

I've just finished a first iteration, c.f PR, (no multi-host backup support) of my proposal and I'm really happy with the first results, I can stop a container and let it move to another host, or I can drain a host, shutdown a host, and the container will magically continue working with the same data on another host. I'll migrate/create other services in the following days to use the same setup.

In my scenario, docker swarm is the orchestrator responsible for knowing where the container is living, and even in cases where we could lose data, we got some snapshots as backup (and probably we should avoid purging these backups too aggressively, but this can be fixed).

Does it make sense to you? Have a look also at the implementation logic I described in #69