RFC: High Availability, Failover & Disaster Recovery

[brandonrc]

Summary

Artifact Keeper has a peer mesh replication system ("Borg Replication") with peer-to-peer swarm distribution, chunked transfers, and network-aware scheduling. However, there's no documented HA strategy, failover behavior, or disaster recovery procedures. For enterprise adoption, operators need to know: what happens when things go wrong?

Current State

What Exists

• Borg Replication: Recursive peer mesh with P2P swarm distribution
• Peer Mesh Network: Full P2P mesh with peer discovery, scored peer selection, chunked swarm transfers, and configurable replication modes (push/pull/mirror)
• Backup Service: Exists in code but no documented restore procedure
• Health Checks: /health endpoint, scanner health monitoring, per-peer heartbeat tracking
• Database: PostgreSQL with 48 migrations (single instance per peer assumed)

What's Missing

• No documented failover behavior when a peer goes down
• No PostgreSQL HA guidance (streaming replication, Patroni, etc.)
• No backup/restore runbook
• No resource limit recommendations
• No capacity planning guide
• Peer behavior during partial mesh partition is undefined
• No split-brain handling documentation for peer mesh

Proposed Architecture Tiers

Tier 1: Single Node (Development / Small Team)

[Client] → [AK Backend] → [PostgreSQL] → [Filesystem]

• Backup: Scheduled pg_dump + filesystem snapshot
• Recovery: Restore from backup
• RTO: Hours | RPO: Last backup

Tier 2: Peer Mesh (Medium Team / Multi-Site)

[Clients] → [Peer A] ←P2P mesh→ [Peer B] ← [Clients]
                ↓                     ↓
          [PostgreSQL A]        [PostgreSQL B]
                ↓                     ↓
              [S3/R2 shared or replicated]

• Peers are equal participants — each can serve reads and writes independently
• Replication via P2P mesh with scored peer selection and configurable modes (push/pull/mirror)
• S3 provides durable storage independent of any single peer
• RTO: Minutes (mesh re-routes) | RPO: Last sync

Tier 3: Active-Active (Enterprise / Global)

[LB] → [AK Peer A] ←P2P mesh→ [AK Peer B] ← [LB]
              ↓                       ↓
        [PG Primary]    →      [PG Replica]
              ↓                       ↓
           [S3 us-east]         [S3 eu-west]

• Multiple peers with shared or cross-replicated storage (S3)
• PostgreSQL streaming replication or Patroni
• Cross-region S3 replication
• Automatic failover via health checks + LB
• RTO: Seconds | RPO: Near-zero

Key Deliverables

1. Failover documentation — What happens when a peer/DB goes down
2. Backup & restore runbook — Step-by-step procedures with verification
3. PostgreSQL HA guide — Patroni, streaming replication, or managed DB recommendations
4. Resource planning guide — CPU/memory/disk recommendations per artifact count
5. Docker Compose HA example — Multi-peer setup with health checks
6. Kubernetes HA manifests — Deployment with PDB, HPA, and anti-affinity
7. Monitoring for HA — Alerts for replication lag, peer health, storage capacity

Questions

1. Should we support active-active writes, or keep a single-writer model with read replicas?
2. Is PostgreSQL HA in scope, or should we recommend managed databases (RDS, Cloud SQL)?
3. Should peers automatically rebalance write responsibility during partial mesh partition?
4. What's the target RTO/RPO for the default deployment?
5. Should we provide a Kubernetes operator for managing HA deployments?

Related

• Borg Replication system: backend/src/replication/
• Peer mesh services: backend/src/services/peer_service.rs, peer_instance_service.rs, sync_worker.rs, transfer_service.rs
• Backup service: backend/src/services/backup_service.rs
• Kubernetes manifests: deploy/k8s/

[todpunk]

In no particular order (sorry, never done github discussions, so they seem foreign for me right now)

• Postgres HA could also just be an option in the helm chart for a Zalando or Percona operator CR.
• I'm personally against ThingA (in this case Artifact Keeper) owning the care and feeding of ThingB and ThingC (in this case S3 configuration and Postgres configuration), but if that's the desired design there's options. Very coupled, though.
• Is this meant to be in addition to what Kubernetes primitives can already handle? If I'm not deploying to K8s there's a lot of questions, but if K8s is the first class citizen, a lot of cloud native pieces become easy to support without extra code, just some optional HPAs and at most an operator (but only if shared state isn't in the DB).
• On the subject of operators, mostly they're to be avoided if they can just use the normal k8s primitives, and that's usually the case unless you have state management operations to consider, like rebalancing indexes when you add/remove nodes, or to prevent downscaling without manual intervention (like strimzi operator is a good example here). I don't know Artifact Keeper super well yet, but it seems like there's no state in the backend that needs to worry about nodes being added/removed from a simple HPA.
• Active-Active Postgres is very difficult. I'd wait for Multigres to handle this if needed, or they can use a managed solution somewhere. I don't think this is a problem for Artifact Keeper to solve specifically.
• Resource Planning guide is awesome, needs real world usage data to start with. Could be a simple calculator page/tool that plugs in the variables into the equation with average defaults, and helps capacity plan. That said, things like Edge nodes complicate this and make it easy to expand in different ways, so the complexity sprawl is quick.

[thimomulder]

As a government organisation with a broad and complex operational landscape, we are seeking to transition from Nexus OOS to a robust, enterprise‑grade solution that supports high availability. You mentioned in your blog "$500k/year on artifact storage", sadly this is quite normal and hope this project will be a competitor for those enterprise solutions.

• Backups and High Availability for the database/S3 data storage layer should be handled at those layers. Of course we can write documentation in a later moment, that describes setting up High Availability on PostgreSQL and/or S3 storage.
• When looking at other applications the following architecture is commonly used. So Artifact-Keeper should focus on make sure the nodes can be uses simultaneously by users.

flowchart TD

    %% ==== Layer 1: Clients ====
    subgraph L1[Client Layer]
        direction TB
        A[Client 1]
        B[Client 2]
        C[Client 3]
    end

    %% ==== Layer 2: Load Balancer ====
    subgraph L2[Load Balancing Layer]
        direction TB
        D[Load Balancer]
    end

    %% ==== Layer 3: Compute Layer (AKS Nodes) ====
    subgraph L3[Compute Layer]
        direction TB
        E[AK Node 1]
        F[AK Node 2]
    end

    %% ==== Layer 4: Data Layer ====
    subgraph L4[Data Layer]
        direction TB
        G[Database]
        H[S3 Storage Layer]
    end

    %% Flow connections
    A --> D
    B --> D
    C --> D

    D --> E
    D --> F

    E --> G
    E --> H
    F --> G
    F --> H

Loading

• /health endpoint needs to make clear that a node is in an unhealthy state, so the Load Balancer knows, not to send requests to that node.

[jstoja]

Hello,

Context: I'm using commercial products and also think their a rip-off. I also operate mission-critical workloads and therefore have experience with Business Continuity and Disaster Recovery more specifically.

Opinion:

• I wouldn't use the peer/mesh replication feature for Business Continuity purposes because I can use the other layer for that. Both PostgreSQL and S3 storage is very well documented on how to achieve High Availability, across regions, etc but also how to restore from backups. Operators are often used to using those technologies and in bigger organizations even have teams to support them (DBAs, Storage team, ...). An organization can then plan for a given RTO/RPO using those technologies, without having to "learn" or test anything related to artifact-keeper.
• I would use the peer/mesh replication feature to offer an local cache for environments that can be down and disconnected from the other instance(s). For example, an environment running on-prem, if connectivity to the peers is lost, it is still up and can be used in that datacenter. You can also synchronize maybe a sub-set of artifacts etc. This can be useful when there's a disconnect between the CI wich pushes artifacts and production that pull artifacts (and you need an always up repository).
• I think this feature is nice, but I would consider very niche and probably not so used.

Best,
J