feat(replication): add lag monitoring, circuit breaker, and audit log…

[marvelousufelix]

📋 Description
In a high-growth financial platform, the database is the ultimate bottleneck. A single primary node cannot sustain the write-throughput required by hundreds of thousands of agents and millions of transactions. This issue moves us from a "Monolithic DB" to a Hybrid Distributed Architecture. We combine Read Replicas for low-latency query performance and Database Sharding for linear write-scalability. This strategy ensures that even during massive market volatility or "swarm spikes," your core ledger remains responsive and robust.

🎯 Objective
Scale the data layer horizontally to support global transaction volumes while maintaining strict ACID compliance for financial records.

🛠️ Technical Requirements
Read-Scaling with Replicas:
Configure a "Leader-Follower" replication topology.
The Primary node handles all INSERT/UPDATE operations (transactions).
Read Replicas (distributed across Availability Zones) handle SELECT queries (reports, dashboards, customer histories), offloading 80-90% of the read load from the Primary.
Horizontal Sharding Strategy:
Partition data across multiple independent database clusters (shards) based on a high-cardinality Shard_Key (e.g., Account_ID or Merchant_ID).
Use Consistent Hashing to ensure even distribution and minimize data reshuffling when adding new shards.
Consistency Management:
Implement Read-Your-Writes consistency for critical API endpoints. Ensure that after a transaction is committed to the Primary, the application routes the immediate subsequent read to a synchronized replica or the primary itself to avoid "stale data" issues in the UI.
Operational Isolation:
By sharding, we isolate failures. If Shard-A experiences a latency spike, customers on Shard-B and Shard-C remain unaffected.
Distributed SQL Alternatives:
Evaluate the migration to a Distributed SQL engine (like CockroachDB or TiDB) which natively handles sharding, replication, and global consensus protocols (Raft/Paxos) without the operational burden of manual sharding logic in the application layer.
✅ Acceptance Criteria
The database throughput scales linearly with the number of added shards.
Read operations are automatically load-balanced across replicas without application-level logic changes.
System maintains strict ACID correctness for all ledger operations, even across distributed nodes.
Database failover is automated (RTO < 30 seconds) with zero data loss (RPO = 0).
The system includes real-time monitoring of "Replication Lag" on all replicas, with automated circuit-breaking if lag exceeds the acceptable threshold (e.g., > 100ms).
🔴 Priority: Critical
Labels: Database, Scalability, Infrastructure, Fintech, Performance
closes #398

[drips-wave]

@marvelousufelix Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits