Add Leader Election Logic for High Availability in ComputeDomain Controller

[Lily922]

Currently, the ComputeDomain (CD) Controller can only run in a single replica mode to avoid conflicts, which creates a risk of downtime if the instance fails. This makes the controller highly vulnerable to failures. A leader election capability should be added to the CD Controller. This would enable it to operate in a multi-replica setup, ensuring that only one replica is responsible for processing resource state changes at any given time, while the other replicas remain idle.

[herb-duan]

Additional Value of This Feature

Beyond enabling high availability (multi-replica deployment), the leader election implementation also fixes a critical data consistency issue:

During rolling upgrades of the compute-domain-controller, old and new replicas run concurrently without leader election, leading to duplicate creation of ResourceClaimTemplate and compute-domain-daemonset resources. These duplicates cannot be cleaned up automatically by the controller (abnormal scenario for controller) and leave permanent data residue (requiring manual cleanup).

The PR #851 resolves this issue by ensuring only one leader replica processes resource changes at any time, eliminating duplicate resource creation during upgrades.

[jgehrcke]

@herb-duan thank you for

During rolling upgrades of the compute-domain-controller, old and new replicas run concurrently [...] leading to duplicate creation [...] resources

We preferred tracking this specific problem in its own issue, and landed a dedicated patch which hopefully addresses it.