Recommended way to configure/run in multi-replica setting

[ahmetb]

I'm looking at using this awesome library in my admission webhook after a long search.

I'm curious if the library has any builtin mechanisms to coordinate first-time cert provisioning or renewals when the webhook itself is deployed as a ReplicaSet with >1 instances (and they race each other and end up with different certs or have write-write conflict on webhookconfiguration caBundle field)?

Or is this concern inherently not valid (maybe because Secrets eventually propagate and processes restart etc)?

[maxsmythe]

Right now we simply let all controllers run without concern for neighboring controllers. This does mean there is a bit of a first-past-the-post race condition, where the first write wins, but the system should eventually converge.

It might be interesting to add leader election or similar in the future, which may make it easier to support gradual rotation of certs as expiry nears.

[ahmetb]

@maxsmythe I appreciate the answer!

Do you mind elaborating a bit on the progression of events? I can probably read the code as well, but I am hoping this serves future readers as well as convey how I'm evaluating the solution.

If N replicas start at the same, do they all realize they need certs and start self-signing a cert, followed by updating the WebhookConfiguration's caBundle?

If that's the case, which one would win (do we only write if caBundle is empty/outdated)?

Presumably, updating the Secret and writing the xWebhookConfiguration objects are not atomic, so is there a chance if two actors (1 & 2) create caBundle1, secret1, caBundle2, secret2, would we potentially end up with TLS secret set to secret1 but xWebhookConfiguration set to caBundle2? What's preventing this from occurring?

Similarly, when a renewal occurs on another replica, how does the restart machinery tell the other instances they need to restart as well?

There's probably an invariant in the reconciliation I'm missing that helps solve these.