Admission webhook using incorrect certificate after restart

[natalian98]

After restarting an ec2 instance with kubeflow deployment, admission webhook seems to be using uncorrect certificate.
This leads to issues with notebook servers: already existing ones can't be started and new ones can't be created due to error Internal error occurred: failed calling webhook "admission-webhook.kubeflow.org": failed to call webhook: Post "https://admission-webhook.kubeflow.svc:4443/apply-poddefault?timeout=10s": x509: certificate signed by unknown authority

The admission-webhook is throwing TLS handshake errors:

I0824 12:11:49.489999       1 main.go:616] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), TLSConfig:(*tls.Config)(0xc0004c6480), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:0, disableKeepAlives:0, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), doneChan:(chan struct {})(nil), onShutdown:[]func()(nil)}
2022/08/24 12:16:24 http: TLS handshake error from 172.31.73.87:38080: remote error: tls: bad certificate
2022/08/24 12:16:24 http: TLS handshake error from 172.31.73.87:34560: remote error: tls: bad certificate

It tries to connect to controller-service's api-server but it's unreachable after restart.

$ juju debug-log --replay --include admission-webhook
application-admission-webhook: 12:08:11 INFO juju.cmd running jujud [2.9.33 e83d2a73f904080c5cdf4aaed2821abd4f58253a gc go1.18.5]
application-admission-webhook: 12:08:11 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=admission-webhook", "--debug"}
application-admission-webhook: 12:08:11 DEBUG juju.agent read agent config, format "2.0"
application-admission-webhook: 12:08:11 INFO juju.worker.upgradesteps upgrade steps for 2.9.33 have already been run.
application-admission-webhook: 12:08:11 INFO juju.cmd.jujud caas operator application-admission-webhook start (2.9.33 [gc])
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "clock" manifold worker started at 2022-08-24 12:08:11.040627232 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-admission-webhook"
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2022-08-24 12:08:11.041395951 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.introspection stats worker now serving
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-24 12:08:11.041710838 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "agent" manifold worker started at 2022-08-24 12:08:11.041746533 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-24 12:08:11.050878063 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2022-08-24 12:08:11.052175801 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2022-08-24 12:08:11.053470761 +0000 UTC
application-admission-webhook: 12:08:11 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:11 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2022-08-24 12:08:11.071644351 +0000 UTC
application-admission-webhook: 12:08:14 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:14 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:14 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:14 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:17 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:20 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:20 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:08:21 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:21 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:21 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:25 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:28 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:28 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:28 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:28 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:34 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:37 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:37 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:08:37 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:37 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:37 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:43 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:46 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:46 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:08:46 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:46 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: try was stopped
stack trace:
try was stopped
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:46 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: try was stopped
application-admission-webhook: 12:08:54 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:08:57 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:08:57 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:08:57 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:08:57 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:08:57 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:06 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:09:09 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:09 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:09:09 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:09:09 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:09:09 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:21 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:09:24 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:24 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:09:24 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:09:24 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:09:24 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:36 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:09:39 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:39 DEBUG juju.api no error, but not connected, probably cancelled before we started
application-admission-webhook: 12:09:39 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:09:39 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:09:39 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:54 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:09:57 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:09:57 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:09:57 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
stack trace:
github.com/juju/juju/api.(*addressProvider).next:803: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/api.recordTryError.func1:1057: 
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:09:57 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
application-admission-webhook: 12:10:15 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:10:15 DEBUG juju.api looked up controller-service.controller-my-controller.svc.cluster.local -> [10.152.183.228]
application-admission-webhook: 12:10:15 DEBUG juju.worker.apicaller [646c5e] failed to connect
application-admission-webhook: 12:10:15 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: unable to connect to API: dial tcp 10.152.183.228:17070: connect: connection refused
stack trace:
dial tcp 10.152.183.228:17070: connect: connection refused
github.com/juju/juju/api.gorillaDialWebsocket:737: 
github.com/juju/juju/api.dialer.dial1:1157: 
github.com/juju/juju/api.dialer.dial:1132: unable to connect to API
github.com/juju/juju/api.dialWebsocketMulti:1028: 
github.com/juju/juju/api.dialAPI:686: 
github.com/juju/juju/api.Open:218: 
github.com/juju/juju/worker/apicaller.connectFallback:161: 
github.com/juju/juju/worker/apicaller.OnlyConnect:58: 
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api
application-admission-webhook: 12:10:15 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: unable to connect to API: dial tcp 10.152.183.228:17070: connect: connection refused
application-admission-webhook: 12:10:38 DEBUG juju.worker.apicaller connecting with old password
application-admission-webhook: 12:10:38 DEBUG juju.api successfully dialed "wss://10.152.183.228:17070/model/646c5e31-e9b1-4440-8487-421d41eeec13/api"
application-admission-webhook: 12:10:38 INFO juju.api cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: operation was canceled
application-admission-webhook: 12:10:38 INFO juju.api connection established to "wss://10.152.183.228:17070/model/646c5e31-e9b1-4440-8487-421d41eeec13/api"
application-admission-webhook: 12:10:38 INFO juju.worker.apicaller [646c5e] "application-admission-webhook" successfully connected to "10.152.183.228:17070"
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2022-08-24 12:10:38.080910968 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-24 12:10:38.089304877 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2022-08-24 12:10:38.09049801 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2022-08-24 12:10:38.091603492 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2022-08-24 12:10:38.091723727 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2022-08-24 12:10:38.091823131 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2022-08-24 12:10:38.097327819 +0000 UTC
application-admission-webhook: 12:10:38 INFO juju.worker.caasupgrader abort check blocked until version event received
application-admission-webhook: 12:10:38 DEBUG juju.worker.caasupgrader current agent binary version: 2.9.33
application-admission-webhook: 12:10:38 INFO juju.worker.caasupgrader unblocking abort check
application-admission-webhook: 12:10:38 INFO juju.worker.migrationminion migration phase is now: NONE
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2022-08-24 12:10:38.107536846 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2022-08-24 12:10:38.108163289 +0000 UTC
application-admission-webhook: 12:10:38 INFO juju.worker.logger logger worker started
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2022-08-24 12:10:38.108504014 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2022-08-24 12:10:38.108835072 +0000 UTC
application-admission-webhook: 12:10:38 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=INFO"
application-admission-webhook: 12:10:38 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
application-admission-webhook: 12:10:38 INFO juju.worker.caasoperator.charm downloading ch:amd64/focal/admission-webhook-42 from API server
application-admission-webhook: 12:10:38 INFO juju.downloader downloading from ch:amd64/focal/admission-webhook-42
application-admission-webhook: 12:10:38 INFO juju.downloader download complete ("ch:amd64/focal/admission-webhook-42")
application-admission-webhook: 12:10:38 INFO juju.downloader download verified ("ch:amd64/focal/admission-webhook-42")
application-admission-webhook: 12:10:57 INFO juju.worker.caasoperator operator "admission-webhook" started
application-admission-webhook: 12:10:57 INFO juju.worker.caasoperator.runner start "admission-webhook/4"
application-admission-webhook: 12:10:57 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-admission-webhook-4
application-admission-webhook: 12:10:57 INFO juju.worker.leadership admission-webhook/4 promoted to leadership of admission-webhook
application-admission-webhook: 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4 unit "admission-webhook/4" started
application-admission-webhook: 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4 hooks are retried true
application-admission-webhook: 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4.charm downloading ch:amd64/focal/admission-webhook-42 from API server
application-admission-webhook: 12:10:57 INFO juju.downloader downloading from ch:amd64/focal/admission-webhook-42
application-admission-webhook: 12:10:57 INFO juju.downloader download complete ("ch:amd64/focal/admission-webhook-42")
application-admission-webhook: 12:10:57 INFO juju.downloader download verified ("ch:amd64/focal/admission-webhook-42")
application-admission-webhook: 12:11:18 INFO juju.worker.caasoperator.uniter.admission-webhook/4 found queued "upgrade-charm" hook
application-admission-webhook: 12:11:24 INFO unit.admission-webhook/4.juju-log Running legacy hooks/upgrade-charm.
application-admission-webhook: 12:11:30 WARNING unit.admission-webhook/4.upgrade-charm Generating RSA private key, 2048 bit long modulus (2 primes)
application-admission-webhook: 12:11:30 WARNING unit.admission-webhook/4.upgrade-charm .............+++++
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm .........................................................................................+++++
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm e is 65537 (0x010001)
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm Generating RSA private key, 2048 bit long modulus (2 primes)
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm ...........+++++
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm ................................................................................+++++
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm e is 65537 (0x010001)
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm Signature ok
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
application-admission-webhook: 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm Getting CA Private Key
application-admission-webhook: 12:11:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "upgrade-charm" hook (via hook dispatching script: dispatch)
application-admission-webhook: 12:11:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4 found queued "config-changed" hook
application-admission-webhook: 12:11:37 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-admission-webhook: 12:11:37 INFO juju.worker.caasoperator started pod init on "admission-webhook/4"
application-admission-webhook: 12:15:42 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-admission-webhook: 12:21:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-admission-webhook: 12:26:02 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-admission-webhook: 12:30:38 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation ran "update-status" hook (via hook dispatching script: dispatch)

api-server logs

2022-08-24 12:10:38 INFO juju.apiserver.connection request_notifier.go:96 agent login: application-admission-webhook for 646c5e31-e9b1-4440-8487-421d41eeec13
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:10:38 INFO juju.apiserver.connection request_notifier.go:96 agent login: application-admission-webhook for 646c5e31-e9b1-4440-8487-421d41eeec13 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 INFO juju.cmd supercommand.go:56 running jujud [2.9.33 e83d2a73f904080c5cdf4aaed2821abd4f58253a gc go1.18.5] 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.cmd supercommand.go:57   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=admission-webhook", "--debug"} 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.agent agent.go:603 read agent config, format "2.0" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 INFO juju.worker.upgradesteps worker.go:60 upgrade steps for 2.9.33 have already been run. 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 INFO juju.cmd.jujud caasoperator.go:204 caas operator application-admission-webhook start (2.9.33 [gc]) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "clock" manifold worker started at 2022-08-24 12:08:11.040627232 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.introspection worker.go:135 introspection worker listening on "@jujud-application-admission-webhook" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "upgrade-steps-gate" manifold worker started at 2022-08-24 12:08:11.041395951 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "caas-units-manager" manifold worker started at 2022-08-24 12:08:11.041710838 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "agent" manifold worker started at 2022-08-24 12:08:11.041746533 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.introspection worker.go:161 stats worker now serving 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:601 "caas-units-manager" manifold worker completed successfully 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "caas-units-manager" manifold worker started at 2022-08-24 12:08:11.050878063 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "api-config-watcher" manifold worker started at 2022-08-24 12:08:11.052175801 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "upgrade-steps-flag" manifold worker started at 2022-08-24 12:08:11.053470761 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:11 DEBUG juju.worker.dependency engine.go:578 "migration-fortress" manifold worker started at 2022-08-24 12:08:11.071644351 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:14 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:14 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:14 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:14 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:17 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:20 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:20 DEBUG juju.api apiclient.go:1137 no error, but not connected, probably cancelled before we started 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:21 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:21 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:21 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:25 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:28 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:28 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:28 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:28 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:34 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:37 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:08:37 DEBUG juju.api apiclient.go:1137 no error, but not connected, probably cancelled before we started 
[multiple same errors]
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:39 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:39 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:39 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:54 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:57 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:57 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:57 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:09:57 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: i/o timeout 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:15 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:15 DEBUG juju.api apiclient.go:806 looked up controller-service.controller-my-controller.svc.cluster.local -> [10.152.183.228] 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:15 DEBUG juju.worker.apicaller connect.go:160 [646c5e] failed to connect 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:15 DEBUG juju.worker.dependency engine.go:616 "api-caller" manifold worker stopped: [646c5e] "application-admission-webhook" cannot open api: unable to connect to API: dial tcp 10.152.183.228:17070: connect: connection refused
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [646c5e] "application-admission-webhook" cannot open api 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:15 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [646c5e] "application-admission-webhook" cannot open api: unable to connect to API: dial tcp 10.152.183.228:17070: connect: connection refused 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.apicaller connect.go:129 connecting with old password 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.api apiclient.go:1153 successfully dialed "wss://10.152.183.228:17070/model/646c5e31-e9b1-4440-8487-421d41eeec13/api" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: operation was canceled 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.api apiclient.go:688 connection established to "wss://10.152.183.228:17070/model/646c5e31-e9b1-4440-8487-421d41eeec13/api" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.apicaller connect.go:163 [646c5e] "application-admission-webhook" successfully connected to "10.152.183.228:17070" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "api-caller" manifold worker started at 2022-08-24 12:10:38.080910968 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:601 "caas-units-manager" manifold worker completed successfully 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "caas-units-manager" manifold worker started at 2022-08-24 12:10:38.089304877 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "upgrader" manifold worker started at 2022-08-24 12:10:38.09049801 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "log-sender" manifold worker started at 2022-08-24 12:10:38.091603492 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "migration-minion" manifold worker started at 2022-08-24 12:10:38.091723727 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "upgrade-steps-runner" manifold worker started at 2022-08-24 12:10:38.091823131 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:601 "upgrade-steps-runner" manifold worker completed successfully 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "migration-inactive-flag" manifold worker started at 2022-08-24 12:10:38.097327819 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.caasupgrader upgrader.go:113 abort check blocked until version event received 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.caasupgrader upgrader.go:128 current agent binary version: 2.9.33 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.caasupgrader upgrader.go:119 unblocking abort check 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "charm-dir" manifold worker started at 2022-08-24 12:10:38.107536846 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.logger logger.go:65 initial log config: "<root>=DEBUG" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "logging-config-updater" manifold worker started at 2022-08-24 12:10:38.108163289 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.logger logger.go:120 logger worker started 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "api-address-updater" manifold worker started at 2022-08-24 12:10:38.108504014 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.dependency engine.go:578 "proxy-config-updater" manifold worker started at 2022-08-24 12:10:38.108835072 +0000 UTC 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 DEBUG juju.worker.logger logger.go:93 reconfiguring logging from "<root>=DEBUG" to "<root>=INFO" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 WARNING juju.worker.proxyupdater proxyupdater.go:282 unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.worker.caasoperator.charm bundles.go:78 downloading ch:amd64/focal/admission-webhook-42 from API server 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.downloader download.go:110 downloading from ch:amd64/focal/admission-webhook-42 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.downloader download.go:93 download complete ("ch:amd64/focal/admission-webhook-42") 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:38 INFO juju.downloader download.go:173 download verified ("ch:amd64/focal/admission-webhook-42") 
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:10:52 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-admission-webhook" with version "v1beta1" 
2022-08-24 12:10:52 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-admission-webhook" with version "v1beta1"
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.caasoperator caasoperator.go:424 operator "admission-webhook" started 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.caasoperator.runner runner.go:556 start "admission-webhook/4" 
2022-08-24 12:10:57 INFO juju.worker.raft.raftforwarder target.go:174 claiming lease "646c5e31-e9b1-4440-8487-421d41eeec13:application-leadership#admission-webhook#" for "admission-webhook/4"
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:10:57 INFO juju.worker.raft.raftforwarder target.go:174 claiming lease "646c5e31-e9b1-4440-8487-421d41eeec13:application-leadership#admission-webhook#" for "admission-webhook/4" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-admission-webhook-4 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.leadership tracker.go:194 admission-webhook/4 promoted to leadership of admission-webhook 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4 uniter.go:326 unit "admission-webhook/4" started 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4 uniter.go:344 hooks are retried true 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.worker.caasoperator.uniter.admission-webhook/4.charm bundles.go:78 downloading ch:amd64/focal/admission-webhook-42 from API server 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.downloader download.go:110 downloading from ch:amd64/focal/admission-webhook-42 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.downloader download.go:93 download complete ("ch:amd64/focal/admission-webhook-42") 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:10:57 INFO juju.downloader download.go:173 download verified ("ch:amd64/focal/admission-webhook-42") 
2022-08-24 12:11:10 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-katib.kubeflow.org" with version "v1beta1"
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:11:10 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-katib.kubeflow.org" with version "v1beta1" 
2022-08-24 12:11:11 INFO juju.kubernetes.provider admissionregistration.go:249 ensuring validating webhook "kubeflow-katib.kubeflow.org" with version "v1beta1"
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:11:11 INFO juju.kubernetes.provider admissionregistration.go:249 ensuring validating webhook "kubeflow-katib.kubeflow.org" with version "v1beta1" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:18 INFO juju.worker.caasoperator.uniter.admission-webhook/4 resolver.go:149 found queued "upgrade-charm" hook 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:24 INFO unit.admission-webhook/4.juju-log server.go:316 Running legacy hooks/upgrade-charm. 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:30 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 Generating RSA private key, 2048 bit long modulus (2 primes) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:30 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 .............+++++ 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 .........................................................................................+++++ 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 e is 65537 (0x010001) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 Generating RSA private key, 2048 bit long modulus (2 primes) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 ...........+++++ 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 ................................................................................+++++ 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 e is 65537 (0x010001) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 Signature ok 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:31 WARNING unit.admission-webhook/4.upgrade-charm logger.go:60 Getting CA Private Key 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "upgrade-charm" hook (via hook dispatching script: dispatch) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4 resolver.go:149 found queued "config-changed" hook 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:37 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:11:37 INFO juju.worker.caasoperator initializer.go:116 started pod init on "admission-webhook/4" 
2022-08-24 12:12:37 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-admission-webhook" with version "v1beta1"
5353a87f-64ff-44da-8664-389bc91fcbf6: controller-0 2022-08-24 12:12:37 INFO juju.kubernetes.provider admissionregistration.go:59 ensuring mutating webhook "kubeflow-admission-webhook" with version "v1beta1" 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:15:42 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:21:33 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:26:02 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch) 
646c5e31-e9b1-4440-8487-421d41eeec13: application-admission-webhook 2022-08-24 12:30:38 INFO juju.worker.caasoperator.uniter.admission-webhook/4.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch) 

Steps to reproduce

Deploy 1.6 bundle
Connect to dashboard
Create a notebook
Restart the instance
Start an existing notebook server or create a new one

Workaround

Restart the deployment:
kubectl rollout restart deployment/admission-webhook -n kubeflow

[DomFleischmann]

We will address this issue when moving to the tls-certificate interface that has been defined by the telco team. See https://github.com/canonical/tls-certificates-operator

[i-chvets]

Jira

[fer-marino]

I'm still seeing this with kubeflow 1.8. The workaround didn't work for me