feat: plumb EventRecorder and emit failure events

[fmuyassarov]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add eventRecorder to the driver and as a starting point we record and publish only the following failure events: ClaimPrepareFailed, NetworkDeviceAttachFailed, RDMADeviceAttachFailed.

One thing I wasn't fully convinced about was the benefit of propagating the event into the driver logs via ...

eventBroadcaster.StartLogging(klog.Infof)

This basically replicates the event into the kubectl logs of the driver. In a way I don't see an issue propagating it to the logs and it may help to discovery the failures a bit faster but at the same time I'm thinking that publishing the failure in the events is enough. I don't have a strong opinion on this and would be happy to hear you think on this.
in practice in looks like this

I0605 09:29:20.122080       1 db.go:592] Device "dummy0" not found in local store, rescanning.
I0605 09:29:20.271155       1 dra_hooks.go:432] claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store
I0605 09:29:20.271283       1 nonblockinggrpcserver.go:161] "handling request succeeded" logger="dra" requestID=4 method="/k8s.io.kubelet.pkg.apis.dra.v1.DRAPlugin/NodePrepareResources" response="claims:{key:\"c8810e22-6955-4690-911b-3251b027bcd1\" value:{error:\"claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store\"}}"
I0605 09:29:20.271407       1 event.go:377] Event(v1.ObjectReference{Kind:"ResourceClaim", Namespace:"default", Name:"dummy-interface-static-ip", UID:"c8810e22-6955-4690-911b-3251b027bcd1", APIVersion:"resource.k8s.io/v1", ResourceVersion:"897", FieldPath:""}): type: 'Warning' reason: 'ClaimPrepareFailed' failed to get network interface name for device dummy0: device dummy0 not found in store
I0605 09:30:38.120584       1 db.go:592] Device "dummy0" not found in local store, rescanning.
I0605 09:30:38.271555       1 dra_hooks.go:432] claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store
I0605 09:30:38.271710       1 nonblockinggrpcserver.go:161] "handling request succeeded" logger="dra" requestID=5 method="/k8s.io.kubelet.pkg.apis.dra.v1.DRAPlugin/NodePrepareResources" response="claims:{key:\"c8810e22-6955-4690-911b-3251b027bcd1\" value:{error:\"claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store\"}}"
I0605 09:30:38.271879       1 event.go:377] Event(v1.ObjectReference{Kind:"ResourceClaim", Namespace:"default", Name:"dummy-interface-static-ip", UID:"c8810e22-6955-4690-911b-3251b027bcd1", APIVersion:"resource.k8s.io/v1", ResourceVersion:"897", FieldPath:""}): type: 'Warning' reason: 'ClaimPrepareFailed' failed to get network interface name for device dummy0: device dummy0 not found in store
I0605 09:31:40.121683       1 db.go:592] Device "dummy0" not found in local store, rescanning.
I0605 09:31:40.268307       1 dra_hooks.go:432] claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store
I0605 09:31:40.268448       1 nonblockinggrpcserver.go:161] "handling request succeeded" logger="dra" requestID=6 method="/k8s.io.kubelet.pkg.apis.dra.v1.DRAPlugin/NodePrepareResources" response="claims:{key:\"c8810e22-6955-4690-911b-3251b027bcd1\" value:{error:\"claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store\"}}"
I0605 09:31:40.268559       1 event.go:377] Event(v1.ObjectReference{Kind:"ResourceClaim", Namespace:"default", Name:"dummy-interface-static-ip", UID:"c8810e22-6955-4690-911b-3251b027bcd1", APIVersion:"resource.k8s.io/v1", ResourceVersion:"897", FieldPath:""}): type: 'Warning' reason: 'ClaimPrepareFailed' failed to get network interface name for device dummy0: device dummy0 not found in store

Which issue(s) this PR is related to:

Fixes: #194

Special notes for your reviewer:

I tried to do bare minimum end to end testing and tried to mimic the failure to test at least ClaimPrepareFailed event on a resourceClaim when it can not prepare a device.

The idea was that I create a dummy interface and it is published in the resourceSlice as available device. Then I install the deviceClass and send SIGSTOP signal to the driver pod so that it can't process requests, this is basically to kind of break/pause the pod. Then I deleted the dummy interface while the driver can't process any changes. Finally, I created a sort of workload pod with resourceClaim. So at this point schedular tries to allocate dummy interface since it is available on the resourceSlice (driver isn't aware or didn't have a chance to rescan the devices since we broke it intentionally). Now kubelet makes nodePrepareResources call and then I send SIGCONT signal to the driver Pod so that it starts handling the request but it fails to find the device now since we deleted the interface and returns an error from which we create an event. Sorry if this was long but I could not come up with better idea of testing.

Below are some snippets from my test if that helps to review:

kubectl get pods
NAME   READY   STATUS              RESTARTS   AGE
pod1   0/1     ContainerCreating   0          111s

kubectl get events -w

8m17s       Normal    NodeAllocatableEnforced         node/dra-worker2                          Updated Node Allocatable limit across pods
8m15s       Normal    Starting                        node/dra-worker2
8m15s       Normal    RegisteredNode                  node/dra-worker2                          Node dra-worker2 event: Registered Node dra-worker2 in Controller
8m2s        Normal    NodeReady                       node/dra-worker2                          Node dra-worker2 status is now: NodeReady
5m19s       Warning   ClaimPrepareFailed              resourceclaim/dummy-interface-static-ip   failed to get netlink to interface dummy0: Link not found
17s         Warning   ClaimPrepareFailed              resourceclaim/dummy-interface-static-ip   failed to get network interface name for device dummy0: device dummy0 not found in store
5m34s       Normal    Scheduled                       pod/pod1                                  Successfully assigned default/pod1 to dra-worker2
5m19s       Warning   FailedPrepareDynamicResources   pod/pod1                                  Failed to prepare dynamic resources: prepare dynamic resources: NodePrepareResources failed for ResourceClaim dummy-interface-static-ip: claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get netlink to interface dummy0: Link not found
17s         Warning   FailedPrepareDynamicResources   pod/pod1                                  Failed to prepare dynamic resources: prepare dynamic resources: NodePrepareResources failed for ResourceClaim dummy-interface-static-ip: claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store

kubectl describe  resourceClaim dummy-interface-static-ip
...
Events:
  Type     Reason              Age   From     Message
  ----     ------              ----  ----     -------
  Warning  ClaimPrepareFailed  43s   dra.net  failed to get netlink to interface dummy0: Link not found

kubectl describe pod pod1

Events:
  Type     Reason                         Age    From               Message
  ----     ------                         ----   ----               -------
  Normal   Scheduled                      2m22s  default-scheduler  Successfully assigned default/pod1 to dra-worker2
  Warning  FailedPrepareDynamicResources  2m7s   kubelet            Failed to prepare dynamic resources: prepare dynamic resources: NodePrepareResources failed for ResourceClaim dummy-interface-static-ip: claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get netlink to interface dummy0: Link not found
  Warning  FailedPrepareDynamicResources  43s    kubelet            Failed to prepare dynamic resources: prepare dynamic resources: NodePrepareResources failed for ResourceClaim dummy-interface-static-ip: claim c8810e22-6955-4690-911b-3251b027bcd1 contain errors: failed to get network interface name for device dummy0: device dummy0 not found in store

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for dranet canceled.

Name	Link
🔨 Latest commit	3b10dbf
🔍 Latest deploy log	https://app.netlify.com/projects/dranet/deploys/6a2a6486b870a9000856b4d2

[fmuyassarov]

/cc @aojea @gauravkghildiyal

[aojea]

@fmuyassarov one last push

[fmuyassarov]

@fmuyassarov one last push

on it

[fmuyassarov]

@fmuyassarov one last push

on it

should be ready now

[fmuyassarov]

/hold
until I make the CI happy

[fmuyassarov]

/hold until I make the CI happy

should be fixed now and is ready to another round of review.
/hold cancel

[aojea]

@fmuyassarov my apologies, I didn't see before the events were sent on clean up, I prefer not to send them there ... you remove those and we merge

[fmuyassarov]

@fmuyassarov my apologies, I didn't see before the events were sent on clean up, I prefer not to send them there ... you remove those and we merge

No worries and that makes sense. It is cleaned up now.

[fmuyassarov]

rebased with required changes after #223 landed

[aojea]

/lgtm
/approve

Thanks for your patience

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, fmuyassarov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fmuyassarov]

/lgtm /approve

Thanks for your patience

thanks for reviewing