Add advanced health check configuration to config file

[elezar]

Enhanced Error-handling config

Current State

See https://docs.nvidia.com/deploy/xid-errors/index.html#xid-error-listing

The NVIDIA GPU Device Plugin

We register for NVML Events of type nvml.EventTypeXidCriticalError | nvml.EventTypeDoubleBitEccError | nvml.EventTypeSingleBitEccError

We treat the following XIDs as non-fatal errors:

XID	Description
13	Graphics Engine Exception
31	GPU memory page fault
43	GPU stopped processing
45	Preemptive cleanup, due to previous errors
68	Video processor exception
109	Context Switch Timeout Error

We allow additional Xids to be specified in the DP_DISABLE_HEALTHCHECKS envvar with the following logic:

• If the value is xids or all we disable healthchecks entirely.
• A comma-separated list of numeric XIDs to ignore: e.g. 109,68

The GKE Device Plugin

See https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/0509b1f9f4b9a357b44ba65e7b508ded8bd5ecf0/pkg/gpu/nvidia/health_check/health_checker.go#L41

By default the following error is checked:

XID	Description
48	Double-bit ECC Error

The XID_CONFIG envvar is used to specifiy a comma-separated list of additional XIDs to treat as critical.

Proposal

Add the following config section:

version: v1
health:
  disabled: false
  eventTypes: [EventTypeXidCriticalError, EventTypeDoubleBitEccError, EventTypeSingleBitEccError]
  ignoredXIDs: [13, 31, 43, 45, 68]
  criticalXIDs: all

GKE defaults:

version: v1
health:
  disabled: false
  eventTypes: [EventTypeXidCriticalError]
  ignoredXIDs: []
  criticalXIDs: [48]

[robertdavidsmith]

Agree it makes sense to move this to config.

ignoredXIDs/criticalXIDs are exact opposites so I'd favor names that make this obvious. Maybe fatalXids/nonFatalXids?

I did wonder if you could have one field but make it a map not an array.
E.g.

  ignoredXIDs:
     "13": true
     "31": true
     "43": true
     "45": true
     "68": true

or

  ignoredXIDs: all

However while it's easy to say remove 45 via helm (just set value to null), replacing a map with string all in helm doesn't work. On balance I'd stick with the two fields.

[blanquicet]

This was partially covered by #1443. However, AFAIU, the config section is still missing.

[blanquicet]

@elezar, I have a question, where I can find what XIDs are classified as EventTypeXidCriticalError? I couldn't find it in the NVML API documentation.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.