[Question]: Make Sharing.MPS.FailRequestsGreaterThanOne authoritative

[radu-malliu]

Currently, Sharing.MPS.FailRequestsGreaterThanOne is a valid config item. However, its value is ignored.

We have a use case whereby we need to be able to request all GPUs available on a node. Device plugin promises around what that request means in terms of time-on-device have no bearing, we know that this workload will have exclusivity when running. So, the only requirement is to be able to request multiple (all) replicas, in order to have access to all GPUs.

Based on this, is there interest in making the value of Sharing.MPS.FailRequestsGreaterThanOne authoritative when MPS is enabled, instead of it being ignored? Happy to PR the change if yes.

[tariq1890]

Device plugin promises around what that request means in terms of time-on-device have no bearing, we know that this workload will have exclusivity when running. So, the only requirement is to be able to request multiple (all) replicas, in order to have access to all GPUs.

So it looks like you want to be able to access all the GPUs while not taking away from the Allocatable GPU count for normal workloads. If I am understanding this correctly, would it suffice if you just set the runtimeClassName: nvidia and set the NVIDIA_VISIBLE_DEVICES=all envar in your pod which needs access to all of the GPUs?

[pamelia]

Thanks, that may work as a debug/runtime escape hatch, but I do not think it is equivalent to what we need here.

For this case we do want the pod GPU access to go through the normal Kubernetes extended-resource allocation path. The verification pod is expected to reserve/request all MPS replicas while it runs, so the allocation is visible to kubelet/scheduler accounting rather than being an out-of-band runtime env override.

There is also an MPS-specific part of the device-plugin Allocate response: when MPS is enabled the plugin injects the MPS pipe/log directory env and mounts for the selected resource. A pod that relies only on runtimeClassName: nvidia and NVIDIA_VISIBLE_DEVICES=all would bypass the multi-replica request validation path this PR is changing, and depending on whether it requests a GPU resource at all, may not get the MPS allocation response data either.

The PR keeps the current default behavior unchanged (failRequestsGreaterThanOne defaults to true for MPS). It only makes an explicit sharing.mps.failRequestsGreaterThanOne: false authoritative for operators that intentionally want multi-replica MPS allocations.

[vrdn-23]

I'm a little confused on maybe how this will work, if someone can explain it for me: MPS currently seems to be soft limiting every client that is connected based on the number of replicas (2 replicas cap each client at 50%, 3 at 33% and so on), so a pod that is asking for 2 replicas (out of 3) will not really get SM utilization of 66%, but still only get 33% (based on the fact this PR never got merged #586 )?

cc @tariq1890 @pamelia