exposing job cpus and gpus

[luccabb]

Summary

exposing a way to get number of cpus and gpus from job information. defaults to local host if not on a slurm cluster

Test Plan

works on slurm and locally:

$ srun python -c "import clusterscope; print(clusterscope.get_job().get_gpus()
); print(clusterscope.get_job().get_cpus())"
...
srun: job 1476042 has been allocated resources
1
2
$ python
Python 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clusterscope
>>> clusterscope.get_job().get_gpus()
WARNING:root:No GPUs found or unable to retrieve GPU information
0
>>> clusterscope.get_job().get_cpus()
96

local node with gpus:

Type "help", "copyright", "credits" or "license" for more information.
>>> import clusterscope
>>> clusterscope.get_job().get_gpus()
2
>>> clusterscope.get_job().get_cpus()
80
>>> exit()

[skalyan]

What is the motivation for this change? Did we have any user requests?

Is the idea to simply hide Slurm env variables and add a wrapper?

[luccabb]

@skalyan yeah, this is to enable: facebookresearch/matrix#105 (comment)

Is the idea to simply hide Slurm env variables and add a wrapper?

it gives the info for slurm or local nodes

[skalyan]

What is the intent of the ask?

Is it to know how many GPUs/CPUs are "present" on the node or "allocated" to this job?

[luccabb]

allocated to the job if slurm job, otherwise what's present in the node

[skalyan]

If I may, I think we should stick to a clear and limited contract for an API - If we want to return GPUs allocated to a job for a given API let's stick to that. I don't see benefits in either allocated or provisioned GPU count coming via the same API.

[luccabb]

@skalyan I think this is fine for now, it matches how other methods from this class behaves. I'm up to change position here as we see how it gets used