Batch device print statement shows only device 0 on multi-GPU setup

[sasank-desaraju]

When running on HPG, the print output that gives us "Training batch is on device _" is only reading "device 0". Is this missing computations on the other GPU (i.e. "device 1"), which the program states is indeed there at the beginning, or is this second GPU not getting used during training for some reason?

[sasank-desaraju]

Oh what lol.
In Wandb, which is where I was checking, there are actually two runs going on. The second one has less information in the logs but just says which device the operations are on and both training and validation are on "device 1", which is the second GPU. Cool! (I think).

[sasank-desaraju]

Okay, now I can't find those initial logs but both runs just show that all batches are on device 1. A question for another day...