cuda out of memory

[yaomingzhang]

When I use v100 32g to run 896896 during seg training, I will encounter obvious deficiencies. However, I still encounter this problem when I change it to 512512. Even if I change it to a smaller one, I still have it. What may be the reason for this?

[yaomingzhang]

add samples_per_gpu=1 , and issue is runtimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 31.75 GiB total capacity; 30.02 GiB already allocated; 15.75 MiB free; 30.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[logicwong]

Hi, which script are you running?

[yaomingzhang]

hello，mask2former_onepeace_adapter_g_896_40k_ade20k_ss.py