CUDA error: device-side assert triggered

[Inosonnia]

After generating the mask by encode_segmap, SpatialClassNLLCriterion.cu reports: Assertion t >= 0 && t < n_classes failed.
It seems that the labels exceed the range of n_classes.
I used the default data mentioned in the src code.
Thank you.

[shahsohil]

@Inosonnia Can you please provide me with more details such as the value of labels which seemed to have exceeded n_classes ?

[Inosonnia]

Since I used the default seg data mentioned in the src code (i.e., VOC12+benchmark_RELEASE), the clses are indeed fall in [0, 20], as I tried to print them out.
It seems that the cls info of data is ok (the mask generation period is fine), and this error occurs even I change n_classes to a big value (i.e., num_classes = 1000, in models/pretrained/sunet.py).

The detailed info is listed here:

pytorch/aten/src/THC/generated/../THCTensorMathCompare.cuh line=82 error=59 : device-side assert triggered
pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [1,0,0], thread: [768,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "train_seg.py", line 340, in
train(args)
File "train_seg.py", line 146, in train
trainmodel(model, optimizer, trainloader, epoch, scheduler, traindata)
File "train_seg.py", line 226, in trainmodel
model(imagesV, labelsV)
File "/python/lib/python2.7/site-packages/torch/nn/modules/module.py", line 479, in call
result = self.forward(*input, **kwargs)
File "/python/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/python/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/python/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generated/../THCTensorMathCompare.cuh:82