Skip to content

dataloader问题 #15

@N1ck1314

Description

@N1ck1314

train: 17, 600 / 1000] FPS: 252.8 (389.1) , Loss/total: 1.08691 , Loss/giou: 0.26345 , Loss/l1: 0.02519 , Loss/location: 0.43331 , Loss/task_class: 0.00042 , IoU: 0.75680
ERROR: Unexpected segmentation fault encountered in worker.
Training crashed at epoch 17
Traceback for the error!
Traceback (most recent call last):
File "/home/lls/sutrack/lib/train/../../lib/train/trainers/base_trainer.py", line 85, in train
self.train_epoch()
File "/home/lls/sutrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 111, in train_epoch
self.cycle_dataset(loader)
File "/home/lls/sutrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 87, in cycle_dataset
torch.nn.utils.clip_grad_norm_(self.actor.net.parameters(), self.settings.grad_clip_norm)
File "/home/lls/miniconda3/envs/sutrack/lib/python3.8/site-packages/torch/nn/utils/clip_grad.py", line 55, in clip_grad_norm_
p.grad.detach().mul_(clip_coef_clamped.to(p.grad.device))
File "/home/lls/miniconda3/envs/sutrack/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 346942) is killed by signal: Segmentation fault.

Restarting training from last epoch ...
Finished training!

请问有遇到这个过数据加载的问题吗 如何解决

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions