Epoch [0/1080]; Iter [0/713880]; Loss 36.86; LR 1.00e-06; Iter time 0.86; ETA 7 days, 2:52:36; Mem 11266.68MB
Epoch [0/1080]; Iter [10/713880]; Loss 39.25; LR 2.17e-06; Iter time 0.47; ETA 3 days, 20:36:27; Mem 11358.37MB
Epoch [0/1080]; Iter [20/713880]; Loss 39.06; LR 3.35e-06; Iter time 0.47; ETA 3 days, 21:07:55; Mem 11358.37MB
Epoch [0/1080]; Iter [30/713880]; Loss 38.90; LR 4.52e-06; Iter time 0.46; ETA 3 days, 19:56:41; Mem 11358.37MB
Epoch [0/1080]; Iter [40/713880]; Loss 39.63; LR 5.70e-06; Iter time 0.47; ETA 3 days, 20:56:11; Mem 11358.37MB
Epoch [0/1080]; Iter [50/713880]; Loss 38.82; LR 6.87e-06; Iter time 0.46; ETA 3 days, 19:59:32; Mem 11358.37MB
Epoch [0/1080]; Iter [60/713880]; Loss 39.38; LR 8.05e-06; Iter time 0.47; ETA 3 days, 21:55:16; Mem 11358.37MB
Epoch [0/1080]; Iter [70/713880]; Loss 38.85; LR 9.22e-06; Iter time 0.47; ETA 3 days, 21:59:19; Mem 11358.37MB
Epoch [0/1080]; Iter [80/713880]; Loss 39.10; LR 1.04e-05; Iter time 0.46; ETA 3 days, 19:05:52; Mem 11358.37MB
Epoch [0/1080]; Iter [90/713880]; Loss 38.01; LR 1.16e-05; Iter time 0.48; ETA 3 days, 22:28:41; Mem 11358.37MB
Epoch [0/1080]; Iter [100/713880]; Loss 37.46; LR 1.27e-05; Iter time 0.48; ETA 3 days, 22:11:35; Mem 11358.37MB
Epoch [0/1080]; Iter [110/713880]; Loss 37.89; LR 1.39e-05; Iter time 0.47; ETA 3 days, 21:41:49; Mem 11358.37MB
Epoch [0/1080]; Iter [120/713880]; Loss 36.78; LR 1.51e-05; Iter time 0.47; ETA 3 days, 22:05:53; Mem 11358.37MB
Epoch [0/1080]; Iter [130/713880]; Loss 37.18; LR 1.63e-05; Iter time 0.45; ETA 3 days, 17:17:56; Mem 11358.37MB
Epoch [0/1080]; Iter [140/713880]; Loss 36.47; LR 1.74e-05; Iter time 0.48; ETA 3 days, 22:44:52; Mem 11358.37MB
Epoch [0/1080]; Iter [150/713880]; Loss 35.66; LR 1.86e-05; Iter time 0.48; ETA 3 days, 23:11:45; Mem 11358.37MB
Epoch [0/1080]; Iter [160/713880]; Loss 36.51; LR 1.98e-05; Iter time 0.48; ETA 3 days, 22:27:10; Mem 11358.37MB
Epoch [0/1080]; Iter [170/713880]; Loss 35.54; LR 2.10e-05; Iter time 0.47; ETA 3 days, 21:35:33; Mem 11358.37MB
Epoch [0/1080]; Iter [180/713880]; Loss 35.42; LR 2.21e-05; Iter time 0.47; ETA 3 days, 20:36:38; Mem 11358.37MB
Epoch [0/1080]; Iter [190/713880]; Loss 35.22; LR 2.33e-05; Iter time 0.46; ETA 3 days, 20:08:43; Mem 11358.37MB
Epoch [0/1080]; Iter [200/713880]; Loss 33.90; LR 2.45e-05; Iter time 0.46; ETA 3 days, 19:36:33; Mem 11358.37MB
Epoch [0/1080]; Iter [210/713880]; Loss 33.86; LR 2.57e-05; Iter time 0.46; ETA 3 days, 20:09:38; Mem 11358.37MB
Epoch [0/1080]; Iter [220/713880]; Loss 32.69; LR 2.68e-05; Iter time 0.46; ETA 3 days, 18:25:42; Mem 11358.37MB
Epoch [0/1080]; Iter [230/713880]; Loss 32.94; LR 2.80e-05; Iter time 0.48; ETA 3 days, 22:45:32; Mem 11358.37MB
Epoch [0/1080]; Iter [240/713880]; Loss 32.65; LR 2.92e-05; Iter time 0.48; ETA 3 days, 22:30:48; Mem 11358.37MB
Epoch [0/1080]; Iter [250/713880]; Loss 32.17; LR 3.04e-05; Iter time 0.48; ETA 3 days, 22:47:54; Mem 11358.37MB
Epoch [0/1080]; Iter [260/713880]; Loss 31.08; LR 3.15e-05; Iter time 0.47; ETA 3 days, 20:29:50; Mem 11358.37MB
Epoch [0/1080]; Iter [270/713880]; Loss 31.69; LR 3.27e-05; Iter time 0.47; ETA 3 days, 22:00:11; Mem 11358.37MB
Epoch [0/1080]; Iter [280/713880]; Loss 30.16; LR 3.39e-05; Iter time 0.49; ETA 4 days, 0:39:25; Mem 11358.37MB
Epoch [0/1080]; Iter [290/713880]; Loss 30.36; LR 3.51e-05; Iter time 0.49; ETA 4 days, 1:34:04; Mem 11358.37MB
Epoch [0/1080]; Iter [300/713880]; Loss 29.33; LR 3.62e-05; Iter time 0.49; ETA 4 days, 0:46:32; Mem 11358.37MB
Epoch [0/1080]; Iter [310/713880]; Loss 29.75; LR 3.74e-05; Iter time 0.50; ETA 4 days, 2:16:41; Mem 11358.37MB
Epoch [0/1080]; Iter [320/713880]; Loss 29.25; LR 3.86e-05; Iter time 0.49; ETA 4 days, 1:36:29; Mem 11358.37MB
Epoch [0/1080]; Iter [330/713880]; Loss 28.91; LR 3.98e-05; Iter time 0.47; ETA 3 days, 21:41:18; Mem 11358.37MB
Epoch [0/1080]; Iter [340/713880]; Loss 28.07; LR 4.09e-05; Iter time 0.48; ETA 3 days, 23:48:39; Mem 11358.37MB
Epoch [0/1080]; Iter [350/713880]; Loss 27.32; LR 4.21e-05; Iter time 0.48; ETA 3 days, 22:10:17; Mem 11358.37MB
Epoch [0/1080]; Iter [360/713880]; Loss 27.46; LR 4.33e-05; Iter time 0.49; ETA 4 days, 0:33:33; Mem 11358.37MB
Epoch [0/1080]; Iter [370/713880]; Loss 26.86; LR 4.45e-05; Iter time 0.51; ETA 4 days, 4:44:19; Mem 11358.37MB
Epoch [0/1080]; Iter [380/713880]; Loss 26.44; LR 4.56e-05; Iter time 0.48; ETA 3 days, 23:03:18; Mem 11358.37MB
Epoch [0/1080]; Iter [390/713880]; Loss 26.25; LR 4.68e-05; Iter time 0.47; ETA 3 days, 21:38:42; Mem 11358.37MB
Epoch [0/1080]; Iter [400/713880]; Loss 25.53; LR 4.80e-05; Iter time 0.46; ETA 3 days, 18:50:20; Mem 11358.37MB
Epoch [0/1080]; Iter [410/713880]; Loss 25.28; LR 4.92e-05; Iter time 0.48; ETA 3 days, 22:38:21; Mem 11358.37MB
Epoch [0/1080]; Iter [420/713880]; Loss 24.53; LR 5.03e-05; Iter time 0.48; ETA 3 days, 23:48:27; Mem 11358.37MB
Epoch [0/1080]; Iter [430/713880]; Loss 23.95; LR 5.15e-05; Iter time 0.47; ETA 3 days, 22:01:22; Mem 11358.37MB
Epoch [0/1080]; Iter [440/713880]; Loss 24.59; LR 5.27e-05; Iter time 0.48; ETA 3 days, 22:51:55; Mem 11358.37MB
Epoch [0/1080]; Iter [450/713880]; Loss 23.92; LR 5.39e-05; Iter time 0.48; ETA 3 days, 22:21:47; Mem 11358.37MB
Training stops on certain point. How should I solve this?
I'm using python=3.6, cuda=11.1 and torch 1.9.0
Epoch [0/1080]; Iter [0/713880]; Loss 36.86; LR 1.00e-06; Iter time 0.86; ETA 7 days, 2:52:36; Mem 11266.68MB
Epoch [0/1080]; Iter [10/713880]; Loss 39.25; LR 2.17e-06; Iter time 0.47; ETA 3 days, 20:36:27; Mem 11358.37MB
Epoch [0/1080]; Iter [20/713880]; Loss 39.06; LR 3.35e-06; Iter time 0.47; ETA 3 days, 21:07:55; Mem 11358.37MB
Epoch [0/1080]; Iter [30/713880]; Loss 38.90; LR 4.52e-06; Iter time 0.46; ETA 3 days, 19:56:41; Mem 11358.37MB
Epoch [0/1080]; Iter [40/713880]; Loss 39.63; LR 5.70e-06; Iter time 0.47; ETA 3 days, 20:56:11; Mem 11358.37MB
Epoch [0/1080]; Iter [50/713880]; Loss 38.82; LR 6.87e-06; Iter time 0.46; ETA 3 days, 19:59:32; Mem 11358.37MB
Epoch [0/1080]; Iter [60/713880]; Loss 39.38; LR 8.05e-06; Iter time 0.47; ETA 3 days, 21:55:16; Mem 11358.37MB
Epoch [0/1080]; Iter [70/713880]; Loss 38.85; LR 9.22e-06; Iter time 0.47; ETA 3 days, 21:59:19; Mem 11358.37MB
Epoch [0/1080]; Iter [80/713880]; Loss 39.10; LR 1.04e-05; Iter time 0.46; ETA 3 days, 19:05:52; Mem 11358.37MB
Epoch [0/1080]; Iter [90/713880]; Loss 38.01; LR 1.16e-05; Iter time 0.48; ETA 3 days, 22:28:41; Mem 11358.37MB
Epoch [0/1080]; Iter [100/713880]; Loss 37.46; LR 1.27e-05; Iter time 0.48; ETA 3 days, 22:11:35; Mem 11358.37MB
Epoch [0/1080]; Iter [110/713880]; Loss 37.89; LR 1.39e-05; Iter time 0.47; ETA 3 days, 21:41:49; Mem 11358.37MB
Epoch [0/1080]; Iter [120/713880]; Loss 36.78; LR 1.51e-05; Iter time 0.47; ETA 3 days, 22:05:53; Mem 11358.37MB
Epoch [0/1080]; Iter [130/713880]; Loss 37.18; LR 1.63e-05; Iter time 0.45; ETA 3 days, 17:17:56; Mem 11358.37MB
Epoch [0/1080]; Iter [140/713880]; Loss 36.47; LR 1.74e-05; Iter time 0.48; ETA 3 days, 22:44:52; Mem 11358.37MB
Epoch [0/1080]; Iter [150/713880]; Loss 35.66; LR 1.86e-05; Iter time 0.48; ETA 3 days, 23:11:45; Mem 11358.37MB
Epoch [0/1080]; Iter [160/713880]; Loss 36.51; LR 1.98e-05; Iter time 0.48; ETA 3 days, 22:27:10; Mem 11358.37MB
Epoch [0/1080]; Iter [170/713880]; Loss 35.54; LR 2.10e-05; Iter time 0.47; ETA 3 days, 21:35:33; Mem 11358.37MB
Epoch [0/1080]; Iter [180/713880]; Loss 35.42; LR 2.21e-05; Iter time 0.47; ETA 3 days, 20:36:38; Mem 11358.37MB
Epoch [0/1080]; Iter [190/713880]; Loss 35.22; LR 2.33e-05; Iter time 0.46; ETA 3 days, 20:08:43; Mem 11358.37MB
Epoch [0/1080]; Iter [200/713880]; Loss 33.90; LR 2.45e-05; Iter time 0.46; ETA 3 days, 19:36:33; Mem 11358.37MB
Epoch [0/1080]; Iter [210/713880]; Loss 33.86; LR 2.57e-05; Iter time 0.46; ETA 3 days, 20:09:38; Mem 11358.37MB
Epoch [0/1080]; Iter [220/713880]; Loss 32.69; LR 2.68e-05; Iter time 0.46; ETA 3 days, 18:25:42; Mem 11358.37MB
Epoch [0/1080]; Iter [230/713880]; Loss 32.94; LR 2.80e-05; Iter time 0.48; ETA 3 days, 22:45:32; Mem 11358.37MB
Epoch [0/1080]; Iter [240/713880]; Loss 32.65; LR 2.92e-05; Iter time 0.48; ETA 3 days, 22:30:48; Mem 11358.37MB
Epoch [0/1080]; Iter [250/713880]; Loss 32.17; LR 3.04e-05; Iter time 0.48; ETA 3 days, 22:47:54; Mem 11358.37MB
Epoch [0/1080]; Iter [260/713880]; Loss 31.08; LR 3.15e-05; Iter time 0.47; ETA 3 days, 20:29:50; Mem 11358.37MB
Epoch [0/1080]; Iter [270/713880]; Loss 31.69; LR 3.27e-05; Iter time 0.47; ETA 3 days, 22:00:11; Mem 11358.37MB
Epoch [0/1080]; Iter [280/713880]; Loss 30.16; LR 3.39e-05; Iter time 0.49; ETA 4 days, 0:39:25; Mem 11358.37MB
Epoch [0/1080]; Iter [290/713880]; Loss 30.36; LR 3.51e-05; Iter time 0.49; ETA 4 days, 1:34:04; Mem 11358.37MB
Epoch [0/1080]; Iter [300/713880]; Loss 29.33; LR 3.62e-05; Iter time 0.49; ETA 4 days, 0:46:32; Mem 11358.37MB
Epoch [0/1080]; Iter [310/713880]; Loss 29.75; LR 3.74e-05; Iter time 0.50; ETA 4 days, 2:16:41; Mem 11358.37MB
Epoch [0/1080]; Iter [320/713880]; Loss 29.25; LR 3.86e-05; Iter time 0.49; ETA 4 days, 1:36:29; Mem 11358.37MB
Epoch [0/1080]; Iter [330/713880]; Loss 28.91; LR 3.98e-05; Iter time 0.47; ETA 3 days, 21:41:18; Mem 11358.37MB
Epoch [0/1080]; Iter [340/713880]; Loss 28.07; LR 4.09e-05; Iter time 0.48; ETA 3 days, 23:48:39; Mem 11358.37MB
Epoch [0/1080]; Iter [350/713880]; Loss 27.32; LR 4.21e-05; Iter time 0.48; ETA 3 days, 22:10:17; Mem 11358.37MB
Epoch [0/1080]; Iter [360/713880]; Loss 27.46; LR 4.33e-05; Iter time 0.49; ETA 4 days, 0:33:33; Mem 11358.37MB
Epoch [0/1080]; Iter [370/713880]; Loss 26.86; LR 4.45e-05; Iter time 0.51; ETA 4 days, 4:44:19; Mem 11358.37MB
Epoch [0/1080]; Iter [380/713880]; Loss 26.44; LR 4.56e-05; Iter time 0.48; ETA 3 days, 23:03:18; Mem 11358.37MB
Epoch [0/1080]; Iter [390/713880]; Loss 26.25; LR 4.68e-05; Iter time 0.47; ETA 3 days, 21:38:42; Mem 11358.37MB
Epoch [0/1080]; Iter [400/713880]; Loss 25.53; LR 4.80e-05; Iter time 0.46; ETA 3 days, 18:50:20; Mem 11358.37MB
Epoch [0/1080]; Iter [410/713880]; Loss 25.28; LR 4.92e-05; Iter time 0.48; ETA 3 days, 22:38:21; Mem 11358.37MB
Epoch [0/1080]; Iter [420/713880]; Loss 24.53; LR 5.03e-05; Iter time 0.48; ETA 3 days, 23:48:27; Mem 11358.37MB
Epoch [0/1080]; Iter [430/713880]; Loss 23.95; LR 5.15e-05; Iter time 0.47; ETA 3 days, 22:01:22; Mem 11358.37MB
Epoch [0/1080]; Iter [440/713880]; Loss 24.59; LR 5.27e-05; Iter time 0.48; ETA 3 days, 22:51:55; Mem 11358.37MB
Epoch [0/1080]; Iter [450/713880]; Loss 23.92; LR 5.39e-05; Iter time 0.48; ETA 3 days, 22:21:47; Mem 11358.37MB
Training stops on certain point. How should I solve this?
I'm using python=3.6, cuda=11.1 and torch 1.9.0