map仅仅只有0.0003

[Hichengdong]

我使用mini数据集进行了训练，在第22个epoch的时候，程序卡死，然后我对最后的两个epoch进行了验证，结果map出奇的低。
================epoch 20=================
AP: 0.0003
mATE: 1.0000
mASE: 1.0000
mAOE: 1.0000
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0001
Eval time: 0.4s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.003 1.000 1.000 1.000 1.000 1.000
truck 0.000 1.000 1.000 1.000 1.000 1.000
bus 0.000 1.000 1.000 1.000 1.000 1.000
trailer 0.000 1.000 1.000 1.000 1.000 1.000
construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000
pedestrian 0.000 1.000 1.000 1.000 1.000 1.000
motorcycle 0.000 1.000 1.000 1.000 1.000 1.000
bicycle 0.000 1.000 1.000 1.000 1.000 1.000
traffic_cone 0.000 1.000 1.000 nan nan nan
barrier 0.000 1.000 1.000 1.000 nan nan
================epoch 21=================
AP: 0.0003
mATE: 1.0000
mASE: 1.0000
mAOE: 1.0000
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0002
Eval time: 0.4s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.003 1.000 1.000 1.000 1.000 1.000
truck 0.000 1.000 1.000 1.000 1.000 1.000
bus 0.000 1.000 1.000 1.000 1.000 1.000
trailer 0.000 1.000 1.000 1.000 1.000 1.000
construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000
pedestrian 0.000 1.000 1.000 1.000 1.000 1.000
motorcycle 0.000 1.000 1.000 1.000 1.000 1.000
bicycle 0.000 1.000 1.000 1.000 1.000 1.000
traffic_cone 0.000 1.000 1.000 nan nan nan
barrier 0.000 1.000 1.000 1.000 nan nan
Testing DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 162/162 [00:29<00:00, 5.46it/s]
TEST Profiler Report

---

| Action | Mean duration (s) | Num calls | Total time (s) | Percentage % |

| Total | - | 2823 | 87.467 | 100 % |

| run_test_evaluation | 31.038 | 1 | 31.038 | 35.486 |
| [Strategy]SingleDeviceStrategy.test_step | 0.15896 | 162 | 25.751 | 29.441 |
| [LightningModule]BEVDepthLightningModel.test_epoch_end | 2.6193 | 1 | 2.6193 | 2.9946 |
| [EvaluationEpochLoop].test_dataloader_idx_0_next | 0.010395 | 162 | 1.6839 | 1.9252 |
| [Strategy]SingleDeviceStrategy.batch_to_device | 0.003483 | 162 | 0.56424 | 0.64509 |
| [LightningModule]BEVDepthLightningModel.test_dataloader | 0.099374 | 1 | 0.099374 | 0.11361 |
| [Callback]TQDMProgressBar.on_test_batch_end | 0.00025195 | 162 | 0.040816 | 0.046664 |
| [Callback]ModelSummary.on_test_batch_end | 9.2947e-06 | 162 | 0.0015057 | 0.0017215 |
| [Callback]TQDMProgressBar.on_test_batch_start | 8.4548e-06 | 162 | 0.0013697 | 0.0015659 |
| [LightningModule]BEVDepthLightningModel.on_test_model_train | 0.0010831 | 1 | 0.0010831 | 0.0012383 |
| [LightningModule]BEVDepthLightningModel.on_test_model_eval | 0.00094745 | 1 | 0.00094745 | 0.0010832 |
| [Callback]EMACallback.on_test_batch_start | 4.0783e-06 | 162 | 0.00066068 | 0.00075535 |
| [Callback]TQDMProgressBar.on_test_start | 0.00061182 | 1 | 0.00061182 | 0.00069948 |
| [Callback]EMACallback.on_test_batch_end | 2.4168e-06 | 162 | 0.00039152 | 0.00044762 |
| [LightningModule]BEVDepthLightningModel.test_step_end | 1.8431e-06 | 162 | 0.00029859 | 0.00034137 |
| [Callback]TQDMProgressBar.on_test_end | 0.00028655 | 1 | 0.00028655 | 0.00032761 |
| [Callback]GradientAccumulationScheduler.on_test_batch_end | 1.4585e-06 | 162 | 0.00023627 | 0.00027012 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_end | 1.4072e-06 | 162 | 0.00022797 | 0.00026063 |
| [Callback]ModelSummary.on_test_batch_start | 1.3618e-06 | 162 | 0.00022062 | 0.00025223 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_start | 1.3274e-06 | 162 | 0.00021504 | 0.00024585 |
| [Callback]GradientAccumulationScheduler.on_test_batch_start | 1.1475e-06 | 162 | 0.0001859 | 0.00021253 |
| [LightningModule]BEVDepthLightningModel.on_test_batch_end | 1.0331e-06 | 162 | 0.00016737 | 0.00019135 |
| [LightningModule]BEVDepthLightningModel.on_test_batch_start | 9.7408e-07 | 162 | 0.0001578 | 0.00018041 |
| [Strategy]SingleDeviceStrategy.test_step_end | 8.8581e-07 | 162 | 0.0001435 | 0.00016406 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup | 8.6112e-05 | 1 | 8.6112e-05 | 9.845e-05 |
| [Callback]TQDMProgressBar.on_test_epoch_start | 1.1989e-05 | 1 | 1.1989e-05 | 1.3707e-05 |
| [Callback]EMACallback.on_test_epoch_end | 9.393e-06 | 1 | 9.393e-06 | 1.0739e-05 |
| [Callback]ModelSummary.on_test_end | 8.157e-06 | 1 | 8.157e-06 | 9.3258e-06 |
| [Callback]ModelSummary.on_test_start | 6.458e-06 | 1 | 6.458e-06 | 7.3833e-06 |
| [Callback]TQDMProgressBar.setup | 5.747e-06 | 1 | 5.747e-06 | 6.5704e-06 |
| [Callback]EMACallback.teardown | 5.065e-06 | 1 | 5.065e-06 | 5.7907e-06 |
| [LightningModule]BEVDepthLightningModel.on_load_checkpoint | 4.902e-06 | 1 | 4.902e-06 | 5.6044e-06 |
| [Callback]EMACallback.on_test_start | 4.671e-06 | 1 | 4.671e-06 | 5.3403e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_epoch_end | 4.038e-06 | 1 | 4.038e-06 | 4.6166e-06 |
| [LightningModule]BEVDepthLightningModel.configure_callbacks | 3.997e-06 | 1 | 3.997e-06 | 4.5697e-06 |
| [Callback]EMACallback.setup | 3.806e-06 | 1 | 3.806e-06 | 4.3513e-06 |
| [LightningModule]BEVDepthLightningModel.configure_sharded_model | 3.26e-06 | 1 | 3.26e-06 | 3.7271e-06 |
| [Callback]EMACallback.on_epoch_end | 3.089e-06 | 1 | 3.089e-06 | 3.5316e-06 |
| [Callback]EMACallback.on_configure_sharded_model | 2.58e-06 | 1 | 2.58e-06 | 2.9497e-06 |
| [Callback]EMACallback.on_epoch_start | 2.052e-06 | 1 | 2.052e-06 | 2.346e-06 |
| [Callback]TQDMProgressBar.on_test_epoch_end | 2.044e-06 | 1 | 2.044e-06 | 2.3369e-06 |
| [Callback]EMACallback.on_test_epoch_start | 2.032e-06 | 1 | 2.032e-06 | 2.3232e-06 |
| [LightningModule]BEVDepthLightningModel.prepare_data | 1.966e-06 | 1 | 1.966e-06 | 2.2477e-06 |
| [Callback]ModelSummary.on_test_epoch_start | 1.854e-06 | 1 | 1.854e-06 | 2.1196e-06 |
| [Callback]EMACallback.on_before_accelerator_backend_setup | 1.802e-06 | 1 | 1.802e-06 | 2.0602e-06 |
| [Callback]EMACallback.on_test_end | 1.636e-06 | 1 | 1.636e-06 | 1.8704e-06 |
| [Callback]TQDMProgressBar.on_epoch_end | 1.597e-06 | 1 | 1.597e-06 | 1.8258e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_start | 1.552e-06 | 1 | 1.552e-06 | 1.7744e-06 |
| [Callback]TQDMProgressBar.teardown | 1.542e-06 | 1 | 1.542e-06 | 1.7629e-06 |
| [Callback]GradientAccumulationScheduler.on_test_start | 1.4e-06 | 1 | 1.4e-06 | 1.6006e-06 |
| [Callback]GradientAccumulationScheduler.on_test_epoch_start | 1.349e-06 | 1 | 1.349e-06 | 1.5423e-06 |
| [Callback]TQDMProgressBar.on_configure_sharded_model | 1.318e-06 | 1 | 1.318e-06 | 1.5068e-06 |
| [Callback]ModelSummary.setup | 1.296e-06 | 1 | 1.296e-06 | 1.4817e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_start | 1.245e-06 | 1 | 1.245e-06 | 1.4234e-06 |
| [Callback]ModelSummary.on_configure_sharded_model | 1.23e-06 | 1 | 1.23e-06 | 1.4062e-06 |
| [Callback]TQDMProgressBar.on_before_accelerator_backend_setup | 1.215e-06 | 1 | 1.215e-06 | 1.3891e-06 |
| [Callback]GradientAccumulationScheduler.setup | 1.214e-06 | 1 | 1.214e-06 | 1.3879e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown | 1.163e-06 | 1 | 1.163e-06 | 1.3296e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_end | 1.148e-06 | 1 | 1.148e-06 | 1.3125e-06 |
| [LightningModule]BEVDepthLightningModel.teardown | 1.147e-06 | 1 | 1.147e-06 | 1.3113e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model | 1.131e-06 | 1 | 1.131e-06 | 1.2931e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup| 1.12e-06 | 1 | 1.12e-06 | 1.2805e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_end | 1.11e-06 | 1 | 1.11e-06 | 1.269e-06 |
| [Callback]GradientAccumulationScheduler.on_test_end | 1.107e-06 | 1 | 1.107e-06 | 1.2656e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start | 1.08e-06 | 1 | 1.08e-06 | 1.2347e-06 |
| [Callback]GradientAccumulationScheduler.on_configure_sharded_model | 1.069e-06 | 1 | 1.069e-06 | 1.2222e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_start | 1.054e-06 | 1 | 1.054e-06 | 1.205e-06 |
| [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup | 1.037e-06 | 1 | 1.037e-06 | 1.1856e-06 |
| [Callback]ModelSummary.on_test_epoch_end | 1.032e-06 | 1 | 1.032e-06 | 1.1799e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end | 1.026e-06 | 1 | 1.026e-06 | 1.173e-06 |
| [Callback]ModelSummary.teardown | 1.023e-06 | 1 | 1.023e-06 | 1.1696e-06 |
| [Callback]ModelSummary.on_before_accelerator_backend_setup | 9.86e-07 | 1 | 9.86e-07 | 1.1273e-06 |
| [Callback]TQDMProgressBar.on_epoch_start | 9.76e-07 | 1 | 9.76e-07 | 1.1158e-06 |
| [LightningModule]BEVDepthLightningModel.setup | 9.64e-07 | 1 | 9.64e-07 | 1.1021e-06 |
| [Callback]GradientAccumulationScheduler.on_epoch_start | 9.51e-07 | 1 | 9.51e-07 | 1.0873e-06 |
| [Callback]GradientAccumulationScheduler.teardown | 9.42e-07 | 1 | 9.42e-07 | 1.077e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_end | 8.99e-07 | 1 | 8.99e-07 | 1.0278e-06 |
| [Callback]ModelSummary.on_epoch_end | 8.9e-07 | 1 | 8.9e-07 | 1.0175e-06 |
| [Callback]ModelSummary.on_epoch_start | 8.88e-07 | 1 | 8.88e-07 | 1.0152e-06 |
| [Callback]GradientAccumulationScheduler.on_test_epoch_end | 8.81e-07 | 1 | 8.81e-07 | 1.0072e-06 |
| [Callback]GradientAccumulationScheduler.on_epoch_end | 8.39e-07 | 1 | 8.39e-07 | 9.5922e-07 |
| [LightningModule]BEVDepthLightningModel.on_test_epoch_start | 7.55e-07 | 1 | 7.55e-07 | 8.6318e-07 |
| [Strategy]SingleDeviceStrategy.on_test_start | 7.51e-07 | 1 | 7.51e-07 | 8.586e-07 |
| [Strategy]SingleDeviceStrategy.on_test_end | 7.13e-07 | 1 | 7.13e-07 | 8.1516e-07 |
| [LightningModule]BEVDepthLightningModel.on_epoch_start | 6.39e-07 | 1 | 6.39e-07 | 7.3056e-07 |
| [LightningModule]BEVDepthLightningModel.on_epoch_end | 5.36e-07 | 1 | 5.36e-07 | 6.128e-07 |

[Hichengdong]

运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py -b 1 --gpus 1 进行训练
运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py --ckpt_path outputs/bev_depth_lss_r50_256x704_128x128_24e_ema/lightning_logs/version_2/21.pth -e -b 1 --gpus 1 进行测试

[wansdd]

have you solved this problems?

[Hichengdong]

have you solved this problems?
Not yet, but I used the full nuScenes dataset later.

[wansdd]

have you solved this problems?
Not yet, but I used the full nuScenes dataset later.

thank you for your replay, when you used the full nuScenes, Are the results will become batter?

[Hichengdong]

have you solved this problems?
Not yet, but I used the full nuScenes dataset later.

thank you for your replay, when you used the full nuScenes, Are the results will become batter?

Yes, after using the complete dataset, the results returned to normal.

[Azitt]

I'm experiencing the same problem with the "mini" dataset. I didn't do any training—I just used the pretrained model from their GitHub repository and evaluated it, but I got the same results as you did. Do I need to train the model myself to achieve the results they mention in the paper? Also, what about the pretrained model they provided on GitHub?

[ZhengyiLi1006]

Hi, I am experiencing the same issue while evaluating in Lyft Dataset. map==0, I'm not sure that where the bug is.