How to determine the optimal model?

[gwuei]

Hello, I saw that a model is saved at each round. When evaluating each model, I found that different metrics may achieve the best performance on different models. For example, for the abs rel metric, the model from the 10th round might be the best, but for rmse, the model from the 15th round might be the best. But how do we determine the optimal model?

[Lavreniuk]

It depends on the target task. Usually, you should choose one primary metric to optimize and select the best checkpoint according to that metric. Different checkpoints may perform best on different metrics (e.g., abs rel vs rmse), which is normal and happens in many ML tasks. So the “optimal” model is the one that performs best on the metric that is most important for your specific application.

[gwuei]

May I ask, how do you determine it during training?

[Lavreniuk]

In my case, the main comparisons with other models (for example, SQLdepth) were done on the KITTI benchmark. Since most works primarily report and compare AbsRel as the main metric on KITTI, I used AbsRel as the primary criterion during training and selected the checkpoint/epoch that achieved the best AbsRel performance.

[gwuei]

Okay, thank you very much.