Just a few notes from my interaction with the sigopt integration.
- We should probably report (best) validation scores, not test scores by defualt
- The runner should check if an experiment exists and is non-archived/deleted before creating a new one.
- In triplet training, the mAP evaluation is costly for large datasets. When doing hyper-parameter optimisation, we are generally only interested in the final value, so one could potentially skip all but the last evaluations. This also prefers a lower number of epochs if the results are comparable, saving on training time. (Maybe only enable this if n_epochs is part of the optimisation process?)
I will update this issue if I come across more points.
Just a few notes from my interaction with the sigopt integration.
I will update this issue if I come across more points.