How do I stop training based on loss value?

[wigging]

In the Training section of the README, it states the following for the MP-20 training:

The validation loss (loss_val) should reach 0.4 after 360 epochs (about 80k steps).

When I train the model using the MP-20 data, the training does not stop until it reaches 900 epochs which is the default value in mp_20.yaml for max_epochs. This occurs even if the loss value is 0.4 or lower. How do I configure the training to stop when the loss value is reached instead of always running to the max number of epochs?

[danielzuegner]

Hi @wigging,

the statement in the README is only to help you determine whether your training job is performing reasonably. It is not meant to indicate that you should stop at this loss value. I am not aware of a pytorch lightning callback that lets you stop at a certain loss value (typically what people use is early stopping if the loss does no longer decrease), but you should be able to write one in a few lines of code if you need training to stop at a certain loss value.

[wigging]

If I wanted the training to stop at a loss value of 0.5 is this what I put in trainer/default.yaml?

callbacks:
  - _target_: pytorch_lightning.callbacks.EarlyStopping
    monitor: loss_val
    mode: min
    stopping_threshold: 0.5

[danielzuegner]

That looks right, though I've never tried this myself.

[wigging]

I ran the MP-20 training with a stopping threshold of 1.5 and the training stopped when the loss value dropped below 1.5. So I can confirm that the YAML snippet shown above does indeed work.