change of loss

[ctxya1207]

How does the loss change during the pre-training phase? My loss is not decreasing very obviously. For example, after several epochs, it goes from 19.4568 to 19.4523.

[pavancm]

Are you using the same training script with the same pretraining data? The loss generally decreases quite a bit in the initial epochs, but towards the later stages the decrease maybe small since the learning rate is also decreasing.

[ctxya1207]

您是否使用相同的训练脚本和相同的预训练数据？损失在初始阶段通常会大幅减少，但在后期，由于学习率也在下降，损失减少幅度可能会很小。

I applied your training framework to my own small dataset for pre-training