This will also allow us to infer the pretrained checkpoint path from the config, as opposed to requiring us to pass the checkpoint in the config.
Thus, the first run with the pretraining config will
- Train the pretrained model.
- Save the pretrained model.
- Train task-specific heads from scratch for the fine-tuning tasks.
Later on, if we want to run additional fine-tuning tasks, we can simply add these to the pretraining config and rerun, which will
- Load the pretrained model
- Train task-specific heads from scratch for the fine-tuning tasks.
This will also allow us to infer the pretrained checkpoint path from the config, as opposed to requiring us to pass the checkpoint in the config.
Thus, the first run with the pretraining config will
Later on, if we want to run additional fine-tuning tasks, we can simply add these to the pretraining config and rerun, which will