Skip to content

Pretraining experiment/task config should incorporate fine-tuning tasks/refer to downstream task configs #137

Description

@alam-shahul

This will also allow us to infer the pretrained checkpoint path from the config, as opposed to requiring us to pass the checkpoint in the config.

Thus, the first run with the pretraining config will

  1. Train the pretrained model.
  2. Save the pretrained model.
  3. Train task-specific heads from scratch for the fine-tuning tasks.

Later on, if we want to run additional fine-tuning tasks, we can simply add these to the pretraining config and rerun, which will

  1. Load the pretrained model
  2. Train task-specific heads from scratch for the fine-tuning tasks.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions