Currently, the platform handles addition of task separately for train and eval. Simplify and unify this.
TODOs:
We want a bigger config combining TrainerRunConfig (from auto-llm) and EvaluatorConfig (from lm-eval-harness) https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/config/evaluate_config.py#L32
- identify common arguments (model args, dataste, templates)
- make the common arguments globally available
- separate trainer and evaluator args (batch sizes, metrics)
- keep them optional (thios lets us run eval and train separately)
Config decomposition: We want to extend config_generator such that it takes this biugger config and generates TrainerRunConfig and EvaluatorConfig. This should allow us to run Trainer and Evaluator with minor changes to the respective classes.
Maybe place this class here: https://github.com/ag-sc/auto-llm/tree/german-ner-datasets/auto_llm/configurator
Branch out from german-ner-datasets
Currently, the platform handles addition of task separately for train and eval. Simplify and unify this.
TODOs:
We want a bigger config combining
TrainerRunConfig(from auto-llm) andEvaluatorConfig(from lm-eval-harness) https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/config/evaluate_config.py#L32Config decomposition: We want to extend
config_generatorsuch that it takes this biugger config and generatesTrainerRunConfigandEvaluatorConfig. This should allow us to run Trainer and Evaluator with minor changes to the respective classes.Maybe place this class here: https://github.com/ag-sc/auto-llm/tree/german-ner-datasets/auto_llm/configurator
Branch out from
german-ner-datasets