Skip to content

Simplify task addition to the platform #35

Description

@vijusudhi

Currently, the platform handles addition of task separately for train and eval. Simplify and unify this.

TODOs:

  • unify train and evaluate configurations
  • simplify eval dataset ingestion to lm-eval-harness

We want a bigger config combining TrainerRunConfig (from auto-llm) and EvaluatorConfig (from lm-eval-harness) https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/config/evaluate_config.py#L32

  • identify common arguments (model args, dataste, templates)
  • make the common arguments globally available
  • separate trainer and evaluator args (batch sizes, metrics)
  • keep them optional (thios lets us run eval and train separately)

Config decomposition: We want to extend config_generator such that it takes this biugger config and generates TrainerRunConfig and EvaluatorConfig. This should allow us to run Trainer and Evaluator with minor changes to the respective classes.
Maybe place this class here: https://github.com/ag-sc/auto-llm/tree/german-ner-datasets/auto_llm/configurator

Branch out from german-ner-datasets

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions