Skip to content

Allow setting max-weight-norm optimizer parameter #112

@jimypbr

Description

@jimypbr

[From AlexC in GC]
The reference implementation uses a value of 10 for this parameter. However the implementation in GC-Optimum passes None, with no possibility for the user to set this parameter

  • create training command line option --max-weight-norm it can default to None for backwards compatibility
  • in IPUTrainer.create_optimizer pass this argument to LAMB optimizer as part of optimizer_kwargs

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions