Skip to content

Automatically batch texts when too long #6

@dennlinger

Description

@dennlinger

For samples that exceed the 512 subword token limit, we currently do not have a strategy in place to deal with this.
This is both unwanted and relatively easy to improve. There are a few considerations with respect to the exact strategy to be used, but it seems like a good starting point to approximate sentences with something like a lightweight spacy model, and then chunk based on approximate max length.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions