What a great job! I would like to ask what the function of if cur_len < tokenizer.model_max_length: assert cur_len == total_len in collate_fn of dataset.cy is in the code, as I have observed that omitting this part of the code can run normally.
What a great job! I would like to ask what the function of
in collate_fn of dataset.cy is in the code, as I have observed that omitting this part of the code can run normally.