Unusually slow training speed

[Iambestfeed]

I am training a tokenizer with a vocab size of 50,368 and have added some special tokens (mask, pad, unk, s, /s)

However, the count pairs step encounters an issue, taking 20 minutes to reach 99% progress, and from then on the speed becomes slower, with an estimated 2-3 hours to complete.

Is this a problem with the Hugging Face tokenizer library, or is it an issue with SuperBPE?

[Iambestfeed]

[PythonNut]

As far as we can tell, this is normal behavior. Training SuperBPE tokenizers is a lot slower than normal BPE tokenizers because the lack of pretokenization prevents the library from compressing the training text into a "bag of words" (with frequencies). If the progress is very slow (multiple days) you may be running out of memory.