Upstreaming minimal set of changes to HF tokenizers to get rid of the need for the fork

[rrenaud]

Hi.

The results in the paper are really nice. It seems like it's time for new LLMs to start having spaces in their tokens. I'd like to put in the leg work to upstream your changes to the tokenizers library.

Is this something you'd like to see happen?

[PythonNut]

Hello! Yes, we are definitely interested in getting SuperBPE into mainline HuggingFace tokenizers, and we think this would greatly improve the ease of model developers interested in experimenting with SuperBPE! The current two stage approach allowed us to iterate quickly, but we think it would be great if people could just take some data and pass it directly to the huggingface trainer with, say, superbpe_transition=0.9 and get a working SuperBPE tokenizer! We would love to stay involved + be helpful during your process.