More comparison

[huyiwen]

Thanks for your work! I wanted to follow up on a question I posted on HF earlier, which may have been overlooked.

While reading your paper, I noticed that when comparing the superbpe and standard BPE models, the superbpe has a shorter context length and requires more training steps (with FLOPs held constant). I'm particularly interested in understanding how the model performs under these two scenarios:

1. The current SuperBPE model under the same number of training steps (early stopped checkpoints).
2. Re-training with a context length of 4096 for the same number of training steps.

I think these might better reflect real training practices.

Looking forward to your insights!

[alisawuffles]

Hi @huyiwen! These are great questions. We are currently training (2), which will probably complete in a week. As a bit of background, we chose to fix effective context size for a more fair comparison. We have seen that at a fixed token position, a SuperBPE model achieves substantially lower bits-per-byte than BPE models due to seeing more raw text as context (left) — this advantage goes away when you instead compare at fixed byte positions (right). However, we understand there are several reasons to keep the context size the same in tokens, which is why we are training this comparison now!

For (1), we can see that if we plot evals against training steps (right) instead of FLOPs (left) on the x-axis, the best SuperBPE model is still ahead of the BPE model, though of course the gain has been massively diminished. This is not quite a "correct" comparison, as the learning rate schedule for the SuperBPE model is not set correctly for the train duration. For model developers who are data-constrained, we recommend scaling up the size of the model which controls for training FLOPs, inference FLOPs, and raw text budget.

[huyiwen]

That makes a lot of sense! I’m really looking forward to using SuperBPE!

[wang99711123]

@alisawuffles Thanks for the great work and discussion! I wonder if experiment (2) has been done yet? Or if there are any conclusions you can share?

[alisawuffles]

Sorry for the delay. We completed experiment (2) and found that indeed, adjusting the sequence length in tokens is crucial to the SuperBPE model's improved performance. Therefore, the benefit of SuperBPE seems to be that we can strike a better tradeoff between the sequence length (in tokens) and number of training steps, without sacrificing true context length.

Therefore, we recommend adjusting the sequence length of the model to match the BPE comparison. To maintain throughput, it is generally advisable to choose a seqlen based on powers of 2. For instance, for a BPE model with seqlen 8096, the SuperBPE counterpart could have a seqlen of 6144. We are in the process of writing a blog post on this finding!

[alisawuffles]

Please see our discussion here! Happy to discuss more. In practice, we have found that people are willing to adjust their max sequence length as long as it preserves the original training throughput. 😄

https://superbpe.github.io/faq.html#context_adjustment