Stage 2 Ignores Outputs from Stage 1?

[narutatsuri]

Did you happen to resolve this/figure out that this was not the case? I'm running into the same question because it seems like Stage 1 is pointless in the current code as Stage 2 does not load the learned tokens.

Originally posted by @narutatsuri in #14

When playing around with the code, I noticed that Stage 2 does not honor the subtokens learned in Stage 1 and instead simply runs tokenization from scratch with the sole difference of allowing cross-space merging. Is this intended? I am assuming it's not based on the description in the paper, which states that Stage 2 is supposed to extend Stage 1 (which I understood as preserve all tokens in Stage 1 but learn new supertokens up until the total number of tokens hits the provided max token count).

Thanks!

[PythonNut]

Stage 2 does start "from scratch," but it should begin by applying the merges from Stage 1. This is necessary because pretokenization discards information about the original order of the pretokens, so we can't form the super tokens during Stage 1. In Stage 2, we relax the pretokenization and replay the merges from Stage 1. Because the pretokenization string is different during the replay, actually the resulting segmentations might not match the result of Stage 1 perfectly but we haven't found this to be an issue. After the replay, we start learning new merges and since in Stage 2 we have relaxed pretok, we can now form the super tokens.

[narutatsuri]

Sorry, my question might have been unclear.

I recognize that Stage 2 starts "from scratch," but my understanding is that Stage 2 should only extend the resulting vocabulary obtained in Stage 1---i.e., (sub)tokens learned in Stage 1 that we decide to inherit should not be destroyed when running Stage 2 (or at the very least the majority of such tokens should be preserved, per what you mentioned above:

Because the pretokenization string is different during the replay, actually the resulting segmentations might not match the result of Stage 1 perfectly but we haven't found this to be an issue.

). But I find this to not be the case when I run the code. For example, when using a final vocab size of 150K, the tokens obtained in Stage 1 and after Stage 2 only share about ~70K tokens and there are 80K tokens in each that do not exist in the other, which was unexpected to me given my understanding of the paper.

[narutatsuri]

Hello! Was wondering if there were any updates on this, really appreciate it!

[alisawuffles]

Hi @narutatsuri, sorry for the trouble, let's figure this out! Stage 2 should inherit all of the tokens learned in stage 1.

Can you elaborate on what you are seeing? How many tokens are you inheriting in stage 1 and how many tokens are you adding in stage 2, and how many tokens do you find to be missing from stage 1?

Can you also share the exact script that you're using for Stage 2?