are our tokenizers initalized correctly?

[kjappelbaum]

perhaps not for batch inference

[n0w0f]

For the llama runs we do not use mattext tokenizers though.

[n0w0f]

Ah I see now.
There was this issue of Llama tokenizer not including pad token.
So we set tokenizer.pad_token = tokenizer.eos_token
ref.

We also tried adding a token, this then resized the vocab and creates a set of problems

[kjappelbaum]

This is not an issue for the serial interface that is in the code at the moment. For batched inference this might be important in the future