In the 143rd sample group of the GPT-OSS-120B dataset, the results obtained by Fastokens and HF tokenizers are inconsistent, as Fastokens employs multithreading during processing, leading to discrepancies compared to the single-threaded text segmentation results. Currently, this appears to be caused by a bug in Fastokens' handling of the O200k tokenization scheme.
In the 143rd sample group of the GPT-OSS-120B dataset, the results obtained by Fastokens and HF tokenizers are inconsistent, as Fastokens employs multithreading during processing, leading to discrepancies compared to the single-threaded text segmentation results. Currently, this appears to be caused by a bug in Fastokens' handling of the O200k tokenization scheme.