[Bug] BertTokenizer custom operator produces non-deterministic alternating outputs due to cross-call state pollution

[YoungVincent-93]

Bug Description

The BertTokenizer custom operator (generated via gen_processing_models) produces non-deterministic and inconsistent tokenization results when called consecutively with the exact same input.
Further investigation reveals that this is not a multi-threading race condition, but rather a cross-call state pollution bug within the C++ kernel implementation. The Compute() method modifies internal member variables (e.g., temporary buffers) during execution and fails to reset them before the next call. Consequently, the N-th inference depends on the dirty state left by the (N-1)-th inference, resulting in a strict alternating pattern of two different tokenization outputs.

Environment

• OS: Windows / Linux (Reproducible on both)
• Python: 3.10.x
• onnx: 1.12.0
• onnxruntime: 1.12.0
• onnxruntime_extensions: 0.15.0
• numpy: 1.24.4
• transformers: 4.49.x

Reproduction Steps

1. Export the Tokenizer to ONNX

import numpy as np
import onnxruntime as ort
from onnxruntime_extensions import gen_processing_models, get_library_path
from transformers import BertTokenizerFast
model_dir = "./pretrained/bert-chinese-sentiment" # Replace with your model
tokenizer = BertTokenizerFast.from_pretrained(model_dir)
# Export
tokenizer_onnx_model = gen_processing_models(tokenizer, pre_kwargs={})[0]
with open("tokenizer.onnx", "wb") as f:
    f.write(tokenizer_onnx_model.SerializeToString())

2. Run Inference in a Loop

sess_opts = ort.SessionOptions()
sess_opts.register_custom_ops_library(get_library_path())
sess = ort.InferenceSession("tokenizer.onnx", sess_options=sess_opts)
test_text = "A0"
for i in range(10):
    feed = {"text": np.array([test_text], dtype=np.string_)}
    input_ids = sess.run(["input_ids"], feed)[0]
    print(f"Run {i}: {input_ids.tolist()}")

Actual Behavior

The output strictly alternates between two completely different tokenization results, even producing different sequence lengths (8 tokens vs. 10 tokens):

Run 0: [101, 1037, 2692, 4305, 100, 1009, 100, 102]          # Variant A (8 tokens)
Run 1: [101, 1037, 2692, 2213, 2480, 2213, 1009, 1009, 100, 102] # Variant B (10 tokens)
Run 2: [101, 1037, 2692, 4305, 100, 1009, 100, 102]          # Variant A
Run 3: [101, 1037, 2692, 2213, 2480, 2213, 1009, 1009, 100, 102] # Variant B
...

Furthermore, the exact token IDs for Variant A and Variant B change if the Python script is restarted, indicating the internal state relies on uninitialized or heap-dependent memory.

Root Cause Analysis

To rule out multi-threading race conditions, I forced ONNX Runtime to run in a strictly single-threaded environment:

sess_opts = ort.SessionOptions()
sess_opts.register_custom_ops_library(get_library_path())
sess_opts.intra_op_num_threads = 1  # Force single thread
sess_opts.inter_op_num_threads = 1  # Force single thread
sess = ort.InferenceSession("tokenizer.onnx", sess_options=sess_opts)

The alternating output persists even in single-threaded mode:

Run 0: [101, 1037, 2692, 2480, 3401, 100, 1022, 100, 102] # Variant C
Run 1: [101, 1037, 2692, 100, 102]                        # Variant D
Run 2: [101, 1037, 2692, 2480, 3401, 100, 1022, 100, 102] # Variant C
Run 3: [101, 1037, 2692, 100, 102]                        # Variant D
...

This proves that the issue is not a concurrency race condition, but rather a deterministic state pollution bug inside the BertTokenizer C++ kernel:

1. The BertTokenizer custom op is instantiated as a global singleton by ONNX Runtime.
2. Inside Compute(), certain member variables (likely string buffers or token vectors used by BasicTokenizer or WordPiece) are modified.
3. These member variables are not reset/cleared at the start of the Compute() method.
4. Thus, Run N reads the dirty state left by Run N-1, resulting in an alternating flip-flop between two internal states.

Workaround

Creating a new InferenceSession for every single inference clears the internal state, but this is completely impractical due to massive performance overhead. Re-creating sessions also yields unpredictable initial states.
The only reliable production workaround currently is to abandon the ONNX tokenizer entirely and use the HuggingFace native tokenizer (which is stateless and thread-safe) for text preprocessing, feeding the resulting Tensors into the ONNX Runtime model for inference.

Related Information

• Attempting to use HfJsonTokenizer (via schema_v2=True in gen_processing_models) does not resolve the issue.
• This bug makes it impossible to build a stable end-to-end ONNX pipeline (Text -> Tokenizer -> BERT -> Logits) using onnxruntime-extensions.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.