Request to expose WordPiece/BERT as a runtime tokenizer type in `OrtxCreateTokenizer` (already exists as graph ops)

[mepiyush2000]

How is this feature request related to a problem?

OrtxCreateTokenizer cannot load a HuggingFace WordPiece/BERT tokenizer.json. Loading one (e.g. sentence-transformers/all-MiniLM-L6-v2, or any tokenizer_class: "BertTokenizer") fails with kOrtxErrorNotImplemented (error code 8). Multiple other models also uses this WordPiece tokeniser and it's not supported by the onnxruntime extension yet.

Onnxruntime-extensions already implements WordPiece and BERT tokenization — but only as graph custom operators (WordpieceTokenizer / BertTokenizer), not as a first-class type in the runtime tokenizer C-API. So users who want a standalone tokenizer (outside an ONNX graph) for BERT-family encoders have no supported path, even though the algorithm already lives in the codebase.

Probable cause (current code)

• operators/tokenizer/tokenizer_jsconfig.hpp — enum class TokenType { kUnknown, kUnigram, kBPE }; has no kWordPiece, and kTokenizerDict maps only BPE + Unigram/SentencePiece class names. BertTokenizer/DistilBertTokenizer/etc. are absent, so GetTokenType() returns kUnknown (after stripping a trailing "Fast"), which surfaces as error 8.
• shared/api/tokenizer_impl.cc — TokenizerImpl::LoadTokenizer only dispatches to the BPE and Unigram/SPM backends.
• Meanwhile the WordPiece logic already exists at operators/tokenizer/bert_tokenizer.{hpp,cc}, operators/tokenizer/wordpiece_tokenizer.{hpp,cc}, and operators/tokenizer/basic_tokenizer.{hpp,cc} (registered for graphs in tokenizers.cc).

Reproduction

extError_t e = OrtxCreateTokenizer(&tokenizer, "path/to/minilm_tokenizer_dir");
// e == kOrtxErrorNotImplemented (8); "Unsupported tokenizer class"

Proposed solution

Wire the existing WordPiece implementation into the runtime tokenizer path:

1. Add kWordPiece to enum class TokenType.
2. Add the BERT-family class names to kTokenizerDict ("BertTokenizer", "DistilBertTokenizer", "MobileBertTokenizer", "ElectraTokenizer", …) → kWordPiece, and/or detect model.type == "WordPiece" from tokenizer.json.
3. Add a kWordPiece case in TokenizerImpl::LoadTokenizer that reuses the existing greedy WordPiece + basic-tokenizer code.
4. Add a test/pp_api_test case that loads a BERT tokenizer.json and checks encode/decode against HF.

Alternatives considered

• Graph custom ops (BertTokenizer/WordpieceTokenizer) — works, but forces embedding tokenization into an ONNX graph; not usable as a standalone tokeniser API.
• BPE/Unigram fallback — not applicable; WordPiece is a distinct algorithm.

Willingness to contribute
Locally, I have implemented the WordPiece tokeniser and verified the output with multiple different test cases. They matches exact same to the Python transformers' AutoTokenizer library tokenizer = AutoTokenizer.from_pretrained("minilm_tokenizer_dir").
Happy to submit a PR implementing the above if the maintainers agree with the approach.