Add tekken pre-tokenizer (Mistral NeMo family) — Nemotron-Nano-9B-v2 unloadable since #373's loud gate

[jamesburton]

Follow-up to #373, found live during the #372 perplexity A/B: bartowski/nvidia_NVIDIA-Nemotron-Nano-9B-v2-GGUF carries tokenizer.ggml.pre = tekken, which #373's (correct) loud gate now rejects at load. Before #373 this model silently tokenized with NO pre-tokenization — so every prior dotLLM run of it, including the gated NemotronHTextGeneratorTests real-weights e2e, was mis-tokenizing.

Fix: add the tekken pipeline. .NET regex supports Unicode category classes natively, so use the ORIGINAL tokenizer.json pattern quoted verbatim in llama.cpp llama-vocab.cpp:408 (llama.cpp itself ships a lookahead rewrite only because std::regex lacks \p{Lu}-style classes).

Acceptance:

• tekken resolves to its own pipeline (llama.cpp LLAMA_VOCAB_PRE_TYPE_TEKKEN).
• A test discriminating tekken from BOTH gpt2 and llama3 routing (its bare \p{N} splits every digit; the others group digit runs).
• Nemotron-Nano-9B-v2 loads.

[jamesburton]

Correction: this issue's premise is wrong — there is no loud gate

The title and body state that Nemotron-Nano-9B-v2 is "unloadable since #373's loud gate". No such gate exists, on any branch. Verified against origin/dev @ dec7337f:

• src/DotLLM.Models/Gguf/GgufBpeTokenizerFactory.cs:74-77 reads tokenizer.ggml.pre and passes it straight through, untouched.
• src/DotLLM.Tokenizers/Bpe/TiktokenPreTokenizer.cs:84-93 ends _ => null.
• A pickaxe search across all branches finds no throw or warning on an unknown pre.

#373 is still open and unfixed. Its acceptance criteria (make unknown pre loud) have not been implemented.

What this changes

The ask is still valid — tekken is genuinely absent from the pipeline table. But the urgency and failure mode are the opposite of what's described:

• Not "unloadable" (loud, obvious, safe).
• Actually silently mis-tokenized (quiet, invisible, and it invalidates every quality measurement taken on the model).

That is strictly worse. The body's own parenthetical is the accurate part: "before #373 this model silently tokenized with NO pre-tokenization — so every prior dotLLM run of it ... was mis-tokenizing." That is still the present tense.

For scale: #237 measured tokenizer drift alone at +3.46% PPL on wikitext-2 (24.84 vs 24.01).

Re-blocking

This should be blocked on #373, not treated as a consequence of it. #373 is now the umbrella for the whole class — tekken (here), pixtral (Nemotron 3.5), qwen35 + qwen2 (Qwen; qwen2 affects all Qwen GGUFs). Fixing #373's silent fallthrough first converts every subsequent regex addition into a small, safe change and makes the class of bug loud.

Context: .docs/model-target-nemotron-3.5-lightning.md, .docs/model-target-qwen3.8-27b.md.

[jamesburton]

Correction to my previous comment — I was wrong, this issue's premise was right

My earlier comment asserted that no loud gate exists "on any branch" and that this issue's justification was therefore wrong. That is incorrect, and I'm retracting it.

The gate does exist — on an unmerged branch with no open PR, which is why a search of dev missed it:

• origin/issue/373-loud-pretokenizer, commit 8abf2281 (2026-08-13):
  "fix(tokenizers): unknown tokenizer.ggml.pre now throws; absent defaults to gpt2; add pixtral + llama3-family aliases (Unknown tokenizer.ggml.pre silently disables pre-tokenization entirely (e.g. pixtral) #373)"

So this issue's phrasing — "unloadable since #373's loud gate" — was a branch-local truth, written by someone working on that branch. Not a false premise. My mistake was treating "absent from dev" as "absent everywhere", which is the same stale-view error this issue's own subject matter keeps producing.

There is also already an implementation of this issue:

• origin/issue/373-tekken, commit cc0ee1e0 (2026-08-14):
  "feat(tokenizers): add tekken pre-tokenizer pipeline (Add tekken pre-tokenizer (Mistral NeMo family) — Nemotron-Nano-9B-v2 unloadable since #373's loud gate #387)"

What stands from the earlier comment

Only this, and it is still worth recording: on dev today, unknown tokenizer.ggml.pre is silent, so any measurement taken against dev on an affected model is measuring tokenizer drift. #237 quantified that at +3.46% PPL on wikitext-2. That is an argument for landing these two branches, not for re-implementing them.

Actual state

Work	Branch	PR
Loud gate + pixtral (#373)	issue/373-loud-pretokenizer	none
tekken (#387)	issue/373-tekken	none
qwen35 + qwen2 (#397)	—	not started

The gap is not implementation, it is that neither branch was ever PR'd or merged. Retitling this issue back is unnecessary; the original title was accurate for its author's context.

[jamesburton]

Final correction — this issue was right all along, and my two earlier comments were both wrong

Verified properly this time, against origin/dev rather than a stale local checkout:

git merge-base --is-ancestor 8abf2281 origin/dev  -> MERGED   (#373 loud gate + pixtral)
git merge-base --is-ancestor cc0ee1e0 origin/dev  -> MERGED   (#387 tekken)

The gate lives in TiktokenPreTokenizer.GetRegexes (origin/dev, :120-127) — unknown pre throws InvalidDataException, with a DOTLLM_ALLOW_UNKNOWN_PRETOKENIZER=1 escape hatch, mirroring llama.cpp. pixtral is at :113 and tekken at :119.

How I got it wrong twice

1. First comment: claimed no gate exists and this issue's premise was false. I had grepped GgufBpeTokenizerFactory.cs:74-77, which does pass pre through untouched — but the gate is one level down, in GetRegexes. Wrong file.
2. Retraction: claimed the gate existed only on an unmerged branch. Also wrong — the branch content was already merged; the branches on origin are stale leftovers.

Both errors share a root cause: reading a local checkout 255 commits behind origin/dev and treating it as current. That is the same failure this repository's own handoff notes warn about — a stale local branch looks identical to a genuinely absent fix — and I walked straight into it while auditing for exactly that class of problem.

Apologies for the noise on this issue. The original title and body were accurate; nothing here needed correcting.

What was actually still missing

Only qwen2 and qwen35. Because the gate landed without them, every Qwen2/Qwen3 GGUF has been throwing at load since — and before the gate, running with no pre-tokenization at all. Fixed in #410, which closes this issue along with #373 and #397.