[chaog992]

What this changes

Ling's tokenizer.json (inclusionAI/Ling-3.1-flash, the model of #850 and #874, and TensorFold's Ling-3.0-flash checkpoints) pre-tokenizes with [^\r\n\p{L}\p{N}]?+\p{L}+ and ?[^\s\p{L}\p{N}]++[\r\n]*. The matcher refused possessive quantifiers (zig/src/core/tokenizer/regex.zig:329), so the tokenizer did not load: error.UnsupportedRegex.

A possessive quantifier now compiles to its greedy repetition inside an atomic step, a lookahead that keeps its match (Inst.look with atomic set): the rest of the pattern goes on from the end of the body's first match and never gives back what it took, as Oniguruma matches it. {n,m}+ keeps Ruby syntax's reading as two quantifiers, and (?>...), which no tokenizer.json I fetched uses, stays refused. A pattern without possessive quantifiers compiles to the same program as before.

It stands on its own: #874, which would add a Ling family, is parked, and none of its files touch the tokenizer.

Receipt

• Environment: base d1018b3 (main, 1.1.1 merged), fix 182afb9. Zig 0.17.0, Linux x86_64 under WSL2, Python tokenizers 0.23.3, no GPU used, no model.
• Exactness: tools/zig/tokenizer_parity.py on the tokenizer.json of inclusionAI/Ling-3.1-flash at revision a4ccd6b4: encode 2784/2784 and decode 5367/5367 equal; on the base the checker stops with error: UnsupportedRegex. TensorFold's five Ling-3.0-flash checkpoints ship the same file (same sha256). inclusionAI/Ling-flash-2.0's at 96f9b7a7, a different file with the same pattern, gives the same counts, and the base refuses it too. The 23 other distinct tokenizer.json files I fetched (listed below the receipt; 11 come from TensorFold's published models) and the tool's six synthetic tokenizers pass, as on the base.
• Decode speed: n/a: tokenization only.
• Prompt speed: no tools/bench_cache.py run (no GPU or model); the change reaches a prompt only through the tokenizer's encode. Encode time on seven other tokenizers (DeepSeek-V3.2-Exp, step3, DeepSeek-V4-Flash, Qwen3-30B-A3B, gpt-oss-20b, GLM-4.6, phi-4), base against fix, with the parity tool's checker on its cases, medians of 12 alternating runs pinned to one core: +0.4 to +2.0% built ReleaseSafe, as the Mac and XPU servers build the tokenizer, and -2.5 to +0.5% built ReleaseFast, as the CUDA server does; a second build of the base, the same machine code, moved -1.6 to +0.9%. That is within code-placement noise: moving the matcher by 16 to 48 bytes moved the base alone by up to 7.6% on the DeepSeek-style tokenizers, and over four such placements the fix against the base averaged -0.8 to -0.6% built ReleaseSafe and -0.4 to -0.3% built ReleaseFast on the tokenizers tried. These runs used builds on 1.1.0; rebuilt on d1018b3, the base and fix checkers have the same .text as those builds.
• Tests run (each exits 0 unless noted):
  • New test in regex.zig: Ling's pattern splits a sample as the tokenizers library does. Five patterns where giving back would match more find what the library's Oniguruma finds (a*+a finds nothing in "aaa", [0-9]?+5 finds only "15" in "15 5", b?+\p{L}+ finds only "bb" in "bb b1", (?:ab)++ab finds nothing in "abab ab", (?:a++)+a finds nothing in "aaa"). Two pin what the atomic step keeps: (?:a|ab)?+b finds "ab" in "ab" (the body's first match, not its longest), and a?a?+a finds "aa" in "aa" (only the quantified element is atomic). a{1,2}+a finds "aaaa", since Ruby syntax reads {n,m}+ as two quantifiers; a plain lookahead still keeps nothing (a(?=b) finds "a" twice in "ab ab"); and (?:a*+)* and (?:a?)++ are refused, as (a*)* is.
  • zig test -O <mode> zig/src/core/tokenizer/regex.zig: 7/7 in Debug, ReleaseSafe and ReleaseFast (6/6 on the base).
  • zig build test test-kernel-contract test-golden fuzz-cortheon: 225/230 steps, 3033/3050 tests, 14 skipped, 3 failed, exit 1 (3033/3049 on the base, 2 failed); server goldens 306/319 and 26/26 equal, 0 unexpected, as on the base. Plain zig build: 24/24 steps.
  • The failures are digest_cache_test's "digest cache detects same-size rewrites even when mtime is restored" and cuda_cost_identity's "cost identity: stable reads reuse a drafter, while a changed payload at the same path invalidates it", which fail often on this machine, on the base too (alone: 16 and 27 of 30 runs): this WSL2 kernel stamps file times on a coarse tick (CONFIG_HZ=100; 11 ms steps here), so a same-size rewrite within one tick keeps the first file times. The third, store_test's "two split ranks make the same decisions on the same calls and each restores its own rows", failed in this run only; the sessions test binary (all its 27 tests) fails it in 1 of 20 runs on the base and 2 of 20 with the fix, for a reason I have not found.
  • CI's zig-windows job fails on other pull requests on d1018b3 too (perf(hip): group routed fc1 and shared-up stream products #1103, nemotron cuda: an encode hook on Vision, and a host test that a request encodes each image once #1104), with compile errors in Windows host code this change does not touch; Windows: main's host tests build again on Windows hosts #1034, still open, fixes part of them. regex.zig's tests compile for x86_64-windows with --test-no-exec, a compile check only.
  • tools/zig/lean_check.py: the same 444 problems as the base, exit 1 on both; the two in the touched file, regex.zig:1 and regex.zig:281 (272 on the base), are inherited. tools/zig/lean_new.py against the base: 0 new. zig fmt --check build.zig zig and git diff --check pass.
• Failing receipt: the new test on the base's regex.zig fails with error.UnsupportedRegex in Debug, ReleaseSafe and ReleaseFast; Debug's trace points at the base's regex.zig:329.
• Not run, and why: no Mac or CUDA run: the tokenizer is host code every backend shares (regex.zig's tests compile for aarch64-macos with -mcpu apple_m1 --test-no-exec, a compile check only). No Mac, so no encode timing on Apple silicon, where the Mac server builds the tokenizer ReleaseSafe. No GPU or model, so no tools/bench_cache.py TTFT and no served Ling request (main has no Ling family yet).

The 23 other tokenizer.json files

TensorFold: DeepSeek-V4-Flash-0731-MXFP4-MLX, DeepSeek-V4.1-Flash-MLX-2bit-MTP, GLM-5.3-Flash-MLX-4bit-MTP, Laguna-S-2.1-MLX-4bit, MiMo-V2.6-Flash-RL-MLX-4bit-MTP, Muse-Glimmer-30B-MLX-4bit, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit, Nex-N2.5-mini-MLX-4bit (the same file as Qwen3.6-35B-A3B-MLX-4bit-MTP), Qwen3.8-27B-MLX-4bit, Qwen3.8-Flash-Next-MLX-4bit, Solar-Open2-250B-MLX-4bit.

Others: LiquidAI/LFM2-8B-A1B, MiniMaxAI/MiniMax-M2, Qwen/Qwen3-30B-A3B (the same file as XiaomiMiMo/MiMo-V2-Flash), baidu/ERNIE-4.5-21B-A3B-PT, deepseek-ai/DeepSeek-V3.2-Exp, ibm-granite/granite-4.0-h-small, microsoft/phi-4, nvidia/NVIDIA-Nemotron-Nano-9B-v2, openai/gpt-oss-20b, stepfun-ai/step3, tencent/Hunyuan-A13B-Instruct, zai-org/GLM-4.6.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (n/a: tokenizer only.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (n/a: tokenizer only; parity against the tokenizers library above.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No arithmetic changes.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (n/a: no kernel changes.)
• Prompt processing is no slower than the last release. (No TTFT run: no GPU or model. The change reaches a prompt only through the tokenizer's encode; see Prompt speed.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (It needs no model or GPU and never skips.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @chaog992, this is landed on main as 79ba3f5, with you as the author. Possessive quantifiers as an atomic step over the greedy repeat is a clean fit for the matcher, and the cases where giving back would match more (a*+a, (?:a|ab)?+b, a{1,2}+a) pin exactly the behaviour the tokenizers library's Oniguruma has. Ling's pre-tokenizer pattern now loads, and no pattern that loaded before changes, since a + after a quantifier used to be refused. The host suite passes on main with it.

- Synthema, Ash's AI Assistant.