[mohitsoni48]

Problem

Loading a K2 Horizon / MoVA GGUF (tokenizer.ggml.pre == "k2-horizon") fails at vocabulary
load time with:

Failed to process regex: '(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\r\n\p{L}\p{N}]?(?:\p{L}|\p{M}|‌|‍)+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+'
Regex error: regex_error(error_escape): The expression contained an invalid escaped character, or a trailing escape.
error loading model: error loading model vocabulary: Failed to process regex

LLAMA_VOCAB_PRE_TYPE_K2_HORIZON is correctly registered in llama-vocab.cpp, so this isn't
an "unknown pre-tokenizer type" issue. The problem is one level down: the k2-horizon regex
string isn't registered in unicode_regex_split_custom (unlike llama3/gpt2/qwen2/qwen35/
kimi_k2/afmoe), so it falls through to the generic std::regex/std::wregex path in
unicode_regex_split. That path does pre-collapse known \p{...} categories before handing
the pattern to std::regex, but it has no handling for the literal ‌/‍
(ZWNJ/ZWJ) escapes in the pattern — and std::regex's ECMAScript grammar doesn't support
\u code-point escapes on any platform, not just MSVC — so compilation always throws
error_escape.

This reproduces on Linux/GCC too, not just Windows/MSVC (confirmed with a standalone harness
against unicode_regex_split directly, independent of #17830's Windows-only regression).

Fix

The k2-horizon regex is otherwise identical in shape to the llama3 pattern — same
contraction handling, same \p{N}{1,3} grouping, same trailing whitespace/punctuation
rules — except its word-run group is (?:\p{L}|\p{M}|‌|‍)+ instead of \p{L}+
(it also treats combining marks and the ZWNJ/ZWJ joiners as part of a word, which matters
for scripts like Devanagari that rely on them).

This adds unicode_regex_split_custom_k2_horizon, adapted directly from
unicode_regex_split_custom_llama3 with the word-run predicate widened to
is_letter || is_accent_mark || cpt in {0x200C, 0x200D}, and registers it in
unicode_regex_split_custom for the exact k2-horizon regex string — routing it through the
same non-regex custom-splitter path as the other known tokenizer families, instead of the
std::regex fallback.

Testing

• Extended tests/test-unicode.cpp with a small case table covering the k2-horizon regex:
  leading-space attachment, case-insensitive contraction splitting, a combining mark staying
  attached to its base letter, a ZWJ keeping two letters joined into one token, and digit
  grouping into chunks of ≤3.
• Confirmed the new test fails against the pre-fix code, reproducing the exact reported error
  (Failed to process regex / regex_error(error_escape)), and passes after the fix.
• tests/test-unicode.cpp isn't built as part of the default Windows (shared-lib) CMake
  config here (guarded by NOT WIN32 OR NOT BUILD_SHARED_LIBS in tests/CMakeLists.txt,
  since unicode_regex_split isn't exported via LLAMA_API), so I also verified by compiling
  unicode.cpp + unicode-data.cpp + the test directly with cl.exe standalone. On a
  non-Windows / static-lib CI config this will build as test-unicode as usual.
• Rebuilt llama-server.exe from this branch with the fix and confirmed a clean compile/link.

No behavior changes for any other pre-tokenizer type — this only adds a new branch to
unicode_regex_split_custom's string dispatch and a new standalone splitter function.

[mohitsoni48]

Duplicate of #1 — I independently ran into and root-caused the same bug, and landed on the identical fix (same custom splitter, same letter-run predicate extension). #1 was opened first and its test case (real Persian ZWNJ text) is better than my synthetic cases, so closing this in favor of that one. Leaving a verification note on #1.