You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Non-ASCII input corrupted: Python binding escapes to \uXXXX (ensure_ascii) and the engine never decodes it #803
Non-ASCII input (accented Latin, Japanese, Korean, …) is corrupted before it reaches the model when using the Python bindings / cactus serve (v2.1.0, macOS ARM64). Severity scales with multi-byte character density: English is unaffected, German nearly so, French partially broken, Japanese/Korean unusable.
Root cause
python/cactus/bindings/cactus.py::_to_json marshals messages/tools with json.dumps(obj). Python's default ensure_ascii=True rewrites every non-ASCII character as \uXXXX, and the engine's messages parser does not decode \u escapes: cactus-engine/src/engine.h::extract_json_string (line ~305) handles only \n \t \r \" \\ and falls through to value += json[pos] for anything else, so \u660e becomes the literal text u660e — the model receives u660eu65e5u306e… for 明日の. Each escaped character costs ~3 tokens and is semantic noise.
Evidence (model-independent)
Same messages+tools through the model's HF tokenizer vs usage.prompt_tokens from cactus serve (prebuilt-graph bundle of LiquidAI/LFM2.5-230M):
lang
HF tokens
cactus tokens
excess beyond the constant en offset (+42)
en
400
442
0
fr
409
467
+16
ko
408
495
+45
ja
403
527
+82
A Python replica of bpe.cpp (its byte map + the bundle's vocab/merges + whole-segment BPE) matches HF exactly for en/fr, so the tokenizer algorithm itself is fine — the corruption happens upstream of encode().
With ensure_ascii=False the excess is a constant +42 in all four languages (en/fr/ja/ko: +42/+42/+44/+42), and on an 808-row multilingual tool-calling eval, ja/ko false-action rates drop from 88–100% back to the bf16 reference (0–16%).
10-second repro
POST any Japanese sentence to /v1/chat/completions and compare usage.prompt_tokens with len(AutoTokenizer.from_pretrained(<same model>)(text).input_ids).
Fix
One line in _to_json: json.dumps(obj, ensure_ascii=False).encode() — PR: #804 (https://github.com/cactus-compute/cactus/pull/804). Alternative/complementary: have the C++ messages JSON reader decode \uXXXX — utils.h already contains a correct decoder used elsewhere.
Second, minor bug found on the way (C++, separate) — now tracked in #806, fixed by #805
bpe.cpp::init_byte_mappings maps bytes 161..255 to identity code points; GPT-2 byte-level BPE excludes 173 (U+00AD), which must map to U+0143 like the other non-printables. UTF-8 sequences containing byte 0xAD tokenize differently from the reference (a Japanese test sentence: 17 tokens with 1 unknown vs 15 in HF). Concrete symptom: 歯 is e6 ad af; cactus run on the stock lfm2.5-230m-cq4 bundle echoes it as �丈 (U+FFFD + wrong kanji) — this mapping bug on the decode side, independent of the escaping bug above. Fix: exclude 173 from the identity byte list AND from the identity branch of the assignment loop (else if (byte >= 161 && byte <= 255 && byte != 173)) — the branch is chosen by byte value, so moving the byte between lists alone is a no-op — and append 173 to remaining_bytes after 160 (→ U+0143). Verified: cactus_tokenize("歯") → [33687] (HF reference) after the change; before: [672, 0, 617]. PR: https://github.com/cactus-compute/cactus/pull/805
Activity
bindings/python: keep non-ASCII text as UTF-8 when marshalling JSON t…