Decode JSON UTF-16 surrogate pairs instead of emitting invalid UTF-8

[cursor]

Problem

json_parse treated each \uXXXX escape as a BMP code point and emitted a 1–3 byte sequence. RFC 8259 encodes supplementary-plane scalars (emoji, etc.) as a UTF-16 surrogate pair. Python json.dumps (default ensure_ascii=True) and many MCP / A2A peers emit that form.

The two 3-byte CESU-8 halves are not UTF-8. json_serialize already rejects overlong / surrogate-range encodings and replaces them with U+FFFD, so a round-trip of "\uD83D\uDE00" (😀) became two replacement characters. MCP tool results, A2A message parts, and HTTP bodies that used escaped emoji were corrupted before they reached the model.

Fix

Decode a high+low surrogate pair into one scalar and emit well-formed UTF-8 (4 bytes for the SMP). Unpaired surrogates become U+FFFD, matching the serializer's replacement policy. BMP \u00e9-style escapes are unchanged.

Tests

unit_json — BMP escape, \uD83D\uDE00 / lowercase pair → 😀, serialize round-trip, unpaired high/low, high+BMP, nested MCP-shaped object, truncated hex. 9/9 passed locally (+ ASan/UBSan).

Independently mergeable against main (0456350). Code hunks (json.cpp, new test_json.cpp, CMakeLists.txt after unit_logger) do not overlap #321–#341. CHANGELOG [Unreleased] will need a rebase after whichever sibling merges first.

  

---

Note

Low Risk
Localized parser change with dedicated tests; improves interoperability for MCP/A2A JSON without altering number/object parsing or API contracts.

Overview
Fixes json_parse so \uXXXX handling matches RFC 8259: adjacent UTF-16 high+low surrogate pairs decode to one supplementary-plane scalar and are emitted as valid UTF-8 (e.g. \uD83D\uDE00 → 😀). Previously each half was encoded as its own 3-byte sequence, which json_serialize then replaced with U+FFFD—breaking emoji in Python json.dumps, MCP tool payloads, and A2A message shapes on parse/round-trip.

Refactors \u parsing with parse_hex4 and append_utf8 (4-byte UTF-8 for SMP; unpaired or out-of-range surrogates → U+FFFD, aligned with the serializer). Adds unit_json / tests/test_json.cpp (BMP, pairs, serialize round-trip, nested MCP-style JSON, truncated hex) and a CHANGELOG entry under [Unreleased].

^{Reviewed by Cursor Bugbot for commit 84fa86c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_3e6798d9-d1d1-42b0-9d31-328ccda8d24c)