Found in a full-repo review of main @ 8e8970a. Both confirmed. Silent-wrong-answer class (#628/#629 family), hit by ordinary API payloads.
1. Surrogate pairs are not combined
x is json_decode of "\"\\ud83d\\ude00\""
print of x
print of (len of x)
Actual: 6 bytes, ED A0 BD ED B8 80, len = 6.
Expected: U+1F600 as 4-byte UTF-8, F0 9F 98 80, len = 4.
eigs_json_parse_string (src/builtins.c:951-980) encodes each \uXXXX independently, so a surrogate pair becomes two 3-byte surrogate encodings — CESU-8, which is not valid UTF-8. Every mainstream JSON encoder emits astral-plane characters (emoji, many CJK extension chars, mathematical alphanumerics) as exactly this escape pair, so any real payload containing one decodes to a string that is invalid UTF-8 and compares unequal to the same text decoded anywhere else.
Note the literal (unescaped) form is handled correctly — json_decode of "\"😀\"" round-trips fine — so this affects only the escaped form, which is the one produced by default by most encoders.
2. `` truncates the rest of the string
x is json_decode of "\"a\\u0000b\""
print of (len of x)
Actual: 1 (the string is "a"). Expected: 3.
cp == 0 appends a NUL byte into the strbuf; make_str(buf.data) at builtins.c:980 is a strlen copy, so everything after the NUL is dropped with rc=0 and no diagnostic.
Suggested fix
Combine a high surrogate (D800–DBFF) with a following low surrogate (DC00–DFFF) into one code point before UTF-8 encoding. Decide explicitly what an unpaired surrogate does — U+FFFD substitution or a loud raise, but not silent CESU-8. For ``, either reject it or carry an explicit length through to make_str so embedded NULs don't truncate; given #416 settled on bytes-forever, a loud rejection is probably the honest call.
Found in a full-repo review of
main@8e8970a. Both confirmed. Silent-wrong-answer class (#628/#629 family), hit by ordinary API payloads.1. Surrogate pairs are not combined
Actual: 6 bytes,
ED A0 BD ED B8 80,len=6.Expected: U+1F600 as 4-byte UTF-8,
F0 9F 98 80,len=4.eigs_json_parse_string(src/builtins.c:951-980) encodes each\uXXXXindependently, so a surrogate pair becomes two 3-byte surrogate encodings — CESU-8, which is not valid UTF-8. Every mainstream JSON encoder emits astral-plane characters (emoji, many CJK extension chars, mathematical alphanumerics) as exactly this escape pair, so any real payload containing one decodes to a string that is invalid UTF-8 and compares unequal to the same text decoded anywhere else.Note the literal (unescaped) form is handled correctly —
json_decode of "\"😀\""round-trips fine — so this affects only the escaped form, which is the one produced by default by most encoders.2. `` truncates the rest of the string
Actual:
1(the string is"a"). Expected: 3.cp == 0appends a NUL byte into the strbuf;make_str(buf.data)atbuiltins.c:980is astrlencopy, so everything after the NUL is dropped with rc=0 and no diagnostic.Suggested fix
Combine a high surrogate (
D800–DBFF) with a following low surrogate (DC00–DFFF) into one code point before UTF-8 encoding. Decide explicitly what an unpaired surrogate does — U+FFFD substitution or a loud raise, but not silent CESU-8. For ``, either reject it or carry an explicit length through tomake_strso embedded NULs don't truncate; given #416 settled on bytes-forever, a loud rejection is probably the honest call.