pypad_glasscoder.py Crashes on Unicode Characters

[dklann]

Platform

Ubuntu (Linux Mint 21.1, up to date)

glasscoder_pypad Version

2.0.1-1

Rivendell Version

4.1.0-1

Expected Behavior

A normal metadata update sent to GlassCoder from Rivendell.

Steps to Reproduce

Send metadata containing, for example, the © symbol. pypad_glasscoder.py exits with non-zero status.

Backtrace

Traceback (most recent call last):
  File "/usr/lib/rivendell/pypad/pypad_glasscoder.py", line 66, in <module>
    rcvr.start(sys.argv[1],int(sys.argv[2]))
  File "/usr/lib/python3/dist-packages/rivendellaudio/pypad.py", line 963, in start
    self.__pypad_Process(Update(jdata,self.__config_parser,rd_config))
  File "/usr/lib/python3/dist-packages/rivendellaudio/pypad.py", line 812, in __pypad_Process
    self.__pad_callback(pad)
  File "/usr/lib/rivendell/pypad/pypad_glasscoder.py", line 51, in ProcessPad
    r = requests.post(req_url, json=json.loads(req_data))
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 187 (char 186)

Additional Thoughts

I see in pypad.py where the JSON is escaped (def __escapeJson()), but it's not clear that this is the right place to address Unicode characters...

[ElvishArtisan]

Unable to reproduce the problem here. This is on CentOS 7, Python 3.6.8.

I see in pypad.py where the JSON is escaped (def __escapeJson()), but it's not clear that this is the right place to address Unicode characters...

None of that is specifically addressing non-Latin1 characters. Strings in Python 3 are Unicode-aware (UTF-8) by default, so non-Latin1 characters should 'just work'.

[dklann]

Huh. It's utterly consistent here. The Cart title with the © symbol causes the crash (per above).

I get that Python3 is Unicode-aware, I'll dig a little deeper to be sure it's actually the Unicode that's causing the issue (maybe there's some other control character in that title).

Thanks for your thoughts Fred!

[ElvishArtisan]

(maybe there's some other control character in that title)

I actually have seen that come up occasionally, particularly with "older" databases dating from v2.x and earlier. Those systems did brain-damaged things when users attempted to enter non-Latin1 (actually, non-Swedish) characters. The symptom is typically an exception raised by the json.loads() call in Receiver(). They can be maddingly elusive to run down.

I was a feckless kid when I did Rivendell v1.x. Character sets?! What are those?? :)

[dklann]

Ha! Right?!?! We all were back in the day! "What else could there be but ASCII and EBCDIC characters?!?!?!"

Just to note: this Rivendell installation is and has always been Rivendell 4 with MariaDB on Ubuntu 22.04 (actually Linux Mint at the moment).

[ElvishArtisan]

EBCDIC? 😱

Now there's a deep rabbit hole...