Skip to content

Simplify Protobuf format #7

@grodtron

Description

@grodtron

Currently protobuf is wrapping all messages in the following global type, and holding the actual message as a serialized string, forcing a second manual serialization and deserializtion of the message itself.

message DebugMsg {

  enum Type {
    Run = 1;
    Next = 2;
    ...
  }

  required Type type = 1;
  required bytes message = 2; // protobuf serialized message
}

This (I believe) was done as a work around for the fact that the nanomsg python library attempts to decode data to be sent as ascii, and crashes on non-ascii characters (e.g. '\xff'). Some testing has shown that if we instead pass the serialized protobuf message as a list of individual characters, it will not attempt to do an ascii decode and will send the raw data correctly:

# In terminal 1
>>> import nnpy
>>> sock = nnpy.Socket(nnpy.AF_SP, nnpy.REQ)
>>> sock.connect("ipc:///tmp/nn_testing")
>>> s = '\xff\xaa\x00 This is a \xBB\xCC test'
>>> sock.send(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/nnpy/socket.py", line 53, in send
    data = data.encode() if isinstance(data, str) else data
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
>>> sock.send([c for c in s])
>>> 
# In second terminal
>>> import nnpy
>>> sock = nnpy.Socket(nnpy.AF_SP, nnpy.REP)
>>> sock.bind("ipc:///tmp/nn_testing")
>>> sock.recv()
'\xff\xaa\x00 This is a \xbb\xcc test'
>>> 

This workaround I believe was also forcing other odd limitations such as using strings to represent boolean or integer fields.

A good amount of the debugger code could be simplified based on this, and would definitely be an improvement in the overall quality of the project.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions