Requests get dropped on high load

[sebschlicht]

Requests get dropped by backend server when pushing too many to the dispatcher and onTimeout gets called.
As discussed this might relate to ZMQ water mark on server side.

[HeinrichHartmann]

Yes, the problem is that Zmq Req-Rep patterns drop messages if the queue at the receiver side is full. This seems to be happening here, since the server cannot keep up with the load.

In the current implementation the only way for the client to know this has happened,
is to call gatherResuts with a timeout, and implement onTimeout callback method.
When used in production, this problem should not be so pressing, since gatherResults is
indeed called with timeout, and no batching of results is necessary.

Approaches:

1. Call gather results more often (e.g. every 1000 requests) this makes the application
   wait for results, before the queues are overflowing.
2. Set HWM to 0 or very high. (Risk: memory overflow and a crash of the server)
3. Change the messaging pattern to send an accepted message when the request is being
   processed by the server. Make dp.execute() block until the accepted message is returned.
   -> Slows down system.

In the mid term the messaging pattern should be changed to something like Majordomo which gives service guarantees.