General questions about capabilities

[ywang-clarify]

Hi @krystianity,

There aren't many rpc-over-redis modules for out there. node-pohl seems promising to me and in your README, you mention a lot of important considerations.

But after playing with it for a day, I have some questions. I have a need for an RPC server that can handle requests of the same task type coming from different RPC clients. The clients should only receive responses to their own requests. With that in mind, any clarifications you can provide on the following questions would be greatly appreciated.

Question 1:

I can't seem to figure out how to send multiple requests using the same RPC client without getting stuck.
RPC Client code:

        rpcClient.sendTask({
            operation: 'test1'
        }, (err, result) => {
            logger.info('test1 result: %s', result.result);
        });
        rpcClient.sendTask({
            operation: 'test2'
        }, (err, result) => {
            logger.info('test2 result: %s', result.result);
        });

RPC Server code:

const receiveTask = function(err, task, callback) {
    if (err) {
        return logger.error(err);
    }

    let taskHandler;

    let operation = _.get(task, 'operation');
    switch (operation) {
        case 'test1':
            taskHandler = function() {
                logger.info('Performing Test1');
                return 'Test1 Complete';
            };
            break;
        case 'test2':
            taskHandler = function() {
                logger.info('Performing Test2');
                return 'Test2 Complete';
            };
            break;
        default:
            return console.warn('Received request for unknown RPC operation %s', operation);
    }
    console.log('got task: ' + JSON.stringify(task));

    Promise.resolve(taskHandler())
        .then(result => {
            task.result = result;
            callback(null, task); //** when you call this.. *magic*
        });
};

rpc.setupTaskReceiver(receiveTask, () => {
    logger.info('Starting to listen for RPC tasks.');
});

Combined output:

debug: Subscribed to redis channel: out:pohl
info: Starting to listen for RPC tasks.
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: subscription ready for inc:pohl
debug: node-pohl: auto-resubscribe is active.
debug: subscription ready for inc:pohl
debug: node-pohl: auto-resubscribe is active.
info: auto re-sub failing, will retry soon. reason: Error: Connection is closed..
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: Subscribed to redis channel: inc:pohl
debug: sending task with id: 50c72726-4978-430c-b692-b50db95d5535 on topic: pohl
debug: publishing ready for out:pohl
debug: publishing redis message on channel:out:pohl
debug: received message on channel: out:pohl
debug: received message from sender {"i":"50c72726-4978-430c-b692-b50db95d5535","p":"{"operation":"test2"}","o":true}
debug: received task and got lock for id: 50c72726-4978-430c-b692-b50db95d5535 on topic: pohl
got task: {"operation":"test2"}
info: Performing Test2
debug: publishing ready for inc:pohl
debug: publishing redis message on channel:inc:pohl
debug: received message on channel: inc:pohl
debug: receiver returned a task: {"i":"50c72726-4978-430c-b692-b50db95d5535","p":"{"operation":"test2","result":"Test2 Complete"}","o":true}
debug: 50c72726-4978-430c-b692-b50db95d5535 has a task send from this sender.
info: test2 result: Test2 Complete
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for inc:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for inc:pohl
info: auto re-sub failing, will retry soon. reason: Error: Connection is closed..
debug: Subscribed to redis channel: inc:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl
debug: node-pohl: auto-resubscribe running.
debug: subscription ready for out:pohl
debug: Subscribed to redis channel: out:pohl

It looks like the request for test1 is never sent because something keeps trying to re-sub while a connection is being reported as closed? I can get the same results by taking one of the included usage-tests and adding a second .sendTask() call.

Question 2:

If I create a new RPC client for every request (seems wasteful), I can run concurrent requests. But if I kick off a lot of concurrent requests, the responses seem to get mixed up and the 'result' passed to the RPC client's callback is sometimes undefined for some reason. Have you ever seen that or have an explanation for why that might happen?

Question 3:

It looks like the clients will actually receive messages for all requests and it's just choosing to ignore the ones that it didn't originate (or timed out). Can you confirm that? Basically, the responses for all client are shared with all other clients? That seems a little dangerous. I think I would have to create a dedicated RPC server instance for each client if I want the responses to be private to that client? That seems difficult since I don't want to write server code to be aware of its client base. I was thinking that maybe it'd be possible to add a replyTo to the task protocol and have the RPC clients listen on a private channel for its response.

[krystianity]

Hi @ywang-clarify, I would love to get you up & running on pohl.

Question 1 -> I am looking into this and will give you an update later. Having auto-redis-res-subscription active is not required, as ioredis is able to handle re-connects itself (it is built in though for firewall and other networking reasons) you can deactive it by passing "autoReconnectInterval" with value: null in the config.

Question 2 -> You should never have to create more than one RPC client per service/application instance (per pohl topic) and if you do create more than one than that is most certainly the reason why you see "mixed up results".

Question 3 -> Yes, currently 2 redis channels are opened per topic inc:topic and out:topic and every instance sender or receiver subscribes to the same topic, the idea was to achieve scalability quickly, by spreading messages to all receivers and making them decide via redlock which one is allowed to work on the task. If you wouldnt want other receivers to listen, you would have to create another rpc client sender and receiver using another topic. Pohl is not built for secure rpc exchange, its goal is to replace http-communication in a microservice world. By sharing high workloads quickly between microservices using redis messages.

[ywang-clarify]

Hi @krystianity,

Thanks for the rapid response.

I tried setting autoReconnectInterval to null. That avoids the repeated attempts to re-sub, but there's still one re-sub failure reported. And test1 still fails to send.

debug: Subscribed to redis channel: out:pohl
info: Starting to listen for RPC tasks.
debug: subscription ready for inc:pohl
debug: node-pohl: auto-resubscribe is not active.
debug: subscription ready for inc:pohl
debug: node-pohl: auto-resubscribe is not active.
info: auto re-sub failing, will retry soon. reason: Error: Connection is closed..
debug: Subscribed to redis channel: inc:pohl
debug: sending task with id: dd1345df-8d1a-4928-ad85-b903bf6dc884 on topic: pohl
debug: publishing ready for out:pohl
debug: publishing redis message on channel:out:pohl
debug: received message on channel: out:pohl
debug: received message from sender {"i":"dd1345df-8d1a-4928-ad85-b903bf6dc884","p":"{"operation":"test2"}","o":true}
debug: received task and got lock for id: dd1345df-8d1a-4928-ad85-b903bf6dc884 on topic: pohl
got task: {"operation":"test2"}
info: Performing Test2
debug: publishing ready for inc:pohl
debug: publishing redis message on channel:inc:pohl
debug: received message on channel: inc:pohl
debug: receiver returned a task: {"i":"dd1345df-8d1a-4928-ad85-b903bf6dc884","p":"{"operation":"test2","result":"Test2 Complete"}","o":true}
debug: dd1345df-8d1a-4928-ad85-b903bf6dc884 has a task send from this sender.
info: test2 result: Test2 Complete

For question 2, that's exactly the answer I was hoping to hear. So hopefully, after resolving the above issue, everything will be great.

For question 3, I think you misunderstood my concern. I like your design for multiple workers which exclusively pick up and work on requests. What concerned me was Pohl.js:182:

    if(!this.stack[message.i]){
        Pohl.DEBUG(message.i + " is not a task send from this sender, or it has timed out already.");
        return;
    } else {
        Pohl.DEBUG(message.i + " has a task send from this sender.");
    }

I'm working on a healthcare application where there are single-tenant services that want to make RPC calls to a multi-tenant service. So if each single-tenant service brings up a RPC client, there's nothing in the pohl configuration that would differentiate them. They would all end up subscribing to the same redis channels and receive each others responses. The quoted code above would ignore the messages that weren't generated by any particular RPC client, but the response data would still flow to that RPC client. The fact that response data meant for a RPC client at one hospital would flow to an RPC client at another hopsital is a problem. I think with the way things are currently working, you'd have to use a different topic for each hospital to avoid this. But it's difficult to work with a server that has to be aware of the list of hospital clients so it could start dedicated listeners for each of them.

It seems better to build dedicated response channels into the protocol. So each client listens for responses on a specified channel. The channel name could even be randomly generated if not specified or defaulted to the topic the way it is right now to maintain the current behavior if not specified. Then in the sendTask(), the task gets a replyTo property that tells the server which redis channel to send the response when it's ready. This way, you can set up a pool of one or more RPC clients listening on the same response channel and apply the same redlock approach to make sure only one of those clients processes the response through the callback. This way, responses for one client pool would never get sent to any other client pool.

[ywang-clarify]

Hi @krystianity, were you able to reproduce the issue in Question 1 where sending two concurrent tasks from the same rpc client fails to run both as expected?