Summary
Using BalancedAll node preference and terminating a Redis master node causes the number of channels in that node to constantly grow when it comes back. I saw this behavior when the master node comes back as a replica. All other nodes maintain a consistent number of channels.
Reproducing the Issue
I was able to reproduce the issue by manually terminating a Redis master node in my cluster.
The Submux version is v0.5.0.
The Submux instance configuration is the following:
- BalancedAll node preference
- No auto-resubscribe (the client handles the signal)
The test that reproduced this issue had this characteristics:
- Multiple Submux instances (one per service pod)
- Around 110 SSubscribeSync per second (some are long lived subs and others last less than 100ms before calling Unsubscribe)
The following graphs shows Redis channels for the different cluster nodes:
I killed two Redis masters during the test, seeing the same behavior in both cases. The first one at 12:30 affecting the node represented with a blue line. The second time at 14:45 affecting the node represented with a green line.
Summary
Using BalancedAll node preference and terminating a Redis master node causes the number of channels in that node to constantly grow when it comes back. I saw this behavior when the master node comes back as a replica. All other nodes maintain a consistent number of channels.
Reproducing the Issue
I was able to reproduce the issue by manually terminating a Redis master node in my cluster.
The Submux version is
v0.5.0.The Submux instance configuration is the following:
The test that reproduced this issue had this characteristics:
The following graphs shows Redis channels for the different cluster nodes:
I killed two Redis masters during the test, seeing the same behavior in both cases. The first one at 12:30 affecting the node represented with a blue line. The second time at 14:45 affecting the node represented with a green line.