Apparent resource leakage when abruptly unsubscribing from all subs

[gtravesedo]

Summary

The number of active Redis channels as reported by the metric redis_pubsubshard_channels doesn't get back to 0 when a wave of Unsubscribe calls happens on all existing Subs. However the metric submux_subscriptions_redis shows 0 subscriptions. Restarting all service instances, and therefore closing and recreating Submux instances, brings the number of Redis channels down to 0. This seems to indicate that some resource leakage is happening in the library.

Reproducing the Issue

I was able to reproduce the issue by terminating all active calls against our service, which in turn causes a wave of Unsubscribe calls on every existing Sub instance.

The Submux version is v0.5.0.

The test that reproduced this issue had this characteristics:

• Multiple Submux instances (one per service pod)
• Around 110 SSubscribeSync per second (some are long lived subs and others last less than 100ms before calling Unsubscribe)
• Sudden termination of all service clients calls, triggering a sudden wave of Unsubscribe calls on all existing Sub instances

The following image shows active Redis channels, the sudden drop when the wave of Unsubscribe calls happens and the fact that some active channels remain in the cluster.

The following image shows the number of subscriptions as reported by submux_subscriptions_redis during the same time window.