Support address mapping for nodes

[kevinbosak]

Description

Currently when connecting to a cluster, the node addresses are given initially. But when a call to Refresh is made, the nodes may report a different IP than those initially used. This can happen, for example, when running the cluster inside of docker and trying to connect from outside of docker. To account for this, a node address mapping could be given, so that redisc will map the addresses reported by the nodes to different addresses. This is currently done in a node redis module: node-redis.

Use case

When developing locally, it can be useful to have a cluster running using docker, but running tests from the host machine. This is currently an issue I am experiencing due to how our integration tests are run.

[kevinbosak]

I hope to make an attempt at this and submit a PR as time permits.

[mna]

Hello Kevin,

Thanks for raising this. Have you tried to use the Redis configuration options for use with Docker? If you take a look at the config file (e.g. this one for redis 6.2: https://raw.githubusercontent.com/redis/redis/6.2/redis.conf) and search for "Docker", you'll find some config options for this use-case.

Ideally that would be something that is solved at this level, as the issue is that redis itself returns addresses that are invalid/inaccessible. For Redis 7, they seem to have added another option to report a different endpoint (in the new CLUSTER SHARDS command, look in https://redis.io/commands/cluster-shards/ for "cluster-preferred-endpoint-type") but redisc does not use that yet.

Hope this helps, let me know how that goes when you get a chance to try it out!
Martin

[kevinbosak]

Oh, great info, thanks! I'll review and close this ticket if it fits my needs.

[kevinbosak]

I've fiddled with this a bit, without any success. It seems the cluster-announce-ip is used to announce the IP to both the client and the other cluster nodes, which doesn't work for my scenario. I have temporarily figured out a workaround by using redis 7 and specifying hostnames for cluster-announce-ip, but that requires updating the hosts file on my machine. This is not an ideal solution.

[mna]

Gotcha, thanks for trying this out and reporting back. Reading more about the docker-related cluster options, it seems it supports setting an IP to announce (which is used for both the clients and the cluster message bus), and distinct ports for clients and message bus. It's unfortunate that it doesn't support setting a distinct IP for clients and message bus, as I think it would've handled your use-case, right?

When you say you tried redis 7 and specifying hostnames for cluster-announce-ip, did you also try using the new (in 7) cluster-announce-hostname configuration? Paired with the new cluster-preferred-endpoint-type config option, I wonder if there's a way to make that work.

It's not all super clear, I think it'd be great to have the redis' folks input on that before committing to any solution to see what is the recommended way to handle that use-case, if you don't mind reaching out on the official repo's issue tracker.

Actually, I found the following issue which seems very similar to what we're trying to achive: redis/redis#7460 and the final comment there seems to suggest that using cluster-announce-hostname you could set the IP in there and set cluster-preferred-endpoint-type hostname to announce the address to clients. Of course, that wouldn't work with redisc as it doesn't use the new CLUSTER SHARDS command and the preferred endpoint address yet, but you can use redis-cli to see how it reports things and if the cluster works properly (that is, if internally it uses the original IP addresses for the message bus).

If that works, than the proper fix would be for redisc to start supporting CLUSTER SHARDS (there's already a ticket for that) and that use-case would be supported for redis 7+.

[kevinbosak]

Thanks again for the response. I added a comment to the redis issue you found, hopefully we can get some traction there. I pointed out that even using hostnames is not ideal as I don't know of a hostname on the host machine that could point to itself except for "localhost", which I doubt can (or should) be overridden in the docker containers. This means another hostname has to be used, and developers using my docker-compose file will have to edit their hosts file, which is what I'm trying to avoid. I do currently have my local setup working by editing my hosts file (one entry per node).

[mna]

Awesome, thanks for reaching out over there, looking forward to any feedback that could help figure this out.