Upstream closed connection - Not reconnecting

[LorenzoRogai]

Hi,

Using the latest version (but also on the previous - we tried up to v2.0.0-rc.8) we're encountering this problem when a node is put into maintenance mode or it's being shutting down - we were also able to replicate it in local (in production we are using AmazonMQ):

2025-05-29T12:54:53.091368Z INFO amq_proxy.channel_pool[client: "192.168.65.1:22398"] Adding upstream connection
2025-05-29T12:54:58.950755Z ERROR amq_proxy.upstream[upstream: "192.168.65.254:5672"] Upstream closed connection: CONNECTION_FORCED - Node was put into maintenance mode 320
2025-05-29T12:54:58.950798Z DEBUG amq_proxy.upstream[upstream: "192.168.65.254:5672"] Upstream connection closed, closing 1 client channels
2025-05-29T12:55:01.138691Z DEBUG amq_proxy.client[client: "192.168.65.1:22398"] Sending heartbeat (last heartbeat 2s ago) <-- heartbeats are still sent
2025-05-29T12:55:03.154244Z DEBUG amq_proxy.client[client: "192.168.65.1:22398"] Sending heartbeat (last heartbeat 2s ago)
....
After some minutes:
2025-05-29T12:58:00.476802Z DEBUG amq_proxy.client[client: "192.168.65.1:22398"] Disconnected #<IO::EOFError:End of file reached> (application process killed with CTRL-C - we expect this to happen when the upstream connection is closed.. clients should be disconnected fully too)

We're using php-amqplib and it seems that the write method never enters the catch block. This doesn't happen with a direct RabbitMQ connection (without using AMQProxy) - with a direct RabbitMQ connection it correctly catches the "Channel connection is closed" error

getLastActivity timestamp is still increasing meaning that AMQProxy is still responding correctly to the client

It seems that the application still sees AMQProxy connection as online and continues to send messages successfully. Additionally when the upstream server is back online AMQProxy does not automatically re-establish a new upstream connection without restarting the whole app from scratch. I expect to see "Adding upstream connection" again

Thank you

[carlhoerberg]

Ok, yes, AMQProxy currently only closes the channels to the upstream server that was disconnected, not the whole client connection. The rationale being that if amqproxy connect to mutiple upstreams, clients can have their channels spread out of several of them, and if only one upstream connection is closed there's no need to close the channels connected to other working upstreams. What the client can do is to get notified on channel close events and then try to reopen a channel.