Local port forwarding not propagating RST packet to server, orphaning the server-side connection

[joeljohansson99]

Version

2.16.0

Bug description

When the client sends a TCP packet with RST flag, i.e. abrupt closure, the Default forwarder is handling this as a read exception and closes the session, and channel immediately without sending a SSH_MSG_CHANNEL_CLOSE to the server. Which causes the server-side connection to stay open while the client side connection is closed. An example of this, and how we found this, is Jedis, who is setting socket.setSoLinger(true, 0), so it sends RST on close, to mitigate port exhaustion. See this discussion Jedis should close tcp connection gracefully #2669

The reason for this happening is due to the DefaultForwarder catching this exception and storing the cause in DefaultForwarder.StaticIoHandler.exceptionCaught():

session.setAttribute(TcpipForwardingExceptionMarker.class, cause);

The cause it then retrieved in DefaultForwarder.StaticIoHandler.sessionClosed() and since the cause is not null, the channel is closed immediately:

Throwable cause = (Throwable) session.removeAttribute(TcpipForwardingExceptionMarker.class);
...
channel.close(cause != null);

Which is then closing the channel without sending SSH_MSG_CHANNEL_CLOSE to the server-side connection.

Thanks for help!

Actual behavior

Client side connection is closed immediately without informing the server-side connection of closure, which is staying in established state until tunnel is closed.

Expected behavior

Client-side connection should, before closing, send an SSH_MSG_CHANNEL_CLOSE to the server-side connection before closing. I guess this can be done by filtering out the connection reset exception or something similar in DefaultForwarder.StaticIoHandler.exceptionCaught(). Let me know what you think!

Relevant log output

[sshd-SshClient[f5b882b]-nio2-thread-6]  DEBUG o.a.sshd.common.io.nio2.Nio2Session - handleReadCycleFailure(Nio2Session[local=/..., remote=/...]) IOException after 73736209 nanos at read cycle=4: Connection reset
[sshd-SshClient[f5b882b]-nio2-thread-6]  DEBUG o.a.sshd.common.io.nio2.Nio2Session - exceptionCaught(Nio2Session[local=/..., remote=/...]) caught IOException[Connection reset] - calling handler
[sshd-SshClient[f5b882b]-nio2-thread-6]  WARN  o.a.s.c.forward.DefaultForwarder - exceptionCaught(Nio2Session[local=/..., remote=/...]) IOException: Connection reset
java.io.IOException: Connection reset
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:425)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:195)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:217)
	at java.base/sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:312)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Other information

This problem only occurs with Apache Mina SSHD. It works fine with OpenSSH (Connections are closed on both the client and the server).

[tomaswolf]

That cause != null was introduced in 2018 in commit 169ff4e, which references SSHD-792, which on the face of it doesn't have anything to do with channel closing.

Code archeology didn't help this time to understand why that change was made :-( .

Seems to me it should just and simply be false. (I.e., always try to close the channel gracefully.)

[joeljohansson99]

Thanks for the response! I agree, it makes sense that it should always try to close the channel gracefully, it will always fall back to closing it immediately if something goes wrong right?

Would you like me to submit a PR, or will you take care of it?

[joeljohansson99]

Hi @tomaswolf, any updates on this?