RATIS-2349. NettyClient#writeAndFlush should support throwing AlreadyClosedException

[leixm]

What changes were proposed in this pull request?

After RATIS-2329, NettyRpcProxy will close connection in exceptionCaught, NettyClient#writeAndFlush should support throwing AlreadyClosedException, otherwise, the following problems will occur.

25/10/29 11:13:52,786 WARN [1@group-47BEDE733167->2-LogAppenderDefault-LogAppenderDaemon] LogAppender: 1@group-47BEDE733167->2-LogAppenderDefault: Failed to installSnapshot SingleFileSnapshotInfo(t:1, i:197600537):[/mnt/ssd/0/celeborn_ratis/c5196f6d-2c34-3ed3-8b8a-47bede733167/sm/snapshot.1_197600537]: java.lang.IllegalStateException: STATE MISMATCHED: In NettyClient, current state CLOSED is not one of the expected states [RUNNING]

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2349

How was this patch tested?

Existing UTs.

[szetszwo]

@leixm , thanks for working no this! Please file another JIRA for the follow up. It would be easier to track the issues.

[OneSizeFitsQuorum]

Hi, Please take a look for the failed UT, could you reproduce it in your local environment?

Error:  org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc.testReadAfterWrite -- Time elapsed: 1.677 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.apache.ratis.ReadOnlyRequestTests.testReadAfterWriteImpl(ReadOnlyRequestTests.java:314)
	at org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:143)
	at org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:121)
	at org.apache.ratis.ReadOnlyRequestTests.testReadAfterWrite(ReadOnlyRequestTests.java:289)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)

[leixm]

Hi, Please take a look for the failed UT, could you reproduce it in your local environment?

Error:  org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc.testReadAfterWrite -- Time elapsed: 1.677 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.apache.ratis.ReadOnlyRequestTests.testReadAfterWriteImpl(ReadOnlyRequestTests.java:314)
	at org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:143)
	at org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:121)
	at org.apache.ratis.ReadOnlyRequestTests.testReadAfterWrite(ReadOnlyRequestTests.java:289)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)

I tried to reproduce this unit test failure locally, this looks wired.

• Running testReadAfterWrite alone resulted in a stable pass.
• Running TestReadOnlyRequestWithGrpc mostly succeeded, with a small chance of failing, but only testFollowerLinearizableReadParallel and testFollowerLeaseReadParallel failed; testReadAfterWrite did not fail.

[szetszwo]

@leixm , @OneSizeFitsQuorum , since this PR only change ratis -netty but not ratis -grpc, the test failure of TestReadOnlyRequestWithGrpc must be unrelated.

Just have submitted a 10x10 run on master: https://github.com/apache/ratis/actions/runs/19023405871

[szetszwo]

+1 the change looks good.

[szetszwo]

Just have submitted a 10x10 run on master: https://github.com/apache/ratis/actions/runs/19023405871

It does fail intermittently; filed RATIS-2350

[szetszwo]

@leixm , thanks for working on this!

@OneSizeFitsQuorum , thanks for reviewing this!