报错信息
2023-05-18 14:18:01.916 ERROR 109202 --- [pool-6-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task.
org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:74) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:268) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.connection.lettuce.LettuceSetCommands.convertLettuceAccessException(LettuceSetCommands.java:520) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.connection.lettuce.LettuceSetCommands.sMembers(LettuceSetCommands.java:245) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.connection.DefaultedRedisConnection.sMembers(DefaultedRedisConnection.java:607) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.DefaultSetOperations.lambda$members$6(DefaultSetOperations.java:158) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.DefaultSetOperations.members(DefaultSetOperations.java:158) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.data.redis.core.DefaultBoundSetOperations.members(DefaultBoundSetOperations.java:152) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
at org.springframework.session.data.redis.RedisSessionExpirationPolicy.cleanExpiredSessions(RedisSessionExpirationPolicy.java:132) ~[spring-session-data-redis-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at org.springframework.session.data.redis.RedisOperationsSessionRepository.cleanupExpiredSessions(RedisOperationsSessionRepository.java:430) ~[spring-session-data-redis-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration.lambda$configureTasks$0(RedisHttpSessionConfiguration.java:248) ~[spring-session-data-redis-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: io.lettuce.core.RedisException: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80) ~[lettuce-core-5.1.8.RELEASE.jar!/:na]
at com.sun.proxy.$Proxy269.smembers(Unknown Source) ~[na:na]
at org.springframework.data.redis.connection.lettuce.LettuceSetCommands.sMembers(LettuceSetCommands.java:243) ~[spring-data-redis-2.1.10.RELEASE.jar!/:2.1.10.RELEASE]
... 19 common frames omitted
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
2023-05-18 14:18:05.678 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:05.678 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:06.579 WARN 109202 --- [2.168.1.35:5672] c.r.c.impl.ForgivingExceptionHandler : An unexpected connection driver error occured (Exception message: Connection reset)
2023-05-18 14:18:06.585 ERROR 109202 --- [2.168.1.35:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: connection error
2023-05-18 14:18:08.139 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:08.140 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:08.272 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:08.272 INFO 109202 --- [lientSelector_1] RocketmqRemoting : closeChannel: close the connection to remote address[192.168.1.41:8876] result: true
2023-05-18 14:18:11.589 ERROR 109202 --- [2.168.1.35:5672] o.s.a.r.c.CachingConnectionFactory : Failed to close auto-recover connection
com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.net.SocketException: Connection reset
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:921) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:1036) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:965) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:949) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.close(AutorecoveringConnection.java:284) [amqp-client-5.4.3.jar!/:5.4.3]
at org.springframework.amqp.rabbit.connection.SimpleConnection.close(SimpleConnection.java:77) ~[spring-rabbit-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory$2.handleRecovery(AbstractConnectionFactory.java:507) [spring-rabbit-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory$2.handleRecoveryStarted(AbstractConnectionFactory.java:501) [spring-rabbit-2.1.8.RELEASE.jar!/:2.1.8.RELEASE]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.notifyRecoveryListenersStarted(AutorecoveringConnection.java:646) [amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:564) [amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.lambda$addAutomaticRecoveryListener$3(AutorecoveringConnection.java:508) [amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.notifyRecoveryCanBeginListeners(AMQConnection.java:748) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.doFinalShutdown(AMQConnection.java:725) ~[amqp-client-5.4.3.jar!/:5.4.3]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:609) ~[amqp-client-5.4.3.jar!/:5.4.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
2023-05-18 17:28:17.586 INFO 109202 --- [nio-8003-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-05-18 17:28:17.586 INFO 109202 --- [nio-8003-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2023-05-18 17:28:17.607 INFO 109202 --- [nio-8003-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 21 ms
2023-05-18 18:15:52.535 INFO 109202 --- [io-8003-exec-25] s.a.m.a.c.distr.DistributorController : distributorBindLog,用户=487437;请求ucode=null,openId=null,sid=66,referrer=null
Connection reset by peer 含义
定义1： 当前服务器接受到了通信对端发送的TCP RST信号，即通信对端已经关闭了连接，通过RST信号希望接收方关闭连接。
定义2：如果一端的Socket被关闭（或主动关闭，或因为异常退出而引起的关闭），另一端仍发送数据，发送的第一个数据包引发该异常
定义3：“Connection reset by peer” is the TCP/IP equivalent of slamming the phone back on the hook. It’s more polite than merely not replying, leaving one hanging. But it’s not the FIN-ACK expected of the truly polite TCP/IP converseur.
此次乌龙事件
原因：esxi虚拟机关机时点击的是挂起,那么开机恢复是内存快照包括TCP状态，但TCP状态对端已经关闭了连接，所以收到RST信号
ha-datacenter 中 esxi44 上的 web44 已打开电源 2023 年 5 月 18 日 星期四 14:17:40 +0800 信息
ha-datacenter 中 esxi44 上的 web44 已恢复 2023 年 5 月 18 日 星期四 14:17:39 +0800 信息
顺便记录下处理过程
查看Jar包超时参数
cat application.yml | grep timeout
timeout: 60000
查看Redis配置
# 查看当前连接数
127.0.0.1:6379> info clients
# Clients
connected_clients:3
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# 查看超时
127.0.0.1:6379> config get time*
1) "timeout"
2) "0"
# 查看最大存活时间
127.0.0.1:6379> config get tcp*
1) "tcp-keepalive"
2) "300"
3) "tcp-backlog"
4) "511"
# 查看最大客户端数
127.0.0.1:6379> config get maxc*
1) "maxclients"
2) "10000"
修改Redis参数
在线生效，注持久有效需修改配置文件
# 最大连接数 3W
config set maxclients 30000
# 超时时间 300 秒
config set timeout 300
# 最大存活时间 60 秒
config set tcp-keepalive 60
参考链接
redis 之 Connection reset by peer
lettuce-core readAddress(..) failed: Connection reset by peer
What does “connection reset by peer” mean?