一、问题描述
Redis日志:
2110:M 24 Apr 02:01:02.058 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
2110:M 24 Apr 02:01:12.862 # Connection with slave client id #58526360 lost.
2110:M 24 Apr 02:01:12.863 # Connection with slave client id #58525913 lost.
sentinal日志:
788:X 24 Apr 02:01:07.970 # +sdown master mymaster 172.16.158.87 6379
1788:X 24 Apr 02:01:08.036 # +odown master mymaster 172.16.158.87 6379 #quorum 2/2
1788:X 24 Apr 02:01:08.037 # +new-epoch 79
1788:X 24 Apr 02:01:08.037 # +try-failover master mymaster 172.16.158.87 6379
1788:X 24 Apr 02:01:12.864 # +vote-for-leader fd5c6228b4f209cfc2c88265a0d47f39879027ab 79
1788:X 24 Apr 02:01:12.865 # +config-update-from sentinel 913c8b2f64f54caf46ff25ac6fa9bdcae86d9d02 172.16.158.85 26379 @ mymaster 172.16.158.87 6379
1788:X 24 Apr 02:01:12.865 # +switch-master mymaster 172.16.158.87 6379 172.16.158.85 6379
1788:X 24 Apr 02:01:12.865 * +slave slave 172.16.158.86:6379 172.16.158.86 6379 @ mymaster 172.16.158.85 6379
1788:X 24 Apr 02:01:12.865 * +slave slave 172.16.158.87:6379 172.16.158.87 6379 @ mymaster 172.16.158.85 6379
1788:X 24 Apr 02:01:12.867 # 913c8b2f64f54caf46ff25ac6fa9bdcae86d9d02 voted for 913c8b2f64f54caf46ff25ac6fa9bdcae86d9d02 79
监控截图
二、问题原因分析
三、解决操作(未验证)
1.修改sysctl vm.dirty_bytes内核参数
echo "vm.dirty_bytes=33554432" >> /etc/sysctl.conf
sysctl -p
2.修改Redis配置参数 repl-backlog-size
repl-backlog-size=200mb
"一劳永逸" 的话,有是有的,而 "一劳永逸" 的事却极少