概述

分析redis在异常情况在的处理流程，redis集群在正常情况下自动主从切换，前提条件master在线多于一半的情况

https://github.com/tair-opensource/RedisShake/wiki

http://www.redis.cn/commands/cluster-failover.html

 环境：

centos7.x

redis-5.x

redis-cluster(3master/3slave)

redis-cluster操作命令

127.0.0.1:3380> cluster help

 1) CLUSTER <subcommand> arg arg ... arg. Subcommands are:

 2) ADDSLOTS <slot> [slot ...] -- Assign slots to current node.

 3) BUMPEPOCH -- Advance the cluster config epoch.

 4) COUNT-failure-reports <node-id> -- Return number of failure reports for <node-id>.

 5) COUNTKEYSINSLOT <slot> - Return the number of keys in <slot>.

 6) DELSLOTS <slot> [slot ...] -- Delete slots information from current node.

 7) FAILOVER [force|takeover] -- Promote current replica node to being a master

l  http://www.redis.cn/commands/cluster-failover.html

 8) FORGET <node-id> -- Remove a node from the cluster.

 9) GETKEYSINSLOT <slot> <count> -- Return key names stored by current node in a slot.

10) FLUSHSLOTS -- Delete current node own slots information.

11) INFO - Return onformation about the cluster.

12) KEYSLOT <key> -- Return the hash slot for <key>.

13) MEET <ip> <port> [bus-port] -- Connect nodes into a working cluster.

14) MYID -- Return the node id.

15) NODES -- Return cluster configuration seen by node. Output format:

16)     <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ... <slot>

17) REPLICATE <node-id> -- Configure current node as replica to <node-id>.

18) RESET [hard|soft] -- Reset current node (default: soft).

19) SET-config-epoch <epoch> - Set config epoch of current node.

20) SETSLOT <slot> (importing|migrating|stable|node <node-id>) -- Set slot state.

21) REPLICAS <node-id> -- Return <node-id> replicas.

22) SLOTS -- Return information about slots range mappings. Each range is made of:

23)     start, end, master and replicas IP addresses, ports and ids

127.0.0.1:3380>

场景

两个master在一台虚机上

查看节点信息 cluster nodes

redis-cli登录到2master在一台机器的slave节点，执行人工切换

cluster failover force

移除异常的redis id

cluster forget xxxx

检查redis集群状态 cluster nodes/cluster info

少于一半master在线

3主3从redis

数据目录存在(redis宿主机不存在文件丢失)

进入节点的服务器，启动redis服务，稍后观察集群是否恢复正常

cluster nodes

cluster info

数据目录丢失

01 人工提升当前redis集群中slave节点到master让集群恢复正常

在线redis的主从关系

人工提升slave->master集群恢复正常

备注：此时redis集群恢复正常，可以继续定位业务故障(抽出另外一个同事处理挂掉redis节点加入集群事宜)

02 重新初始化redis安装，然后以slave节点形式加入到集群

01 脚本初始化集群或者copy当前集群内的redis脚本、配置(修正正确)，启动

脚本初始化redis(注释掉集群初始化slot)

copy其他节点修正ip/port

使用脚本启动redis

02 redis节点加入到集群中

新增slave，根据redis信息执行加入集群

./bin/redis-cli -h 172.24.20.31 -p 6379 -a xx

cluster nodes  //查看节点映射信息

./bin/redis-cli -h 172.24.20.31 -p 6379 -a xxx  --cluster add-node 172.24.20.30:3001 172.24.20.31:6379 --cluster-slave

注释：

172.24.20.30:3001   要加入集群的redis信息

172.24.20.31:6379   当前集群在线的节点任意一个

--cluster-slave  以slave身份加入到集群中自动寻找master作为依附

172.24.20.202:3380> cluster forget be91cd62eec29df6a95da23f59d01bb92bbd4656

OK

注释：

be91cd62eec29df6a95da23f59d01bb92bbd4656  redis节点id标识(master fail)

172.24.20.202:3380> cluster forget 6901e19a1924395bc9c3190992e1f25bbfc51577

OK

在查看集群恢复正常

cluster  info