Redis Cluster in K3S

学习资料

https://www.cnblogs.com/cheyunhua/p/15619317.html
https://blog.csdn.net/cqnaqjy/article/details/126001999
https://segmentfault.com/a/1190000039196137

背景说明

周末要求了下Redis Cluster in Docker 和 in file
搭建了六主六从的Redis集群(在两个服务器上面)
并且简单验证了机器的性能. 
发现本地文件比docker稍微好一些
但是因为docker 使用的 --net=host的模式搭建的
模拟不出来网络转发和分层的损耗,所以今天研究了下redis Cluster in K8S
自己K8S能力只是在于原理, 实操比较弱. 所以耗时比较久
这里简单记录一下整个过程, 备忘

需要说明的是, 仅是搭建出来还没有应用去访问和连接. 
我也不清楚集群重启之后pod地址发生变化应该如何处理. 

第一步 共享存储的搭建 NFS

yum install nfs* -y
yum install rpcbind* -y
systemctl enable nfs && systemctl enable rpcbind
systemctl restart nfs && systemctl restart rpcbind 

# 创建目录, 注意不要选择根目录下一级目录.貌似会有问题. 
mkdir -p /usr/local/k8s/redis/pv
# 编辑配置文件
vim /etc/exports
/usr/local/k8s/redis/pv *(rw,sync,no_root_squash)
# 注意测试环境这么写.生产可能需要注意安全.
# 注意保存后需要重启一下 nfs
systemctl restart nfs
exportfs 
# 就可以查看.

K8S创建sc等

• 可以保存文件为 redis-sc.yaml
• 部署的方式为 kubectl apply -f redis-sc.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: redis
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: redis-nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: redis/nfs
reclaimPolicy: Retain
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: gmoney23/nfs-client-provisioner
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: redis/nfs
            - name: NFS_SERVER
              value: 10.110.139.191 ## 指定自己nfs服务器地址
            - name: NFS_PATH  
              value: /usr/local/k8s/redis/pv2  ## nfs服务器共享的目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.110.139.191
            path: /usr/local/k8s/redis/pv2
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

创建redis配置文件

• 注意命令为 需要将文件保存为 redis.conf 的文件:
• kubectl create configmap redis-conf --from-file=redis.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
  namespace: default
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 10000
    protected-mode no
    daemonize no
    pidfile /var/run/redis.pid
    port 6379
    tcp-backlog 511
    bind 0.0.0.0
    timeout 3600
    tcp-keepalive 1
    loglevel verbose
    logfile /data/redis.log
    databases 16
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    requirepass Test20131127
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    lua-time-limit 20000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    #rename-command FLUSHALL  ""
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-entries 512
    list-max-ziplist-value 64
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes

创建Redis Cluster的集群

• 注意执行命令就可以, 可以保存为 redis-cluster.yaml
• kubectl apply -f redis-cluster.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: default
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 12
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: harbor.gscloud.online/gscloud/redis:7.0.5
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "redis-nfs-storage"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

Redis 集群创建

• 第一步 拉出来K8S内的pod信息

kubectl get pods -l app=redis-cluster  -o jsonpath='{range.items[*]}{.status.podIP}:6379 '
# 注意这个里面 多最后一个 :6379 需要清理掉

# 然后需要进入某一个容器
kubectl exec -it redis-cluster-01 bash
执行命令为:
redis-cli -h 127.0.0.1 -p 6379 --cluster create \
10.42.236.144:6379 10.42.73.71:6379 10.42.236.149:6379 \
10.42.73.76:6379 10.42.236.145:6379 10.42.73.72:6379 \
10.42.236.146:6379 10.42.73.73:6379 10.42.236.147:6379  \
10.42.73.74:6379 10.42.236.148:6379 10.42.73.75:6379 \
--cluster-replicas 1

Redis 集群检查

kubectl exec -it redis-cluster-1 bash
#执行命令
#注意一开始我没设置密码
redis-cli cluster nodes
redis-cli cluster info

注意 Cluster集群的测试命令为:
redis-benchmark -h 127.0.0.1 --cluster >K8S1.txt
redis-benchmark -h 127.0.0.1 -c 100 --cluster >K8S2.txt
redis-benchmark -h 127.0.0.1 -c 100 -d 1024 --cluster >K8S3.txt
获取部分有效信息的语句为:
cat file1.txt |grep -E "requests per second|====="

Redis集群测试结果