使用Velero备份、恢复、迁移Kubernetes集群

使用Velero备份、恢复、迁移Kubernetes集群

Velero
Velero属于VMWare开源的Kubernetes集群备份、恢复、迁移工具.
可以提供Kubernetes 备份功能更,在Kubernetes集群出现问题之后,能够快速的恢复.
并且也提供了集群迁移功能,可以将Kubernetes资源迁移到其他集群.
Velero 将备份的信息在对象存储中,默认情况下可以使用 AWS、Azure、GCP 的对象存储,
本案例中使用mino自建存储

#使用场景
灾备场景:提供备份恢复k8s集群的能力
迁移场景:提供拷贝集群资源到其他集群的能力(复制同步开发、测试、生产环境的集群)

Velero 地址：https://github.com/vmware-tanzu/velero

1.安装 minio
mkdir -p /minio/data

#下载镜像
docker pull minio/minio

#启动镜像
docker run -p 9000:9000 \
--name minio-server \
-v /minio/data:/data \
minio/minio server /data

#默认用户名与密码
minioadmin/minioadmin

#访问web
http://192.168.91.11:9000

#创建bucket
velero

2.部署Velero
服务端：目标Kubernetes集群
客户端: 本地的命令行的工具,需要配合kubernetes认证使用.

#项目地址
https://github.com/vmware-tanzu/velero

#解压
[root@master-1 opt]# tar xvf velero-v1.3.2-linux-amd64.tar.gz
[root@master-1 opt]# mv velero-v1.3.2-linux-amd64/velero /usr/bin/
[root@master-1 opt]# chmod +x /usr/bin/velero

#创建连接s3
[root@master-1 opt]# cat /opt/credentials-velero
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin

#创建授权文件
cd /root/kubernetes
vi user-csr.json
{
"CN": "awsuser",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}

#生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes user-csr.json | cfssljson -bare awsuser

#复制证书
[root@master-1 kubernetes]# cp awsuser-key.pem /etc/kubernetes/ssl/
[root@master-1 kubernetes]# cp awsuser.pem /etc/kubernetes/ssl/

创建 kubeconfig 文件

# 设置集群参数
cd /root/config/
export KUBE_APISERVER="https://192.168.91.254:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=awsuser.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials awsuser \
--client-certificate=/etc/kubernetes/ssl/awsuser.pem \
--client-key=/etc/kubernetes/ssl/awsuser-key.pem \
--embed-certs=true \
--kubeconfig=awsuser.kubeconfig

# 设置上下文参数
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=awsuser \
--namespace=velero-system \
--kubeconfig=awsuser.kubeconfig

# 设置默认上下文
kubectl config use-context kubernetes --kubeconfig=awsuser.kubeconfig

#赋值权限
kubectl create clusterrolebinding awsuser --clusterrole=cluster-admin --user=awsuser

#部署velero
kubectl create ns velero-system
velero --kubeconfig /root/config/awsuser.kubeconfig \
install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.1.0 \
--bucket velero \
--secret-file /opt/credentials-velero \
--use-volume-snapshots=false \
--namespace velero-system \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.91.11:9000

#安装过程
[root@master-1 config]# velero --kubeconfig /root/config/awsuser.kubeconfig install --provider aws --plugins velero/velero-plugin-for-aws:v1.1.0 --bucket velero --secret-file /opt/credentials-velero --use-volume-snapshots=false --namespace velero-system --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.91.11:9000
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero-system: attempting to create resource
Namespace/velero-system: already exists, proceeding
Namespace/velero-system: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero-system' to view the status.

#创建备份
#备份default空间,备份名称为:default-backup
[root@master-1 config]# velero backup create default-backup \
--include-namespaces default \
--kubeconfig=/root/config/awsuser.kubeconfig \
--namespace velero-system

#查看备份
velero backup describe default-backup \
--kubeconfig=/root/config/awsuser.kubeconfig \
--namespace velero-system

#查看S3是否有存储文件

#删除default 空间下的nginx
kubectl delete deployment nginx
kubectl delete pods nginx
kubectl delete svc -l run=nginx
kubectl delete deployment.apps/nginx

#还原nginx
velero restore create --from-backup default-backup --wait \
--kubeconfig=/root/config/awsuser.kubeconfig \
--namespace velero-system

#执行命令
[root@master-1 config]# velero restore create --from-backup default-backup --wait \
> --kubeconfig=/root/config/awsuser.kubeconfig \
> --namespace velero-system
Restore request "default-backup-20201019191046" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
....
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe default-backup-20201019191046` and `velero restore logs default-backup-20201019191046`.

#查看pod 状态(正在创建)
[root@master-1 config]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nacos-0 1/1 Running 1 25h
nacos-1 1/1 Running 1 25h
nacos-2 1/1 Running 1 25h
nfs-client-provisioner-6bb8946b87-k7ndx 1/1 Running 3 35d
nginx-7bb7cd8db5-4vr2q 0/1 ContainerCreating 0 8s
nginx-7bb7cd8db5-shkrj 0/1 ContainerCreating 0 8s

4.定时备份
定时备份
对集群资源进行定时备份，则可在发生意外的情况下，进行恢复（默认情况下，备份保留 30 天）。
# 每日1点进行备份
velero create schedule <SCHEDULE NAME> --schedule="0 1 * * *"
# 每日1点进行备份，备份保留48小时
velero create schedule <SCHEDULE NAME> --schedule="0 1 * * *" --ttl 48h
# 每6小时进行一次备份
velero create schedule <SCHEDULE NAME> --schedule="@every 6h"
# 每日对 web namespace 进行一次备份
velero create schedule <SCHEDULE NAME> --schedule="@every 24h" --include-namespaces web

5.资源查看
velero backup get #备份查看
velero schedule get #查看定时备份
velero restore get #查看可恢复备份