Kubernetes云原生存储解决方案openebs部署实践-4.0.1版本（helm部署）
简介
OpenEBS 是一种开源云原生存储解决方案。OpenEBS 可以将 Kubernetes 工作节点可用的任何存储转化为本地或复制的 Kubernetes 持久卷。OpenEBS 帮助应用和平台团队轻松地部署需要快速、持久耐用、可靠且可扩展的容器原生存储的 Kubernetes 有状态工作负载。
OpenEBS 也是基于 NVMe 存储部署的首选方案之一。
OpenEBS 最初由 MayaData 构建，并捐赠给了云原生计算基金会（Cloud Native Computing Foundation）。
OpenEBS 工作流:
安装部署
另一篇文章部署实践了3.10.0版本，本文部署最新的版本，新版本提供了更多的功能。
- 设置helm仓库
helm repo add openebs https://openebs.github.io/openebs
helm repo update
- 使用默认值安装OpenEBS helm chart
helm install openebs --namespace openebs openebs/openebs --create-namespace
上面的命令会安装OpenEBS Local PV Hostpath, OpenEBS Local PV LVM, OpenEBS Local PV ZFS, and OpenEBS Replicated Storage组件。
下面的命令选择不安装openebs replicated Storage:
helm install openebs --namespace openebs openebs/openebs --set engines.replicated.mayastor.enabled=false --create-namespace
- 查看安装的chart
helm ls -n openebs
- 安装情况确认
kubectl get pods -n openebs
如果是全部组件安装：
NAME READY STATUS RESTARTS AGE
openebs-agent-core-674f784df5-7szbm 2/2 Running 0 11m
openebs-agent-ha-node-nnkmv 1/1 Running 0 11m
openebs-agent-ha-node-pvcrr 1/1 Running 0 11m
openebs-agent-ha-node-rqkkk 1/1 Running 0 11m
openebs-api-rest-79556897c8-b824j 1/1 Running 0 11m
openebs-csi-controller-b5c47d49-5t5zd 6/6 Running 0 11m
openebs-csi-node-flq49 2/2 Running 0 11m
openebs-csi-node-k8d7h 2/2 Running 0 11m
openebs-csi-node-v7jfh 2/2 Running 0 11m
openebs-etcd-0 1/1 Running 0 11m
openebs-etcd-1 1/1 Running 0 11m
openebs-etcd-2 1/1 Running 0 11m
openebs-io-engine-7t6tf 2/2 Running 0 11m
openebs-io-engine-9df6r 2/2 Running 0 11m
openebs-io-engine-rqph4 2/2 Running 0 11m
openebs-localpv-provisioner-6ddf7c7978-4fkvs 1/1 Running 0 11m
openebs-loki-0 1/1 Running 0 11m
openebs-lvm-localpv-controller-7b6d6b4665-fk78q 5/5 Running 0 11m
openebs-lvm-localpv-node-mcch4 2/2 Running 0 11m
openebs-lvm-localpv-node-pdt88 2/2 Running 0 11m
openebs-lvm-localpv-node-r9jn2 2/2 Running 0 11m
openebs-nats-0 3/3 Running 0 11m
openebs-nats-1 3/3 Running 0 11m
openebs-nats-2 3/3 Running 0 11m
openebs-obs-callhome-854bc967-5f879 2/2 Running 0 11m
openebs-operator-diskpool-5586b65c-cwpr8 1/1 Running 0 11m
openebs-promtail-2vrzk 1/1 Running 0 11m
openebs-promtail-mwxk8 1/1 Running 0 11m
openebs-promtail-w7b8k 1/1 Running 0 11m
openebs-zfs-localpv-controller-f78f7467c-blr7q 5/5 Running 0 11m
openebs-zfs-localpv-node-h46m5 2/2 Running 0 11m
openebs-zfs-localpv-node-svfgq 2/2 Running 0 11m
openebs-zfs-localpv-node-wm9ks 2/2 Running 0 11m
问题记录：
本文在进行实际部署后，出现了openebs-csi-node无法正常启动的问题，详情如下：
root@master1:~# kubectl get pod -n openebs
...
openebs-csi-node-jk7wj 0/2 Init:0/1 1 23h
openebs-csi-node-k98dr 0/2 Init:0/1 1 23h
openebs-csi-node-vrhsj 0/2 Init:0/1 1 23h
openebs-csi-node-wp7z2 0/2 Init:0/1 1 23h
...
# 查看日志pod卡在初始化阶段
root@master1:/etc# kubectl logs -n openebs openebs-csi-node-jk7wj
Defaulted container "csi-node" out of: csi-node, csi-driver-registrar, nvme-tcp-probe (init)
Error from server (BadRequest): container "csi-node" in pod "openebs-csi-node-jk7wj" is waiting to start: PodInitializing
出现了该问题的原因为linux内核没有加载nvme_tcp模块，各个节点使用命令modprobe nvme_tcp加载后，删除pod重建启动正常。
将模块加载命令写入到配置文件中，重启系统后自动加载：
echo "nvme_tcp" | sudo tee /etc/modules-load.d/openebs.conf
如果禁用了Replicated Storage，输出类似如下：
NAME READY STATUS RESTARTS AGE
openebs-localpv-provisioner-6ddf7c7978-jsstg 1/1 Running 0 3m9s
openebs-lvm-localpv-controller-7b6d6b4665-wfw64 5/5 Running 0 3m9s
openebs-lvm-localpv-node-62lnq 2/2 Running 0 3m9s
openebs-lvm-localpv-node-lhndx 2/2 Running 0 3m9s
openebs-lvm-localpv-node-tlcqv 2/2 Running 0 3m9s
openebs-zfs-localpv-controller-f78f7467c-k7ldb 5/5 Running 0 3m9s
openebs-zfs-localpv-node-5mwbz 2/2 Running 0 3m9s
openebs-zfs-localpv-node-g45ft 2/2 Running 0 3m9s
openebs-zfs-localpv-node-g77g6 2/2 Running 0 3m9s
说明：
OpenEBS中的Replicated Storage是指通过将数据复制到多个存储节点来提供高可用性和数据冗余的一种存储方案。
demo测试
openebs部署完成后会自动创建存储类，我们使用openebs-hostpath这个StorageClass来创建PVC。
存储类：
root@master1:~# kubectl get sc -n openebs
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
mayastor-etcd-localpv openebs.io/local Delete WaitForFirstConsumer false 2d
mayastor-loki-localpv openebs.io/local Delete WaitForFirstConsumer false 2d
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 2d
openebs-single-replica io.openebs.csi-mayastor Delete Immediate true 2d
创建一个示例应用挂载openebs提供的存储进行测试。资源清单文件openebs-test.yaml，包括pvc和pod：
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-hostpath-pvc
spec:
storageClassName: openebs-hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
name: hello-local-hostpath-pod
spec:
volumes:
- name: local-storage
persistentVolumeClaim:
claimName: local-hostpath-pvc
containers:
- name: hello-container
image: busybox
command:
- sh
- -c
- 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
volumeMounts:
- mountPath: /mnt/store
name: local-storage
创建资源：kubectl create -f openebs-test.yaml，等待创建完成。
root@master1:~# kubectl get pod -o wide | grep hello
hello-local-hostpath-pod 1/1 Running 0 33s 10.244.2.35 master3 <none> <none>
root@master1:~# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
local-hostpath-pvc Bound pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410 5Gi RWO openebs-hostpath 78s
root@master1:~# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410 5Gi RWO Delete Bound default/local-hostpath-pvc openebs-hostpath 77s
# 检查node1节点的本地路径
test@master3:~$ cat /var/openebs/local/pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410/greet.txt
Wed Jul 3 14:04:49 UTC 2024 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.