首页 > 其他分享 >在线方式部署k8s+prometheus集群(kubesphere环境)

在线方式部署k8s+prometheus集群(kubesphere环境)

时间:2024-06-22 13:43:38浏览次数:26  
标签:name kubesphere server metrics prometheus nfs io k8s

前言:半月前在公司生产环境上离线部署了k8s集群 和 Prometheus+Grafana监控平台的搭建,下面我租用3台华为云服务器演示在线方式部署k8s(单master节点)+prometheus集群。下期再出一版离线方式部署k8s(双master节点)集群。
安装步骤:
安装Docker
安装Kubernetes
安装KubeSphere前置环境
安装KubeSphere

1.安装Docker

配置docker的yum源地址
yum -y install wget 
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum provides docker-ce
安装指定的docker版本
yum install -y docker-ce-20.10.7 docker-ce-cli-20.10.7 containerd.io-1.4.6
# 启动&开机启动docker
systemctl enable docker --now
​
# docker加速配置
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

2.安装kubernetes环境
2.1、基本环境

#设置每个机器自己的hostname
hostnamectl set-hostname xxx
hostnamectl set-hostname k8s-master1  &&  bash
hostnamectl set-hostname k8s-node1 && bash
hostnamectl set-hostname k8s-node2 && bash

添加hosts解析(所有节点操作)
cat >> /etc/hosts << EOF
192.168.0.182  k8s-master1
192.168.0.145  k8s-node1
192.168.0.28   k8s-node2
EOF

# 将 SELinux 设置为 permissive 模式(相当于将其禁用)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

#关闭swap
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

#将桥接的IPv4流量传递到iptables的链:
vim etc/sysctl.d/k8s.conf
添加:
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

sysctl --system  

时间同步:
yum install ntpdate -y
ntpdate time.windows.com

2.2、安装kubelet、kubeadm、kubectl

配置kubelet kubeadm kubectl 的yum源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装指定版本的kubeadm kubectl kubelet
sudo yum install -y kubelet-1.20.9 kubeadm-1.20.9 kubectl-1.20.9

启动kubelet
systemctl enable kubelet

2.3.初始化master节点
2.3.1初始化
kubeadm init --apiserver-advertise-address=192.168.0.182 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.20.9 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

2.3.2记录关键的信息

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.182:6443 --token nbm1b1.zbd9dx9c1yvp1ebn \
--discovery-token-ca-cert-hash sha256:98c66d83855f52326939d916c21fc254359c771285932d72d15733c3f4972f52

2.3.3、安装Calico网络插件

curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml

2.3.4、加入worker节点
将master节点初始化的结果命令如下,复制到各个节点
kubeadm join 192.168.0.182:6443 --token nbm1b1.zbd9dx9c1yvp1ebn --discovery-token-ca-cert-hash sha256:98c66d83855f52326939d916c21fc254359c771285932d72d15733c3f4972f52

3、安装KubeSphere前置环境
3.1、nfs文件系统
3.1.1、安装nfs-server

# 在每个机器。
yum install -y nfs-utils

## 创建了一个存储类,文件命名为sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  archiveOnDelete: "true"  ## 删除pv的时候,pv的内容是否要备份

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
          # resources:
          #    limits:
          #      cpu: 10m
          #    requests:
          #      cpu: 10m
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.0.182 ## 指定自己nfs服务器地址
            - name: NFS_PATH  
              value: /nfs/data  ## nfs服务器共享的目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.0.182
            path: /nfs/data
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

#执行kubectl apply -f sc.yaml
#查看相关的存储,确认配置是否生效
kubectl get sc

3.2、安装metrics-server
集群指标监控组件,文件命名为metrics-server.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --kubelet-insecure-tls
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

#执行 kubectl apply -f metrics-server.yaml

4、安装KubeSphere
4.1下载相关的配置文件

wget https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/kubesphere-installer.yaml
wget https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/cluster-configuration.yaml

4.2、修改cluster-configuration
在 cluster-configuration.yaml中指定我们需要开启的功能
参照官网“启用可插拔组件”
https://kubesphere.com.cn/docs/pluggable-components/overview/

4.3、执行安装

kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml


这里的报错是和API版本有关系
解决方法:vim kubesphere-installer.yaml 将找到的每个 apiextensions.k8s.io/v1beta1 更改为 apiextensions.k8s.io/v1

启动cluster-configuration.yaml无报错,忘了截图

4.4、查看安装进度
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

访问 : http://192.168.0.182:30880
账号 : admin
密码 : P@88w0rd

4.5、查看集群状态和pod
kubectl get nodes

kubectl get pods -A

部署完成!

标签:name,kubesphere,server,metrics,prometheus,nfs,io,k8s
From: https://www.cnblogs.com/zhoutuo/p/18249196

相关文章

  • 3.3 Ubuntu24使用kubeadm部署高可用K8S集群
    Ubuntu24使用kubeadm部署高可用K8S集群使用kubeadm部署一个k8s集群,3个master+1个worker节点。1.环境信息操作系统:ubuntu24.04内存:2GBCPU:2网络:能够互访,能够访问互联网hostnameip备注k8s-master1192.168.0.51master1k8s-master2192.168.0.52master2k8s-master3192.......
  • Centos7.9使用kubeadm部署K8S 1.27.6集群环境(内网通过代理部署)
    Centos7.9使用kubeadm部署K8S1.27.6集群环境(内网通过代理部署)在内网借助代理服务器,使用kubeadm部署一个k8s集群,单master+2worker节点,K8S版本为1.7.6,使用containerd作为容器运行时。1.环境信息操作系统:CentOS7.9.2009内存:8GBCPU:4网络:节点通过代理进行访问。host......
  • Centos7.9使用kubeadm部署K8S 1.27.6集群环境(内网通过代理部署)
    Centos7.9使用kubeadm部署K8S1.27.6集群环境(内网通过代理部署)在内网借助代理服务器,使用kubeadm部署一个k8s集群,单master+2worker节点,K8S版本为1.7.6,使用containerd作为容器运行时。1.环境信息操作系统:CentOS7.9.2009内存:8GBCPU:4网络:节点通过代理进行访问。ho......
  • Ubuntu24使用kubeadm部署高可用K8S集群
    Ubuntu24使用kubeadm部署高可用K8S集群使用kubeadm部署一个k8s集群,3个master+1个worker节点。1.环境信息操作系统:ubuntu24.04内存:2GBCPU:2网络:能够互访,能够访问互联网hostnameip备注k8s-master1192.168.0.51master1k8s-master2192.168.0.52master......
  • Jenkins + K8s 实现动态 slave 配置
    环境介绍本次jenkins部署在本地服务器上,下面我们开始动态slave配置。k8s创建RBAC##首先需要创建命名空间pipeline[root@master1~]#catpipeline-acount.yamlapiVersion:v1kind:ServiceAccountmetadata:name:jenkins-slavenamespace:pipeline---ki......
  • KubeSphere 社区双周报|2024.06.07-06.20
    KubeSphere社区双周报主要整理展示新增的贡献者名单和证书、新增的讲师证书以及两周内提交过commit的贡献者,并对近期重要的PR进行解析,同时还包含了线上/线下活动和布道推广等一系列社区动态。本次双周报涵盖时间为:2024.06.07-06.20。贡献者名单新晋KubeSpherecontribu......
  • prometheus 中文说明告警指标
    https://blog.51cto.com/qiangsh/1977449主机和硬件监控可用内存指标主机中可用内存容量不足10%-alert:HostOutOfMemoryexpr:node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes*100<10for:5mlabels:severity:warningannotations:......
  • linux可视化监控:Grafana+Prometheus+node_exporter
    原理node_exporter:它的作用主要是收集性能测试的数据,如cpu、内存磁盘网络等信息,然后将数据保存到prometheus,相当于将数据存入到数据库中;想要监控不同的内容,就使用不同的exporter,比如mysqld_exporterPrometheus:虽然说是监控平台,但是实际上是一套数据库,用于做数据存储,不能做展......
  • k8s探针类型及探针配置
    探针类型:存活探针(LivenessProbe):用于判断容器是否存活(running状态),如果LivenessProbe探针探测到容器不健康,则kubelet杀掉该容器,并根据容器的重启策略做相应的处理。如果一个容器不包含LivenessProbe探针,则kubelet认为该容器的LivenessProbe探针返回的值永远是“Success”。......
  • k8s容器启动不了,一直重启, 报红提示Not Ready
    k8s容器启动不了,一直重启,报红提示NotReady反复多次重启后,才能够启动成功。发现是启动时间过长,不断达到了失败阈值,于是会不断重启。将failureThreshold、initialDelaySeconds、periodSeconds这几个参数设置大一些,就可以启动了。k8s探针类型及探针配置:详情见:https://blo......