2、K8S集群监控

标签：__ K8S name kubernetes labels label 集群监控 kube

一、KubeStateMetrics简介
二、集群组件
- 2.0、部署 KubeStateMetrics
- 2.1、集群组件监控
三、cAdvisor
- 3.1、Prometheus 添加 cAdvisor 配置

一、KubeStateMetrics简介

kube-state-metrics 是一个 Kubernetes 组件，它通过查询 Kubernetes 的 API 服务器，收集关于 Kubernetes 中各种资源（如节点、pod、服务等）的状态信息，并将这些信息转换成 Prometheus 可以使用的指标。
kube-state-metrics 主要功能：

节点状态信息，如节点 CPU 和内存的使用情况、节点状态、节点标签等。
Pod 的状态信息，如 Pod 状态、容器状态、容器镜像信息、Pod 的标签和注释等。
Deployment、Daemonset、Statefulset 和 ReplicaSet 等控制器的状态信息，如副本数、副本状态、创建时间等。
Service 的状态信息，如服务类型、服务 IP 和端口等。
存储卷的状态信息，如存储卷类型、存储卷容量等。
Kubernetes 的 API 服务器状态信息，如 API 服务器的状态、请求次数、响应时间等。

通过 kube-state-metrics 可以方便的对 Kubernetes 集群进行监控，发现问题，以及提前预警。

二、集群组件

2.0、部署 KubeStateMetrics

包含ServiceAccount、ClusterRole、ClusterRoleBinding、Deployment、ConfigMap、Service 六类YAML文件

2.0.1 部署RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  labels:
    k8s-app: kube-state-metrics
rules:
- apiGroups: [""]
  resources: ["configmaps","secrets","nodes","pods",
              "services","resourcequotas",
              "replicationcontrollers","limitranges",
              "persistentvolumeclaims","persistentvolumes",
              "namespaces","endpoints"]
  verbs: ["list","watch"]
- apiGroups: ["extensions"]
  resources: ["daemonsets","deployments","replicasets"]
  verbs: ["list","watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets","daemonsets","deployments","replicasets"]
  verbs: ["list","watch"]
- apiGroups: ["batch"]
  resources: ["cronjobs","jobs"]
  verbs: ["list","watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["list","watch"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["subjectaccessreviews"]
  verbs: ["create"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["list","watch"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["list","watch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses","volumeattachments"]
  verbs: ["list","watch"]
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations","validatingwebhookconfigurations"]
  verbs: ["list","watch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies","ingresses"]
  verbs: ["list","watch"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  labels:
    app: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

2.0.2 部署deploy、svc

apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
    app.kubernetes.io/name: kube-state-metrics   ##不能删除此注解,该注解用于Prometheus自动发现
spec:
  type: ClusterIP
  ports:
  - name: http-metrics
    port: 8080
    targetPort: 8080
  - name: telemetry
    port: 8081
    targetPort: 8081
  selector:
    k8s-app: kube-state-metrics
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: bitnami/kube-state-metrics:2.0.0
        #image: registry.cn-shenzhen.aliyuncs.com/starsl/kube-state-metrics:v2.3.0
        securityContext:
          runAsUser: 65534
        ports:
        - name: http-metrics    ##用于公开kubernetes的指标数据的端口
          containerPort: 8080
        - name: telemetry       ##用于公开自身kube-state-metrics的指标数据的端口
          containerPort: 8081
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5

2.0.3 确认验证

kubectl  -n kube-system  get  pods | grep  kube-state

curl -kL  $(kubectl get service -n kube-system | grep kube-state-metrics |awk '{ print $3 }'):8080/metrics | tail -20

2.1、集群组件监控

在 prometheus-config.yaml 一次添加如下采集数据：

2.1.1、kube-apiserver

需要注意的是使用https访问时，需要tls相关配置，可以指定ca证书路径或者 insecure_skip_verify: true跳过证书验证。
除此之外，还要指定 bearer_token_file，否则会提示 server returned HTTP status 400 Bad Request;


    - job_name: kube-apiserver
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        action: keep
        regex: default;kubernetes
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

更新配置

kubectl apply -f prometheus-cm.yaml

导入grafana模板

15761

2.1.2、controller-manager

查看controller-manager信息（名称可能不太一样，大家注意一下）

xxxxxxxxxx


# kubectl describe pod -n kube-system kube-controller-manager-master1
Name:                 kube-controller-manager-k8s-master
Namespace:            kube-system
……
Labels:               component=kube-controller-manager
                      tier=control-plane
……
Containers:
  kube-controller-manager:
    ……
    Command:
      kube-controller-manager
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
      --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
      --bind-address=127.0.0.1
      ……

由上可知，匹配pod对象，lable标签为component=kube-controller-manager即可，但需注意的是controller-manager默认只运行127.0.0.1访问，因此还需先修改所用master节点的controller-manager配置.

修改 /etc/kubernetes/manifests/kube-controller-manager.yaml

xxxxxxxxxx


# cat /etc/kubernetes/manifests/kube-controller-manager.yaml 
……
  - command:
    - --bind-address=0.0.0.0 # 端口改为0.0.0.0
    #- --port=0 # 注释0端口
……

编写prometheus配置文件，需要注意的是，他默认匹配到的是80端口，需要手动指定为10252端口;


    - job_name: kube-controller-manager
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: kube-controller-manager
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10252
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

2.1.3、scheduler

xxxxxxxxxx


[root@tiaoban prometheus]# kubectl describe pod -n kube-system kube-scheduler-master1
Name:                 kube-scheduler-k8s-master
Namespace:            kube-system
……
Labels:               component=kube-scheduler
                      tier=control-plane
……

由上可知，匹配pod对象，lable标签为component=kube-scheduler即可scheduler和controller-manager一样，默认监听0端口，需要注释
修改所有master节点 /etc/kubernetes/manifests/kube-scheduler.yaml

xxxxxxxxxx


# cat /etc/kubernetes/manifests/kube-scheduler.yaml
……
  - command:
    - --bind-address=0.0.0.0 # 端口改为0.0.0.0
    #- --port=0 # 注释0端口
……

编写prometheus配置文件，需要注意的是，他默认匹配到的是80端口，需要手动指定为10251端口,同时指定token，否则会提示 server returned HTTP status 400 Bad Request;


    - job_name: kube-scheduler
      kubernetes_sd_configs:
      - role: pod
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: kube-scheduler
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10251
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

2.1.4、kube-state-metrics

编写prometheus配置文件，需要注意的是，他默认匹配到的是8080和801两个端口，需要手动指定为8080端口;

    - job_name: "kube-state-metrics"
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ["kube-system"]
      relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
        regex: kube-state-metrics
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:8080
      - action: replace
        source_labels: [__meta_kubernetes_namespace]
        target_label: k8s_namespace
      - action: replace
        source_labels: [__meta_kubernetes_service_name]
        target_label: k8s_sname

2.1.5、coredns

编写prometheus配置文件，需要注意的是，他默认匹配到的是53端口，需要手动指定为9153端口


    - job_name: coredns
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels:
          - __meta_kubernetes_service_label_k8s_app
        regex: kube-dns
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:9153
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

2.1.6、etcd


# kubectl describe pod -n kube-system etcd-master1
Name:                 etcd-master1
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 master1/192.10.192.158
Start Time:           Mon, 30 Jan 2023 15:06:35 +0800
Labels:               component=etcd
                      tier=control-plane
···
    Command:
      etcd
      --advertise-client-urls=https://192.10.192.158:2379
      --cert-file=/etc/kubernetes/pki/etcd/server.crt
      --client-cert-auth=true
      --data-dir=/var/lib/etcd
      --initial-advertise-peer-urls=https://192.10.192.158:2380
      --initial-cluster=master1=https://192.10.192.158:2380
      --key-file=/etc/kubernetes/pki/etcd/server.key
      --listen-client-urls=https://127.0.0.1:2379,https://192.10.192.158:2379
      --listen-metrics-urls=http://127.0.0.1:2381
      --listen-peer-urls=https://192.10.192.158:2380
      --name=master1
      --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
      --peer-client-cert-auth=true
      --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
      --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
      --snapshot-count=10000
      --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
···

由上可知，启动参数里面有一个 --listen-metrics-urls=http://127.0.0.1:2381 的配置，该参数就是来指定 Metrics 接口运行在 2381 端口下面的，而且是 http 的协议，所以也不需要什么证书配置，这就比以前的版本要简单许多了，以前的版本需要用 https 协议访问，所以要配置对应的证书。但是还需修改配置文件，地址改为0.0.0.0

编写prometheus配置文件，需要注意的是，他默认匹配到的是2379端口，需要手动指定为2381端口


    - job_name: etcd
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_label_component
        regex: etcd
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:2381
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

上面部分参数简介如下:

kubernetes_sd_configs: 设置发现模式为 Kubernetes 动态服务发现
kubernetes_sd_configs.role: 指定 Kubernetes 的服务发现模式，这里设置为 endpoints 的服务发现模式，该模式下会调用 kube-apiserver 中的接口获取指标数据。并且还限定只获取 kube-state-metrics 所在 - Namespace 的空间 kube-system 中的 Endpoints 信息
kubernetes_sd_configs.namespace: 指定只在配置的 Namespace 中进行 endpoints 服务发现
relabel_configs: 用于对采集的标签进行重新标记

热加载prometheus，使configmap配置文件生效（也可以等待prometheus的自动热加载）：

curl -XPOST http://prometheus.kubernets.cn/-/reload

旧版本监控etcd需要证书

通过secret创建证书
:::tips
kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/server.crt --from-file=/etc/kubernetes/pki/etcd/server.key -n monitor
:::
创建ServiceEndpoints资源

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
    app.kubernetes.io/name: etcd
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: port
    port: 2379
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 10.10.10.11
  - ip: 10.10.10.12
  - ip: 10.10.10.13
  ports:
  - port: 2379

修改prometheus配置

    - job_name: "kubernetes-etcd"
      scheme: https
      tls_config:
        ca_file: /certs/ca.crt
        cert_file: /certs/server.crt
        key_file: /certs/server.key
        insecure_skip_verify: false
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ["kube-system"]
      relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
        regex: etcd

通过deploy挂载证书

        - name: certs # 挂载证书配置
          readOnly: true
          mountPath: /certs
          
      - name: certs
        secret:
          secretName: etcd-certs # 存储etcd证书Secret名称
          
kubectl apply -f prometheus_deploy.yaml

检查

导入模板grafana模板

2.1.7、kubelet


[root@master01 prometheus-all]# ss -tunlp | grep kubelet
tcp    LISTEN     0      16384  127.0.0.1:26594                 *:*                   users:(("kubelet",pid=1133,fd=15))
tcp    LISTEN     0      16384  127.0.0.1:10248                 *:*                   users:(("kubelet",pid=1133,fd=33))
tcp    LISTEN     0      16384  [::]:10250              [::]:*                   users:(("kubelet",pid=1133,fd=28))

由上可知，kubelet在每个node节点都有运行，端口为10250，因此role使用node即可。

编写prometheus配置文件，需要注意的是，他的指标采集地址为/metrics/cadvisor，需要配置https访问，可以设置insecure_skip_verify: true跳过证书验证

添加配置

- job_name: kubelet
  metrics_path: /metrics/cadvisor
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_endpoints_name]
    action: replace
    target_label: endpoint
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
    
  或者，采用云原生grafna面版的配置
    - job_name: 'k8s-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

更新cm文件

 kubectl apply -f prometheus-cm.yaml 
 kubectl apply -f prometheus-deploy.yaml

查看Prometheus配置

2.1.8、node

什么是node export

:::tips
Node Exporter是一个用于监控Linux系统性能指标的开源软件工具。它是Prometheus生态系统中的一部分，用于收集和暴露有关主机（通常是运行Linux操作系统的服务器）的各种性能指标和系统统计信息。Node Exporter可以帮助系统管理员、运维团队和开发人员实时监控服务器的性能，并在出现问题时提供警报和数据以进行故障排除。
Node Exporter可以收集的一些常见性能指标包括：

CPU使用率：包括总体CPU使用率、每个核心的使用率等。
内存使用情况：包括可用内存、已使用内存、缓冲区和缓存等信息。
磁盘空间：监控硬盘分区的空闲空间、使用情况和I/O活动。
网络统计：包括网络接口的数据传输速率、错误率和丢包率等信息。
文件系统统计：关于文件系统的信息，如inode使用情况、文件句柄数等。
系统负载：系统的平均负载，用于衡量系统的忙碌程度。
进程信息：有关运行的进程的统计信息，如进程数、线程数等。
温度传感器数据：如果硬件支持，可以监控温度传感器的数据。
:::

DaemonSet运行node_exporter

[root@master01 prometheus-all]# cat node-exporter.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    name: node-exporter
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      name: node-exporter
  template:
    metadata:
      labels:
        name: node-exporter
        app: node-exporter
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v0.16.0
        ports:
        - containerPort: 9100
        resources:
          requests:
            cpu: 0.15
        securityContext:
          privileged: true
        args:
        - --path.procfs
        - /host/proc
        - --path.sysfs
        - /host/sys
        - --collector.filesystem.ignored-mount-points
        - '"^/(sys|proc|dev|host|etc)($|/)"'
        volumeMounts:
        - name: dev
          mountPath: /host/dev
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
        - name: rootfs
          mountPath: /rootfs
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: dev
          hostPath:
            path: /dev
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /

创建node

kubectl apply -f node-exporter.yaml

添加cm配置

    - job_name: "kubernetes-nodes"
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:9100'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

更新配置

:::tips
kubectl apply -f prometheus-cm.yaml
kubectl apply -f prometheus-deploy.yaml
:::

三、cAdvisor

cAdvisor 主要功能：

对容器资源的使用情况和性能进行监控。它以守护进程方式运行，用于收集、聚合、处理和导出正在运行容器的有关信息。
cAdvisor 本身就对 Docker 容器支持，并且还对其它类型的容器尽可能的提供支持，力求兼容与适配所有类型的容器。
Kubernetes 已经默认将其与 Kubelet 融合，所以我们无需再单独部署 cAdvisor 组件来暴露节点中容器运行的信息。

3.1、Prometheus 添加 cAdvisor 配置

由于 Kubelet 中已经默认集成 cAdvisor 组件，所以无需部署该组件。需要注意的是，他的指标采集地址为 /metrics/cadvisor，需要配置https访问，可以设置 insecure_skip_verify: true 跳过证书验证;

    - job_name: 'k8s-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
      
      
    - job_name: 'k8s-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      metric_relabel_configs:
      - source_labels: [instance]
        separator: ;
        regex: (.+)
        target_label: node
        replacement: $1
        action: replace

热加载prometheus，使configmap配置文件生效：


curl -XPOST http://prometheus.kubernets.cn/-/reload

标签：__,K8S,name,kubernetes,labels,label,集群,监控,kube
From： https://www.cnblogs.com/Unstoppable9527/p/18327996

一、KubeStateMetrics简介

二、集群组件

2.0、部署 KubeStateMetrics

2.0.1 部署RBAC

2.0.2 部署deploy、svc

2.0.3 确认验证

2.1、集群组件监控

2.1.1、kube-apiserver

更新配置

导入grafana模板

2.1.2、controller-manager

2.1.3、scheduler

2.1.4、kube-state-metrics

2.1.5、coredns

2.1.6、etcd

旧版本监控etcd需要证书

2.1.7、kubelet

添加配置

更新cm文件

2.1.8、node

什么是node export

DaemonSet运行node_exporter

创建node

添加cm配置

更新配置

三、cAdvisor

3.1、Prometheus 添加 cAdvisor 配置

相关文章

赞助商

阅读排行