问题描述

监控系统搭建好后，kube-controller-manager和kube-scheduler没有数据：

背景描述

Kubernetes 版本：1.20.1
kube-prometheus 版本：0.7
版本说明：https://github.com/prometheus-operator/kube-prometheus/tree/release-0.7#apply-the-kube-prometheus-stack

排查过程

检查kube-controller-manager和kube-scheduler服务端口

在k8s-master01执行
# netstat -nltp|grep sche
# netstat -nltp|grep contr

从图可以看出，端口已经对0.0.0.0放开。

查看serviceMonitor的标签

kube-controller-manager的serviceMonitor文件在kube-prometheus/manifests/prometheus-serviceMonitorKubeControllerManager.yaml
kube-scheduler的serviceMonitor文件在kube-prometheus/manifests/prometheus-serviceMonitorKubeScheduler.yaml

# cd kube-prometheus/manifests/
# grep "selector" prometheus-serviceMonitorKubeControllerManager.yaml -A 3
# grep "selector" prometheus-serviceMonitorKubeScheduler.yaml -A 3

或者用命令检查：

kubectl get servicemonitors.monitoring.coreos.com -n monitoring kube-controller-manager -oyaml
kubectl get servicemonitors.monitoring.coreos.com -n monitoring kube-scheduler -oyaml

从截图可以看出，kube-controller-manager标签是：k8s-app: kube-controller-manager
同样方式，查看出kube-scheduler的标签是：k8s-app: kube-scheduler

查看kube-controller-manager和kube-scheduler的svc

此处说明下，kube-controller-manager和kube-scheduler是二进制方式安装，svc和endpoint需要自己新增并创建，注意标签需要与serviceMonitor的标签一致

添加kube-controller-manager和kube-scheduler的svc

文件prometheus-kubeControllerManagerService.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: kube-controller-manager
  name: kube-controller-manager
  namespace: kube-system
spec:
  type: ClusterIP
  sessionAffinity: None
  ports:
  - name: http-metrics
    port: 10252
    protocol: TCP
    targetPort: 10252
---
#二进制部署还得创建对应的 Endpoints 对象将两个组件挂入到 kubernetes 集群内，然后通过 Service 提供访问，才能让 Prometheus 监控到。
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: kube-controller-manager
  name: kube-controller-manager
  namespace: kube-system
subsets:
- addresses:
  - ip: 10.10.246.36
  - ip: 10.10.246.37
  - ip: 10.10.246.38
  ports:
  - name: http-metrics
    port: 10252
    protocol: TCP

文件prometheus-kubeSchedulerService.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: kube-system
spec:
  type: ClusterIP
  ports:
  - name: http-metrics
    port: 10251
    protocol: TCP
    targetPort: 10251
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: kube-system
subsets:
- addresses:
  - ip: 10.10.246.36
  - ip: 10.10.246.37
  - ip: 10.10.246.38
  ports:
  - name: http-metrics
    port: 10251
    protocol: TCP

检查svc标签是否与serviceMonitor的标签一致

# kubectl get svc -n kube-system -l k8s-app=kube-controller-manager
# kubectl get svc -n kube-system -l k8s-app=kube-scheduler