首页 > 其他分享 >63、Prometheus-独立部署的Prometheus监控K8S集群

63、Prometheus-独立部署的Prometheus监控K8S集群

时间：2023-04-18 19:58:00浏览次数：60  
标签：__ kubernetes prometheus server Prometheus 63 meta token K8S

Kubernetes学习目录

1、简介

1.1、原因

这里我们以prometheus的配置解析如获取各各所需的文件和相关的原理问题，不会细写通过标签如果去获取数据的规则，先把获取K8S的数据链路打通，有助于后面的深入。

研究四五天，网上搜了，获取相关token和ca.crt文件这块都是忽略了事，踏了不少坑。

1.2、Prometheus版本

prometheus]# prometheus --version
prometheus, version 2.43.0 (branch: HEAD, revision: edfc3bcd025dd6fe296c167a14a216cab1e552ee)
  build user:       root@8a0ee342e522
  build date:       20230321-12:56:07
  go version:       go1.19.7
  platform:         linux/amd64
  tags:             netgo,builtinassets

2、Promtheus配置

2.1、复制ca.crt到prometheus server linux系统授信此CA

2.1.1、不复制授信CA报错

Apr 18 09:56:18 prometheus-server prometheus: ts=2023-04-18T01:56:18.116Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.10.26:6443/api/v1/pods?limit=500&resourceVersion=0\": x509: certificate signed by unknown authority"

2.1.2、如何获取ca.crt

# 在apiserver操作
[root@master1 ~]# ll /etc/kubernetes/pki/ca.*
-rw-r--r-- 1 root root 1099 Apr 14 16:52 /etc/kubernetes/pki/ca.crt
-rw------- 1 root root 1675 Apr 14 16:52 /etc/kubernetes/pki/ca.key
-rw-r--r-- 1 root root   17 Apr 16 23:58 /etc/kubernetes/pki/ca.srl

2.1.3、复制为linux系统授信

# 复制好，无需做其它操作
[root@master1 ~]# scp /etc/kubernetes/pki/ca.crt [email protected]:/etc/ssl/certs/k8s.crt

2.2、监控apiserver的示例解析-prometheus.yml

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:

- job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://192.168.10.26:6443
    # 下面部分没有配置的话，会报使用匿名用户获取，没有RBAC权限的问题
    bearer_token_file: /data/server/prometheus/etc/token
    tls_config:
      insecure_skip_verify: true
  relabel_configs:
  - action: keep
    regex: default;kubernetes;https
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  scheme: https
  # 这部分没有配置的话，会报：server returned HTTP status 403 Forbidden
  bearer_token_file: /data/server/prometheus/etc/token
  tls_config:
    insecure_skip_verify: true

# 这里监控为apiserver为例，切记标红新版本的需要，不然会引起失败，

2.3、serviceAccount与token关联

2.3.1、查询serviceAccount信息

master1 ~]# kubectl get sa my-prom-prometheus-server 
NAME                        SECRETS   AGE
my-prom-prometheus-server   0         21h

2.3.2、创建secret并且与sa关联

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: prometheus
  annotations:
    kubernetes.io/service-account.name: my-prom-prometheus-server
type: kubernetes.io/service-account-token
EOF

2.3.3、查询sa关联的信息

[root@master1 ~]# kubectl describe sa my-prom-prometheus-server 
Name:                my-prom-prometheus-server
Namespace:           default
Labels:              app=prometheus
                     app.kubernetes.io/managed-by=Helm
                     chart=prometheus-20.2.0
                     component=server
                     heritage=Helm
                     release=my-prom
Annotations:         meta.helm.sh/release-name: my-prom
                     meta.helm.sh/release-namespace: default
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              prometheus
Events:              <none>

2.3.4、获取token值

[root@master1 ~]# kubectl get secrets prometheus -o go-template --template='{{.data.token}}' | base64 -d > /tmp/token
[root@master1 ~]# cat /tmp/token 
eyJhbGciOiJSUzI1NiIsImtpZCI6ImxlSDFNY1UtX2hWUzlkZVU5a3BEdGhXeFBqbmdLcXBrS3QydWJxOHdGQzgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibXktcHJvbS1wcm9tZXRoZXVzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY1MGIxOGNmLTYzNTYtNDNiNy1hYWIzLTA5YTJlMWI0ODVmOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Om15LXByb20tcHJvbWV0aGV1cy1zZXJ2ZXIifQ.hKV62eieWNogKDurt96e0MKVQEcmdWKuvBBEurgxB097-eRXIxLIExY-qkMjHsxkjsyFdmbw0iXU-I6UR6ScwoUIEDEF93yzlkObV5b4go4pMwvuBTbniK3onoe6AZuAuU-SJY6mxwf2oDFGPhXiTNq7RRKD-cD5romGNHB-gczJlbUuzagUdl0ovDyQfhAdDST1ce4aYuE5OQlaaP67tn6wSXwSb3gT8uqQGCw_hHUjWS6CSzTGlOejTj3FAewjo0P-lNFfILOoGj2UlCwmKvI3rZD-9KAO-EJQOUA6FjJU1803QKdcqaN1r3eELUy7S3msMO_qmpYIGWtXbb0o9Q

2.3.5、将token复制到prometheus目录

scp /tmp/token [email protected]:/data/server/prometheus/etc/

2.3.6、在prometheus主机上测试toke是否可以使用

token=$(cat /data/server/prometheus/etc/token)
curl https://192.168.10.26:6443/metrics -H "Authorization: Bearer ${token}"  -k

2.4、重启prometheus服务并且查看效果

2.4.1、重启prometheus服务

systemctl restart prometheus

2.4.2、查看监控效果

说明监控K8S链路已经打通

3、Prometheus RBAC流程分析

3.1、创建serviceAccount【用户帐号】

cat > serviceaccount.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-prom-prometheus-server
  namespace: default
EOF

3.2、创建ClusterRole【集群角色和权限】

cat > clusterrole.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: release-name-prometheus-server
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - nodes/proxy
      - pods/proxy
      - nodes/metrics
      - services
      - endpoints
      - pods
      - ingresses
      - configmaps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
      - ingresses
    verbs:
      - get
      - list
      - watch
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
EOF

3.3、创建ClusterRoleBinding【ServiceAccount与ClusterRole绑定关系】

cat > clusterrolebinding.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: release-name-prometheus-server
subjects:
  - kind: ServiceAccount
    name: my-prom-prometheus-server
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: release-name-prometheus-server
EOF

3.4、ServiceAccount与token关联【认证阶段】

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: prometheus
  annotations:
    kubernetes.io/service-account.name: my-prom-prometheus-server
type: kubernetes.io/service-account-token
EOF

3.5、用户整个流程总结

用户/接口 -> token+sa -> 认证阶段 -> 鉴权阶段 ->ClusterRoleBinding【查询sa与clusterRole关联信息】 -> clusterRole【查询是否有权限，有权限就通过，没有的话，返回403】

4、增加监控K8S-node节点-示例

4.1、配置prometheus

- job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - role: node
    api_server: https://192.168.10.26:6443
    bearer_token_file: /data/server/prometheus/etc/token
    tls_config:
      insecure_skip_verify: true
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: 192.168.10.26:6443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/$1/proxy/metrics
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  scheme: https
  bearer_token_file: /data/server/prometheus/etc/token
  tls_config:
    insecure_skip_verify: true

4.2、重启prometheus服务

systemctl restart prometheus

4.3、查看监控效果

5、增加kube-state-metrics-示例

需要资源pods/proxy权限，记得增加哦

 

5.1、配置prometheus

- honor_labels: true
  job_name: kube-state-metrics
  kubernetes_sd_configs:
  - role: pod
    api_server: https://192.168.10.26:6443
    bearer_token_file: /data/server/prometheus/etc/token
    tls_config:
      insecure_skip_verify: true
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      action: keep
      regex: 'kube-state-metrics'
    - source_labels: [__address__]
      action: replace
      target_label: instance
    - target_label: __address__
      replacement: 192.168.10.26:6443
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
      regex: ([^;]+);([^;]+);([^;]+)
      target_label: __metrics_path__
      replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name
  scheme: https
  bearer_token_file: /data/server/prometheus/etc/token
  tls_config:
    insecure_skip_verify: true

5.2、重启prometheus服务

systemctl restart prometheus

5.3、查看监控效果

6、增加coreDNS-示例

需要资源pods/proxy权限，记得增加哦

 

6.1、配置prometheus

- honor_labels: true
  job_name: kubernetes-coreDNS
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://192.168.10.26:6443
    bearer_token_file: /data/server/prometheus/etc/token
  relabel_configs:
    - source_labels: [__meta_kubernetes_endpoints_name]
      action: keep
      regex: 'kube-dns'
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      action: keep
      regex: '9153'
    - target_label: __address__
      replacement: 192.168.10.26:6443
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
      regex: ([^;]+);([^;]+);([^;]+)
      target_label: __metrics_path__
      replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
  scheme: https
  bearer_token_file: /data/server/prometheus/etc/token
  tls_config:
    insecure_skip_verify: true

6.2、开放CoreDNS Metrics

]# kubectl -n kube-system edit svc kube-dns 
...
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS

6.3、重启prometheus服务

systemctl restart prometheus

6.4、查看监控效果

 

标签：__,kubernetes,prometheus,server,Prometheus,63,meta,token,K8S
From： https://www.cnblogs.com/ygbh/p/17328667.html

相关文章

  • 69、K8S-Helm-template导出独立的yaml文件
    1、将helm项目导出为独立yaml文件-实践1.1、需求有时候，我们需要导出yaml分析yaml编写情况，而不是直接部署到k8s，这个时候，就需要使用template来实现了1.2、开始操作1.2.1、创建存放yaml文件的目录helm_prometheus]#cd/opt/helm_prometheus/&&mkdirprometheus-tplhelm_p......
  • ASEMI代理ADI亚德诺AD8638ARJZ-REEL7车规级芯片
    编辑-ZAD8638ARJZ-REEL7芯片参数：型号：AD8638ARJZ-REEL7偏移电压：3μV输入偏置电流：1.5pA输入失调电流：7pA输入电压范围：−0.1~+3V共模抑制比：133dB输入电阻：22.5TΩ差模输入电容：4pF共模输入电容：1.7pF输出电压高：4.985V输出电压低：7.5mV短路电流：±19mA闭环输出阻抗：4.2Ω电源抑制比：14......
  • 动态规划：剑指 Offer 63. 股票的最大利润
    题目描述：假设把某股票的价格按照时间先后顺序存储在数组中，请问买卖该股票一次可能获得的最大利润是多少？ 限制：0<=数组长度<=10^5    classSolution{publicintmaxProfit(intprices[]){//状态定义：dp[i]记为利润profit//前i......
  • k8s概念
             ......
  • zabbix 集成 prometheus 数据
    一、概述Zabbix和Prometheus都是开源监控系统，它们具有不同的特点和优势，因此很多人希望将它们集成在一起，以便充分利用它们的功能。以下是将Zabbix和Prometheus集成的一些步骤：安装和配置Prometheus：在安装和配置Prometheus之前，需要先确定您要监控的目标。可以是主机、容器、服务等......
  • 入手一块pm863 1.92T 检查
    入手一块pm8631.92T检查SMART信息CDM测速AS-SSD测速HD-TunePro测速......
  • k8s 工作原理以及常用基础
    k8s是什么☛可移植:支持公有云，私有云，混合云，多重云☛可扩展:模块化，插件化，可挂载，可组合☛自动化：自动部署，自动重启，自动复制，自动伸缩/扩展k8s核实组件1）主要组件●etcd：保存了整个集群的状态；●apiserver：提供了资源操作的唯一入口，并提供访问控制、API注册和发现等机制......
  • 入手一块sm863 1.92T 检查
    入手一块sm8631.92T检查SMART信息CDM测速AS-SSD测速HD-TunePro测速......
  • k8s 认证
    整体过程简述：请求发起方进行K8sAPI请求，经过Authentication（认证）、Authorization（鉴权）、AdmissionControl（准入控制）三个阶段的校验，最后把请求转化为对K8s对象的变更操作持久化至etcd中。认证阶段。当请求发起方建立与APIServer的安全连接后，进入请求的认证阶段（图中步骤1......
  • Kubernetes（k8s）健康检查详解与实战演示（就绪性探针 和 存活性探针）
    一、概述Kubernetes中的健康检查主要使用就绪性探针（readinessProbes）和存活性探针（livenessProbes）来实现，service即为负载均衡，k8s保证service后面的pod都可用，是k8s中自愈能力的主要手段，主要基于这两种探测机制，可以实现如下需求：异常实例自动剔除，并重启新实例多种类型探针检......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99