1. 背景.
我打算在K8S集群部署一套Prometheus监控系统，以监控系统和各应用的各项指标，如资源、性能及自定义监控指标，具体部署方案和细节就不和大家详细说了，后面再和大家分享，这次先说我遇到问题。在Prometheus各组件都部署成功时候，我发现grafana的service的类型为 "ClusterIP"，这意味着我无法在浏览器访问，于是我决定通过编辑grafana的yaml，将"ClusterIP" 改为 "NodePort" 类型，却发现在浏览器使用nodeip+端口方式还是访问不了。
2. 执行 "kubectl get all -n monitoring" 发现grafana为ClusterIP类型.
# kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-main-0 2/2 Running 0 4h47m
pod/blackbox-exporter-5d668b5c6-f9fds 3/3 Running 0 5h17m
pod/grafana-68fd49fd99-jhs25 1/1 Running 0 5h17m
pod/kube-state-metrics-78ddfd78fd-8blqk 3/3 Running 0 5h17m
pod/node-exporter-6lvps 2/2 Running 0 5h17m
pod/node-exporter-8pw78 2/2 Running 0 5h17m
pod/node-exporter-mmnbc 2/2 Running 0 5h17m
pod/node-exporter-p49nq 2/2 Running 1 (5h13m ago) 5h17m
pod/node-exporter-v7fvb 2/2 Running 0 5h17m
pod/prometheus-adapter-5485575f49-8r4gd 1/1 Running 0 5h17m
pod/prometheus-adapter-5485575f49-974m8 1/1 Running 0 5h17m
pod/prometheus-k8s-0 2/2 Running 0 5h15m
pod/prometheus-operator-5b687bfbb8-7djk2 2/2 Running 0 5h17m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-main ClusterIP 10.96.23.99 <none> 9093/TCP,8080/TCP 5h17m
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 5h15m
service/blackbox-exporter ClusterIP 10.96.38.236 <none> 9115/TCP,19115/TCP 5h17m
service/grafana ClusterIP 10.96.112.113 <none> 3000/TCP 5h17m
service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 5h17m
service/node-exporter ClusterIP None <none> 9100/TCP 5h17m
service/prometheus-adapter ClusterIP 10.96.216.177 <none> 443/TCP 5h17m
service/prometheus-k8s ClusterIP 10.96.113.84 <none> 9090/TCP,8080/TCP 5h17m
service/prometheus-operated ClusterIP None <none> 9090/TCP 5h15m
service/prometheus-operator ClusterIP None <none> 8443/TCP 5h17m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 5 5 5 5 5 kubernetes.io/os=linux 5h17m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/blackbox-exporter 1/1 1 1 5h17m
deployment.apps/grafana 1/1 1 1 5h17m
deployment.apps/kube-state-metrics 1/1 1 1 5h17m
deployment.apps/prometheus-adapter 2/2 2 2 5h17m
deployment.apps/prometheus-operator 1/1 1 1 5h17m
NAME DESIRED CURRENT READY AGE
replicaset.apps/blackbox-exporter-5d668b5c6 1 1 1 5h17m
replicaset.apps/grafana-68fd49fd99 1 1 1 5h17m
replicaset.apps/kube-state-metrics-78ddfd78fd 1 1 1 5h17m
replicaset.apps/prometheus-adapter-5485575f49 2 2 2 5h17m
replicaset.apps/prometheus-operator-5b687bfbb8 1 1 1 5h17m
NAME READY AGE
statefulset.apps/alertmanager-main 1/1 5h15m
statefulset.apps/prometheus-k8s 1/1 5h15m
3. 将 grafana的"ClusterIP" 改为 "NodePort" 类型.
# kubectl edit svc grafana -n monitoring
service/grafana edited
# kubectl get svc grafana -n monitoring -oyaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2024-04-10T01:16:27Z"
labels:
app.kubernetes.io/component: grafana
app.kubernetes.io/name: grafana
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 10.4.0
name: grafana
namespace: monitoring
resourceVersion: "804268"
uid: c6b751b4-9710-4159-8051-0e73660577ca
spec:
clusterIP: 10.96.112.113
clusterIPs:
- 10.96.112.113
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 32440
port: 3000
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/component: grafana
app.kubernetes.io/name: grafana
app.kubernetes.io/part-of: kube-prometheus
sessionAffinity: None
type: NodePort
###将 type: NodePort 改为 clusterIP 即可.
###将 type: NodePort 改为 clusterIP 即可.
###将 type: NodePort 改为 clusterIP 即可.
status:
loadBalancer: {}
4. 执行 "kubectl get all -n monitoring" 发现grafana变为了NodePort类型.
# kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-main-0 2/2 Running 0 5h1m
pod/blackbox-exporter-5d668b5c6-f9fds 3/3 Running 0 5h32m
pod/grafana-68fd49fd99-jhs25 1/1 Running 0 5h32m
pod/kube-state-metrics-78ddfd78fd-8blqk 3/3 Running 0 5h32m
pod/node-exporter-6lvps 2/2 Running 0 5h31m
pod/node-exporter-8pw78 2/2 Running 0 5h31m
pod/node-exporter-mmnbc 2/2 Running 0 5h31m
pod/node-exporter-p49nq 2/2 Running 1 (5h27m ago) 5h31m
pod/node-exporter-v7fvb 2/2 Running 0 5h31m
pod/prometheus-adapter-5485575f49-8r4gd 1/1 Running 0 5h31m
pod/prometheus-adapter-5485575f49-974m8 1/1 Running 0 5h31m
pod/prometheus-k8s-0 2/2 Running 0 5h29m
pod/prometheus-operator-5b687bfbb8-7djk2 2/2 Running 0 5h31m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-main ClusterIP 10.96.23.99 <none> 9093/TCP,8080/TCP 5h32m
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 5h29m
service/blackbox-exporter ClusterIP 10.96.38.236 <none> 9115/TCP,19115/TCP 5h32m
service/grafana NodePort 10.96.112.113 <none> 3000:31735/TCP 5h32m
service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 5h32m
service/node-exporter ClusterIP None <none> 9100/TCP 5h31m
service/prometheus-adapter ClusterIP 10.96.216.177 <none> 443/TCP 5h31m
service/prometheus-k8s ClusterIP 10.96.113.84 <none> 9090/TCP,8080/TCP 5h31m
service/prometheus-operated ClusterIP None <none> 9090/TCP 5h29m
service/prometheus-operator ClusterIP None <none> 8443/TCP 5h31m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 5 5 5 5 5 kubernetes.io/os=linux 5h32m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/blackbox-exporter 1/1 1 1 5h32m
deployment.apps/grafana 1/1 1 1 5h32m
deployment.apps/kube-state-metrics 1/1 1 1 5h32m
deployment.apps/prometheus-adapter 2/2 2 2 5h31m
deployment.apps/prometheus-operator 1/1 1 1 5h31m
NAME DESIRED CURRENT READY AGE
replicaset.apps/blackbox-exporter-5d668b5c6 1 1 1 5h32m
replicaset.apps/grafana-68fd49fd99 1 1 1 5h32m
replicaset.apps/kube-state-metrics-78ddfd78fd 1 1 1 5h32m
replicaset.apps/prometheus-adapter-5485575f49 2 2 2 5h31m
replicaset.apps/prometheus-operator-5b687bfbb8 1 1 1 5h31m
NAME READY AGE
statefulset.apps/alertmanager-main 1/1 5h29m
statefulset.apps/prometheus-k8s 1/1 5h29m
4. 采用10.0.0.104+31735端口方式访问grafana发现访问失败.
我节点ip地址为 10.0.0.104，你需要使用你自己集群节点是ip+自己NodePort端口访问grafana服务。
我节点ip地址为 10.0.0.104，你需要使用你自己集群节点是ip+自己NodePort端口访问grafana服务。
我节点ip地址为 10.0.0.104，你需要使用你自己集群节点是ip+自己NodePort端口访问grafana服务。
5. 我开始在网络上查找资料，发现是网络限制原因.
解决方法是删除monitoring命名空间下的网络策略让其从新加载pod间网络，稍微等待一会哦，在浏览器就可以正常访问了。
kubectl delete networkpolicy --all -n monitoring
6. 在浏览器使用节点ip+端口访问测试，发现没有问题.
