rancher prometheus监控

标签：container name sum namespace instance prometheus 监控 rancher total

 

集群指标

集群 CPU 利用率

目录    表达式
详细信息    1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))
摘要       1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))

集群平均负载

load1    sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)
load5    sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)
load15    sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)

 

load1    sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})
load5    sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})
load15    sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})

集群内存利用率

详细信息    1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)
摘要       1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)

 

集群磁盘利用率

目录    表达式
详细信息    (sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)
摘要       (sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})

 

集群磁盘 I/O

目录    表达式
详细信息    
read    sum(rate(node_disk_read_bytes_total[5m])) by (instance)
written    sum(rate(node_disk_written_bytes_total[5m])) by (instance)
摘要    
read    sum(rate(node_disk_read_bytes_total[5m]))
written    sum(rate(node_disk_written_bytes_total[5m]))

 

 

节点指标

节点 CPU 利用率

目录    　　表达式
详细信息    avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)
摘要    　　1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))

节点平均负载

目录    　　表达式
详细信息    
load1    sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})
load5    sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})
load15    sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})
摘要    
load1    sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})
load5    sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})
load15    sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})

 

节点内存利用率

目录    　　表达式
详细信息    1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})
摘要   　　 1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})

节点磁盘利用率

目录    　　表达式
详细信息    (sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)
摘要    　　(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})

 

节点磁盘 I/O

目录    　　表达式
详细信息    
read    　　sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))
written    sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))
摘要    
read    　　sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))
written    sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))

 

Etcd 指标

Etcd Leader

max(etcd_server_has_leader)

Leader 改变次数

max(etcd_server_leader_changes_seen_total)

 

失败 Proposals 次数

sum(etcd_server_proposals_failed_total)

 

GRPC 客户端流量

目录    表达式
详细信息    
in    　　sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)
out    　 sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)
摘要    
in     sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))
out    sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))

对等流量

目录    表达式
详细信息    
in    sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)
out    sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)
摘要    
in    sum(rate(etcd_network_peer_received_bytes_total[5m]))
out    sum(rate(etcd_network_peer_sent_bytes_total[5m]))

 

DB 大小

目录    表达式
详细信息    sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)
摘要    sum(etcd_debugging_mvcc_db_total_size_in_bytes)

 

RPC 速率

目录    表达式
详细信息    
total    sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)
fail    sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)
摘要    
total    sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))
fail    sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))

 

磁盘操作

目录    表达式
详细信息    
commit-called-by-backend    sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)
fsync-called-by-wal   　　   sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)
摘要    
commit-called-by-backend    sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))
fsync-called-by-wal   　　   sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))

 

磁盘同步持续时间

目录    表达式
详细信息    
wal    histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))
db    histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))
摘要    
wal    sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))
db    sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))

 

Kubernetes 组件指标

API Server 请求延迟

目录    　　表达式
详细信息    avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06
摘要    　　avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06

 

API Server 请求率

目录    表达式
详细信息    sum(rate(apiserver_request_count[5m])) by (instance, code)
摘要    　　sum(rate(apiserver_request_count[5m])) by (instance)

 

调度失败的 Pod

目录    表达式
详细信息    sum(kube_pod_status_scheduled{condition="false"})
摘要    　　sum(kube_pod_status_scheduled{condition="false"})

 

控制器管理器队列深度

目录    表达式
详细信息    
volumes    sum(volumes_depth) by instance
deployment    sum(deployment_depth) by instance
replicaset    sum(replicaset_depth) by instance
service    sum(service_depth) by instance
serviceaccount    sum(serviceaccount_depth) by instance
endpoint    sum(endpoint_depth) by instance
daemonset    sum(daemonset_depth) by instance
statefulset    sum(statefulset_depth) by instance
replicationmanager    sum(replicationmanager_depth) by instance
摘要    
volumes    sum(volumes_depth)
deployment    sum(deployment_depth)
replicaset    sum(replicaset_depth)
service    sum(service_depth)
serviceaccount    sum(serviceaccount_depth)
endpoint    sum(endpoint_depth)
daemonset    sum(daemonset_depth)
statefulset    sum(statefulset_depth)
replicationmanager    sum(replicationmanager_depth)

 

调度器 E2E 调度延迟

目录    表达式
详细信息    histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06
摘要   　　 sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)

 

调度程序抢占尝试

目录    表达式
详细信息    sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)
摘要   　　 sum(rate(scheduler_total_preemption_attempts[5m]))

Ingress Controller 连接数

目录    表达式
详细信息    
reading    sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)
waiting    sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)
writing    sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)
accepted    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)
active    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)
handled    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)
摘要    
reading    sum(nginx_ingress_controller_nginx_process_connections{state="reading"})
waiting    sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})
writing    sum(nginx_ingress_controller_nginx_process_connections{state="writing"})
accepted    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))
active    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))
handled    sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))

 

Ingress Controller 请求处理时间

目录    　　表达式
详细信息    topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))
摘要    　　topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))

 

Workload 指标

Workload CPU 利用率

目录    　　　　　　　　　　表达式
详细信息    
cfs throttled seconds    sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
user seconds    　　　　  sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
system seconds    　　　　sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
usage seconds    　　　　 sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
摘要    
cfs throttled seconds    sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
user seconds    　　　　  sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
system seconds    　　　　sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
usage seconds    　　　　 sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))

 

Workload 内存利用率

目录    　　表达式
详细信息    sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)
摘要    　　sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})

Workload 网络数据包

目录    表达式
详细信息    
receive-packets    sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
receive-dropped    sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
receive-errors    sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
transmit-packets    sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
transmit-dropped    sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
transmit-errors    sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)
摘要    
receive-packets    sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
receive-dropped    sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
receive-errors    sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
transmit-packets    sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
transmit-dropped    sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))
transmit-errors    sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))

 

Pod 指标

Pod CPU 利用率

目录    表达式
详细信息    
cfs throttled seconds    sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)
usage seconds    sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)
system seconds    sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)
user seconds    sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)
摘要    
cfs throttled seconds    sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))
usage seconds    sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))
system seconds    sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))
user seconds    sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))

 

Pod 内存利用率

目录   　　 表达式
详细信息    sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)
摘要    　　sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})

 

Pod 网络数据包

目录    表达式
详细信息    
receive-packets    sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
receive-dropped    sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
receive-errors    sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-packets    sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-dropped    sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-errors    sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
摘要    
receive-packets    sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
receive-dropped    sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
receive-errors    sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-packets    sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-dropped    sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))
transmit-errors    sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))

 

Container 指标

Container CPU 利用率

目录    表达式
cfs throttled seconds    sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))
usage seconds    sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))
system seconds    sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))
user seconds    sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))

 

Container 内存利用率

sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})

 

Container 磁盘 I/O

目录    表达式
read    sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))
write    sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))

 

摘自

https://docs.rancher.cn/docs/rancher2.5/monitoring-alerting/expression/_index

 

标签：container,name,sum,namespace,instance,prometheus,监控,rancher,total
From： https://www.cnblogs.com/fengjian2016/p/16969040.html