一：安装部分：
https://prometheus.io/download/ ###下载源码解压即可
https://grafana.com/grafana/dashboards ###搜索数据源为prometheus的
这里下载了：prometheus、node_exporter、alertmanager、pushgateway
同时机器需要安装docker
yum install docker -y systemctl start docker.service安装gragana：
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.3.4-1.x86_64.rpm yum localinstall grafana-5.3.4-1.x86_64.rpm systemctl start grafana-server 二 ：配置:1、prometheus配置：
global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. scrape_timeout: 10s alerting: alertmanagers: - static_configs: - targets: - 127.0.0.1:9093 rule_files: - "./rules/rule_*.yml" # - "second_rules.yml" scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' ###这个必须配置，这个地址抓取的所有数据会自动加上`job=prometheus`的标签 # metrics_path defaults to '/metrics' #抓取监控目标的路径，默认是/metrics 可以根据自己业务的需要进行修改 # scheme defaults to 'http'. static_configs: #这是通过静态文件的配置方法：这种方法直接指定要抓去目标的ip和端口 - targets: ['localhost:9090'] - job_name: gateway static_configs: - targets: ['127.0.0.1:9091'] labels: ## 打上标签，instance会被指定为‘gataway’ instance: gataway - job_name: node_export file_sd_configs: #refresh_interval: 1m #刷新发现文件的时间间隔 - files: - /data/prometheus-2.12.0.linux-amd64/node_discovery.json - job_name: mysql_discovery file_sd_configs: #refresh_interval: 1m #刷新发现文件的时间间隔 - files: - /data/prometheus-2.12.0.linux-amd64/mysql_discovery.json - job_name: redis_discovery file_sd_configs: #refresh_interval: 1m #刷新发现文件的时间间隔 - files: - /data/prometheus-2.12.0.linux-amd64/redis_discovery.json各种discovery的模版为：
[{"targets": ["127.0.0.1:9100"],"labels": {"instance": "test","idc": "beijing"}},{"targets": ["127.0.0.1:9101"],"labels": {"instance": "test2","idc": "beijing"}}]
2、alertmanager配置：
global: resolve_timeout: 5m #templates: # - 'demo.tmpl' route: receiver: webhook group_wait: 30s group_interval: 5m repeat_interval: 10m group_by: ['alertname'] routes: - receiver: webhook group_wait: 10s match: job_name: mysql|kubernetes - receiver: 'webhook-kafka' group_by: [instance, alertname] match_re: instance: ^kafka-(.*) receivers: - name: webhook webhook_configs: - url: http://localhost:8060/dingtalk/ops_dingding/send send_resolved: true - name: webhook-kafka webhook_configs: - url: http://localhost:8062/dingtalk/ops_59/send send_resolved: true
####备注：尽量使用动态发现的配置，以免配置文件过长
- job_name: mysql_discovery file_sd_configs: #refresh_interval: 1m #刷新发现文件的时间间隔 - files: - /data/prometheus-2.12.0.linux-amd64/redis_discovery.json 三：启动服务：1、promethues的server启动：
nohup sh start.sh 2>&1 > prometheus.log &--start.sh内容为
./prometheus --storage.tsdb.path=./data --storage.tsdb.retention.time=168h --web.enable-lifecycle --storage.tsdb.no-lockfil
2、alertmanager的启动：
nohup sh start.sh 2>&1 > alertmanager.log &--start.sh内容为
./alertmanager --config.file="alertmanager.yml"其余服务的启动
nohup ./node_exporter & nohup ./pushgateway & -------------------------- 启动云数据库的采集服务： docker run -d \ -p 9104:9104 \ -e DATA_SOURCE_NAME="user:password@(url)/" \ prom/mysqld-exporter docker run -d \ -p 9121:9121 \ -e REDIS_ADDR="redis://url:port" \ -e REDIS_PASSWORD="password" \ oliver006/redis_exporter 如果有多组 只要修改外部暴漏端口和连接信息等等就可以了
-----报警规则配置：
rule_node.yml
groups: - name: 主机状态-监控告警 rules: - alert: 主机状态 expr: up == 0 for: 1m labels: status: 非常严重 annotations: summary: "{{$labels.instance}}:服务器宕机" description: "{{$labels.instance}}:服务器延时超过5分钟" - alert: CPU使用情况 expr: 100-(avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance)* 100) > 60 for: 1m labels: status: 一般告警 annotations: summary: "{{$labels.mountpoint}} CPU使用率过高！" description: "{{$labels.mountpoint }} CPU使用大于60%(目前使用:{{$value}}%)" - alert: 内存使用 expr: 100 -(node_memory_MemTotal_bytes -node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes ) / node_memory_MemTotal_bytes * 100> 80 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} 内存使用率过高！" description: "{{$labels.mountpoint }} 内存使用大于80%(目前使用:{{$value}}%)" - alert: IO性能 expr: 100-(avg(irate(node_disk_io_time_seconds_total[1m])) by(instance)* 100) < 60 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} 流入磁盘IO使用率过高！" description: "{{$labels.mountpoint }} 流入磁盘IO大于60%(目前使用:{{$value}})" - alert: 网络 expr: ((sum(rate (node_network_receive_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} 流入网络带宽过高！" description: "{{$labels.mountpoint }}流入网络带宽持续2分钟高于100M. RX带宽使用率{{$value}}" - alert: 网络 expr: ((sum(rate (node_network_transmit_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} 流出网络带宽过高！" description: "{{$labels.mountpoint }}流出网络带宽持续2分钟高于100M. RX带宽使用率{{$value}}" - alert: TCP会话 expr: node_netstat_Tcp_CurrEstab > 1000 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} TCP_ESTABLISHED过高！" description: "{{$labels.mountpoint }} TCP_ESTABLISHED大于1000%(目前使用:{{$value}}%)" - alert: 磁盘容量 expr: 100-(node_filesystem_free_bytes{fstype=~"ext4|xfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs"}*100) > 80 for: 1m labels: status: 严重告警 annotations: summary: "{{$labels.mountpoint}} 磁盘分区使用率过高！" description: "{{$labels.mountpoint }} 磁盘分区使用大于80%(目前使用:{{$value}}%)"
rule_mysql.yml
groups: - name: MySQLStatsAlert rules: - alert: MySQL is down expr: up{job="mysql-discorvery"} == 0 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} MySQL is down" description: "MySQL database is down. This requires immediate action!" - alert: Read buffer size is bigger than max. allowed packet size expr: mysql_global_variables_read_buffer_size > mysql_global_variables_slave_max_allowed_packet for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Read buffer size is bigger than max. allowed packet size" description: "Read buffer size (read_buffer_size) is bigger than max. allowed packet size (max_allowed_packet).This can break your replication." - alert: Sort buffer possibly missconfigured expr: mysql_global_variables_innodb_sort_buffer_size <256*1024 or mysql_global_variables_read_buffer_size > 4*1024*1024 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Sort buffer possibly missconfigured" description: "Sort buffer size is either too big or too small. A good value for sort_buffer_size is between 256k and 4M." - alert: Thread stack size is too small expr: mysql_global_variables_thread_stack <196608 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Thread stack size is too small" description: "Thread stack size is too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size." - alert: Used more than 90% of max connections limited expr: mysql_global_status_threads_connected > mysql_global_variables_max_connections * 0.8 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Used more than 80% of max connections limited" description: "Used more than 80% of max connections limited" - alert: InnoDB Force Recovery is enabled expr: mysql_global_variables_innodb_force_recovery != 0 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} InnoDB Force Recovery is enabled" description: "InnoDB Force Recovery is enabled. This mode should be used for data recovery purposes only. It prohibits writing to the data." - alert: InnoDB Log File size is too small expr: mysql_global_variables_innodb_log_file_size < 16777216 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} InnoDB Log File size is too small" description: "The InnoDB Log File size is possibly too small. Choosing a small InnoDB Log File size can have significant performance impacts." - alert: Table definition cache too small expr: mysql_global_status_open_table_definitions > mysql_global_variables_table_definition_cache for: 1m labels: severity: page annotations: summary: "Instance {{ $labels.instance }} Table definition cache too small" description: "Your Table Definition Cache is possibly too small. If it is much too small this can have significant performance impacts!" - alert: Thread stack size is possibly too small expr: mysql_global_variables_thread_stack < 262144 for: 1m labels: severity: page annotations: summary: "Instance {{ $labels.instance }} Thread stack size is possibly too small" description: "Thread stack size is possibly too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size." - alert: InnoDB Plugin is enabled expr: mysql_global_variables_ignore_builtin_innodb == 1 for: 1m labels: severity: page annotations: summary: "Instance {{ $labels.instance }} InnoDB Plugin is enabled" description: "InnoDB Plugin is enabled" - alert: Binary Log is disabled expr: mysql_global_variables_log_bin != 1 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Binary Log is disabled" description: "Binary Log is disabled. This prohibits you to do Point in Time Recovery (PiTR)." - alert: IO thread stopped expr: mysql_slave_status_slave_io_running != 1 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} IO thread stopped" description: "IO thread has stopped. This is usually because it cannot connect to the Master any more." - alert: SQL thread stopped expr: mysql_slave_status_slave_sql_running == 0 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} SQL thread stopped" description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master." - alert: SQL thread stopped expr: mysql_slave_status_slave_sql_running != 1 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} Sync Binlog is enabled" description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master." - alert: Slave lagging behind Master expr: rate(mysql_slave_status_seconds_behind_master[1m]) >30 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} Slave lagging behind Master" description: "Slave is lagging behind Master. Please check if Slave threads are running and if there are some performance issues!" - alert: Instance has slow logs expr: irate(mysql_global_status_slow_queries[5m]) > 10 for: 1m labels: severity: worning annotations: summary: "Instance {{ $labels.instance }} has slow log" description: "slow log"rule_redis.yml:
groups: - name: MySQLStatsAlert rules: - alert: redis is down expr: up{job="redis-discorvery"} == 0 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} MySQL is down" description: "MySQL database is down. This requires immediate action!" - alert: redis memory alert expr: 100 * (redis_memory_used_bytes{instance !~ "pro-sas|pro-redis-dun"} / redis_config_maxmemory{instance !~ "pro-sas|pro-redis-dun"} ) > 90 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} memory userd over 90" description: "redis memory short"写在最后：
prometheus监控k8s:
http://note.youdao.com/noteshare?id=dbbc868d32835cab5eac5a455df243ed
prometheus钉钉报警:
https://github.com/timonwong/prometheus-webhook-dingtalk
标签：实战,description,labels,alert,instance,prometheus,1m,annotations From： https://www.cnblogs.com/xuxuxu-2022/p/17571555.html