Alertmanager钉钉告警

一.部署alertmanager
1.1 创建目录

mkdir -p /deploy/app/alertmanager/config
chmod -R 777 /deploy/app/alertmanager/config

1.2部署alertmanager，也需要个配置文件，先部署临时容器，然后把配置文件复制下来

docker run -d -p 9093:9093 --name alertmanager prom/alertmanager
docker cp alertmanager:/etc/alertmanager/alertmanager.yml /deploy/app/alertmanager/config
docker rm -f alertmanager

1.3修改配置文件

global:
  #旧的告警3分钟没有更新，则认为告警解决        
  resolve_timeout: 3m

route:
  group_by: ['alertname'] #根据告警规则组名进行分组
  group_wait: 0s       #在组内等待所配置的时间，如果同组内，30秒内出现相同报警，在一个组内出现
  group_interval: 10s   #控制告警组的发送频率，一条告警消息发送后，等待10秒，发送第二组告警
  repeat_interval: 1h   #发送报警间隔，如果指定时间内没有修复，则重新发送报警
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs: #可设置多个接收方
  - url: 'http://192.168.80.86:8060/dingtalk/webhook1/send'
  - url: 'http://192.168.80.86:8060/dingtalk/webhook2/send' 
    send_resolved: true     # 表示服务恢复后会收到恢复告警

inhibit_rules:
  - source_match:    #配置了仰制告警
      severity: '非常严重'
    target_match:
      severity: '严重'
    equal: ['alertname',"target","job","instance"]

1.4正式部署

docker run -d \
-p 9093:9093 \
--name alertmanager \
--network-alias=alertmanager \
--restart=always \
-v "/deploy/app/alertmanager/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml" \
prom/alertmanager:v0.25.0

http://192.168.80.86:9093/#/status 打开链接能看到自己的配置即可

二.安装钉钉插件
2.1安装钉钉插件prometheus-webhook-dingtalk，和上述一样，就不解释了

创建目录
mkdir -p /deploy/app/prometheus-webhook-dingtalk/templates
chmod 777 -R /deploy/app/prometheus-webhook-dingtalk/templates

docker run -d -p 8060:8060 --name=dingding timonwong/prometheus-webhook-dingtalk
docker cp dingding:/etc/prometheus-webhook-dingtalk/config.yml /deploy/app/prometheus-webhook-dingtalk/
docker rm -f dingding

2.2修改/deploy/app/prometheus-webhook-dingtalk/config.yml

templates:
  - /etc/prometheus-webhook-dingtalk/templates/templates.tmpl

targets: #配置多个接收方
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=2170a4079a1008d52320d32d6d895a98e0ee70f93c409
    secret: SEC48c5d9fbe1b1634fde69b28027eebb86ecfeeccb74c0af5290e
    message:
      text: '{{ template "email.to.message" . }}'
  webhook2:
    url: https://oapi.dingtalk.com/robot/send?access_token=55e44b618ae703b3c3cde84d39404024618af0b77926dc227d6
    secret: SECc7d467845f4c44803a1f4b50935415afea42e0299183bc805506
    message:
      text: '{{ template "email.to.message" . }}'

2.3 修改/deploy/app/prometheus-webhook-dingtalk/templates/templates.tmpl

{{ define "email.to.message" }}
  
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}

<font color=#FF0000>=========  **监控告警** ========= </font>

**告警类型:**    {{ $alert.Labels.alertname }}
**告警级别:**    {{ $alert.Labels.severity }} 级
**故障主机:**    {{ $alert.Labels.instance }} {{ $alert.Labels.device }}
**告警详情:**    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}

**故障时间:**    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{ end }}{{ end }}

{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}

<font color=#00800>=========  **告警恢复** ========= </font>

**告警主机:**    {{ $alert.Labels.instance }}
**告警类型:**    {{ $alert.Labels.alertname }}
**告警级别:**    {{ $alert.Labels.severity }} 级
**告警详情:**    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}
**故障时间:**    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}

**恢复时间:**    {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}

{{ end }}{{ end }}{{ end }}

2.4正式部署

docker run -d \
-p 8060:8060 \
--name=dingding \
--restart=always \
--network-alias=dingding \
-v /deploy/app/prometheus-webhook-dingtalk:/etc/prometheus-webhook-dingtalk \
timonwong/prometheus-webhook-dingtalk

三,修改prometheus的配置
3.1编辑prometheus的告警规则文件，更多规则请点击下方链接
https://awesome-prometheus-alerts.grep.to/rules#host-and-hardware
vim /deploy/app/prometheus/rules/node_exporter.yml

vim /deploy/app/prometheus/rules/node_exporter.yml


groups:
- name: 服务器主机信息监控告警
  rules:
  - alert: 公司内部服务器监控
    expr: up {job="公司内部服务器"} == 0
    for: 0m
    labels:
      severity: 非常严重
    annotations:
      description: "监控的目标已丢失，请检查服务器自身或node_exporter服务"


  - alert: "内存报警"
    expr: 100 - ((node_memory_MemAvailable_bytes * 100) / node_memory_MemTotal_bytes) > 95
    for: 3m  # 告警持续时间，超过这个时间才会发送给alertmanager
    labels:
      severity: 严重
    annotations:
      summary: "{{ $labels.instance }} 内存使用率过高，请尽快处理！"
      description: "{{ $labels.instance }}内存使用率超过95%,当前使用率{{ $value }}%."


  - alert: "磁盘空间报警"
    expr: (1 - node_filesystem_avail_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100 > 95
    for: 60s
    labels:
      severity: 严重
    annotations:
      summary: "{{ $labels.instance }}磁盘空间使用超过95%了"
      description: "{{ $labels.instance }}磁盘使用率超过95%,当前使用率{{ $value }}%."

  - alert: "CPU报警"
    expr: 100-(avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance)* 100) > 95
    for: 120s
    labels:
      severity: 严重
      instance: "{{ $labels.instance }}"
    annotations:
      summary: "{{$labels.instance}}CPU使用率超过95%了"
      description: "{{ $labels.instance }}CPU使用率超过95%,当前使用率{{ $value }}%."

  - alert: "磁盘IO性能报警"
    expr: ((irate(node_disk_io_time_seconds_total[30m]))* 100) > 95
    for: 3m
    labels:
      severity: 严重
    annotations:
      summary: "{{$labels.instance}} 流入磁盘IO使用率过高，请尽快处理！"
      description: "{{$labels.instance}} 流入磁盘IO大于95%,当前使用率{{ $value }}%."

3.2

vim /deploy/app/prometheus/config/prometheus.yml

global:
  # 数据采集间隔
  scrape_interval:     45s
  # 告警检测间隔
  evaluation_interval: 45s


alerting:
  alertmanagers:
    - static_configs:
        - targets: ["192.168.80.86:9093"]

# 告警规则
rule_files:
  # 这里匹配指定目录下所有的.rules文件
  - /prometheus/rules/*.yml

# 采集配置
scrape_configs:
  # 采集项(prometheus)
  #- job_name: 'prometheus'
  #  static_configs:
  #    # prometheus自带了对自身的exporter监控程序，所以不需额外安装exporter就可配置采集项
  #    - targets: ['localhost:9090']

  - job_name: "公司内部服务器"
    file_sd_configs:    #基于文件发现服务方式，也可以基于Consul服务发现等，其它方式不讲解，又不给工资费劲
      - files:
        - /prometheus/ClientAll/*.json  #用json格式文件方式发现服务，下面的是用yaml格式文件方式，都可以，反正两个都部署了，随便你用哪个
        refresh_interval: 10m
      - files:
        - /prometheus/ClientAll/*.yaml
        refresh_interval: 10m

热加载

curl -X POST http://192.168.80.86:9090/-/reload

测试一下 停止某个节点的node_exporter
即可看到告警通知