标签：Haproxy exporter tomcat Tomcat redis Nginx prometheus root string

服务监控

可以根据prometheus官网找到对应服务的exporter

https://prometheus.io/docs/instrumenting/exporters/

监控Tomcat容器

基于第三方 exporter 实现对目的服务的监控

github项目地址：https://github.com/nlighten/tomcat_exporter

所需要的Jar包获取

选择对应版本采集服务程序包，点击下载按钮，下载则是最新版本

如果想下载以往版本解决兼容某些tomcat问题，则点击版本后进入详细版本页面进行下载

根据测试，simpleclient程序包使用0.12.0开始以后版本不对8.5.x 的tomcat兼容，无法显示metrics页面，但tomcat-exporter程序包可以使用最新版本

Jar监控程序包列表：

simpleclient-0.8.0.jar

simpleclient_common-0.8.0.jar

simpleclient_hotspot-0.8.0.jar

simpleclient_servlet-0.8.0.jar

tomcat_exporter_client-0.0.17.jar

tomcat_exporter_servlet-0.0.17.war

并将tomcat_exporter_servlet-0.0.17.war修改为metrics进行tomcat页面发布

监控 tomcat 的活跃连接数、堆栈内存等信息：

# TYPE tomcat_connections_active_total gauge
tomcat_connections_active_total{name="http-nio-8080",} 2.0
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 2.4451216E7

制作tomcat镜像，将包含metrics监控指标的jar包导入tomcat镜像内

root@deploy:~/dockerfile\# cat Dockerfile 
FROM harbor.cncf.net/web/tomcat:8.5.43
  
MAINTAINER LXH

LABEL Description="tomcat-8.5.43-app1"


ADD metrics.war /usr/local/tomcat/webapps/

ADD simpleclient-0.8.0.jar /usr/local/tomcat/lib

ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib

ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib

ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib

ADD tomcat_exporter_client-0.0.17.jar /usr/local/tomcat/lib


EXPOSE 8080 8443

容器镜像构建脚本

root@deploy:~/dockerfile\# cat build.sh 
#!/bin/bash
version=$1
docker build -t tomcatapp:${version} ./
docker tag tomcatapp:${version} harbor.cncf.net/web/tomcatapp:${version}
docker push harbor.cncf.net/web/tomcatapp:${version}

编写tomcat k8s yaml文件

k8s部署监控相关的组件的时候需要初始化很多linux的内核参数。但是文件系统挂载到pod容器中就会变成read-only，难以进行操作实现需求。所以需要给POD privileged权限，然后在容器的初始化脚本或代码中去修改sysctl参数。给容器的spec指定`securityContext.privileged=true`参数

prometheus-server如果使用k8s内部部署并配置服务发现，则需要添加prometheus的scrape注解`annotations.prometheus.io/scrape='true'`

root@master1:~/yaml/tomcat-exporter# cat tomcat-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: default
spec:
  selector: 
    matchLabels: 
     app: tomcat
  replicas: 1 
  template: 
    metadata:
      labels:
        app: tomcat
      annotations:
        prometheus.io/scrape: 'true'     #添加prometheus服务发现的抓取注解
    spec:
      containers:
      - name: tomcat
        image: harbor.cncf.net/web/tomcatapp:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        securityContext: 
          privileged: true             #给POD privileged权限

部署tomcat server，如果svc也想被prometheus-server服务发现采集，也许要添加prometheus-server服务注解

root@master1:~/yaml/tomcat-exporter# cat tomcat-svc.yaml 
kind: Service 
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'   #添加prometheus服务发现的抓取注解
  name: tomcat-service
spec:
  selector:
    app: tomcat
  ports:
  - nodePort: 31080
    port: 80
    protocol: TCP
    targetPort: 8080
  type: NodePort

验证：

root@master1:~/yaml/tomcat-exporter\# kubectl get pods -n default 
NAME                                READY   STATUS    RESTARTS   AGE
tomcat-deployment-b8c8457cf-fgztx   1/1     Running   0          14s
root@master1:~/yaml/tomcat-exporter\# kubectl get svc -n default 
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   17d

访问tomcat页面和指标

配置prometheus-server配置文件，添加tomcat-exporter

root@prometheus:/usr/local/prometheus# vim prometheus.yml
- job_name: "tomcat"
    static_configs:
      - targets: ["192.168.100.3:31080"]
      
root@prometheus:/usr/local/prometheus# systemctl restart prometheus

部署导入grafana官方模板

https://github.com/nlighten/tomcat_exporter/tree/master/dashboard

复制example.json导入grafana

监控Redis

通过 redis_exporter 监控 redis 服务状态。

官方exporter：

prometheus官网 https://prometheus.io/docs/instrumenting/exporters/ ,找到redis_exporter

redis_exporter的GIthub项目地址 https://github.com/oliver006/redis_exporter

下载redis-exporter

https://github.com/oliver006/redis_exporter/releases

解压部署redis_exporter

[root@redis1 local]\# tar xf redis_exporter-v1.43.1.linux-amd64.tar.gz

redis-exporter会默认监听本机localhost:6379端口，来返回指标数据/metrics。启动默认监听9121端口

命令帮助信息

root@redis1 redis_exporter]\# ./redis_exporter --help
Usage of ./redis_exporter:
  -check-key-groups string
    	Comma separated list of lua regex for grouping keys
  -check-keys string
    	Comma separated list of key-patterns to export value and length/size, searched for with SCAN
  -check-keys-batch-size int
    	Approximate number of keys to process in each execution, larger value speeds up scanning.
    	WARNING: Still Redis is a single-threaded app, huge COUNT can affect production environment. (default 1000)
  -check-single-keys string
    	Comma separated list of single keys to export value and length/size
  -check-single-streams string
    	Comma separated list of single streams to export info about streams, groups and consumers
  -check-streams string
    	Comma separated list of stream-patterns to export info about streams, groups and consumers, searched for with SCAN
  -config-command string
    	What to use for the CONFIG command (default "CONFIG")
  -connection-timeout string
    	Timeout for connection to Redis instance (default "15s")
  -count-keys string
    	Comma separated list of patterns to count (eg: 'db0=production_*,db3=sessions:*'), searched for with SCAN
  -debug
    	Output verbose debug information
  -export-client-list
    	Whether to scrape Client List specific metrics
  -export-client-port
    	Whether to include the client's port when exporting the client list. Warning: including the port increases the number of metrics generated and will make your Prometheus server take up more memory
  -include-config-metrics
    	Whether to include all config settings as metrics
  -include-system-metrics
    	Whether to include system metrics like e.g. redis_total_system_memory_bytes
  -is-cluster
    	Whether this is a redis cluster (Enable this if you need to fetch key level data on a Redis Cluster).
  -is-tile38
    	Whether to scrape Tile38 specific metrics
  -log-format string
    	Log format, valid options are txt and json (default "txt")
  -max-distinct-key-groups int
    	The maximum number of distinct key groups with the most memory utilization to present as distinct metrics per database, the leftover key groups will be aggregated in the 'overflow' bucket (default 100)
  -namespace string
    	Namespace for metrics (default "redis")
  -ping-on-connect
    	Whether to ping the redis instance after connecting
  -redact-config-metrics
    	Whether to redact config settings that include potentially sensitive information like passwords (default true)
  -redis-only-metrics
    	Whether to also export go runtime metrics
  -redis.addr string
    	Address of the Redis instance to scrape (default "redis://localhost:6379")
  -redis.password string
    	Password of the Redis instance to scrape
  -redis.password-file string
    	Password file of the Redis instance to scrape
  -redis.user string
    	User name to use for authentication (Redis ACL for Redis 6.0 and newer)
  -script string
    	Path to Lua Redis script for collecting extra metrics
  -set-client-name
    	Whether to set client name to redis_exporter (default true)
  -skip-tls-verification
    	Whether to to skip TLS verification
  -tls-ca-cert-file string
    	Name of the CA certificate file (including full path) if the server requires TLS client authentication
  -tls-client-cert-file string
    	Name of the client certificate file (including full path) if the server requires TLS client authentication
  -tls-client-key-file string
    	Name of the client key file (including full path) if the server requires TLS client authentication
  -tls-server-ca-cert-file string
    	Name of the CA certificate file (including full path) if the web interface and telemetry should require TLS client authentication
  -tls-server-cert-file string
    	Name of the server certificate file (including full path) if the web interface and telemetry should use TLS
  -tls-server-key-file string
    	Name of the server key file (including full path) if the web interface and telemetry should use TLS
  -tls-server-min-version string
    	Minimum TLS version that is acceptable by the web interface and telemetry when using TLS (default "TLS1.2")
  -version
    	Show version information and exit
  -web.listen-address string
    	Address to listen on for web interface and telemetry. (default ":9121")
  -web.telemetry-path string
    	Path under which to expose metrics. (default "/metrics")

常用选项：

-redis.addr：连接redis地址:端口

-redis.password：连接redis的密码

-web.listen-address：指定redis_exporter的服务端口，格式为-web.listen-address ":9121

启动脚本

使redis_exporter监听在自定义端口9333,默认端口监听为9121

[root@redis1 redis_exporter]\# cat start.sh 
#!/bin/bash
cmd=/usr/local/redis_exporter
ip=192.168.100.104   #reidis地址
password=123456      #redis密码
nohup $cmd/redis_exporter -redis.addr $ip:6379 -redis.password $password -web.listen-address ":9333"  > /dev/null 2>&1 &
[ $? -eq 0 ] &&  echo "start success!" || echo "start failed"

停止脚本

root@redis1 redis_exporter]\# cat stop.sh 
#!/bin/bash
PID=$(ps -ef|grep redis_exporter|grep -v grep|awk '{print $2}')

if [ -z ${PID} ];then
  echo "redis_expoter isn't running"
else 
  kill $PID
  sleep 2s
  [ ! -z ${PID} ]
  echo "redis_expoter is stopped"
fi

监控redis单实例

编辑启动脚本后，运行启动脚本

[root@redis1 redis_exporter]\# cat start.sh
#!/bin/bash
cmd=/usr/local/redis_exporter
ip=192.168.100.104   #reidis地址
password=123456      #redis密码
nohup $cmd/redis_exporter -redis.addr $ip:6379 -redis.password $password -web.listen-address ":9333"  > /dev/null 2>&1 &
[ $? -eq 0 ] &&  echo "start success!" || echo "start failed"

[root@redis1 redis_exporter]\# ./start.sh 
start success!

访问验证

prometheus-server配置

scrape_configs:
  - job_name: redis_exporter
    static_configs:
    - targets: ['192.168.100.104:9333']

监控redis集群多实例

查看集群信息

分别为192.168.100.104上的6379-6384端口的6个redis为集群

官网示例：

https://github.com/oliver006/redis_exporter

Using below config instead of the /metric endpoint the /scrape endpoint will be used by prometheus. As an example the first target will be queried with this web request:

http://exporterhost:9121/scrape?target=first-redis-host:6379

解释

1、创建一个名称为redis_exporter_targets job，来获取每个redis节点的实例。

将采集redis_expoter Http URI修改为/scrape，原因是默认prometheus会收集目标target的/metrics URI，所以需要在prometheus-server配置文件中指定metrics_path 的路径，并进行redis_exporter的标签重写，将\__address\__ 标签内容为`redis实例ip:6379、6380、6381`节点实例,替换为替换为instance标签，并将value修改为`redis_exporter http`的访问地址。最终prometheus-server就能获得每个redis的target：`http://exporterhost:9121/scrape?target=first-redis-host:6379`

2、还需要单独添加一个名称为redis_exporter job来抓取redis_exporter的地址

scrape_configs:
  ## config for the multiple Redis targets that the exporter will scrape
  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://first-redis-host:6379
        - redis://second-redis-host:6379
        - redis://second-redis-host:6380
        - redis://second-redis-host:6381
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121

  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - <<REDIS-EXPORTER-HOSTNAME>>:9121

redis_expoter配置

redis_expoter只要能连接到一个集群的一个节点，因此只需要配置连接集群中一个redis节点就能查询其他节点的指标

[root@redis1 redis_exporter]\# cat start.sh 
#!/bin/bash
cmd=/usr/local/redis_exporter
ip=192.168.100.104
password=123456
nohup $cmd/redis_exporter -redis.addr $ip:6379 -redis.password $password -web.listen-address ":9333"  > /dev/null 2>&1 &
[ $? -eq 0 ] &&  echo "start success!" || echo "start failed"

prometheus-server配置

root@prometheus:~\# cat /usr/local/prometheus/prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:

  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://192.168.100.104:6379
        - redis://192.168.100.104:6380
        - redis://192.168.100.104:6381
        - redis://192.168.100.104:6382
        - redis://192.168.100.104:6383
        - redis://192.168.100.104:6384
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.100.104:9333
  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - 192.168.100.104:9333

访问prometheus-server web页面

将标签重写后，URI修改为/scripe?target=redis节点:6379

配置grafana导入redis-exporter模板

ID：11835

监控k8s redis集群

这里每个redis节点使用statefulset部署，并暴露集群端口供redis_exporter使用。redis_expoter单独使用deployment部署

部署redis集群参考kubernetes 案例redis集群配置

验证redis集群

查看redis集群pod

查看集群状态

redis_exporter镜像构建

root@deploy:~/dockerfile/redis-exporter\# cat Dockerfile 
FROM harbor.cncf.net/os/ubuntu:20.04

ADD redis_exporter.tar.gz /usr/local/

ENTRYPOINT ["/usr/local/redis_exporter/start.sh"]

准备redis_exporter程序,下载后解压并配置

root@deploy:~/dockerfile/redis-exporter\# ll redis_exporter/
total 8892
drwxr-xr-x 2 root root      91 Sep 24 03:29 ./
drwxr-xr-x 3 root root      91 Sep 24 03:30 ../
-rw-r--r-- 1 root root    1063 Sep 24 03:10 LICENSE
-rw-r--r-- 1 root root   43382 Sep 24 03:10 README.md
-rwxr-xr-x 1 root root 9048064 Sep 24 03:10 redis_exporter*
-rwx------ 1 root root     202 Sep 24 03:29 start.sh*
-rwx------ 1 root root     224 Sep 24 03:10 stop.sh*

准备redis_exporter启动脚本

redis的ip地址修改为k8s中redis集群中的master节点的pod服务名称，redis-0.redis.redis-test.svc.cluster.local

root@deploy:~/dockerfile/redis-exporter\# cat redis_exporter/start.sh 
#!/bin/bash
cmd=/usr/local/redis_exporter
ip="redis-0.redis.redis-test.svc.cluster.local"
password=123456
$cmd/redis_exporter -redis.addr $ip:6379 -redis.password $password -web.listen-address ":9121"

准备镜像构建脚本

root@deploy:~/dockerfile/redis-exporter\# cat build.sh 
#!/bin/bash
version=$1
docker build -t harbor.cncf.net/project/redis_exporter:$version ./
docker push harbor.cncf.net/project/redis_exporter:$version

部署redis_exporter

创建deployment，与redis集群在同一个ns下

root@master1:~/yaml/redis-exporter# cat redis-exporter.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-exporter
  namespace: redis-test 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-exporter
  template:
    metadata:
      labels:
        app: redis-exporter
    spec:
      containers:
      - name: redis-exporter
        image: harbor.cncf.net/project/redis_exporter:1.7
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121

创建svc

root@master1:~/yaml/redis-exporter# cat redis-exporter-svc.yaml 
kind: Service  #service 类型
apiVersion: v1
metadata:
  name: redis-redis-service
  namespace: redis-test 
spec:
  selector:
    app: redis-exporter
  ports:
  - nodePort: 39121
    name: redis-exporter
    port: 9121
    protocol: TCP
    targetPort: 9121
  type: NodePort

验证部署

root@master1:~/yaml\# kubectl get svc -n redis-test redis-redis-service 
NAME                  TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
redis-redis-service   NodePort   10.100.135.230   <none>        9121:39121/TCP   99m

root@master1:~/yaml\# kubectl get pods -n redis-test 
NAME                             READY   STATUS    RESTARTS   AGE
redis-0                          1/1     Running   0          134m
redis-1                          1/1     Running   0          142m
redis-2                          1/1     Running   0          142m
redis-3                          1/1     Running   0          142m
redis-4                          1/1     Running   0          142m
redis-5                          1/1     Running   0          126m
redis-exporter-bf6bf8576-k9p4x   1/1     Running   0          101m

验证查看redis_exporter web界面

配置k8s中的prometheus-server，添加redis-exporter服务发现

target为redis-exporter的svc名称

    - job_name: 'redis_exporter'
      static_configs:
        - targets:
          - redis-redis-service.redis-test:9121

配置后重新应用promentheus configmap并删除prometheus-server pod重启

root@master1:~/yaml\# kubectl apply -f prometheus-cfg.yaml 
configmap/prometheus-config configured
root@master1:~/yaml\# kubectl delete -f prometheus-deployment.yaml 
deployment.apps "prometheus-server" deleted
root@master1:~/yaml\# kubectl apply -f prometheus-deployment.yaml

查看prometheus target界面验证target

查看grafana导入dashboard

监控mysql

通过 mysqld_exporter 监控 MySQL 服务的运行状态

官方exporter：

prometheus官网: https://prometheus.io/docs/instrumenting/exporters/，找到mysql_exporer

Github项目地址：https://github.com/prometheus/mysqld_exporter

监听MySQL单实例

1、确保监控的mysql实例正常运行，添加授权监控账号权限。

已经安装mysql服务，端口监听在3307

CREATE USER 'exporter'@'localhost' IDENTIFIED BY '123456' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

测试账号权限

root@mysql1:~\# mysql -u exporter -p
#输入密码
mysql> select user();
+--------------------+
| user()             |
+--------------------+
| exporter@localhost |
+--------------------+
1 row in set (0.00 sec)

2、部署mysql_expoter，安装在mysql服务本地

下载地址：https://github.com/prometheus/mysqld_exporter/releases

解压安装

root@mysql1:/usr/local\# tar xf mysqld_exporter-0.14.0.linux-amd64.tar.gz
root@mysql1:/usr/local\# ln -s mysqld_exporter-0.14.0.linux-amd64 mysql_exporter

添加mysql客户端文件，添加监控账户密码信息和mysql服务端信息，使mysql_exporter免交互登录mysql

root@mysql1:~\# vim /root/.my.cnf
[client]
host=localhost
port=3307
user = exporter
password = 123456

标签：Haproxy,exporter,tomcat,Tomcat,redis,Nginx,prometheus,root,string
From： https://www.cnblogs.com/punchlinux/p/16856793.html

prometheus 监控案例-Tomcat、Redis、Mysql、Haproxy、Nginx