1.收集nginx日志
学习背景:access.log,error.log目前日志混杂在一个es索引下。
改进filebeat配置
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html
加上日志输入文件的判断,采用不同的索引
1 filebeat.inputs:
2 - type: log
3 enabled: true
4 paths:
5 - /var/log/nginx/access.log
6 json.keys_uner_root: true
7 json.overwrite_keys: true
8
9
10 - type: log
11 enabled: true
12 paths:
13 - /var/log/nginx/error.log
14
15
16 output.elasticsearch:
17 hosts: ["http://localhost:9200"]
18 indices:
19 - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}"
20 when.contains:
21 log.file.path: "/var/log/nginx/access.log"
22 - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
23 when.contains:
24 log.file.path: "/var/log/nginx/error.log"
25
26
27 setup.ilm.enabled: false
28 setup.template.enabled: false
29
30 logging.level: info
31 logging.to_files: true
32 logging.files:
33 path: /var/log/filebeat
34 name: filebeat
35 keepfiles: 7
36 permissions: 0644
查看es数据
access.log
error.log
完成nginx日志拆分index
curl 127.0.0.1/laoliu
curl 127.0.0.1/laoliu
2.Kibana图形化定制
柱状图
选择可视化
选择图形模式
柱状图、x、y轴图形数据
多个客户端,记得点击save,保存图形
data table
可视化存档
饼图
- 客户端类型
- 操作系统类型
- http状态码
编辑dashboard
3.使用ES预处理节点转换Nginx日志
背景
1.我们可以提提前修改nginx日志为json格式(常见做法,省事)
2.不修改源数据,使用es提供的grok转换器(太麻烦)
3.更高级用法是用filebeat模块
https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html
先恢复你nginx日志为基础格式
systemctl stop filebeat
> /var/log/nginx/access.log
vim /etc/nginx/nginx.conf
systemctl restart nginx
curl 127.0.0.1
cat /var/log/nginx/access.log
nginx基础日志
127.0.0.1 - - [19/Feb/2023:11:53:59 +0800] "GET / HTTP/1.1" 200 9205 "-" "curl/7.29.0" "-"
127.0.0.1 - - [19/Feb/2023:11:53:59 +0800] "GET / HTTP/1.1" 200 9205 "-" "curl/7.29.0" "-"
Grok
https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html
GROK是一种采用组合多个预定义的正则表达式,用来匹配分割文本并映射到关键字的工具。通常用来对日志数据进行处理。本文档主要介绍GROK的模式说明以及常用语法。
GROK模式及说明如下表所示。
https://help.aliyun.com/document_detail/129387.html
https://developer.qiniu.com/insight/4759/grok-parser
测试
grok转换语法
127.0.0.1 ==> %{IP:clientip}
- ==> -
- ==> -
[08/Oct/2020:16:34:40 +0800] ==> \\[%{HTTPDATE:nginx.access.time}\\]
"GET / HTTP/1.1" ==> "%{DATA:nginx.access.info}"
200 ==> %{NUMBER:http.response.status_code:long}
5 ==> %{NUMBER:http.response.body.bytes:long}
"-" ==> "(-|%{DATA:http.request.referrer})"
"curl/7.29.0" ==> "(-|%{DATA:user_agent.original})"
"-" ==> "(-|%{IP:clientip})"
字符对应grok模式
%{IP:clientip} - - \[%{HTTPDATE:nginx.access.time}\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\"
转换结果
1.创建pipeline
GET _ingest/pipeline
PUT _ingest/pipeline/pipeline-nginx-access
{
"description" : "nginx access log",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{IP:clientip} - - \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\""]
}
},{
"remove": {
"field": "message"
}
}
]
}
2.修改filebeat
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
processors:
- drop_fields:
fields: ["ecs","log"] //删除不需要的字段
output.elasticsearch:
hosts: ["10.0.0.18:9200"]
pipelines:
- pipeline: "pipeline-nginx-access"
when.contains:
tags: "access"
indices:
- index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
setup.ilm.enabled: false
setup.template.enabled: false
logging.level: info
logging.to_files: true
例如删除如下字段
3.测试pipeline转换的nginx日志
清空,删除所有es 索引,注意测试用法,生产别执行。
[root@es-node1 ~]#curl -X DELETE 'http://localhost:9200/_all'
{"acknowledged":true}
[root@es-node1 ~]#systemctl restart filebeat.service
[root@es-node1 ~]#
[root@es-node1 ~]#systemctl restart filebeat.service
[root@es-node1 ~]#
注意再去访问nginx,产生日志,此时filebeat会抓取新日志数据,写入es,生成索引
[root@es-node1 ~]#curl 127.0.0.1
[root@es-node1 ~]#curl 127.0.0.1
[root@es-node1 ~]#curl 127.0.0.1
4.kibana可视化新格式es数据
5.小结
只有专门维护ELK的高级工程师,只维护ELK,需要大量且复杂的维护日志系统,需要手工写grok转换规则。
标签:ELK,filebeat,log,收集,access,nginx,日志,es From: https://www.cnblogs.com/sxy-blog/p/18128280