Logstash分析Nginx日志

六、Logstash分析Nginx日志

实现思路：
1.将 Nginx 普通日志转换为 json
2.将 Nginx 日志的时间格式进行格式化输出
3.将 Nginx 日志的来源IP进行地域分析
4.将 Nginx 日志的 user-agent 字段进行分析
5.将 Nginx 日志的 bytes 修改为整数
6.移除没有用的字段，message、headers

6.1 架构

6.2 实现

日志格式一：
14.145.74.175 - - [10/Nov/2020:00:01:53 +0800] "POST /course/ajaxmediauser/ HTTP/1.1" 200 54 "www.oldxu.com" 
"http://www.oldxu.com/video/678" mid=678&time=60&learn_time=551.5 
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" "-" 10.100.136.64:80 200 0.014 0.014
所用的grok语法
 grok {
            match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:hostname} (?:%{QS:referrer}|-) (?:%{NOTSPACE:post_args}|-) %{QS:useragent} (?:%{QS:x_forward_for}|-) (?:%{URIHOST:upstream_host}|-) (?:%{NUMBER:upstream_response_code}|-) (?:%{NUMBER:upstream_response_time}|-) (?:%{NUMBER:response_time}|-)" }
 }

日志格式2：
123.150.183.45 - - [22/Nov/2015:12:01:01 +0800] "GET /online/ppjonline/images/forms/validatePass.png HTTP/1.1" 200 
370 "http://www.papaonline.com.cn/online/ppjonline/order/orderNow.jsp" 
"Mozilla/5.0 (Linux; U; Android 4.3; zh-CN; SCH-N719 Build/JSS15J) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 UCBrowser/9.9.5.489 U3/0.8.0 Mobile Safari/533.1"
所用的grok语法：
	grok {
		match => {
			"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:useragent}"
		}
	}

6.2.1 配置filebeat

cat filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths: /var/log/nginx/access.log
  tags: ["inginx-access"]
- type: log
  enabled: true
  paths: /var/log/nginx/error.log
  tags: ["nginx-error"]

output.logstash:
  hosts: ["172.16.1.151:5044"]

6.2.2 配置logstash

cat nginx_beat_logstash_es.conf 
input {

	beats {
		port => 5044
	}
}

filter {
  if "nginx-access" in [tags][0]{	
	grok {
		match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:hostname} (?:%{QS:referrer}|-) (?:%{NOTSPACE:post_args}|-) %{QS:useragent} (?:%{QS:x_forward_for}|-) (?:%{URIHOST:upstream_host}|-) (?:%{NUMBER:upstream_response_code}|-) (?:%{NUMBER:upstream_response_time}|-) (?:%{NUMBER:response_time}|-)"}
	}
	useragent {
		source => "useragent"
		target => "useragent"
	}
	geoip {
		source => "clientip"
	}
        date {
            match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
            timezone => "Asia/Shanghai"
        }
        
        mutate {
         convert => ["bytes","integer"]
         convert => ["response_time", "float"]
         convert => ["upstream_response_time", "float"]
         remove_field => ["message","agent"]
         add_field => { "target_index" => "logstash-nginx-access-%{+YYYY.MM.dd}"}
	}
   }else if "nginx-error" in [tags][0]{
	mutate {
		add_field => { "target_index" => "logstash-nginx-error-%{+YYYY.MM.dd}" }
		remove_field => ["agent"]
	}     
  }
}

output {
	stdout {
		codec => rubydebug
	}
	elasticsearch {
		hosts => ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
		index => "%{[target_index]}"
	}
}

6.2.3 kibana生成图

1.总访问次数
2.独立ip总量
3.总产生流量
4.柱状图，时间段访问次数
5.柱状图，时间段流量趋势
6.饼图，浏览器设备、状态码、手机设备