首页 > 其他分享 >Logstash 入门实战(2)--安装及使用

Logstash 入门实战(2)--安装及使用

时间：2022-09-25 09:33:34浏览次数：47  
标签：入门 GET -- like 537.36 1.1 Logstash 104.0 10.49

本文主要介绍 Logstash 的安装及简单的使用，相关的环境及软件信息如下：CentOS 7.9、Logstash 8.2.2、Elasticsearch 8.2.2。

1、安装

根据环境下载对应的安装包：https://www.elastic.co/cn/downloads/logstash，这里选择 Linux x86_64 版本；下载完成后在服务器上解压即可：

tar zxvf logstash-8.2.2-linux-x86_64.tar.gz

2、简单使用

这里使用 Logstash 来收集 Nginx 访问日志并保存到 Elasticsearch 中。

2.1、创建索引

先创建用于存放 Nginx 访问日志的索引。

curl -X PUT -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index' -d '
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "ip": {
        "type": "text"
      },
      "remoteUser": {
        "type": "text"
      },
      "accessTime": {
        "type": "date"
      },
      "method": {
        "type": "keyword"
      },
      "path": {
        "type": "text"
      },
      "protocal": {
        "type": "keyword"
      },
      "version": {
        "type": "keyword"
      },
      "status": {
        "type": "integer"
      },
      "bytes": {
        "type": "integer"
      },
      "referer": {
        "type": "text"
      },
      "userAgent": {
        "type": "text"
      }
    }
  }
}'

2.2、Logstash 配置输入

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

这里指定了 Nginx 日志文件的路径。

2.3、Logstash 配置过滤器

我们需要对日志进行处理，提取出我们需要的字段。

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}

grok 插件通过正则表达把原始日志拆分成相应的字段；date 插件把字段转成日期格式；mutate 插件把字段转成我们需要的类型；prune 插件过滤出不需要存到 Elasticsearch 的字段。

2.4、Logstash 配置输出

配置输出到本地的 Elasticsearch。

output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}

2.5、完整配置

input {
  file {
    path => ["/home/hadoop/app/nginx-1.8.0/logs/access.log"]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{IP:ip} - %{USER:remoteUser} \[%{HTTPDATE:accessTimeStr}\] \"%{WORD:method} %{URIPATHPARAM:path} %{WORD:
protocal}/%{NUMBER:version}\" %{INT:status} %{INT:bytes} \"%{DATA:referer}\" \"%{DATA:userAgent}\"" }
  }

  if [tags][0] == '_grokparsefailure' {
    drop{}
  }
  
  date {
    match => ["accessTimeStr", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "accessTime"
  }
  
  mutate {
    convert => {
      "bytes" => "integer"
      "status" => "integer"
    }
  }
  
  prune {
    blacklist_names => ["log","@version","host","@timestamp","accessTimeStr","event"]
  }
}


output {
  stdout { }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-index"
  }
}
nginx.conf

2.6、运行 Logstash

bin/logstash -f nginx.conf

2.7、验证

Nginx 的访问日志信息如下：

10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
...

Logstash 的控制台日志信息如下：

{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:15Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
{
         "bytes" => 0,
       "referer" => "-",
      "protocal" => "HTTP",
     "userAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
            "ip" => "10.49.196.1",
    "accessTime" => 2022-09-07T03:04:16Z,
        "method" => "GET",
       "message" => "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
    "remoteUser" => "-",
        "status" => 304,
       "version" => "1.1",
          "path" => "/"
}
...

Elasticsearch 中查询数据：

curl -X GET -H 'Content-Type:application/json' 'http://10.49.196.11:9200/nginx-index'

结果如下：

{
  "took": 530,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "nginx-index",
        "_id": "nSjnFYMB-RPngHUTzpDo",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:15Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:15 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nijnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      },
      {
        "_index": "nginx-index",
        "_id": "nyjnFYMB-RPngHUT0JCK",
        "_score": 1.0,
        "_source": {
          "bytes": 0,
          "referer": "-",
          "protocal": "HTTP",
          "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
          "ip": "10.49.196.1",
          "accessTime": "2022-09-07T03:04:16Z",
          "method": "GET",
          "message": "10.49.196.1 - - [07/Sep/2022:11:04:16 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36\"",
          "remoteUser": "-",
          "status": 304,
          "version": "1.1",
          "path": "/"
        }
      }
    ]
  }
}

 

 

 

 

标签：入门,GET,--,like,537.36,1.1,Logstash,104.0,10.49
From： https://www.cnblogs.com/wuyongyin/p/16664309.html

相关文章

  • C++源码压缩程序
    C++源码压缩程序一、必做内容:(一）压缩器1．为了提高C++源程序的可读性，C++程序在书写过程中加入了空行、空格、缩进、注释等。假设你想牺牲可读性，以节省磁盘空间，那么你可以......
  • 斐波那契查找算法
    斐波那契也称黄金分割法，通过黄金分割点找到mid值，即mid=low+F(k-1)-1 （F代表斐波那契数列）对F(k-1)-1的理解由斐波那契数列F[k]=F[k-1]+F[k-2]的性质，可以得到 （F[k]-1......
  • matlab_
    一、安装工具包SVM：由于libsvm版本较高，如以下需要三个参数了，要添加上dec_value，不然结果为空[predict_class,accuracy]=libsvmpredict(output_test,input1_test,model);......
  • bottle库上传文件
    安装bottle库pipinstallbottle上传代码[email protected]('/upload')defupload_get():returnbottle.static_file('index.html','d:/web')@bottl......
  • 二--5.经典进程的同步问题
    先申请资源信号量，再申请互斥信号量1.生产者-消费者问题   1.1利用记录型信号量......
  • 什么是C语言
    什么是C语言？C语言是一门计算机语言计算机语言是什么呢？人和计算机交流的语言，如C/C++、Java、python 语言的发展？二进制语言（硬件-电-正电1/负电01010100101001100......
  • 【补题计划】NOI Online 2022
    【NOIOnline2022】补题记录入门组T1[NOIOnline2022]王国比赛lj小模拟一遍过（都没编译就交了）点击查看代码#include<iostream>#include<cstdio>#include<cmath>......
  • 第四周学习总结
    2022-2023-120221418《计算机基础与程序设计》第四周学习总结这个作业属于哪个课程2022-2023-1-计算机基础与程序设计这个作业要求在哪里<作业要求的链接>[20......
  • WPF开发中遇到的新知识 -- 1
    前后台同时启动的方式目的：希望在WPF前台启动后，带动ASP.NETCore后台服务一同启动，在前台关闭后，也一起关闭方法：在打开窗口之前，首先手动打开ASP.NETCore子进程，然后注册......
  • NodeJS的安装
    前言虽然这些东西很基本也很简单，但是过段时间就会遗忘，有空记录下吧，反正也不耗费多少时间，后期至少比百度快点。安装步骤Linux下的安装下载安装包下载地址：http://nodejs......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99