Logstash入门教程

前言

在线调试工具：​​http://grokdebug.herokuapp.com/​​ 一个例子

input {
    file {
        path => "/data/mosh/logstash-6.2.4/logs/test.log"
        # 下面2个配置是为了从头开始读
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}
filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:tmp_log_date} \|- %{WORD} %{JAVACLASS} \[%{JAVAFILE}\]  - taskId %{BASE10NUM:tmp_task_id} step %{INT:tmp_task_step}"]
        add_field => {
            "log_date" => "%{tmp_log_date}"
            "task_id" => "%{tmp_task_id}"
            "task_step" => "%{tmp_task_step}"
        }
    }
}
output {
    # 不满足筛选条件的就不写入数据库了
    if "_grokparsefailure" not in [tags] {
        mongodb {
            uri => "mongodb://username:userpassword@mad134:27019"
            database => "cis-ws-monitor"
            collection => "task"
        }
    }
}

输出ERROR级别的日志

input {
    file {
        path => "/data/mosh/logstash-6.2.4/logs/server.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        codec => multiline {
            pattern => "%{TIMESTAMP_ISO8601:logdate} \|-\s*%{LOGLEVEL}"
            negate => true
            what => "previous"
            auto_flush_interval => 30  # 如果在规定时候内没有新的日志事件就不等待后面的日志事件
        }
    }
}
filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logdate} \|-\s*ERROR"]
    }
}
output { 
    if "_grokparsefailure" not in [tags] {
        stdout {
            codec => rubydebug
        }
    }
}

读取多个路径下的日志

input {
  file {
    path => ["/data/server.log","/data/server2.log"]
  }
}

参考博客

每次从头读日志
[2]​​​https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns​​​ 查看插件和安装插件

[5]​​http://www.mamicode.com/info-detail-1693015.html​​ 自定义输出的json格式