Logstash简介
logstash是一个开源的数据采集工具,通过数据源采集数据.然后进行过滤,并自定义格式输出到目的地。
数据分为:
- 结构化数据 如:mysql数据库里的表等
- 半结构化数据 如: xml,yaml,json等
- 非结构化数据 如:文档,图片,音频,视频等
logstash可以采集任何格式的数据,当然我们这里主要是讨论采集系统日志,服务日志等日志类型数据。
官方产品介绍:https://www.elastic.co/cn/products/logstash
input插件: 用于导入日志源
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
filter插件: 用于过滤
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
output插件: 用于导出
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
logstash部署
在logstash服务器上确认openjdk安装
[root@test3 ~]# java -version
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)
在logstash服务器上安装logstash
[root@test3 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.2.rpm
[root@test3 ~]# rpm -ivh logstash-6.5.2.rpm
配置logstash主配置文件
[root@3 ~]# cat /etc/logstash/logstash.yml |grep -v '#' |grep -v '^$'
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/ 打开注释,并加上配置目录路径
http.host: "10.1.1.13" 打开注释,并改为本机IP(这是用于xpack监控用,但要收费,所以在这里不配置也可以)
path.logs: /var/log/logstash
启动测试
[root@test3 ~]# cd /usr/share/logstash/bin
使用下面的空输入和空输出启动测试一下
[root@test3 bin]# ./logstash -e 'input {stdin {}} output {stdout {}}'
运行后,输入字符将被stdout做为标准输出内容输出
运行后,输入字符将被stdout做为标准输出内容输出
关闭启动
测试能启动成功后,ctrl+c取消,则关闭了
另一种验证方法:
[root@test3]# vim /etc/logstash/conf.d/test.conf
input {
stdin {
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
[root@test3 bin]# pwd
/usr/share/logstash/bin
[root@test3 bin]# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/test.conf -r
......
Config Validation Result: OK. Exiting Logstash
--path.settings 指定logstash主配置文件目录
-f 指定片段配置文件
-t 测试配置文件是否正确
codec => rubydebug这句可写可不定,默认就是这种输出方式
[root@test3 bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf
......
haha
{
"@timestamp" => 2019-07-02T10:40:10.839Z,
"message" => "haha",
"host" => "vm3.cluster.com",
"@version" => "1"
}
hehe
{
"@timestamp" => 2019-07-02T10:40:11.794Z,
"message" => "hehe",
"host" => "vm3.cluster.com",
"@version" => "1"
}
-r参数很强大,会动态装载配置文件,也就是说启动后,可以不用重启修改配置文件
日志采集
采集messages日志
这里以/var/log/messages为例,只定义input输入和output输出,不考虑过滤
[root@test3 bin]# vim /etc/logstash/conf.d/test.conf
input {
file {
path => "/var/log/messages"
start_position => "beginning"
}
}
output {
elasticsearch{
hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
index => "messages-%{+YYYY.MM.dd}"
}
stdout {
}
}
[root@test3 bin]# ./logstash --path.settings /etc/logstash/ -r -f /etc/logstash/conf.d/test.conf &
后台运行如果要杀掉,请使用pkill java或ps查看PID再kill -9清除
通过浏览器访问es-head验证
采集多日志源
[root@test3 bin]# vim /etc/logstash/conf.d/test.conf
input {
file {
path => "/var/log/messages"
start_position => "beginning"
type => "messages"
}
file {
path => "/var/log/yum.log"
start_position => "beginning"
type => "yum"
}
}
filter {
}
output {
if [type] == "messages" {
elasticsearch {
hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
index => "messages-%{+YYYY-MM-dd}"
}
}
if [type] == "yum" {
elasticsearch {
hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
index => "yum-%{+YYYY-MM-dd}"
}
}
}