Filebeat多目录收集日志

参考：
https://www.jianshu.com/p/d889aae7c72e

filebeat日志通过redis传递至logstash在输出至elasticsearch参考
https://www.cnblogs.com/minseo/p/9185423.html

1. 场景需求说明
   在同一台主机有多个日志需要区分不同index输出至elasticsearch
2. filebeat配置

# cat /etc/filebeat/filebeat.yml
# 给不同日志打不同tags用于区分
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/fastchat/*.log
  tags: ["psych-log-0388"]
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/bert/*.log
  tags: ["bert-log-0388"]
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
# 输出至redis 
# key是自定义
output.redis:
  hosts: ["192.168.3.65:46379"]
  db: "3"
  password: "password"
  key: "0388"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

3. logstash配置

# cat /etc/logstash/conf.d/psych-bert-log-redis-to-elasticsearch.conf 
# key和filebeat设置为一致
input{
    redis {
    host => "192.168.3.65"
    port => "46379"
    password => "password"
    db => "3"
    data_type => "list"
    key => "0388"
 
}
}

# 通过在filebeat中自定义tags来区分不同日志并使用不同的index输出至elasticsearch 
output{
    if "psych-log-0388" in [tags]  {
       elasticsearch {
            hosts => ["192.168.3.59:9200"]
            index => "psych-log-0388-%{+YYYY.MM.dd}"
        }
        #stdout{
        #   codec => rubydebug
        #}
    }
    if "bert-log-0388" in [tags]  {
       elasticsearch {
            hosts => ["192.168.3.59:9200"]
            index => "bert-log-0388-%{+YYYY.MM.dd}"
        }
        #stdout{
        #   codec => rubydebug
        #}
    }
}

启动logstash

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/psych-bert-log-redis-to-elasticsearch.conf

查看elasticsearch是否收到日志

# curl http://192.168.3.59:9200/_cat/indices|grep psych

在kibana添加对应日志不详述