部署logstash收集daemonset filebeat发过去的kafka日志

实现：部署如下图红框 位置logstash

架构：

主机：10.0.7.56

部署logstash

yum -y install java-1.8.0-openjdk
yum -y install https://mirror.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.9.0/logstash-7.9.0.rpm
systemctl start logstash
systemctl enable logstash

创建配置文件

从kafka接收数据，发送到es

vim /etc/logstash/conf.d/daemonset-filebeat-kafka-to-es.conf
input {
  kafka {
    bootstrap_servers => "10.0.7.53:9092,10.0.7.54:9092,10.0.7.55:9092"
    topics => ["daemonset-pod-console-log"]
    codec => json {
        charset => "UTF-8"
    }
  }
}

output {
  if [fields][log_topic] == "daemonset-pod-console-log"  {
    elasticsearch {
      hosts => ["10.0.7.46:9200","10.0.7.47:9200","10.0.7.48:9200"]
      index => "daemonset-pod-console-log-%{+YYYY.MM.dd}"
    }}
}

启动并查看日志

systemctl restart logstash
tail -f /var/log/logstash/logstash-plain.log