ELK日志收集搭建

elasticsearch的版本要注意兼容,以清华开源网站为例,不同版本的es有对应适应版本的filebeat、kibana等,这里以8.50为例

 我这里用filebeat直接代替logstash,架构是需要收集日志的机器上安装filebeat,多台es机器组成集群,数量最好不要低于3台\

es:

官方文档链接:https://www.elastic.co/guide/en/elasticsearch/reference/8.5/settings.html

wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-8.x/8.5.0/elasticsearch-8.5.0-x86_64.rpm

cd /etc/elasticsearch/ && cp elasticsearch.yml{,.bak} && vi elasticsearch.yml

#集群 多台是添加此配置
cluster.name: my-application

node.name: node-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 172.22.16.17,127.0.0.1

#http.port: 9200
#添加es主机,可以是ip:端口,或者域名
discovery.seed_hosts: ["172.1.1.1", "172.1.1.2", "172.1.1.3"]
#指定master节点列表,当1挂掉提升2
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

xpack.security.enabled: false
#默认
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

http.host: 0.0.0.0

需要收集日志的机器安装Filebeat:

官方文档链接:https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-8.x/8.5.0/filebeat-8.5.0-x86_64.rpm

修改配置文件:

cd /etc/filebeat && cp filebeat.yml{,.bak}&& vi filebeat.yml

filebeat.inputs:
#7.16版本之后已经启用log,现改filestream,同一机器下多个日志ID不能相同,创建一个tags,后边有tags区分要写入不同的索引当中
- type: filestream
  id: my-online-id
  enabled: true
  paths:
    - /var/log/a.log
  tags: ["log-a"]

- type: filestream
  id: my-flyshenhe-id
  enabled: true
  paths:
    - /var/log/b.log
  tags: ["log-b"]

setup.ilm.enabled: false
setup.template.name: "log"
setup.template.pattern: "log-*"
#可设置分片和副本数,这是默认的
setup.template.settings:
  index.number_of_shards: 1

#输出到es集群,hosts填入主机ip
output.elasticsearch:
  hosts: ["172.1.1.1:9200", "172.1.1.2:9200", "172.1.1.3:9200"]
  indices:
    - index: "log-a-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "log-a"
    - index: "log-b-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "log-b"

展示到kibana:

cd /etc/kibana/ && cp kibana.yml{,.bak} && vi kibana.yml