谷歌一下,搜索结果有各种样的博客文章,看了很多篇,反而被绕晕,经过自己的实践搭建完成获取到的经验最适合自己,在这里把整个过程记录下来,也希望可以帮助到一些需要的人。
搭建前,请阅读官方文档,虽然是英文,如果能看懂你能学到更精髓的部分。英文好坏也是技术和技术之间的分水岭。好的技术资料很多都是英文的。
官方文档地址:
https://www.elastic.co/guide/en/enterprise-search/8.6/start.html
本文档是使用elasticsearch 8.6,操作系统centos 7.9, 使用了三台服务器搭建集群。
博文大纲
1.elasticsearch 8集群搭建
2.kibana安装并连接elasticearch
3.filebeat安装和配置
4.logstash安装以及cdn日志解析配置
正文:
以下所有组件都是使用yum安装。
1.elasticsearch 8集群搭建
根据官方文档:
https://www.elastic.co/guide/en/elasticsearch/reference/8.6/install-elasticsearch.html
以下操作在都有节点执行。
导入elasticsearch gpg
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
添加yum仓库
vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
清除yum缓存
yum clean all
安装elasticsearch
yum install --enablerepo=elasticsearch elasticsearch -y
编辑主节点配置文件:
cluster.name: cdn-logs #集群名称
node.name: node01 #当前节点名称
path.data: /var/lib/elasticsearch #修改成自己的数据存储目录
path.logs: /var/log/elasticsearch #修改成自己的日志文件存储目录
network.host: 10.108.0.2 #对外提供访问IP
http.port: 9200 #对外提供访问端口
#以下配置是elasticsearch安装时自动添加的关于ssl的配置,不需要手动添加
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node01"] #配置初始master节点为当前节点
http.host: 0.0.0.0 #允许任何IP访问该该节点
transport.host: 10.108.0.2 #节点之间通讯IP
transport.port: 9300 #节点之间通讯端口
使用调试模式启动elasticsearch
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch
启动完成后,会生成如下信息,这些信息很重要,后面将从节点接入集群,将kiaban加入集群都会用到:
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.
ℹ️ HTTP CA certificate SHA-256 fingerprint:
c777ed87ee9f9a66d6382df34d0b931a56cbaa7a7023d02ce4d7b198521a5f29
ℹ️ Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
#将kiban连接到elasticseatch集群所使用的密钥
eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiR0ttN21ZY0JtcnBqS0xYVGFqaHc6TUROMEIxWUVRdkNneGlfeklXZ3dsZyJ9
ℹ️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
#从节点使用这个密钥加入elasticsearch集群
eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9
If you're running in Docker, copy the enrollment token and run:
`docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.7.0`
编辑从节点node02配置文件:
cluster.name: cdn-logs
node.name: node02
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.108.0.3
http.port: 9200
transport.host: 10.108.0.3
transport.port: 9300
编辑从节点node03配置文件:
cluster.name: cdn-logs
node.name: node03
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.108.0.4
http.port: 9200
transport.host: 10.108.0.4
transport.port: 9300
在从节点执行如下命令加入elasticsearch集群:
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjN1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9
This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:
- Security auto configuration will be removed from elasticsearch.yml
- The [certs] config directory will be removed
- Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y
编辑从节点node03配置文件:
启动elasticsearch