首页 > 其他分享 >elasticsearch 8集群搭建并完成CDN日志收集和解析

elasticsearch 8集群搭建并完成CDN日志收集和解析

时间:2023-04-19 22:32:04浏览次数:46  
标签:CDN cluster elasticsearch enrollment path 日志 节点 transport

  谷歌一下,搜索结果有各种样的博客文章,看了很多篇,反而被绕晕,经过自己的实践搭建完成获取到的经验最适合自己,在这里把整个过程记录下来,也希望可以帮助到一些需要的人。

搭建前,请阅读官方文档,虽然是英文,如果能看懂你能学到更精髓的部分。英文好坏也是技术和技术之间的分水岭。好的技术资料很多都是英文的。

官方文档地址:

   https://www.elastic.co/guide/en/enterprise-search/8.6/start.html

本文档是使用elasticsearch 8.6,操作系统centos 7.9, 使用了三台服务器搭建集群。

博文大纲

1.elasticsearch 8集群搭建

2.kibana安装并连接elasticearch

3.filebeat安装和配置

4.logstash安装以及cdn日志解析配置

正文:

 以下所有组件都是使用yum安装。

1.elasticsearch 8集群搭建

根据官方文档:

     https://www.elastic.co/guide/en/elasticsearch/reference/8.6/install-elasticsearch.html

以下操作在都有节点执行。

导入elasticsearch gpg

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

添加yum仓库

vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

清除yum缓存

yum clean all

安装elasticsearch

yum install --enablerepo=elasticsearch elasticsearch -y

编辑主节点配置文件:

cluster.name: cdn-logs    #集群名称
node.name: node01         #当前节点名称
path.data: /var/lib/elasticsearch   #修改成自己的数据存储目录
path.logs:  /var/log/elasticsearch  #修改成自己的日志文件存储目录
network.host: 10.108.0.2   #对外提供访问IP
http.port: 9200            #对外提供访问端口
#以下配置是elasticsearch安装时自动添加的关于ssl的配置,不需要手动添加
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
 xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node01"] #配置初始master节点为当前节点
http.host: 0.0.0.0   #允许任何IP访问该该节点
transport.host: 10.108.0.2 #节点之间通讯IP
transport.port: 9300       #节点之间通讯端口

使用调试模式启动elasticsearch

sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch

启动完成后,会生成如下信息,这些信息很重要,后面将从节点接入集群,将kiaban加入集群都会用到:

✅ Elasticsearch security features have been automatically configured!

✅ Authentication is enabled and cluster connections are encrypted.


ℹ️  HTTP CA certificate SHA-256 fingerprint:

  c777ed87ee9f9a66d6382df34d0b931a56cbaa7a7023d02ce4d7b198521a5f29


ℹ️  Configure Kibana to use this cluster:

• Run Kibana and click the configuration link in the terminal when Kibana starts.

• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
#将kiban连接到elasticseatch集群所使用的密钥
  eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiR0ttN21ZY0JtcnBqS0xYVGFqaHc6TUROMEIxWUVRdkNneGlfeklXZ3dsZyJ9

ℹ️ Configure other nodes to join this cluster:

• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
#从节点使用这个密钥加入elasticsearch集群
  eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9


  If you're running in Docker, copy the enrollment token and run:

  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.7.0`


编辑从节点node02配置文件:

cluster.name: cdn-logs
node.name: node02
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.108.0.3
http.port: 9200
transport.host: 10.108.0.3
transport.port: 9300

编辑从节点node03配置文件:

cluster.name: cdn-logs
node.name: node03
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.108.0.4
http.port: 9200
transport.host: 10.108.0.4
transport.port: 9300

在从节点执行如下命令加入elasticsearch集群:

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjNzc3ZWQ4N2VlOWY5YTY2ZDYzODJkZjM0ZDBiOTMxYTU2Y2JhYTdhNzAyM2QwMmNlNGQ3YjE5ODUyMWE1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9


/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjcuMCIsImFkciI6WyIxNTcuMjQ1LjIwOS4xODg6OTIwMCJdLCJmZ3IiOiJjN1ZjI5Iiwia2V5IjoiRnFtN21ZY0JtcnBqS0xYVGFqaFc6Z3FqeWtobThTQy1XTWJ5NXlVQkNnUSJ9

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:  
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

elasticsearch 8集群搭建并完成CDN日志收集和解析_elasticsearch

编辑从节点node03配置文件:

启动elasticsearch


标签:CDN,cluster,elasticsearch,enrollment,path,日志,节点,transport
From: https://blog.51cto.com/zjhrunnnnggo/6207527

相关文章

  • 助你掌握搜索神器,10个实用的Elasticsearch查询技巧
    前言  Elasticsearch是一个非常流行的搜索引擎,已经成为了许多企业的首选解决方案。然而,我们要想成为一个优秀的程序员,就必须掌握各种查询技巧。本文将向大家介绍10个实用的Elasticsearch查询技巧,并配上详细的代码示例,帮助我们更好地掌握Elasticsearch的查询语法。示例1.匹......
  • python编写CDN刷新脚本
    脚本刷新根目录[root@jenkinscdn]#catdns_flush.py#!/usr/bin/python3fromaliyunsdkcore.clientimportAcsClientfromaliyunsdkcdn.request.v20180510.RefreshObjectCachesRequestimportRefreshObjectCachesRequestimportsysiflen(sys.argv)<2:print(�......
  • ELK日志分析系统
    拓扑图:推荐步骤:搭建ElaticSearch群集,通过浏览器查看和图形管理群集配置客户端装apache和logstash服务配置采集apache日志配置kibana监听elasticsearch服务器日志实验步骤:一、搭建Elaticsearch群集,通过浏览器查看和图形管理群集1、配置第一台Elaticsearch服务器(1)修改hosts文件复制......
  • 挖/填坑日志
    我是挖坑大师!写一点做了但是没有完全做完的事情:小说(框架搭建了一堆,but实际本上只有三个案子,网文甚至没更到那个阶段())第二本小说啊对对对其实曾经干过一个时间干两本小说的,但是一本在无名作文本上丢了,还好没有更很多()Witness_C++模拟器可以来这个地方看进度,有进展会更新Natu......
  • 简单的python3脚本:从日志中提取信息
    命名:log_extractor.pyordownload_stats_extractor.py#coding:utf-8#!/usr/bin/python3deffilter_line(line,contains,contains_not):ifall(cinlineforcincontains)andnotany(ninlinefornincontains_not):ifint(line.split()[8])==2......
  • AOP实现日志打印
    packagecom.youmu.framework.love.interfaces;importjava.lang.annotation.*;/***@Author:guodong*@CreateTime:2023-04-1916:38*@Description:自定义注解日志类*@Version:1.0*/@Documented@Target(ElementType.METHOD)//注解的作用类型为方法@Rete......
  • Docker安装ElasticSearch
    1安装说明在平时工作的时候,开发环境大多数会安装单机ElasticSearch,但生产环境基本会安装ElasticSearch集群版,所以我们接下来实现一下ElasticSearch单机安装,下一节实现集群安装,但安装也大多数采用Docker安装。不过中文搜索,会实现分词器集成,可以采用IK分词器。ElasticSearch采用......
  • TFA-收集日志及分析
    下载https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=272133523880062&id=1513912.1&_afrWindowMode=0&_adf.ctrl-state=fghvcgapa_617a安装[root@rhel75~]#./ahf_setupAHFInstallerforPlatformLinuxArchitecturex86_64AHFIns......
  • LInux单机部署ELK日志收集
    LInux单机部署ELK日志收集一、环境准备centos7cpu:1核内存:8G#安装vim,wget,net-tools设置主机名:vim/etc/hosts127.0.0.1localhostlocalhost.localdomainlocalhost4localhost4.localdomain4::1localhostlocalhost.localdomainlocalhost6localhost6.local......
  • Elasticsearch搜索功能的实现(五)-- 实战
    实战环境elasticsearch8.5.0+kibna8.5.0+springboot3.0.2+springdataelasticsearch5.0.2+jdk17一、集成springdataelasticsearch1添加依赖<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-st......