ELK_8.2.0
安装及部署手册
目录
安装环境
系统环境
操作系统：CentOS 7
软件环境
基础软件
ELK所需基础服务及版本信息，如下表所示：
|
序号 |
软件名称 |
软件版本 |
备注 |
|
1 |
jdk |
jdk-11.0.18 |
|
|
2 |
elasticsearch |
8.2.0 |
|
|
3 |
filebeat |
8.2.0 |
|
|
4 |
logstash |
8.2.0 |
|
|
5 |
kibana |
8.2.0 |
|
|
6 |
ELK服务
优化系统
|
序号 |
服务名称 |
操作命令 |
备注 |
|
0 |
修改主机名 |
hostnamectl set-hostname es01（主机名称） |
需要重启机器 |
|
1 |
关闭selinux |
setenforce 0 |
vi /etc/sysconfig/selinux 里边 enforcing 改为disabled 重启动机器生效 |
|
2 |
关闭firewalld |
systemctl stop firewalld systemctl disable firewalld |
|
|
3 |
打开句柄数限制 |
vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 并执行命令： sysctl -p |
|
|
4 |
配置域名解析 |
/etc/hosts 10.209.22.105 esserver1 10.209.22.106 esserver2 |
|
JDK安装配置
我们要将jdk-17.0.8_linux-aarch64_bin.tar.gz解压到linux系统中，用于es使用。（解压位置自定义为：/ ）。
使用命令 tar -zxvf /opt/jdk-17.0.8_linux-aarch64_bin.tar.gz -C / 将opt文件夹下的jdk-17.0.8解压到根目录 /home/es 下。
|
export JAVA_HOME=/home/es/jdk-17.0.6 PATH=$JAVA_HOME/bin:$PATH CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME export PATH export CLASSPATH |
ES生成证书
如下操作在其中一个node节点执行即可，生成完证书传到集群其他节点即可
|
bin/elasticsearch-certutil ca bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 |
两条命令es8.x开启xpark认证步骤均一路回车即可，不需要给秘钥再添加密码。
证书创建完成之后，一定放到elasticsearch的config目录下，放到其他目录会报路径无权限的错误！
ES配置文件更改
elasticsearch.yml
|
# ---------------------------------- Cluster ----------------------------------- # 集群中所有节点必须保持一致 cluster.name: elk-application # ------------------------------------ Node ------------------------------------ # 需要随节点更改 node.name: es01 node.attr.rack: r1 # ----------------------------------- Paths ------------------------------------ path.data: /home/es/elasticsearch-8.2.0/data path.logs: /home/es/elasticsearch-8.2.0/logs # ----------------------------------- Memory ----------------------------------- bootstrap.memory_lock: false # ---------------------------------- Network ----------------------------------- # 默认情况下，Elasticsearch 仅仅绑定回环地址，比如127.0.0.1 和[::1] # 需要随节点更改 network.host: 10.20.12.103 http.port: 9200 transport.port: 9300 # 开启安全防护 http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type # 集群所有 master-eligible 节点 # 对应旧版中的 discovery.zen.ping.unicast.hosts discovery.seed_hosts: ["es01:9300", "es02:9300", "es03:9300"] # Elasticsearch 7.0新引入的配置项 # 集群第一次启动达到这个数量后就开始引导 cluster.initial_master_nodes: ["es01", "es02", "es03"] xpack.security.enabled: true xpack.license.self_generated.type: basic xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 |
启动es
|
bin/elasticsearch -d |
es生成管理密码
|
bin/elasticsearch-setup-passwords interactive |
jvm.options
|
-Xms8g -Xmx8g |
Filebeat配置
|
#=========================== Filebeat inputs ============================= filebeat.inputs: - type: log enabled: true paths: - /home/rsyslog/logs/2*/*.log ##获取路径下所有包含secure开头的文件。 #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["10.20.12.106:5044"] ##logstash服务地址和端口 #----------------------------- elasticsearch output -------------------------------- #output.elasticsearch: # hosts: ["10.20.12.103:9200", "10.20.12.104:9200", "10.20.12.105:9200"] # username: "elastic" # password: "Elastic0309#" # indices: # - index: "test-index-%{+yyyy.MM.dd}" # allow_older_versions: true #兼容旧es #----------------------------- console output -------------------------------- #output.console: # pretty: true # enable: true |
Filebeat启动
|
Nohup ./filebeat -e & |
Syslog server配置
|
# Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 $template Remote,"/home/rsyslog/logs/%$YEAR%-%$MONTH%-%$DAY%/%fromhost-ip%.log" # 设置远程日志存放路径和文件格式 :fromhost-ip, !isequal, "127.0.0.1" ?Remote # 如果是本机日志则不记录 & ~ #丢弃无匹配日志 |
编辑/etc/sysconfig/rsyslog加入“-m 0 -r”
|
SYSLOGD_OPTIONS=”-m 0 -r” |
Syslog client配置
|
*.* @@10.20.12.106:514 #发送到syslog server服务日志 |
如何记录操作命令：需配置/etc/bashrc
|
export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; }); logger $(date "+%Y-%m-%d %H:%M:%S"):$user:$(hostname -I):$msg:$(who am i|cut -d\( -f2|cut -d\) -f1); }' |
Logstash配置
|
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { beats { port => 5044 } } # -------清洗数据grok-------- filter{ grok{ match => {"message" => "(?<logTime>(%{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}(?<Time>%{TIME}))) (?<localname>.*?) (?<sysuser>.*?) (?<timestamp>(%{TIMESTAMP_ISO8601}))::%{IPV4:local_ip} :(?<shell>.*?):%{IPV4:login_ip}"} } } #-----------输出到es-------- output { elasticsearch { hosts => ["10.20.12.103:9200", "10.20.12.104:9200", "10.20.12.105:9200"] index => "sys-log-%{+YYYY.MM.dd}" user => "elastic" password => "Elastic0309#" } stdout { codec => rubydebug } } #output { # stdout { # codec => rubydebug # } #} |
Logstash启动
|
Nohup bin/logstash -f config/logstash-sample.conf & |
Logstash清洗linux主机日志
|
日志样例： "message": "Mar 14 18:30:06 es03 root: 2023-03-14 18:30:06::10.20.12.105 :more bashrc:10.2.11.85" Grok规则：(?<logTime>(%{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}(?<Time>%{TIME}))) (?<localname>.*?) (?<sysuser>.*?) (?<timestamp>(%{TIMESTAMP_ISO8601}))::%{IPV4:localip} :(?<shell>.*?):%{IPV4:clientip} |
Kibana配置
kibana.yml
|
server.port: 5601 server.host: "10.20.12.106" elasticsearch.hosts: ["http://10.20.12.103:9200", "http://10.20.12.104:9200", "http://10.20.12.105:9200"] elasticsearch.username: "kibana_system" elasticsearch.password: "XXXXXXX" |
Kibana使用手册
Discover需要点+显示单列
Stack Management 配置索引集
KQL与GROK调试工具
Webhook告警
VSFTP搭建部署
rpm -ivh vsftpd-*
配置文件
|
pasv_enable=YES #pasv_address=172.17.69.121 #请修改为您的 Linux 云服务器公网 IP #pasv_address=10.244.244.9 #请修改为您的 Linux 云服务器公网 IP pasv_min_port=41001 pasv_max_port=43030 anonymous_enable=NO local_enable=YES write_enable=YES ls_recurse_enable=YES local_umask=022 anon_umask=022 max_clients=50 dirmessage_enable=YES xferlog_enable=YES connect_from_port_20=YES xferlog_std_format=YES chroot_local_user=NO chroot_list_enable=YES chroot_list_file=/etc/vsftpd/chroot_list listen=YES listen_ipv6=NO pam_service_name=vsftpd userlist_enable=yes userlist_deny=NO # 禁止未加入白名单用户登录 userlist_file=/etc/vsftpd/user_list # 用户名单 tcp_wrappers=YES local_root=/app/admin # ftp根目录 chroot_local_user=YES #anon_root=/app/admin #allow_writeable_chroot=YES #加入此配置，用户可以访问上层目录 user_config_dir=/etc/vsftpd/vsftpd # 用户单独配置 ftpd_banner=” Authorized users only. All activity may be monitored and reported.” reverse_lookup_enable=No #ftp登录过慢问题 |
/etc/vsftpd/vsftpd用户单独配置选项
|
local_root=/app/admin/yinmengfei #FTP用户家目录 cmds_denied=DELE #禁止用户删除 #cmds_allowed=ABOR,CWD,LIST,MDTM,NLST,PASS,PASV,PORT,PWD,QUIT,RETR,RNFR,RNTO,SIZE,TYPE,USER,ACCT,HELP,MODE,NOOP,REIN,STAT,STOU,STRU,SYST #允许用户操作 |