1 Keepalived 架构和安装
2.1 Keepalived 架构
Keepalived进程树
Keepalived <-- Parent process monitoring children \_ Keepalived <-- VRRP child \_ Keepalived <-- Healthchecking child
2.2 Keepalived 环境准备
#环境准备 #两台keepalive机器分别配一个单独网卡用于keepalive做心跳监测 : eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> ... inet 192.168.10.151/24 brd 192.168.10.255 scope global eth1 #两台后端服务器服务准备 [root@web01 ~]#apt update & apt install nginx mysql-server -y #各节点时间必须同步,ubuntu自带时间同步功能
2.3 Keepalived 安装
包安装
#CentOS [root@centos ~]#yum -y install keepalived #ubuntu [root@ubuntu1804 ~]#apt update && apt -y install keepalived
编译安装
#Ubuntu20.04和22.04安装相关包 [root@ka1 ~]#apt update && apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev #keepalived官网下载 keepalived-2.2.8.tar.gz #下载解压 [root@ka1 ~]#wget https://keepalived.org/software/keepalived-2.2.8.tar.gz [root@ka1 ~]#tar xvf keepalived-2.2.8.tar.gz -C /usr/local/src [root@ka1 ~]#cd /usr/local/src/keepalived-2.2.8/ #编译 [root@ka1 keepalived-2.2.8]#./configure --prefix=/usr/local/keepalived [root@ka1 keepalived-2.2.8]#make && make install #拷贝已经生成的service文件 [root@ka1 keepalived-2.2.8]#cp keepalived/keepalived.service /lib/systemd/system/ [root@ka1 keepalived-2.2.8]#systemctl daemon-reload #创建配置文件,默认配置路径可以是编译路径/usr/local/keepalived/etc/keepalived/keepalived.conf(优先高)或者/etc/keepalived/keepalived.conf(优先级低) [root@ka1 keepalived-2.2.8]#mkdir /etc/keepalived [root@ka1 keepalived-2.2.8]#cp /usr/local/keepalived/etc/keepalived/keepalived.conf.sample /etc/keepalived/keepalived.conf [root@ka1 keepalived-2.2.8]#systemctl enable --now keepalived.service #ping不通自己的vip,因新版内核嵌了新防火墙nft(centos8开始有了,ubuntu也有) #新版:nftables规则替代iptables规则 [root@ubuntu2204 ~]#nft list ruleset table ip keepalived { set vips { type ipv4_addr elements = { 192.168.200.16, 192.168.200.17, 192.168.200.18 } } chain out { type filter hook output priority filter - 1; policy accept; ip saddr @vips drop } chain in { type filter hook input priority filter - 1; policy accept; ip daddr @vips drop } } #清空nft规则 [root@ubuntu2204 ~]#nft flush ruleset [root@ubuntu2204 ~]#nft list ruleset #可以ping通自己的vip
容器 Docker 安装
Docker的官方仓库上没有提供Keepalived的容器镜像,可以利用在源码目录中提供的Dockerfile自行制作
2.4 KeepAlived 配置说明
全局配置
#/etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost #keepalived 发生故障切换时邮件发送的目标邮箱,可以按行区分写多个 [email protected] [email protected] } notification_email_from keepalived@localhost #发邮件的地址 smtp_server 127.0.0.1 #邮件服务器地址 smtp_connect_timeout 30 #邮件服务器连接timeout router_id ka1.example.com #每个keepalived主机唯一标识,建议使用当前主机名,如果多节点重名可能会影响切换脚本执行 vrrp_skip_check_adv_addr #默认会对所有通告报文都检查,会比较消耗性能,启用此配置后,如果收到的通告报文和上一个报文是同一个路由器,则跳过检查 vrrp_strict #严格遵守VRRP协议,启用此项后以下状况将无法启动服务或工作异常:1.无VIP地址 2.配置了单播邻居 3.在VRRP版本2中有IPv6地址,开启动此项并且没有配置vrrp_iptables时会自动开启iptables(旧内核)或者nft(新内核)的防火墙规则,默认导致VIP无法访问,建议不加此项配置 vrrp_garp_interval 0 #gratuitous ARP messages 报文发送延迟,0表示不延迟 vrrp_gna_interval 0 #unsolicited NA messages (不请自来)消息发送延迟 vrrp_mcast_group4 224.0.0.18 #此为默认多播地址,可以指定。指定组播IP地址范围:224.0.0.0到239.255.255.255,默认值:224.0.0.18,如果配置了单播,此项失效 vrrp_iptables #此项和vrrp_strict同时开启时,则不会添加防火墙规则,如果无配置vrrp_strict项,则无需启用此项配置,注意:新版加此项仍有iptables(旧内核)或者nft(新内核)规则 }
精简配置
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs { router_id ka1 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.200.16 192.168.200.17 192.168.200.18 } } [root@ka1 ~]#systemctl reload keepalived.service
配置虚拟路由器
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs { router_id ka1 } vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 66 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { #子网掩码默认32,别人会ping不通,这里改24 10.0.0.10/24 dev eth0 label eth0:1 #绑定网卡(业务网卡),定个标签别名 } } [root@ka1 ~]#systemctl reload keepalived.service #查看 [root@ka1 ~]#ip a #ka2服务器安装keepalived,修改配置 [root@ka2 ~]#vim /etc/keepalived/keepalived.conf global_defs { router_id ka2 } vrrp_instance VI_1 { state BACKUP interface eth1 virtual_router_id 66 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } } [root@ka1 ~]#systemctl reload keepalived.service
2.5 实现 Keepalived 独立子配置文件
当生产环境复杂时, /etc/keepalived/keepalived.conf 文件中保存所有集群的配置会导致内容过多,不易管理
可以将不同集群的配置,比如:不同集群的VIP配置放在独立的子配置文件中
利用include 指令可以实现包含子配置文件
include /path/file
范例
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs { router_id ka1 } include /etc/keepalived/conf.d/*.conf #将VRRP相关配置放在子配置文件中 [root@ka1 ~]#mkdir /etc/keepalived/conf.d [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 66 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } } [root@ka1 ~]#systemctl reload keepalived.service #可能有问题,重启下 [root@ka2 keepalived-2.2.8]#systemctl restart keepalived.service
查看是否脑裂
#通过arping命令,如果返回多条,说明脑裂,多个mac地址有该ip -c表示测几次 [root@ka2 ~]#arping -c1 -I eth0 10.0.0.10
2.6 启用 Keepalived 日志功能
注意: 编译安装方式如果实现有问题,可以重启主机可以解决
#根据keepalived的service文件,在启动的环境变量里配置或者直接在启动命令里加选项 [root@ka1 ~]#vim /usr/local/keepalived/etc/sysconfig/keepalived KEEPALIVED_OPTIONS="-D -S 6" #0-7选个没人用的,这里写local6 #去查看local6有没有定义 [root@ka1 ~]#vim /etc/rsyslog.conf #上面没有,到上面的子配置中查找 [root@ka1 ~]#cd /etc/rsyslog.d [root@ka1 rsyslog.d]#vim 50-default.conf #追加 local6.* /var/log/keepalived.log #重启服务 [root@ka1 rsyslog.d]#systemctl restart keepalived.service rsyslog.service #查看(没有就重启) [root@ka1 rsyslog.d]#tail /var/log/keepalived.log
3 Keepalived 实现 VRRP
3.1 实现Master/Backup的 Keepalived 单主架构
上面已经展示过了
3.2 抢占模式和非抢占模式
默认为抢占模式
非抢占模式 nopreempt
注意:要关闭 VIP抢占,必须将各 Keepalived 服务器 state 配置为 BACKUP
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf #ha1主机配置 vrrp_instance VI_1 { state BACKUP #都为BACKUP interface eth1 virtual_router_id 66 priority 100 #优先级高 advert_int 1 nopreempt #添加此行,设为nopreempt #ha2主机配置 vrrp_instance VI_1 { state BACKUP #都为BACKUP interface eth1 virtual_router_id 66 priority 80 #优先级低 advert_int 1 #nopreempt #注意:如果ka2主机也是非抢占式,会导致ka1即使优先级降低于ka2,VIP也不会切换至ka2 [root@ka1 ~]#systemctl reload keepalived.service
抢占延迟模式 preempt_delay
抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回 VIP
但是如果低优先级的主机down机,则立即抢占VIP地址,而不再延迟
preempt_delay # #指定抢占延迟时间为#s,默认延迟300s,在优先级高的节点配置
注意:需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict
范例:
#ka1主机配置 vrrp_instance VI_1 { state BACKUP #都为BACKUP interface eth0 virtual_router_id 66 priority 100 #优先级高 advert_int 1 preempt_delay 60 #抢占延迟模式,默认延迟300s #ka2主机配置 vrrp_instance VI_1 { state BACKUP #都为BACKUP interface eth0 virtual_router_id 66 priority 80 #优先级低 advert_int 1
3.3 VIP 单播配置
默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以设置为单播,减少网络流量
另外:有些公有云不支持多播,可以利用单播实现
单播优先与多播,即同时配置,单播生效
注意:启用 vrrp_strict 时,不能启用单播
#在所有节点vrrp_instance语句块中设置对方主机的IP,建议设置为专用于对应心跳线网络的地址,而非使 用业务网络 unicast_src_ip <IPADDR> #指定发送单播的源IP unicast_peer { <IPADDR> #指定接收单播的对方目标主机IP(多个就写多个) ...... }
范例:
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 66 priority 100 advert_int 1 unicast_src_ip 192.168.10.151 unicast_peer { 192.168.10.152 } authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } } [root@ka1 ~]#systemctl restart keepalived.service [root@ka2 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 { state BACKUP interface eth1 virtual_router_id 66 priority 80 advert_int 1 unicast_src_ip 192.168.10.152 unicast_peer { 192.168.10.151 } authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } } [root@ka2 ~]#systemctl restart keepalived.service
3.4 Keepalived 通知脚本配置
当keepalived的状态变化时,可以自动触发脚本的执行,比如:发邮件通知用户
默认以用户keepalived_script身份执行脚本,如果此用户不存在,以root执行脚本
可以用下面指令指定脚本执行用户的身份
global_defs { ...... script_user <USER> ...... }
通知脚本类型
当前节点成为主节点时触发的脚本
notify_master <STRING>|<QUOTED-STRING>
当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>
当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>
通用格式的通知触发机制,一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>
当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>
脚本的调用方法
在 vrrp_instance VI_1 语句块的末尾加下面行
#记得给脚本添加执行权限 notify_master "/etc/keepalived/notify.sh master" #执行的命令可以自己更改 notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault"
范例
#两台keepalive机器结尾都追加,发送邮箱脚本 [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 66 priority 100 advert_int 1 unicast_src_ip 192.168.10.151 unicast_peer { 192.168.10.152 } authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } notify_master "/etc/keepalived/notify.sh master" #追加 notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } [root@ka1 ~]#systemctl reload keepalived.service
3.5 实现 Master/Master 的 Keepalived 双主架构
访问量大,不要用这种模式
#复制一份新的子配置文件 [root@ka1 keepalived]#vim /etc/keepalived/conf.d/www.magedu.net.conf vrrp_instance VI_2 { #修改名称 state BACKUP #修改角色 interface eth1 virtual_router_id 88 #修改 priority 80 #修改优先级 advert_int 1 unicast_src_ip 192.168.10.151 #可共用vrrp单播,同一个心跳网络 unicast_peer { 192.168.10.152 } authentication { auth_type PASS auth_pass 654321 } virtual_ipaddress { 10.0.0.20/24 dev eth0 label eth0:2 #修改vip,修改别名 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } [root@ka1 ~]#systemctl restart keepalived.service [root@ka2 keepalived]#vim /etc/keepalived/conf.d/www.magedu.net.conf vrrp_instance VI_2 { #修改名称 state MASTER #修改角色 interface eth1 virtual_router_id 88 #修改 priority 100 #修改优先级 advert_int 1 unicast_src_ip 192.168.10.151 #可共用vrrp单播,同一个心跳网络 unicast_peer { 192.168.10.152 } authentication { auth_type PASS auth_pass 654321 } virtual_ipaddress { 10.0.0.20/24 dev eth0 label eth0:2 #修改vip,修改别名 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } [root@ka2 ~]#systemctl restart keepalived.service
3.6 实现多主模架构
不放心,可以多配几个从节点。比如8台机器,可以配成一主两从
3.7 同步组 (了解)
LVS NAT 模型VIP和DIP需要同步,需要同步组
vrrp_sync_group VG_1 { group { #作为一组一起漂移 VI_1 # name of vrrp_instance (below) VI_2 # One for each moveable IP } } vrrp_instance VI_1 { eth0 vip } vrrp_instance VI_2 { eth1 dip }
4 实现 IPVS 的高可用性
通过keepalive自动实现lvs的调用规则(keepalived和lvs是黄金搭档, 要实现lvs通过keepalived实现比较好)
解决lvs两大问题 1.高可用问题 2.监测后端服务是否正常,如果异常就剔除该服务,恢复再加入 #443属于https,加密七层。一般使用四层检查
lvs内核监控,外部无法看到其对应监控的端口
#查看lvs规则 ipvsadm -Ln
5 基于 VRRP Script 实现其它应用的高可用性
5.1 VRRP Script 配置
定义 VRRP script
vrrp_script <SCRIPT_NAME> { #定义一个检测脚本,在global_defs 之外配置 script <STRING>|<QUOTED-STRING> #shell命令或脚本路径 interval <INTEGER> #间隔时间,单位为秒,默认1秒 timeout <INTEGER> #超时时间 weight <INTEGER:-254..254> #默认为0,如果设置此值为负数,当上面脚本返回值为非0时,会将此值与本节点权重相加可以降低本节点权重,即表示fall. 如果是正数,当脚本返回值为0,会将此值与本节点权重相加可以提高本节点权重,即表示 rise.通常使用负值 fall <INTEGER> #执行脚本连续几次都失败,则转换为失败,建议设为2以上 rise <INTEGER> #执行脚本连续几次都成功,把服务器从失败标记为成功 user USERNAME [GROUPNAME] #执行监测脚本的用户或组 init_fail #设置默认标记为失败状态,监测成功之后再转换为成功状态 }
调用 VRRP script
vrrp_instance VI_1 { … track_script { <SCRIPT_NAME> } }
tcpdump实时查看vip漂移情况
[root@ka1 ~]#tcpdump -i eth1 -nn hsot 192.168.10.151
5.2 实战案例:利用脚本实现主从角色切换
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf global_defs { ... } vrrp_script check_down {#存在文件为假就降优先级为70,当不存在该文件又恢复优先级为100 script "[ ! -f /etc/keepalived/down ]" #/etc/keepalived/down存在时返回非0,触发权重-30 interval 1 weight -30 #优先级prio-30,从而出发地址漂移 fall 3 rise 2 timeout 2 } vrrp_instance VI_1 { state MASTER #在另一个节点为BACKUP interface eth0 virtual_router_id 66 priority 100 #在另一个节点为80 advert_int 1 authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } track_interface { eth0 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" track_script { check_down #调用前面定义的脚本 } }
实现 HAProxy 高可用(haproxy利用脚本实现主从角色切换)
#两台keepalived服务器安装haproxy [root@ka1 ~]#apt install haproxy [root@ka1 ~]#vim /etc/haproxy/haproxy.cfg listen status #追加状态页 mode http bind 0.0.0.0:9999 stats enable log global stats uri /haproxy-status listen webservers bind 10.0.0.10:80 server web01 10.0.0.153:80 check server web02 10.0.0.154:80 check [root@ka1 ~]#systemctl restart haproxy.service #在另一个台配置haproxy.cfg时,vip10.0.0.10不在上面,导致绑定不上去报错,要改内核参数(两台都要改) #查看内核参数 [root@ka2 ~]#sysctl -a|grep bind net.ipv4.ip_nonlocal_bind = 0 [root@ka2 ~]#vim /etc/sysctl.conf #追加 net.ipv4.ip_nonlocal_bind = 1 #允许绑定不存在的ip端口,允许使用 #从配置文件“/etc/sysctl.conf”加载内核参数设置 [root@ka2 ~]#sysctl -p net.ipv4.ip_nonlocal_bind = 1 #再修改/etc/haproxy/haproxy.cfg即可绑定 #在第一个keepalived节点配,第二个节点不用配(减了也没人替) [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_script chk_haproxy { #严谨判断写脚本判断状态页 script "killall -0 haproxy" #检查haproxy(只能监测进程没事,不能说明进程健康) interval 1 weight -30 fall 2 rise 2 } vrrp_instance VI_1 { state MASTER interface eth1 virtual_router_id 66 priority 100 advert_int 1 unicast_src_ip 192.168.10.151 unicast_peer { 192.168.10.152 } authentication { auth_type PASS auth_pass 123456 } virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1 } track_script { chk_haproxy #调用前面定义的脚本 } } [root@ka1 ~]#systemctl restart keepalived.service #测试关闭ka1上的keepalived,看nginx端请求是否来自ka2; #启动ka1上的keepalived,看nginx端请求是否来自ka1(ka1又抢回vip)
nginx同上(内核参数可以不改),把检查haproxy换成检查nginx killall -0 nginx
探测端口是否打开
#探测127.0.0.1的9999端口是否打开(当然端口打开,进程也有可能假死,也不算特别严谨) [root@ka1 ~]#</dev/tcp/127.0.0.1/9999 #读取数据到当前终端 [root@ka1 ~]#echo $? 0 [root@ka1 ~]#</dev/tcp/127.0.0.1/9998 -bash: connect: Connection refused -bash: /dev/tcp/127.0.0.1/9998: Connection refused [root@ka1 ~]#echo $? 1
检查脚本可以改成尝试重启,如果失败再降低优先级进行切换
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_script chk_nginx { script "/etc/keepalived/check_nginx.sh" interval 1 weight -30 fall 2 rise 2 } [root@ka1 ~]#vim /etc/keepalived/check_nginx.sh #如检查nginx成功,返回成功;失败重启nginx,成功返回成功,失败返回失败 killall -0 nginx || systemctl restart nginx
标签:KEEPALIVED,keepalived,vrrp,etc,ka1,集群,conf,ubuntu,root From: https://www.cnblogs.com/ludingchao/p/18440234