kubeadm在单master k8s集群中添加新节点

1.安装及配置nginx+keepalived

　　需要安装nginx(haproxy)+keepalived 为apiserver提供高可用master的vip。可以在master节点直接安装nginx+keepalive，但是由于80/443端口会被ingress占用，所以在本机安装只能提供apiserver的负载均衡，而使用另外的两台机器安装，就可以在nginx同时提供k8s集群所有node节点的80http和443https的负载均衡，用这个vip添加域名解析，不会有域名解析的单点故障。

　　由于测试服务器是华为云服务器，所以vip需要进行注册，否则不能解析，在弹性负载均衡-子网中申请虚拟IP地址，并对两台lb服务器进行绑定。

1)安装keepalived

yum install -y conntrack-tools libseccomp libtool-ltdl

yum -y install keepalived
2)配置keepalive

编辑keepalived配置文件

在lb1服务器:

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
     script_user root
     enable_script_security
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script nginx_check {
        script "/etc/keepalived/nginx_health.sh"
        interval 2
        weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.38.0.144
    }
    track_script {
        nginx_check
    }
}

在lb2服务器：
! Configuration File for keepalived

global_defs {
   notification_email {
   [email protected]
     [email protected]
     [email protected]
     script_user root
     enable_script_security
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script nginx_check {
        script "/etc/keepalived/nginx_health.sh"
        interval 2
        weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.38.0.144
    }
    track_script {
        nginx_check
    }
}
其中定义自定义资源监控脚本（vrrp_script）：nginx_check，通过调用（track_script）这个具体的脚本/etc/keepalived/nginx_health.sh来实现对nginx的监控,并根据监控的结果实现动态调整
在lb1及lb2两台服务器添加监控脚本,如果nginx进程数量为0那么重启nginx,过两秒后再次查询进程数量，如果进程数量仍为0则关闭keepalived.
vim /etc/keepalived/nginx_health.sh
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    systemctl restart nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
       systemctl stop keepalived
    fi
fi
3)安装nginx
yum install -y pcre  pcre-devel zlib  zlib-devel openssl openssl-devel
wget http://nginx.org/download/nginx-1.23.4.tar.gz
tar -zxvf nginx-1.23.4.tar.gz
cd nginx-1.23.4
./configure --prefix=/data/nginx --with-http_stub_status_module --with-http_ssl_module --with-stream
make && make install
4)配置nginx
在lb1,lb2两台服务器上
vim /data/nginx/conf/nginx.conf

user root;
worker_processes auto;
error_log logs/error.log;
pid logs/nginx.pid;
#include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  logs/k8s-access.log  main;

    upstream k8s-http {
       server 10.38.0.50:80;
       server 10.38.0.58:80;
       server 10.38.0.166:80;
       server 10.38.0.77:80;

    }
    upstream k8s-https {
       server 10.38.0.50:443;
       server 10.38.0.58:443;
       server 10.38.0.166:443;
       server 10.38.0.77:443;
    }
    upstream k8s-apiserver {
       server 10.38.0.50:6443;
       server 10.38.0.58:6443;
       server 10.38.0.166:6443;
    }
    server {
       listen 80;
       proxy_connect_timeout 2s;
       proxy_timeout 5m;
       proxy_upload_rate 0;
       proxy_download_rate 0;
       proxy_buffer_size 4k;
       proxy_pass k8s-http;
    }
    server {
       listen 443;
       proxy_connect_timeout 2s;
       proxy_timeout 5m;
       proxy_upload_rate 0;
       proxy_download_rate 0;
       proxy_buffer_size 4k;
       proxy_pass k8s-https;
    }
   server {
       listen 26443;
       proxy_connect_timeout 2s;
       proxy_timeout 5m;
       proxy_upload_rate 0;
       proxy_download_rate 0;
       proxy_buffer_size 4k;
       proxy_pass k8s-apiserver;
    }
}

添加nginx到systemd服务
vim /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx
After=network.target

[Service]
Type=forking
PIDFile=/data/nginx/logs/nginx.pid
ExecStart=/data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
ExecReload=/data/nginx/sbin/nginx -s reload
ExecStop=/data/nginx/sbin/nginx -s quit
PrivateTmp=true

[Install]
WantedBy=multi-user.target
添加后可使用systemctl start nginx进行nginx的服务的启动