二机制安装Kubernetes 1.29 高可用集群(1)--系统初始配置

软件环境描述说明：

• OS：openEuler 22.03 LTS SP3
• kubernetes：1.29.2
• etcd：3.5.12
• calico：3.27.0
• cri-docker：0.3.10
• cfssl/cfssljson/cfssl-certinfo：1.64
• haproxy：2.2.6
• keepalived：2.2.4

网段规划：

物理主机：192.168.83.0/24       Service IP：10.66.0.0/16      Pod IP：172.31.0.0/16

服务器规划：

参考文档：

https://www.cnblogs.com/xmwan/p/17940137

https://gitee.com/cby-inc/Kubernetes/blob/main/doc/v1.28.3-CentOS-binary-install-IPv6-IPv4-Three-Masters-Two-Slaves-Offline.md

1. 将所有节点openEuler系统显示时间修改为24小时制

cat >> /etc/profile << EOF
export LC_TIME=POSIX
EOF

source /etc/profile

2. 所有节点关闭防火墙和selinux

systemctl stop firewalld && systemctl disable firewalld
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0
sestatus

3.1 所有k8s节点安装软件包

dnf -y install net-tools lrzsz nmap lsof bash-completion ipset ipvsadm chrony socat conntrack ebtables tar

cat > /etc/bash.bashrc << EOF
# enable bash completion in interactive shells
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
EOF

source /etc/bash.bashrc

3.2 所有haproxy和etcd节点安装软件包

dnf -y install net-tools lrzsz nmap lsof chrony tar

4. 1 所有k8s-node节点安装docker

#添加docker安装源
dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

#因docker源中没有openEuler的版本，需要将$releasever替换为openEuler兼容的CentOS8
sed -i s/\$releasever/8/g /etc/yum.repos.d/docker-ce.repo

#安装docker-ce
dnf -y install docker-ce

#添加docker镜像加速源
cat > /etc/docker/daemon.json << EOF
{
        "exec-opts": ["native.cgroupdriver=systemd"],
        "registry-mirrors":["https://hub-mirror.c.163.com",
                            "https://docker.mirrors.ustc.edu.cn"]
}
EOF

4.2 启动所有k8s-node节点docker

systemctl enable --now docker
docke info

5.1 将两个haproxy节点配置为NTP Server

vi /etc/chrony.conf
pool ntp.aliyun.com iburst
pool ntp.tencent.com iburst
allow 192.168.83.0/24

systemctl enable --now chronyd

#验证同步状态
chronyc sourcestats -v

5.2 其它各节点与两个haproxy节点做时间同步 

vi /etc/chrony.conf
pool 192.168.83.201 iburst
pool 192.168.83.202 iburst

systemctl enable --now chronyd 

chronyc sourcestats -v

6. 所有节点配置hosts

cat >> /etc/hosts << EOF
192.168.83.201 k8s-haproxy01
192.168.83.202 k8s-haproxy02
192.168.83.210 k8s-master-etcd01
192.168.83.211 k8s-master-etcd02
192.168.83.212 k8s-etcd03
192.168.83.220 k8s-node01
192.168.83.221 k8s-node02
EOF

7. 所有k8s节点设置ulimit进程资源限制参数

ulimit -SHn 65535

cat >> /etc/security/limits.conf << EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF

8.1 所有k8s节点加载containerd所需的br_netfilter和overlay模块

cat > /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

#加载模块
modprobe overlay && modprobe br_netfilter

#查看是否加载
lsmod | grep br_netfilter
lsmod | grep overlay

8.2 所有k8s节点配置内核路由转发及网桥过滤并关闭swap

vi /etc/sysctl.conf
将net.ipv4.ip_forward=0
改为
net.ipv4.ip_forward=1
#最后添加以下内容
###k8s Config###
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
vm.swappiness=0
user.max_user_namespaces=28633

sysctl -p

9 .所有k8s节点配置ipvsadm模块加载

cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

#授权并运行
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules
echo 'bash /etc/sysconfig/modules/ipvs.modules' >> /etc/rc.local && chmod a+x /etc/rc.d/rc.local

#查看是否加载
lsmod | grep -e ip_vs -e nf_conntrack

10. 所有k8s节点关闭swap分区

vi /etc/fstab
......
#在加载swap的行前添加#号
#/dev/mapper/openeuler-swap none  swap defaults  0 0

#重启后生效，不重启临时关闭命令
swapoff -

11. 设置从k8s-master-etcd01节点免密登录到其他节点

[root@k8s-master-etcd01]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:D9YQOzG0M90OWqnQxZCMCTDd4zLVebxk5WEupwKUKjQ root@k8s-master-etcd01
The key's randomart image is:
+---[RSA 3072]----+
|  oo.o.B**..+    |
|  E...B.=O** .   |
| . . +o.O+*.=    |
|  . + .o X.*     |
|   . o  S o .    |
|       . +       |
|          .      |
|                 |
|                 |
+----[SHA256]-----+

[root@k8s-master-etcd01]# ssh-copy-id root@k8s-master-etcd02
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-etcd03
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-node01
[root@k8s-master-etcd01]# ssh-copy-id root@k8s-node02