一、实验环境准备
镜像选择:CentOS-7-x86_64-DVD-2009.iso
配置:4核、6G内存、80G硬盘
兼容性:ESXI 7.0及更高版本
服务器信息:
k8s集群角色 | ip | 主机名 | 安装的组件 |
---|---|---|---|
控制节点 | 10.104.26.192 | hqs-master | apiserver、controller-manager、scheduler、etcd、kube-proxy、docker、calico |
工作节点 | 10.104.26.193 | hqs-node1 | kubelet、kube-proxy、docker、calico、coredns |
工作节点 | 10.104.26.194 | hqs-node2 | kubelet、kube-proxy、docker、calico、coredns |
1、kubeadm 和二进制安装 k8s 适用场景分析
kubeadm是官方提供的开源工具,是一个开源项目,用于快速搭建 kubernetes 集群。
kubeadmin init
和 kubeadm join
是 kubeadm 中最重要的两个命令,前者用于初始化集群,后者用于加入节点。
kubeadm 初始化 k8s 的过程中,会创建 kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kube-proxy 等组件,这些组件都是以 Pod 的形式运行在 k8s 集群中的,具备故障自恢复能力。
二进制安装 k8s,需要自己手动创建这些组件,而且二进制安装的 k8s 集群,不具备故障自恢复能力。
二进制安装的 k8s 集群,不具备故障自恢复能力,而且二进制安装的 k8s 集群,不支持 kubeadm 升级。所以,kubeadm 适合用于生产环境,二进制安装适合用于测试环境。
2、初始化部署环境
(1)修改机器IP为静态IP
# 控制节点
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.192
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24
# 工作节点1
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.193
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24
# 工作节点2
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.194
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24
(2)修改主机名
# 控制节点
[root@localhost ~]# hostnamectl set-hostname hqs-master && bash
[root@hqs-master ~]#
# 工作节点1
[root@localhost ~]# hostnamectl set-hostname hqs-node1 && bash
[root@hqs-node1 ~]#
# 工作节点2
[root@localhost ~]# hostnamectl set-hostname hqs-node2 && bash
[root@hqs-node2 ~]#
(3)修改hosts文件
让各个节点都能够通过主机名访问到其他节点。修改每个机器的 /etc/hosts
文件,执行如下内容:
echo '10.104.26.192 hqs-master
10.104.26.193 hqs-node1
10.104.26.194 hqs-node2' >> /etc/hosts
(4)配置主机间免密登录
k8s 集群中的各个节点之间需要通过 ssh 进行通信,所以需要配置主机间免密登录。
控制节点执行如下命令:
# 生成密钥(一路回车)
[root@hqs-master ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:/GSg0F2vu/qUv1TqjaHDrsm4Fgj08wwmzjw2HbqBLB8 root@hqs-master
The keys randomart image is:
+---[RSA 2048]----+
| . |
| . . . . . |
| . .. . o . |
| o *. o . . |
|.= * B. S + . |
|o.E o + + o o |
|.o * . .= + |
| o .o oo* + |
| .o.===.=.. |
+----[SHA256]-----+
# 将本地生成的密钥文件和私钥文件拷贝到其他节点
[root@hqs-master ~]# ssh-copy-id hqs-master
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-master (10.104.26.192)' cant be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-masters password: <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hqs-master'"
and check to make sure that only the key(s) you wanted were added.
[root@hqs-master ~]# ssh-copy-id hqs-node1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-node1 (10.104.26.193)' can't be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-node1's password: <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hqs-node1'"
and check to make sure that only the key(s) you wanted were added.
[root@hqs-master ~]# ssh-copy-id hqs-node2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-node2 (10.104.26.194)' can't be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-node2's password: <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hqs-node2'"
and check to make sure that only the key(s) you wanted were added.
node1 和 node2 也执行如上命令,将密钥文件拷贝到其他节点。
# node1
[root@hqs-node1 ~]# ssh-keygen
[root@hqs-node1 ~]# ssh-copy-id hqs-master
[root@hqs-node1 ~]# ssh-copy-id hqs-node1
[root@hqs-node1 ~]# ssh-copy-id hqs-node2
# node2
[root@hqs-node2 ~]# ssh-keygen
[root@hqs-node2 ~]# ssh-copy-id hqs-master
[root@hqs-node2 ~]# ssh-copy-id hqs-node1
[root@hqs-node2 ~]# ssh-copy-id hqs-node2
(5)关闭交换分区swap
swap交换分区,在机器内存不够时会使用,但是swap分区的性能较低,k8s设计时为了提升性能,默认不允许使用交换分区。
kubeadm初始化的时候会检查是否开启了swap分区,如果开启了,会报错,所以需要关闭swap分区。如果不想关闭交换分区,可以使用--ignore-preflight-errors=Swap
参数忽略检查。
# 临时关闭
[root@hqs-master ~]# swapoff -a
[root@hqs-node1 ~]# swapoff -a
[root@hqs-node2 ~]# swapoff -a
# 永久关闭————注释swap挂载
[root@hqs-master ~]# vim /etc/fstab
#/dev/mapper/centos-swap swap swap defaults 0 0
# 这两台因为时克隆的机器,还需要删除UUID
[root@hqs-node1 ~]# vim /etc/fstab
#UUID=b64332af-5acd-4202-8dbe-8dc83c50bfae /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
[root@hqs-node2 ~]# vim /etc/fstab
#UUID=b64332af-5acd-4202-8dbe-8dc83c50bfae /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
(6)修改机器内核参数
k8s需要修改机器内核参数,使其支持路由转发和桥接,否则会报错。
# 1.master节点修改
# 临时修改,加载 br_netfilter 模块(桥接模块)
[root@hqs-master ~]# modprobe br_netfilter
# 永久修改
[root@hqs-master ~]# echo "modprobe br_netfilter" >> /etc/profile
# k8s.conf作用是修改内核参数,使其支持路由转发和桥接
[root@hqs-master ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 使配置生效
# sysctl 命令用于在运行时动态地修改内核的运行参数,可用于修改网络相关的参数
[root@hqs-master ~]# sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
# 查看内核参数
[root@hqs-master ~]# sysctl net.bridge.bridge-nf-call-ip6tables
# 2.node1节点修改
[root@hqs-node1 ~]# modprobe br_netfilter
[root@hqs-node1 ~]# echo "modprobe br_netfilter" >> /etc/profile
[root@hqs-node1 ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
[root@hqs-node1 ~]# sysctl -p /etc/sysctl.d/k8s.conf
# 3.node2节点修改
[root@hqs-node2 ~]# modprobe br_netfilter
[root@hqs-node2 ~]# echo "modprobe br_netfilter" >> /etc/profile
[root@hqs-node2 ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
[root@hqs-node2 ~]# sysctl -p /etc/sysctl.d/k8s.conf
修改上述内核参数主要是为了解决以下问题:
- 问题1:执行
sysctl -p /etc/sysctl.d/k8s.conf
出现报错:sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
。该问题是因为没有加载br_netfilter
模块,执行modprobe br_netfilter
加载该模块即可。 - 问题2:安装docker后,执行
docker info
出现报错:WARNING: bridge-nf-call-iptables is disabled. WARNING: bridge-nf-call-ip6tables is disabled.
。该问题是因为没有开启路由转发,执行sysctl net.bridge.bridge-nf-call-ip6tables=1
和sysctl net.bridge.bridge-nf-call-iptables=1
开启路由转发即可。 - 问题3:kubeadm初始化k8s报错:
ERROR FileContent--proc-sys-net-ipv4-ip_forward contents are not set to 1
。该问题是因为没有开启路由转发,执行sysctl net.ipv4.ip_forward=1
开启路由转发即可。
(7)关闭防火墙
关闭防火墙,或者开放k8s需要的端口。
[root@hqs-master ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@hqs-node1 ~]# systemctl stop firewalld && systemctl disable firewalld
[root@hqs-node2 ~]# systemctl stop firewalld && systemctl disable firewalld
(8)关闭selinux
关闭selinux,或者设置为permissive模式。
# 修改配置文件
[root@hqs-master ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@hqs-node1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@hqs-node2 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
# 修改selinux配置后,需要重启机器,配置才能永久生效
[root@hqs-master ~]# reboot
[root@hqs-node1 ~]# reboot
[root@hqs-node2 ~]# reboot
# 查看selinux状态
[root@hqs-master ~]# getenforce
Disabled
(9)配置阿里云yum源
配置阿里云yum源,或者使用其他yum源。
# 删除原有yum源
[root@hqs-master ~]# rm -rf /etc/yum.repos.d/*.repo
[root@hqs-node1 ~]# rm -rf /etc/yum.repos.d/*.repo
[root@hqs-node2 ~]# rm -rf /etc/yum.repos.d/*.repo
# 下载阿里云yum源
[root@hqs-master ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 如没有wget命令,可上传CentOS-Base.repo文件到/etc/yum.repos.d/目录下
[root@hqs-master yum.repos.d]# scp CentOS-Base.repo root@hqs-node1:/etc/yum.repos.d/
CentOS-Base.repo 100% 2523 253.5KB/s 00:00
[root@hqs-master yum.repos.d]# scp CentOS-Base.repo root@hqs-node2:/etc/yum.repos.d/
CentOS-Base.repo 100% 2523 253.5KB/s 00:00
# 清除缓存
[root@hqs-master ~]# yum clean all && yum makecache
[root@hqs-node1 ~]# yum clean all && yum makecache
[root@hqs-node2 ~]# yum clean all && yum makecache
# 安装lrzsz、scp、vim、wget、net-tools
[root@hqs-master ~]# yum install -y lrzsz scp vim wget net-tools yum-utils
[root@hqs-node1 ~]# yum install -y lrzsz scp vim wget net-tools yum-utils
[root@hqs-node2 ~]# yum install -y lrzsz scp vim wget net-tools yum-utils
# 配置国内docker的repo源
[root@hqs-master ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@hqs-master ~]# ls /etc/yum.repos.d/
CentOS-Base.repo docker-ce.repo
[root@hqs-node1 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@hqs-node2 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 配置epel源
# 下载epel源
[root@hqs-master ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
--2023-06-21 13:47:03-- http://mirrors.aliyun.com/repo/epel-7.repo
Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 119.96.33.219, 182.40.59.176, 182.40.41.199, ...
Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|119.96.33.219|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 664 [application/octet-stream]
Saving to: ‘/etc/yum.repos.d/epel.repo’
100%[=======================================================================================================>] 664 --.-K/s in 0s
2023-06-21 13:47:03 (147 MB/s) - ‘/etc/yum.repos.d/epel.repo’ saved [664/664]
[root@hqs-master ~]# scp /etc/yum.repos.d/epel.repo hqs-node1:/etc/yum.repos.d/
epel.repo 100% 664 1.0MB/s 00:00
[root@hqs-master ~]# scp /etc/yum.repos.d/epel.repo hqs-node2:/etc/yum.repos.d/
epel.repo 100% 664 1.2MB/s 00:00
(10)配置安装k8s组件的yum源
配置k8s组件的yum源,或者使用其他yum源。
# 编写kubernetes.repo文件
[root@hqs-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
[root@hqs-master ~]# ls /etc/yum.repos.d/
CentOS-Base.repo docker-ce.repo epel.repo kubernetes.repo
# 将kubernetes.repo文件分发到其他节点
[root@hqs-master ~]# scp /etc/yum.repos.d/kubernetes.repo hqs-node1:/etc/yum.repos.d/
kubernetes.repo 100% 129 145.5KB/s 00:00
[root@hqs-master ~]# scp /etc/yum.repos.d/kubernetes.repo hqs-node2:/etc/yum.repos.d/
kubernetes.repo 100% 129 237.0KB/s 00:00
(11)配置时间同步
使用ntpdate同步时间。
# 安装ntpdate
[root@hqs-master ~]# yum install -y ntpdate
[root@hqs-node1 ~]# yum install -y ntpdate
[root@hqs-node2 ~]# yum install -y ntpdate
# 与ntp服务器同步时间
[root@hqs-master ~]# ntpdate cn.pool.ntp.org
[root@hqs-node1 ~]# ntpdate cn.pool.ntp.org
[root@hqs-node2 ~]# ntpdate cn.pool.ntp.org
# 将时间同步命令写入crontab
[root@hqs-master ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
[root@hqs-node1 ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
[root@hqs-node2 ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
# 重启crontab服务
[root@hqs-master ~]# systemctl restart crond
[root@hqs-node1 ~]# systemctl restart crond
[root@hqs-node2 ~]# systemctl restart crond
(12)开启ipvs
开启ipvs,需要在所有节点上执行。
ipvs(IP Vertual Server):实现了传输层的负载均衡,是一种高性能、可扩展的负载均衡方案。
- Linux内核的一部分,承担着负载均衡的功能。
- 通过ipvsadm工具来配置ipvs。
- ipvs可基于TCP和UDP的服务请求转发到真实服务器上
ipvs和iptables的区别:
- ipvs是在传输层工作,iptables是在网络层工作。
- ipvs是基于内容的负载均衡,iptables是基于地址的负载均衡。
- ipvs为大型集群提供了高性能的负载均衡,iptables为小型集群提供了负载均衡。
- ipvs支持更多的负载均衡算法,iptables只支持轮询算法。
- ipvs支持服务器健康检查、连接重试等功能,iptables不支持。
# 准备ipvs.modules文件
[root@hqs-master ~]# cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in ${ipvs_modules}; do
/sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1
if [ 0 -eq 0 ]; then
/sbin/modprobe ${kernel_module}
fi
done
EOF
# 修改ipvs.modules文件权限并执行
[root@hqs-master ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
# 将ipvs.modules文件分发到其他节点
[root@hqs-master ~]# scp /etc/sysconfig/modules/ipvs.modules hqs-node1:/etc/sysconfig/modules/
ipvs.modules 100% 164 188.5KB/s 00:00
[root@hqs-master ~]# scp /etc/sysconfig/modules/ipvs.modules hqs-node2:/etc/sysconfig/modules/
ipvs.modules 100% 164 188.5KB/s 00:00
# 修改node节点 ipvs.modules文件权限并执行
[root@hqs-node1 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
[root@hqs-node2 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
(13)安装基础软件包
[root@hqs-master ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet
[root@hqs-node1 ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet
[root@hqs-node2 ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet