环境条件
- 升级内核
- 修改内核参数
- 配置cgroup
- 安装CRI
内核参数
- 配置网络流量转发
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF
- 启用模块
modprobe overlay modprobe br_netfilter
- 调整内核参数net.ipv4.ip_forward
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF
- 启用内核参数without reboot
[23:07:40] root@kubernetes-template-os:~ # sysctl --system * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... kernel.yama.ptrace_scope = 0 * Applying /usr/lib/sysctl.d/50-coredump.conf ... kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e kernel.core_pipe_limit = 16 * Applying /usr/lib/sysctl.d/50-default.conf ... kernel.sysrq = 16 kernel.core_uses_pid = 1 kernel.kptr_restrict = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.promote_secondaries = 1 net.core.default_qdisc = fq_codel fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... net.core.optmem_max = 81920 * Applying /usr/lib/sysctl.d/50-pid-max.conf ... kernel.pid_max = 4194304 * Applying /etc/sysctl.d/60-gce-network-security.conf ... net.ipv4.tcp_syncookies = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 1 net.ipv4.conf.default.secure_redirects = 1 net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.conf.all.log_martians = 1 net.ipv4.conf.default.log_martians = 1 kernel.randomize_va_space = 2 kernel.panic = 10 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.d/k8s.conf ... net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 * Applying /etc/sysctl.conf ... - 结果验证
[23:09:10] root@kubernetes-template-os:~ # sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 ## 验证 br_netfilter, overlay [23:09:14] root@kubernetes-template-os:~ # lsmod | grep br_netfilter br_netfilter 32768 0 bridge 393216 1 br_netfilter [23:10:06] root@kubernetes-template-os:~ # lsmod | grep overlay overlay 188416 0
配置cgroup驱动
在Linux中CGroup的功能是用于限制分配给进程的资源，Kubernetes中的Kubelet与CRI都需要对cgroup控制组强势执行Pod的资源控制
注意：kubelet与CRI必须使用同一个cgroup，避免资源压力大造成的不稳定
Kubernetes支持二种cgroup
-
- cgroupfs是kubelet默认使用的cgroup驱动，但是当使用systemd初始化系统时，不建议使用cgroupfs
systemd- systemd 与 cgroup 集成紧密，并将为每个 systemd 单元分配一个 cgroup。 因此，如果你
systemd用作初始化系统，同时使用cgroupfs驱动，则系统中会存在两个不同的 cgroup 管理器 - 修改kubelet的cgroup驱动，编辑
KubeletConfiguration的cgroupDriver选项，并将其设置为systemd。例如：apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration ... cgroupDriver: systemd
- systemd 与 cgroup 集成紧密，并将为每个 systemd 单元分配一个 cgroup。 因此，如果你
说明：从 v1.22 开始，在使用 kubeadm 创建集群时，如果用户没有在
KubeletConfiguration下设置cgroupDriver字段，kubeadm 默认使用systemd
CRI安装
CRI版本说明
Kubernetes 从 1.26 版本开始仅适用于 v1 版本的容器运行时（CRI）API。早期版本默认为 v1 版本， 但是如果容器运行时不支持 v1 版本的 API， 则 kubelet 会回退到使用（已弃用的）v1alpha2 版本的 API。
CRI支持选择
Containerd配置
选择使用containerd作为CRI，官方文档提供了三种安装方式
- 官方二进制，https://github.com/containerd/containerd/blob/main/docs/getting-started.md#option-1-from-the-official-binaries，共需要安装三个组件
- Download the
containerd-<VERSION>-<OS>-<ARCH>.tar.gzarchive from https://github.com/containerd/containerd/releases , verify its sha256sum, and extract it under/usr/local - Download the
runc.<ARCH>binary from https://github.com/opencontainers/runc/releases , verify its sha256sum, and install it as/usr/local/sbin/runc. - Download the
cni-plugins-<OS>-<ARCH>-<VERSION>.tgzarchive from https://github.com/containernetworking/plugins/releases , verify its sha256sum, and extract it under/opt/cni/bin
- Download the
- 使用apt-get 或者 dnf（docker官方YUM源）
- Docker官方YUM源的Containerd集成了runcf，但是不包含cni-plugins
- 源码安装
YUM安装
- 配置YUM仓库
[01:29:01] root@kubernetes-template-os:~ # dnf install -y yum-utils Last metadata expiration check: 1:32:41 ago on Sun 15 Oct 2023 11:56:31 PM CST. Dependencies resolved. ============================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================ Installing: yum-utils noarch 4.0.21-23.el8 baseos 80 k Transaction Summary ============================================================================================================================================================================================================ Install 1 Package Total download size: 80 k Installed size: 23 k Downloading Packages: yum-utils-4.0.21-23.el8.noarch.rpm 760 kB/s | 80 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 107 kB/s | 80 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : yum-utils-4.0.21-23.el8.noarch 1/1 Running scriptlet: yum-utils-4.0.21-23.el8.noarch 1/1 Verifying : yum-utils-4.0.21-23.el8.noarch 1/1 Installed: yum-utils-4.0.21-23.el8.noarch Complete! [01:33:07] root@kubernetes-template-os:~ # yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
- 安装最新版本Docker engine
[01:33:16] root@kubernetes-template-os:~ # dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Docker CE Stable - x86_64 509 kB/s | 50 kB 00:00 Last metadata expiration check: 0:00:01 ago on Mon 16 Oct 2023 01:38:21 AM CST. Dependencies resolved. ============================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================ Installing: containerd.io x86_64 1.6.24-3.1.el8 docker-ce-stable 34 M docker-buildx-plugin x86_64 0.11.2-1.el8 docker-ce-stable 13 M docker-ce x86_64 3:24.0.6-1.el8 docker-ce-stable 24 M docker-ce-cli x86_64 1:24.0.6-1.el8 docker-ce-stable 7.2 M docker-compose-plugin x86_64 2.21.0-1.el8 docker-ce-stable 13 M Installing dependencies: checkpolicy x86_64 2.9-1.el8 baseos 348 k container-selinux noarch 2:2.221.0-1.module_el8+653+feef7bfe appstream 69 k docker-ce-rootless-extras x86_64 24.0.6-1.el8 docker-ce-stable 4.9 M fuse-common x86_64 3.3.0-17.el8 baseos 22 k fuse-overlayfs x86_64 1.12-1.module_el8+454+d7ef4b8d appstream 70 k fuse3 x86_64 3.3.0-17.el8 baseos 54 k fuse3-libs x86_64 3.3.0-17.el8 baseos 95 k libcgroup x86_64 0.41-19.el8 baseos 70 k libslirp x86_64 4.4.0-1.module_el8+487+8e42a277 appstream 70 k policycoreutils-python-utils noarch 2.9-24.el8 baseos 260 k python3-audit x86_64 3.0.7-5.el8 baseos 87 k python3-libsemanage x86_64 2.9-9.el8 baseos 128 k python3-policycoreutils noarch 2.9-24.el8 baseos 2.3 M python3-setools x86_64 4.3.0-5.el8 baseos 627 k slirp4netns x86_64 1.2.1-1.module_el8+661+d1afb926 appstream 56 k Enabling module streams: container-tools rhel8 Transaction Summary ============================================================================================================================================================================================================ Install 20 Packages
- 也可以安装指定版本
[01:41:23] root@kubernetes-template-os:~ # dnf list docker-ce --showduplicates | sort -r Last metadata expiration check: 0:03:11 ago on Mon 16 Oct 2023 01:38:21 AM CST. Installed Packages docker-ce.x86_64 3:24.0.6-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.6-1.el8 @docker-ce-stable docker-ce.x86_64 3:24.0.5-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.4-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.3-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.2-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.1-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.0-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.6-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.5-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.4-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.3-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.2-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.1-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.0-1.el8 docker-ce-stable dnf install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io docker-buildx-plugin docker-compose-plugin
Containerd配置文件
- 生成默认配置城文件
[03:04:54] root@kubernetes-template-os:~ # containerd config default > /etc/containerd/config.toml
- 配置systemd cgroup
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] ... [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true
安装Kubernetes软件包
- 配置YUM源
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF ## 查看YUM源 [03:11:26] root@kubernetes-template-os:~ # cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF ## 查看YUM源 [03:11:26] root@kubernetes-template-os:~ # cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl
- 关闭selinux
- 方法一，setenforce 0
- 方法二，sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config
- 安装kubelet、kubectl、kubeadm
dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet
- Kubernetes shell自动补全
echo 'source <(kubectl completion bash)' >>~/.bashrc echo 'source <(kubeadm completion bash)' >>~/.bashrc source ~/.bashrc