在Centos7 安装最新版本 Kubernetes + Docker

1.容器运行时

说明：v1.24 之前的 Kubernetes 版本直接集成了 Docker Engine 的一个组件，名为 dockershim。自 1.24 版起，Dockershim 已从 Kubernetes 项目中移除。

需要在集群内每个节点上安装一个 容器运行时 以使 Pod 可以运行在上面。
Kubernetes 1.29 要求使用符合容器运行时接口（CRI）的运行时

卸载旧版本的 Docker

sudo yum remove docker*
sudo rm -rf /var/lib/docker/*

安装 Docker Engine、containerd、Docker Compose:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

启动 Docker Engine

    sudo systemctl start docker
    sudo systemctl enable docker
    sudo systemctl status docker

验证 Docker 安装是否成功

    sudo docker run hello-world

修改 docker 的cgroup

kubernetes 默认的 cgroup 驱动是 systemd

修改方式一：修改 Docker 的服务文件

启动命令追加一个参数 --exec-opt native.cgroupdrive=systemd

vi /usr/lib/systemd/system/docker.service
··ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd··

或者 使用 sed 替换
sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service 

修改方式二：修改 Docker 的守护进程配置

cat >> /etc/docker/daemon.json <<-EOF 
{ 
"exec-opts": ["native.cgroupdriver=systemd"] 
} 
EOF

systemctl restart docker
docker info | grep -i cgroup

2.配置 CRI 环境

Docker 使用的容器运行时接口为 cri-docker

获取软件

mkdir /data/softs && cd /data/softs wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.2/cri-dockerd-0.3.2.amd64.tgz

解压软件

tar xf cri-dockerd-0.3.2.amd64.tgz 
mv cri-dockerd/cri-dockerd /usr/local/bin/ 

检查效果

cri-dockerd --version

cri-docker Service 定制配置

cat > /etc/systemd/system/cri-dockerd.service<<-EOF 
[Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com 
After=network-online.target firewalld.service docker.service 
Wants=network-online.target 
[Service] Type=notify ExecStart=/usr/local/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --cri-dockerd-root-directory=/var/lib/dockershim --docker-endpoint=unix:///var/run/docker.sock --cri-dockerd-root-directory=/var/lib/docker ExecReload=/bin/kill -s HUP $MAINPID 
TimeoutSec=0 
RestartSec=2 
Restart=always 
StartLimitBurst=3 
StartLimitInterval=60s 
LimitNOFILE=infinity 
LimitNPROC=infinity 
LimitCORE=infinity 
TasksMax=infinity 
Delegate=yes 
KillMode=process 
[Install] WantedBy=multi-user.target EOF

cri-docker Socket 定制配置

cat > /etc/systemd/system/cri-dockerd.socket <<-EOF 
[Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service 
[Socket] ListenStream=/var/run/cri-dockerd.sock 
SocketMode=0660 
SocketUser=root 
SocketGroup=docker 
[Install] WantedBy=sockets.target EOF

设置服务开机自启动

systemctl daemon-reload 
systemctl enable cri-dockerd.service 
systemctl restart cri-dockerd.service

3.Kubernetes 环境初始化

关闭 SELinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

关闭swap分区或者禁用swap文件

方式一：带 swap 的行打 # 号
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a 

方式二：备份 /etc/fstab 然后 拷贝 除 swap 以外的 行
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak | grep -v swap > /etc/fstab

转发 IPv4 并让 iptables 看到桥接流量

cat <<EOF |  tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

设置所需的 sysctl 参数，参数在重新启动后保持不变

cat <<EOF |  tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

应用 sysctl 参数而不重新启动

sysctl --system

通过运行以下指令确认 br_netfilter 和 overlay 模块被加载：

lsmod | grep br_netfilter
lsmod | grep overlay

通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1：

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

启用内核模块,用于实现负载均衡和网络连接跟踪等功能

cat <<EOF |  tee /etc/sysconfig/modules/ipvs.modues
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

通过运行以下指令确认

cut -f1 -d " " /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4 

关闭防火墙

systemctl stop firewalld 
systemctl disable firewalld
systemctl mask firewalld
systemctl status firewalld

安装kubernetes集群工具 kubectl、kubeadm、kubelet

Kubernetes yum repository

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

更新软件源

yum makecache fast

Master 及所有节点部署

安装 kubelet, kubeadm and kubectl:

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
(Optional) Enable the kubelet service before running kubeadm:
systemctl enable --now kubelet
systemctl start kubelet
systemctl status kubelet ，查看服务状态，服务状态应该为Error（255）

err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml
原因：还没有开始搭建 kubernetes集群，所以该配置文件不存在
其他错误信息 journalctl -xe

Kubelet 配置 cgroup 驱动

cat >/etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

查看需要的镜像

kubeadm config images list  

无法访问才手动下载

在 Dockerhub（https://hub.docker.com/u/kubeimage）上查看相应的镜像及版本,可以忽略 
docker pull kubeimage/kube-apiserver-amd64:v1.29.3
docker pull kubeimage/kube-controller-manager-amd64:v1.29.3
docker pull kubeimage/kube-scheduler-amd64:v1.29.3
docker pull kubeimage/kube-proxy-amd64:v1.29.3
docker pull coredns/coredns:1.11.1
docker pull cnagent/pause:3.9

对下载下来的镜像重新打 tag

docker tag kubeimage/kube-apiserver-amd64:v1.29.3 kubeimage/kube-apiserver:v1.29.3
docker tag kubeimage/kube-controller-manager-amd64:v1.29.3 kubeimage/kube-controller-manager:v1.29.3
docker tag kubeimage/kube-scheduler-amd64:v1.29.3 kubeimage/kube-scheduler:v1.29.3
docker tag kubeimage/kube-proxy-amd64:v1.29.3 kubeimage/kube-proxy:v1.29.3

docker image ls

移除不需要的镜像

docker rmi {registry.k8s.io/kube-apiserver:v1.29.3,registry.k8s.io/kube-controller-manager:v1.29.3,registry.k8s.io/kube-scheduler:v1.29.3,registry.k8s.io/kube-proxy:v1.29.3}

导入镜像（手动）

docker image load < k8s-images.zip

4.Master节点初始化

单maseter节点集群

mkdir -p /k8sdata/log/
kubeadm init --apiserver-advertise-address=master节点的IP \
--kubernetes-version=v1.29.3 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=Swap \ 
--cri-socket=unix:///var/run/cri-dockerd.sock | tee /k8sdata/log/kubeadm-init.log

多master集群

mkdir -p /k8sdata/log/
kubeadm init --control-plane-endpoint=负载均衡IP \
--upload-certs \
--kubernetes-version=v1.29.3 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--cri-socket=unix:///var/run/cri-dockerd.sock | tee /k8sdata/log/kubeadm-init.log

定制kubernetes的登录权限

mkdir -p $HOME/.kube 
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
chown $(id -u):$(id -g) $HOME/.kube/config

命令补全 tab 可以补全可以输入的命令

! grep -q kubectl "$HOME/.bashrc" && echo "source /usr/share/bash-completion/bash_completion" >>"$HOME/.bashrc"
! grep -q kubectl "$HOME/.bashrc" && echo "source <(kubectl completion bash)" >>"$HOME/.bashrc"
! grep -q kubeadm "$HOME/.bashrc" && echo "source <(kubeadm completion bash)" >>"$HOME/.bashrc"
! grep -q crictl "$HOME/.bashrc" && echo "source <(crictl completion bash)" >>"$HOME/.bashrc"
source "$HOME/.bashrc"

安装网络系统（二选一）

Flannel
mkdir -p /k8sdata/network/
wget --no-check-certificate -O /k8sdata/network/flannelkube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl create -f /k8sdata/network/flannelkube-flannel.yml
或者
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Calico
mkdir -p /k8sdata/network/
wget --no-check-certificate -O /k8sdata/network/calico.yml https://docs.projectcalico.org/manifests/calico.yaml
kubectl create -f /k8sdata/network/calico.yml

5.Kubernetes 常用命令

# 获取节点
    kubectl get nodes -o wide

# 实时查询nodes状态
    watch kubectl get nodes -o wide

# 获取pod
    kubectl get pods --all-namespaces -o wide

# 查看镜像列表
    kubeadm config images list

# 节点加入集群
    kubeadm token create --print-join-command
    kubeadm join 192.168.254.130:6443 --token sgdjaz.1rzpwz4wdcox0l03 --discovery-token-ca-cert-hash sha256:1745a7ad9531b935f54d7b6187c1635ac950b6fbd3b0dfeda415b20914aaefe7

# 描述node
    kubectl describe node k8s-master
    
# 描述pod
    kubectl describe pod --namespace=kube-flannel