底层网络沙箱容器pause无法下载报错

#如出现报错，是因为底层沙箱容器pause无法下载导致出现的问题
Events:
  Type     Reason                  Age                     From           Message
  ----     ------                  ----                    ----           -------
  Warning  FailedCreatePodSandBox  2m51s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3d1bfa6d85d4f6f8564afeabb38dbcd98fd747843e6baff8a3ddd8e09d16c519" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m50s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a95717455f40fc6ca915da95a30c68403aa9d79d6091d41d8aba4207ae0c082a" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m49s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ee9bacea2ded971659178d6b571141591e6b16afd2ba287d343813b0a01bd0c2" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m48s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "205c9fea36af5c48db74fe8c225604fea0419eec06a2d129bec67c0fa2073883" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m47s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "74e067c04f2f0b37ee6c2fc805cdb98dbd0b1113d39a507dc98cfe6c128f2683" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m46s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "32b368d478f645836d913e7a18e1e76de61864ae9e816e6eb54f527a19859e24" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m45s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8061bf7c355c2156b681310cb3b661d67732dbc56d7a786d7eeb752d1542ae99" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m43s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5e59dd8a3def744bb49bb8c9f8733438f10c53d185dc3c2841ee795bc92e8fa9" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  FailedCreatePodSandBox  2m42s                   kubelet        Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "96d47ca1517051bdf48655eb2c7c5739b376be7e7251ee8546840c874ef4a7b4" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Normal   SandboxChanged          2m39s (x12 over 2m50s)  kubelet        Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  2m38s (x4 over 2m41s)   kubelet        (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fb9d0c03fed84d94d69a86f2fd2776658966e6f81abe91ac46089cfb0bf776a6" network for pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d": networkPlugin cni failed to set up pod "myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d_myserver" network: plugin type="hybridnet" failed (add): request ip return 400 failed to wait for pod myserver-tomcat-app1-deployment-underlay-b4ff64fd8-r6j2d/myserver to be coupled with ip, expect 1 and get 0
  Warning  IPAllocationFail        8s (x16 over 2m52s)     PodController  unable to select network: unable to find underlay network for node node3, should check webhook liveness


#解决
生成containerd的配置文件
mkdir /etc/containerd -p 
生成配置文件
containerd config default > /etc/containerd/config.toml
编辑配置文件
vim /etc/containerd/config.toml
# sandbox_image = "k8s.gcr.io/pause:3.6"
改为：
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"
SystemdCgroup = false 改为 SystemdCgroup = true#因为使用docker这里可以不修改如果使用containerd需要修改
#重启
systemctl restart containerd
因为底层网络沙箱容器pause:3.9下载的问题使用cri-docker的underlay网络部署失败

kubeadm初始化报错

初始化报错
root@master1:~# kubeadm init --apiserver-advertise-address=10.0.0.30 --apiserver-bind-port=6443 --kubernetes-version=v1.27.6 --pod-network-cidr=10.200.0.0/16 --service-cidr=10.0.10.0/16 --service-dns-domain=cluster.local --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap --cri-socket unix:///var/run/cri-dockerd.sock
[init] Using Kubernetes version: v1.27.6
[preflight] Running pre-flight checks
    [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
[preflight] Pulling images required for setting up a Kubernetes cluster

#报错信息如下
W1215 17:24:20.372975    4553 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.6, falling back to the nearest etcd version (3.5.7-0)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

Unfortunately, an error has occurred:
    timed out waiting for the condition

This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
    - 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

检查kubelet

root@master1:~# journalctl -xeu kubelet
░░
░░ The job identifier is 9109.
Dec 15 17:29:38 master1 kubelet[5073]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for mor>
Dec 15 17:29:38 master1 kubelet[5073]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.249499 5073 server.go:199] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.255491 5073 server.go:415] "Kubelet version" kubeletVersion="v1.27.2"
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.255716 5073 server.go:417] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.256157 5073 server.go:837] "Client rotation is on, will bootstrap in background"
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.258528 5073 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.260752 5073 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Dec 15 17:29:38 master1 kubelet[5073]: I1215 17:29:38.270224 5073 server.go:662] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Dec 15 17:29:38 master1 kubelet[5073]: E1215 17:29:38.270459 5073 run.go:74] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename\t\t\t\tType\t\>
Dec 15 17:29:38 master1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

#command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps

contained统计swap交换空间没有关闭

解决方法

swapoff -a

sed -ri 's/.*swap.*/#&/' /etc/fstab