kubelet.go 2466 Error getting node not found

kubeadm初始化kubernetes集群报错，kubelet服务提示：kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found处理记录。

0. ENV

kubernetes 1.22.x/1.23.x/1.24.x(不限于所列版本)

CentOS 7.x/Ubuntu 22.04(不限于所列版本)

1. 问题现象

kubeadm初始化master失败

[root@k8s3-master ~]# kubeadm init --apiserver-advertise-address=192.168.90.125 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.23.8 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.23.8
[preflight] Running pre-flight checks
...
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.      #执行超过40s


  Unfortunately, an error has occurred:
    timed out waiting for the condition


  This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)


  If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'


  Additionally, a control plane component may have crashed or exited when started by the container runtime.
  To troubleshoot, list all containers using your preferred container runtimes CLI.


  Here is one example how you may list all Kubernetes containers running in docker:
    - 'docker ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'docker logs CONTAINERID'


error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster      #初始化失败
To see the stack trace of this error execute with --v=5 or higher

2. 分析

根据提示查看kubelet服务

[root@k8s3-master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2022-07-06 17:27:00 CST; 49s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 80133 (kubelet)
    Tasks: 26
   Memory: 32.1M
   CGroup: system.slice/kubelet.service
           └─80133 usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=regist...


Jul 06 17:27:49 k8s3-master kubelet[80133]: E0706 17:27:49.660120   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:49 k8s3-master kubelet[80133]: E0706 17:27:49.760811   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:49 k8s3-master kubelet[80133]: E0706 17:27:49.861505   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:49 k8s3-master kubelet[80133]: I0706 17:27:49.957202   80133 kubelet_node_status.go:70] "Attempting to register node" node="k8s3-master"
Jul 06 17:27:49 k8s3-master kubelet[80133]: E0706 17:27:49.962534   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:49 k8s3-master kubelet[80133]: E0706 17:27:49.998194   80133 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://192.168.90.125:6443/api/v1/nodes\": dial tcp 192.168....ode="k8s3-master"
Jul 06 17:27:50 k8s3-master kubelet[80133]: E0706 17:27:50.063267   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:50 k8s3-master kubelet[80133]: E0706 17:27:50.163691   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:50 k8s3-master kubelet[80133]: E0706 17:27:50.264587   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"
Jul 06 17:27:50 k8s3-master kubelet[80133]: E0706 17:27:50.365572   80133 kubelet.go:2466] "Error getting node" err="node \"k8s3-master\" not found"

服务虽然已经启动，但有报错，提示节点k8s3-master不能发现。

3. 进一步查看主机名配置

1) 查看当前主机名为k8s3-master

[root@k8s3-master ~]# hostname
k8s3-master

2) hostname文件中显示为k8s3-master无误

[root@k8s3-master ~]# cat etc/hostname 
k8s3-master

3) etc/hosts中显示无误

[root@k8s3-master ~]# cat etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


192.168.80.125 k8s3-master
192.168.80.126 k8s3-node1
192.168.80.127 k8s3-node2

当前确认主机名和对应IP无误，如果不正确，请及时更正。

4. 重置kubeadm并重新加载kubelet服务

当前master节点重置kubeadm

[root@k8s3-master ~]# kubeadm reset -f

重新加载kubelet服务

systemctl daemon-reload && systemctl restart kubelet && systemctl status kubelet

5. 因疏忽所在的问题原因-问题解决

最后发现kubeadm初始化IP地址写错,将错误的192.168.90.125更正为192.168.80.125,再次初始化成功。

[root@k8s3-master ~]# kubeadm init --apiserver-advertise-address=192.168.80.125 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.23.8 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --v=5
...
Your Kubernetes control-plane has initialized successfully!
...

6. 小结

kubeadm初始化master时，kubelet提示主机名未发现，需要确定主机名配置、hosts中(或DNS)的主机名解析是否匹配，另外还需要确认kubeadm init中IP正确，kubeadm使用配置文件初始化时，也需要核实配置文件中IP和主机名是否正确。

配置文件中初始化参考：

  advertiseAddress: 192.168.80.125  #1. 更改为master ip
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: master1                    #2. 更改为master主机名