docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.2
kubeadm init --kubernetes-version=v1.23.2 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=172.16.0.0/16 --service-cidr=10.96.0.0/12 --apiserver-advertise-address=192.168.1.198 --ignore-preflight-errors=Swap --ignore-preflight-errors=all --v=5
I0222 17:18:35.573883 14392 initconfiguration.go:117] detected and using CRI socket: /var/run/dockershim.sock
I0222 17:18:35.573973 14392 kubelet.go:217] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
ignore-preflight-errors: Invalid value: "all,swap": don't specify individual checks if 'all' is used
[root@qbkj-k8s-master01 kubernetes]# kubeadm init --kubernetes-version=v1.23.2 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=172.16.0.0/16 --service-cidr=10.96.0.0/12 --apiserver-advertise-address=192.168.1.198 --ignore-preflight-errors=all --v=5
I0222 17:18:57.253598 14430 initconfiguration.go:117] detected and using CRI socket: /var/run/dockershim.sock
I0222 17:18:57.253668 14430 kubelet.go:217] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
[init] Using Kubernetes version: v1.23.2
[preflight] Running pre-flight checks
I0222 17:18:57.257119 14430 checks.go:578] validating Kubernetes and kubeadm version
I0222 17:18:57.257142 14430 checks.go:171] validating if the firewall is enabled and active
I0222 17:18:57.264077 14430 checks.go:206] validating availability of port 6443
I0222 17:18:57.264447 14430 checks.go:206] validating availability of port 10259
I0222 17:18:57.264475 14430 checks.go:206] validating availability of port 10257
I0222 17:18:57.264499 14430 checks.go:283] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0222 17:18:57.264512 14430 checks.go:283] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0222 17:18:57.264549 14430 checks.go:283] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0222 17:18:57.264562 14430 checks.go:283] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0222 17:18:57.264575 14430 checks.go:433] validating if the connectivity type is via proxy or direct
I0222 17:18:57.264609 14430 checks.go:472] validating http connectivity to first IP address in the CIDR
I0222 17:18:57.264627 14430 checks.go:472] validating http connectivity to first IP address in the CIDR
I0222 17:18:57.264644 14430 checks.go:107] validating the container runtime
I0222 17:18:57.386519 14430 checks.go:133] validating if the "docker" service is enabled and active
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
I0222 17:18:57.396420 14430 checks.go:332] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0222 17:18:57.396454 14430 checks.go:332] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0222 17:18:57.396485 14430 checks.go:654] validating whether swap is enabled or not
I0222 17:18:57.396513 14430 checks.go:373] validating the presence of executable conntrack
I0222 17:18:57.396528 14430 checks.go:373] validating the presence of executable ip
I0222 17:18:57.396540 14430 checks.go:373] validating the presence of executable iptables
I0222 17:18:57.396558 14430 checks.go:373] validating the presence of executable mount
I0222 17:18:57.396572 14430 checks.go:373] validating the presence of executable nsenter
I0222 17:18:57.396592 14430 checks.go:373] validating the presence of executable ebtables
I0222 17:18:57.396605 14430 checks.go:373] validating the presence of executable ethtool
I0222 17:18:57.396619 14430 checks.go:373] validating the presence of executable socat
I0222 17:18:57.396634 14430 checks.go:373] validating the presence of executable tc
I0222 17:18:57.396647 14430 checks.go:373] validating the presence of executable touch
I0222 17:18:57.396661 14430 checks.go:521] running all checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 20.10
I0222 17:18:57.520021 14430 checks.go:404] checking whether the given node name is valid and reachable using net.LookupHost
I0222 17:18:57.520231 14430 checks.go:620] validating kubelet version
I0222 17:18:57.569110 14430 checks.go:133] validating if the "kubelet" service is enabled and active
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0222 17:18:57.575964 14430 checks.go:206] validating availability of port 10250
I0222 17:18:57.576004 14430 checks.go:206] validating availability of port 2379
I0222 17:18:57.576026 14430 checks.go:206] validating availability of port 2380
I0222 17:18:57.576048 14430 checks.go:246] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0222 17:18:57.576123 14430 checks.go:842] using image pull policy: IfNotPresent
I0222 17:18:57.641754 14430 checks.go:851] image exists: registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.2
I0222 17:18:57.707357 14430 checks.go:851] image exists: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.2
I0222 17:18:57.772058 14430 checks.go:851] image exists: registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.2
I0222 17:18:57.836981 14430 checks.go:851] image exists: registry.aliyuncs.com/google_containers/kube-proxy:v1.23.2
I0222 17:18:57.900527 14430 checks.go:851] image exists: registry.aliyuncs.com/google_containers/pause:3.6
I0222 17:18:57.973392 14430 checks.go:859] pulling: registry.aliyuncs.com/google_containers/etcd:3.5.1-0
I0222 17:21:22.159131 14430 checks.go:859] pulling: registry.aliyuncs.com/google_containers/coredns:v1.8.6
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0222 17:22:04.318903 14430 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0222 17:22:04.446610 14430 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local qbkj-k8s-master01] and IPs [10.96.0.1 192.168.1.198]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0222 17:22:04.671763 14430 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0222 17:22:04.733925 14430 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0222 17:22:04.947372 14430 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0222 17:22:05.083449 14430 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost qbkj-k8s-master01] and IPs [192.168.1.198 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost qbkj-k8s-master01] and IPs [192.168.1.198 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0222 17:22:05.889171 14430 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0222 17:22:06.046254 14430 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0222 17:22:06.342144 14430 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0222 17:22:06.423619 14430 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0222 17:22:06.494803 14430 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0222 17:22:06.924578 14430 kubelet.go:65] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0222 17:22:07.024546 14430 manifests.go:99] [control-plane] getting StaticPodSpecs
I0222 17:22:07.024814 14430 certs.go:522] validating certificate period for CA certificate
I0222 17:22:07.024880 14430 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0222 17:22:07.024889 14430 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I0222 17:22:07.024896 14430 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0222 17:22:07.027035 14430 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0222 17:22:07.027051 14430 manifests.go:99] [control-plane] getting StaticPodSpecs
I0222 17:22:07.027227 14430 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0222 17:22:07.027236 14430 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I0222 17:22:07.027245 14430 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0222 17:22:07.027250 14430 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0222 17:22:07.027255 14430 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0222 17:22:07.028671 14430 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0222 17:22:07.028690 14430 manifests.go:99] [control-plane] getting StaticPodSpecs
I0222 17:22:07.028866 14430 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0222 17:22:07.029281 14430 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0222 17:22:07.029877 14430 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0222 17:22:07.029893 14430 waitcontrolplane.go:91] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.502142 seconds
I0222 17:22:13.533255 14430 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0222 17:22:13.548416 14430 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
I0222 17:22:13.560379 14430 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I0222 17:22:13.560398 14430 patchnode.go:31] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "qbkj-k8s-master01" as an annotation
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node qbkj-k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node qbkj-k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ysiq1e.sey7vmxp6jmey5t2
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0222 17:22:14.612302 14430 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I0222 17:22:14.612664 14430 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0222 17:22:14.612844 14430 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0222 17:22:14.614496 14430 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0222 17:22:14.624406 14430 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0222 17:22:14.625145 14430 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons] Applied essential addon: CoreDNS
I0222 17:22:14.995671 14430 request.go:597] Waited for 192.777895ms due to client-side throttling, not priority and fairness, request: POST:https://192.168.1.198:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.198:6443 --token ysiq1e.sey7vmxp6jmey5t2 \
--discovery-token-ca-cert-hash sha256:6e92ec003089ff087f2b0ccf1e094fb43c636c7a8436dce378ec050425fec1c3