本次集群部署采用的容器技术是containerd
系统版本:CentOs8.1
k8s版本:v1.20.4
containerd版本:ctr containerd.io 1.4.3
master: 192.168.43.151
node1: 192.168.43.152
node2: 192.168.43.153
1、初始化配置
#下载CentOS 8的repo文件(用阿里的)
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
安装 kubelet kubeadm kubectl
master、node节点都需要执行
添加 kubernetes.repo 文件
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 关闭SElinux
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
systemctl enable --now kubelet # 开机启动kubelet
systemctl disable --now firewalld # 关闭防火墙
# k8s要求关闭swap
swapoff -a && sysctl -w vm.swappiness=0 # 关闭swap
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab # 取消开机挂载swap
2,安装containerd
安装前配置
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# 设置必需的 sysctl 参数,这些参数在重新启动后仍然存在。
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# Apply sysctl params without reboot
sysctl --system
### 安装所需包
dnf install -y yum-utils device-mapper-persistent-data lvm2
### 新增 Docker 仓库
dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
## 安装 containerd
dnf install -y containerd.io
# 配置 containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 重启 containerd
systemctl restart containerd
3、安装kubelet kubeadm kubectl
dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
#--disableexcludes=kubernetes 禁掉除了这个之外的别的仓库
#kubelet配置修改
# config kubelet cgroup
cat > /etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock
EOF
# config CRI
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
vi /etc/containerd/config.toml
# [plugins."io.containerd.grpc.v1.cri"] 下的 sandbox_image
# 将k8s.gcr.io/pause:3.2修改为如下
sandbox_image="registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"
# 加上 SystemdCgroup = true
在[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]中加入
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
# 重启 containerd
sudo systemctl restart containerd
# 重启 kubelet
sudo systemctl restart kubelet
4,准备镜像
kubeadm config images pull
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.4
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.4
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.20.4
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.20.4
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
ctr image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
ctr images list
5,启动集群
master启动
kubeadm init --kubernetes-version=v1.20.4 --apiserver-advertise-address=192.168.43.151 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-network-cidr=192.168.0.0/16
# --image-repository:指定镜像源
-----执行结果----
[root@ck8s1 ~]# kubeadm init --kubernetes-version=v1.20.4 --apiserver-advertise-address=192.168.43.151 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-network-cidr=192.168.0.0/16
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ck8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 192.168.43.151]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ck8s1 localhost] and IPs [192.168.43.151 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ck8s1 localhost] and IPs [192.168.43.151 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 80.003966 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ck8s1 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node ck8s1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7pw2tl.x2121e9p3dgr7ktb
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.43.151:6443 --token 7pw2tl.x2121e9p3dgr7ktb \
--discovery-token-ca-cert-hash sha256:694e2924e2cf2211cefba492faf0259d0810d6afb72c5e941120b2911d8d18ab
----------------------------------------------
#启动之后,需要再执行以下,否则在使用kubectl的时候会报8080端口错误:
# 非root用户
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
# root用户
export KUBECONFIG=/etc/kubernetes/admin.conf
6,应用calico网络
#kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
[root@ck8s1 ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
7,node节点加入:
kubeadm join 192.168.43.151:6443 --token 7pw2tl.x2121e9p3dgr7ktb \
--discovery-token-ca-cert-hash sha256:694e2924e2cf2211cefba492faf0259d0810d6afb72c5e941120b2911d8d18ab --cri-socket /run/containerd/containerd.sock
#前半部分是master输出的语句,最后添加cri的参数
--cri-socket /run/containerd/containerd.sock 的作用是指定cri为containerd
#查看node状态
#kubectl get node
[root@ck8s1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
ck8s1 Ready control-plane,master 172m v1.20.4
ck8s2 Ready <none> 84m v1.20.4
ck8s3 Ready <none> 80m v1.20.4
# 查看pod状态
kubectl get pod -A
[root@ck8s1 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6949477b58-zzfnb 1/1 Running 0 98m
kube-system calico-node-7w7p5 1/1 Running 0 83m
kube-system calico-node-9j57n 1/1 Running 0 98m
kube-system calico-node-zdxtq 0/1 Running 0 80m
kube-system coredns-54d67798b7-d426f 1/1 Running 0 172m
kube-system coredns-54d67798b7-pzbgf 1/1 Running 0 172m
kube-system etcd-ck8s1 1/1 Running 0 172m
kube-system kube-apiserver-ck8s1 1/1 Running 0 172m
kube-system kube-controller-manager-ck8s1 1/1 Running 0 172m
kube-system kube-proxy-4pbzn 1/1 Running 0 83m
kube-system kube-proxy-5cbcr 1/1 Running 0 172m
kube-system kube-proxy-75c6f 1/1 Running 0 80m
kube-system kube-scheduler-ck8s1 1/1 Running 0 172m
8,错误排查和解决方法
--报错1
[preflight] Running pre-flight checks
[WARNING FileExisting-tc]: tc not found in system path
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
解决方法:
dnf install tc -y
modprobe br_netfilter && echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
--报错2:
Warning FailedCreatePodSandBox 53m (x9 over 73m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.2": failed to pull image "k8s.gcr.io/pause:3.2": failed to pull and unpack image "k8s.gcr.io/pause:3.2": failed to resolve reference "k8s.gcr.io/pause:3.2": failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.2: dial tcp 64.233.187.82:443: i/o timeout
Warning FailedCreatePodSandBox 43m (x3 over 44m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.2": failed to pull image "k8s.gcr.io/pause:3.2": failed to pull and unpack image "k8s.gcr.io/pause:3.2": failed to resolve reference "k8s.gcr.io/pause:3.2": failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.2: dial tcp 74.125.204.82:443: i/o timeout
解决方法:
第一种:
vim /etc/containerd/config.toml
# [plugins."io.containerd.grpc.v1.cri"] 下的 sandbox_image
# 将k8s.gcr.io/pause:3.2修改为如下(替换为可下载的镜像地址)
sandbox_image="registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"
其实还有其它的方法也能规避掉此问题,不用改config.toml文件,ctr image pull 把镜像下来,指定namespace 例如:
第二种:
下载镜像
ctr -n k8s.io image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
将下载的镜像打上k8s.gcr.io的tag
ctr -n k8s.io image tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
标签:kube,kubernetes,--,containerd,v1.20,io,k8s From: https://blog.51cto.com/u_15941303/6008796