使用KubeKey安装K8s集群
准备 Linux 主机
对主机的各种要求见官方文档多节点安装,下面只列一些重要的操作步骤
升级内核版本
# 如果使用Kube-proxy使用的是ipvs模式,一定的升级内核版本到4.1及以上
安装依赖
yum install -y socat conntrack ebtables ipset ipvsadm bash-completion
配置dns
解决无法拉取docker镜像的问题
# 在 /etc/resolv.conf 最后添加一行 nameserver 114.114.114.114 使用下面的命令即可完成
nmcli c modify ens33 ipv4.dns 114.114.114.114,8.8.8.8
nmcli c up ens33
问题日志:
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.4.1 --platform amd64"
Error response from daemon: Get "https://registry.cn-beijing.aliyuncs.com/v2/": dial tcp: lookup registry.cn-beijing.aliyuncs.com on 10.191.9.2:53: server misbehaving: Process exited with status 1
RedHat(CentOS7) 需要关闭SELinux
# 永久关闭SELinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# 临时生效
setenforce 0 #切换成宽容模式
#关闭后需要重启系统
reboot
# 查看SELinux的状态
getenforce
关闭防火墙
systemctl stop firewalld.service && systemctl disable firewalld && systemctl status firewalld
修改主机名
# 在每个节点上修改主机名,保证集群内的每个节点主机名称唯一
nmcli g hostname k8s.master
# 在 执行 ./kk create cluster -f config-sample.yaml 命令的节点上
# 修改 /etc/hosts 添加下面一行,来解决无法完成ssh 连接的问题
127.0.0.1 k8s.master
问题日志:
08:29:10 CST [GreetingsModule] Greetings
08:29:11 CST message: [localhost.k8s.18]
Greetings, KubeKey!
08:30:40 CST success: [localhost.k8s.18]
08:30:40 CST failed: [localhost.k8s.17]
08:30:40 CST failed: [localhost.k8s.16]
08:48:40 CST [GreetingsModule] Greetings
08:50:10 CST failed: [localhost.k8s.16]
08:50:10 CST failed: [localhost.k8s.18]
08:50:10 CST failed: [localhost.k8s.17]
下载 KubeKey
# 无法访问外网时 先执行以下命令以确保您从正确的区域下载 KubeKey
export KKZONE=cn
# 执行以下命令下载并解压 KubeKey
curl -sfL https://get-kk.kubesphere.io | VERSION=v3.1.6 sh -
创建集群
1.创建示例配置文件
# 创建配置文件 ,如果不带 --with-kubesphere 参数,就只会安装一个K8s集群,
# 后续可以通过其他方式直接在现有的k8s集群上安装 kubesphere
./kk create config --with-kubernetes v1.21.14 --with-kubesphere v3.4.1
2. 编辑配置文件
直接参照官方文档即可, 主要根据你主机的实际情况修改 hosts, roleGroups 的配置
3. 使用配置文件创建集群
chmod 755 kk && ./kk create cluster -f config-sample.yaml -y
问题:
下载镜像失败:
所有节点修改配置文件 /etc/docker/daemon.json
{
"registry-mirrors": [
"https://docker.rainbond.cc"
],
"log-opts": {
"max-size": "5m",
"max-file": "3"
},
"exec-opts": [
"native.cgroupdriver=systemd"
]
}
service的nodeport端口无法访问
# 查看kube-proxy 启动模式
kubectl get cm kube-proxy -n kube-system -o yaml | grep mode
# 如果为 iptables 模式
# 看看service对应的iptables规则是否生效 ks-console 为 service的名称
iptables-save |grep ks-console
## 输出结果为空，表示service没有正确生效
# 如果为 ipvs 模式 ,可以使用如下命令查看负载规则,如果为空的,表示有问题
ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.233.0.1:443 rr
-> 10.191.9.145:6443 Masq 1 4 0
TCP 10.233.0.3:53 rr
-> 10.233.87.1:53 Masq 1 0 0
-> 10.233.104.1:53 Masq 1 0 0
TCP 10.233.0.3:9153 rr
-> 10.233.87.1:9153 Masq 1 0 0
-> 10.233.104.1:9153 Masq 1 0 0
UDP 10.233.0.3:53 rr
-> 10.233.87.1:53 Masq 1 0 0
-> 10.233.104.1:53 Masq 1 0 0
kube-proxy 安装失败
# 找到kube-proxy的Pod
kubectl get pods -A |grep kube-proxy
# 查看proxy的启动日志
kubectl logs kube-proxy-5lql9 -n kube-system
## 下面日志可以看出来使用的是 IPVS 模式，
I1029 11:46:04.120985 1 node.go:172] Successfully retrieved node IP: 10.191.9.18
I1029 11:46:04.121536 1 server_others.go:140] Detected node IP 10.191.9.18
I1029 11:46:04.293286 1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I1029 11:46:04.293395 1 server_others.go:274] Using ipvs Proxier.
I1029 11:46:04.293415 1 server_others.go:276] creating dualStackProxier for ipvs.
W1029 11:46:04.293669 1 server_others.go:495] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
## 下面这一行可以看出来，内核版本过低，无法需要升级到最低 4.1版本
E1029 11:46:04.294302 1 proxier.go:390] can't set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1
W1029 11:46:04.294924 1 proxier.go:449] IPVS scheduler not specified, use rr by default
E1029 11:46:04.295269 1 proxier.go:390] can't set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1
W1029 11:46:04.295573 1 proxier.go:449] IPVS scheduler not specified, use rr by default
kube-proxy使用ipvs模式需要ipvsadm支持
# 错误日志
W1030 00:57:26.101775 1 server_others.go:495] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
#所有节点 安装依赖 建议重启下集群所有节点，否则iptables规则不会更新。
yum install -y ipvsadm
kubectl get pod -A |grep kube-proxy |awk '{print $2}'
添加命令自动补全
yum -y install bash-completion && \
echo 'source <(kubectl completion bash)' >>~/.bashrc && \
kubectl completion bash >/etc/bash_completion.d/kubectl && \
source /etc/profile
验证集群
检查所有pod是否是running状态
kubectl get pod -A
检查kube-proxy启动日志
kubectl -n kube-system logs kube-proxy-kw6dh -f
# 一般没有E开头的就表示正常
I1101 07:14:31.834660 1 node.go:172] Successfully retrieved node IP: 10.191.9.135
I1101 07:14:31.834935 1 server_others.go:140] Detected node IP 10.191.9.135
I1101 07:14:31.913553 1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I1101 07:14:31.913703 1 server_others.go:274] Using ipvs Proxier.
I1101 07:14:31.913780 1 server_others.go:276] creating dualStackProxier for ipvs.
W1101 07:14:31.913960 1 server_others.go:495] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
W1101 07:14:31.915501 1 proxier.go:449] IPVS scheduler not specified, use rr by default
W1101 07:14:31.916538 1 proxier.go:449] IPVS scheduler not specified, use rr by default
W1101 07:14:31.916637 1 ipset.go:113] ipset name truncated; [KUBE-6-LOAD-BALANCER-SOURCE-CIDR] -> [KUBE-6-LOAD-BALANCER-SOURCE-CID]
W1101 07:14:31.916664 1 ipset.go:113] ipset name truncated; [KUBE-6-NODE-PORT-LOCAL-SCTP-HASH] -> [KUBE-6-NODE-PORT-LOCAL-SCTP-HAS]
I1101 07:14:31.917043 1 server.go:647] Version: v1.21.14
I1101 07:14:31.939266 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
I1101 07:14:31.939467 1 conntrack.go:52] Setting nf_conntrack_max to 262144
I1101 07:14:31.940395 1 conntrack.go:83] Setting conntrack hashsize to 65536
I1101 07:14:31.946238 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1101 07:14:31.946562 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1101 07:14:31.947077 1 config.go:315] Starting service config controller
I1101 07:14:31.947140 1 shared_informer.go:240] Waiting for caches to sync for service config
I1101 07:14:31.947413 1 config.go:224] Starting endpoint slice config controller
I1101 07:14:31.947439 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
W1101 07:14:31.953614 1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W1101 07:14:31.958174 1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
I1101 07:14:32.047653 1 shared_informer.go:247] Caches are synced for endpoint slice config
I1101 07:14:32.047756 1 shared_informer.go:247] Caches are synced for service config
安装一个nginx服务
配置文件
apiVersion: v1
kind: Namespace
metadata:
name: nginx-namespace
---
apiVersion: v1
kind: Pod
metadata:
namespace: nginx-namespace
name: nginx-pod-1 ##pod名称
labels:
app: nginx ###写上这个pod的标签，方便svc连接
spec:
containers:
- name: nginx-stable-pod-1 ##pod容器名称
image: nginx:stable ###镜像
imagePullPolicy: IfNotPresent ###镜像拉取策略
ports:
- containerPort: 80 ###容器端口
---
apiVersion: v1
kind: Pod
metadata:
namespace: nginx-namespace
name: nginx-pod ##pod名称
labels:
app: nginx ###写上这个pod的标签，方便svc连接
spec:
containers:
- name: nginx-stable-pod ##pod容器名称
image: nginx:stable ###镜像
imagePullPolicy: IfNotPresent ###镜像拉取策略
ports:
- containerPort: 80 ###容器端口
---
apiVersion: v1
kind: Service
metadata:
namespace: nginx-namespace
name: nginx-service
spec:
selector:
app: nginx ###关联容器标签
ports:
- port: 80 ###容器端口
targetPort: 80
nodePort: 30002 ###nodeport映射为30002端口，便于外部主机访问
type: NodePort ###svc类型为nodeport
安装和验证
kubectl apply -f nginx-service.yaml
# 查看容器是否正常启动
kubectl -n nginx-namespace get pod
# ip为任意节点的宿主机IP
curl 10.191.9.21:30002