目录
问题现象
开发环境集群突然无法连接到apiserver，报错如下
排错思路
查看kubelet状态
从kubelet状态中可以看见无法连接6443端口
查看apiserver容器日志
可以看见提示About to store object in database（将要在数据库中存储对象）之后开始报错无法将对象或者在数据库获取对象，那么接下来需要查看etcd状态
查看etcd状态
果不其然可以看到etcd证书过期could not connect:x509:certificate has expired or is not yet valid
由于etcd证书过期导致apiserver无法存储到etcd数据库导致启动失败
查看证书到期时间
etcd集群证书更新
查看证书路径
systemctl cat etcd
所有节点备份证书
cd /etc/etcd/ssl
mkdir /tmp/etcd_cert_backup
mv ./* /tmp/etcd_cert_backup
所有节点备份etcd数据
停止Etcd操作
systemctl stop etcd
备份Etcd数据
cd /var/lib
tar -zvcf etcd.tar.gz etcd/
生成etcd集群配置文件
配置文件，在证书目录创建openssl.conf，最下面一行，如果是三台机器构建的Etcd集群，则三台ip都需要加上
[root@k8s-master1 ssl]# vi openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names
[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer
[alt_names]
DNS.1 = localhost
DNS.2 = etcd.kube-system.svc.cluster.local
DNS.3 = etcd.kube-system.svc
DNS.4 = etcd.kube-system
DNS.5 = etcd
DNS.6 = lb.kubesphere.local
DNS.7 = k8s-master1
DNS.8 = k8s-master2
DNS.9 = k8s-master3
IP.1 = 127.0.0.1
IP.2 = 10.100.1.31
IP.3 = 10.100.1.32
IP.4 = 10.100.1.33
生成ca证书
openssl genrsa -out ca-key.pem 2048 > / dev/null 2>&1
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 -out ca.pem -subj "/CN=etcd-ca" > /dev/null 2>&1
生成etcd证书
for host in node1 node2 node3; do
# Member key
# 用于 etcd 节点之间的通信
openssl genrsa -out member-${host}-key.pem 2048 > /dev/null 2>&1
openssl req -new -key member-${host}-key.pem -out member-${host}.csr -subj "/CN=etcd-member-${host}" -config openssl.conf > /dev/null 2>&1
openssl x509 -req -in member-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-${host}.pem -days 3650 -extensions ssl_client -extfile openssl.conf > /dev/null 2>&1
# Admin key
# 用于 etcd 集群的管理
openssl genrsa -out admin-${host}-key.pem 2048 > /dev/null 2>&1
openssl req -new -key admin-${host}-key.pem -out admin-${host}.csr -subj "/CN=etcd-admin-${host}" > /dev/null 2>&1
openssl x509 -req -in admin-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin-${host}.pem -days 3650 -extensions ssl_client -extfile openssl.conf > /dev/null 2>&1
# Node keys
# 用于 etcd 节点与客户端之间的通信
openssl genrsa -out node-${host}-key.pem 2048 > /dev/null 2>&1
openssl req -new -key node-${host}-key.pem -out node-${host}.csr -subj "/CN=etcd-node-${host}" > /dev/null 2>&1
openssl x509 -req -in node-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out node-${host}.pem -days 3650 -extensions ssl_client -extfile openssl.conf > /dev/null 2>&1
# 将证书复制到每个节点
scp -r /etc/etcd/ssl/* root@${host}:/etc/etcd/ssl/
done
查看新证书
openssl x509 -in ca.pem -noout -text | grep 'Not'
openssl x509 -in member-k8s-master1.pem -noout -text | grep 'Not'
openssl x509 -in admin-k8s-master1.pem -noout -text | grep 'Not'
openssl x509 -in node-k8s-master1.pem -noout -text | grep 'Not'
修改etcd启动配置文件
master1
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
--name=k8s-master1 \
--cert-file=/etc/etcd/ssl/member-k8s-master1.pem \
--key-file=/etc/etcd/ssl/member-k8s-master1-key.pem \
--peer-cert-file=/etc/etcd/ssl/member-k8s-master1.pem \
--peer-key-file=/etc/etcd/ssl/member-k8s-master1-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls=https://10.100.1.31:2380 \
--listen-peer-urls=https://10.100.1.31:2380 \
--listen-client-urls=https://10.100.1.31:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://10.100.1.31:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=k8s-master1=https://10.100.1.31:2380,k8s-master2=https://10.100.1.32:2380,k8s-master3=https://10.100.1.33:2380 \
--initial-cluster-state=new \
--data-dir=/var/lib/etcd
master2
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
--name=k8s-master2 \
--cert-file=/etc/etcd/ssl/member-k8s-master2.pem \
--key-file=/etc/etcd/ssl/member-k8s-master2-key.pem \
--peer-cert-file=/etc/etcd/ssl/member-k8s-master2.pem \
--peer-key-file=/etc/etcd/ssl/member-k8s-master2-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls=https://10.100.1.32:2380 \
--listen-peer-urls=https://10.100.1.32:2380 \
--listen-client-urls=https://10.100.1.32:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://10.100.1.32:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=k8s-master1=https://10.100.1.31:2380,k8s-master2=https://10.100.1.32:2380,k8s-master3=https://10.100.1.33:2380 \
--initial-cluster-state=new \
--data-dir=/var/lib/etcd
master3
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
--name=k8s-master3 \
--cert-file=/etc/etcd/ssl/member-k8s-master3.pem \
--key-file=/etc/etcd/ssl/member-k8s-master3-key.pem \
--peer-cert-file=/etc/etcd/ssl/member-k8s-master3.pem \
--peer-key-file=/etc/etcd/ssl/member-k8s-master3-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--initial-advertise-peer-urls=https://10.100.1.33:2380 \
--listen-peer-urls=https://10.100.1.33:2380 \
--listen-client-urls=https://10.100.1.33:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://10.100.1.33:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=k8s-master1=https://10.100.1.31:2380,k8s-master2=https://10.100.1.32:2380,k8s-master3=https://10.100.1.33:2380 \
--initial-cluster-state=new \
--data-dir=/var/lib/etcd
所有节点同时启动etcd
记住，要3个节点一起重启，不然会hang住
systemctl daemon-reload
systemctl restart etcd