首页 > 其他分享 >记一次阿里云搭建K8S在恢复镜像快照之后etcd一个节点无法启动问题

记一次阿里云搭建K8S在恢复镜像快照之后etcd一个节点无法启动问题

时间:2024-09-06 10:02:52浏览次数:9  
标签:快照 etcd -- but environment ETCD corresponding K8S

  1. 环境查看
    系统环境
# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
# uname -a
Linux CentOS7K8SMaster01005101 3.10.0-1160.114.2.el7.x86_64 #1 SMP Wed Mar 20 15:54:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

软件环境

# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
# /opt/etcd/bin/etcd --version
etcd Version: 3.3.10
Git SHA: 27fc7e2
Go Version: go1.10.4
Go OS/Arch: linux/amd64
  1. 故障现象
    阿里云一台服务器出现挖矿之后恢复了磁盘快照etcd无法启动
    image

查看/var/log/messages日志发现以下日志

etcd: member fa1b88064457e060 has already been bootstrapped
systemd: etcd.service: main process exited, code=exited, status=1/FAILURE
systemd: Unit etcd.service entered failed state.
systemd: etcd.service failed.
systemd: etcd.service holdoff time over, scheduling restart.
etcd: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_LISTEN_PEER_URLS, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_ADVERTISE_CLIENT_URLS, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_INITIAL_CLUSTER, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_INITIAL_CLUSTER_TOKEN, but unused: shadowed by corresponding flag
etcd: recognized environment variable ETCD_INITIAL_CLUSTER_STATE, but unused: shadowed by corresponding flag
etcd: etcd Version: 3.3.10
  1. 原因分析
    恢复快照之后数据不同步导致
  2. 修复步骤
    删除/var/lib/etcd/重启etcd无法修复
    需要修改etcd启动文件
    image
    完整的配置文件如下
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=existing \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

重启

# systemctl reload-daemon
# systemctl restart etcd

查看是否修复

# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   

标签:快照,etcd,--,but,environment,ETCD,corresponding,K8S
From: https://www.cnblogs.com/minseo/p/18399706

相关文章

  • 基础网络安全——K8S关键概念及搭建过程中遇到的问题补充
    一、K8S集群基本概念             k8s是一组服务器集群,是一个分布式的容器编排系统,对运行在集群上的容器进行管理,K8S集群包括控制平面(ControlPlane)以及1个或者多个工作节点(workernode),而控制平面包括masternode以及etcd节点。并且在这两种类型的节点上运行多种不......
  • K8S 中的 Node 管理
    1.Node的扩缩容在实际生产系统中,经常遇到服务器容量不足的情况,这时就需要购买新的服务器,对应用系统进行水平扩展以实现扩容。在k8s中,对一个新的node的加入非常简单,只需要在node节点上安装Docker、kubelet和kube-proxy服务,然后将kubelet和kube-proxy的启动参数中的master url指定为......
  • k8s介绍
    ⼀、编排分类单机容器编排:docker-compose容器集群编排:dockerswarm、mesos+marathon、kubernetes应⽤编排:ansible(模块,剧本,⻆⾊)⼆、系统管理进化史1.传统部署时代早期,各个组织是在物理服务器上运⾏应⽤程序。由于⽆法限制在物理服务器中运⾏的应⽤程序资源使⽤,......
  • 部署K8S集群(三):子节点部署及加入集群
    子节点:安装kubeletkubeadmkubectlyuminstall-ykubeletkubeadmkubectl安装containerd下载地址:https://github.com/containerd/containerd/releases找到标签为Latest的包,根据系统类型下载对应的包#解压tarCxzvf/usr/localcontainerd-1.7.21-linux-amd64.tar.g......
  • 云计算-k8s超配理论
    在很多商业k8s方案中,存在所谓的超配,或者超卖,超售比的配置,只要开启就能大幅度提高k8s资源利用率。这里总结了一些原理概念概念当前很多业务有波峰和波谷,部署服务时,为了保证服务的性能和稳定性,通常会按照波峰时需要的资源申请,但是波峰的时间可能很短,这样在非波峰时段就有资源浪费。另......
  • 部署K8S集群(一):环境准备
    环境介绍:以下操作使用三台基于Hyper-v的OpenEuler虚拟机IP主机名系统版本172.17.48.27masteropeneuler-22.03LTSSP3172.17.48.28worker01openeuler-22.03LTSSP3172.17.48.29worker02openeuler-22.03LTSSP3虚拟机准备操作:以下操作需要在全部三......
  • 部署K8S集群(二):主节点初始化
    主节点:以下操作只在master进行安装kubeletkubeadmkubectlyuminstall-ykubeletkubeadmkubectl安装containerd下载地址:https://github.com/containerd/containerd/releases找到标签为Latest的包,根据系统类型下载对应的包#解压tarCxzvf/usr/localcontainerd-1......
  • k8s之存储类
    介绍在Kubernetes中,StorageClass是一种API对象,它定义了一种存储卷的动态供应方式。StorageClass允许管理员描述不同类型的存储系统或细节,例如不同类型的本地存储、不同类型的云存储等。当用户需要持久化存储时,他们可以请求特定类型的StorageClass,Kubernetes将根据这个描......
  • K8S:Yaml
    学习自:​K8s中yaml文件详解(超详细)​_yamllist-CSDN博客0、前言YAML文件是一种类似JSON的配置文件,相比JSON,YAML使用起来更简洁、强大、方便。它实质上是一种通用的数据串行化格式。1、语法规则区分大小写用缩进表明层级,同一缩进代表同一层级;只允许空格缩进,不允许TAB;可以......
  • k8s创建nfs+StorageClass
    什么是StorageClass前置准备nfsservernfsserver地址:192.168.2.152nsfpath:/datak8s集群集群版本:1.22容器:containerd1.6创建rbac参考https://github.com/kubernetes-retired/external-storage/blob/master/nfs-client/deploy/rbac.yamlapiVersion:v1kind:Ser......