首页 > 编程语言 >记录一次生产环境因磁盘空间不足驱逐pod造成pod重建The node had condition: [DiskPressure]

记录一次生产环境因磁盘空间不足驱逐pod造成pod重建The node had condition: [DiskPressure]

时间：2023-11-27 17:56:14浏览次数：41  
标签：node DiskPressure 6987c5cb8d deploy 54m msg pod cms

     #记录一次生产报The node had condition: [DiskPressure]造成pod无限重启的监控不停的报警

#进入k8s的管理机检查发现msg的pod重启重建pod多次
[root@VM_248_6_centos ~]# kubectl get pod -n cms-v2-prod
NAME                                            READY   STATUS    RESTARTS   AGE
省略......
cms-msg-deploy-6987c5cb8d-6fczf                 1/1     Running   0          9d
cms-msg-deploy-6987c5cb8d-867ls                 1/1     Running   0          69m刚刚被重建的pod
cms-msg-deploy-6987c5cb8d-btlxd                 1/1     Running   0          8h
cms-msg-deploy-6987c5cb8d-k96xk                 1/1     Running   0          4h18m
cms-msg-deploy-6987c5cb8d-pjkx2                 1/1     Running   0          165m
cms-msg-deploy-6987c5cb8d-r4hdd                 1/1     Running   0          55m刚刚被重建的pod
省略。。。。。。。。。。。。。。。。。

#查看看k8s集群的events事件,可以看到pod发生重建都是在10.169.248.131和10.169.248.132的服务器
[root@VM_248_6_centos ~]# kubectl get event
LAST SEEN   TYPE      REASON             OBJECT                                 MESSAGE
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-4nk44    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-4nk44 to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-4nk44    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-85wgh    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-85wgh to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-85wgh    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-8929w    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-8929w to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-8929w    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-8tvtx    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-8tvtx to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-8tvtx    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-92l6g    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-92l6g to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-92l6g    The node had condition: [DiskPressure].
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-bcqrs    The node was low on resource: ephemeral-storage. Container sidecar-jdk was using 38073452Ki, which exceeds its request of 0.
54m         Normal    Killing            pod/cms-msg-deploy-6987c5cb8d-bcqrs    Stopping container sidecar-jdk
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-lvktl    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-lvktl to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-lvktl    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-m664z    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-m664z to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-m664z    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-mfdw2    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-mfdw2 to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-mfdw2    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-nd9rt    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-nd9rt to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-nd9rt    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-ngvhw    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-ngvhw to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-ngvhw    The node had condition: [DiskPressure].
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-r4hdd    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-r4hdd to 10.169.248.131
54m         Normal    Pulled             pod/cms-msg-deploy-6987c5cb8d-r4hdd    Container image "ccr.yxyun.yuexiu.com/idc1-yxhq-ump-registry/cms-v2-msg:release-2.0.27.20231010" already present on machine
54m         Normal    Created            pod/cms-msg-deploy-6987c5cb8d-r4hdd    Created container cms-msg-container
54m         Normal    Started            pod/cms-msg-deploy-6987c5cb8d-r4hdd    Started container cms-msg-container
54m         Normal    Pulled             pod/cms-msg-deploy-6987c5cb8d-r4hdd    Container image "ccr.yxyun.yuexiu.com/idc1-yxhq-ump-registry/openjdk:8u232-stretch-yak-dubbo-cmsapm" already present on machine
54m         Normal    Created            pod/cms-msg-deploy-6987c5cb8d-r4hdd    Created container sidecar-jdk
54m         Normal    Started            pod/cms-msg-deploy-6987c5cb8d-r4hdd    Started container sidecar-jdk
<unknown>   Normal    Scheduled          pod/cms-msg-deploy-6987c5cb8d-w8xsq    Successfully assigned cms-v2-prod/cms-msg-deploy-6987c5cb8d-w8xsq to 10.169.248.132
54m         Warning   Evicted            pod/cms-msg-deploy-6987c5cb8d-w8xsq    The node had condition: [DiskPressure].
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-m664z
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-lvktl
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-8tvtx
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-mfdw2
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-ngvhw
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-4nk44
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-w8xsq
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-nd9rt
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   Created pod: cms-msg-deploy-6987c5cb8d-85wgh
54m         Normal    SuccessfulCreate   replicaset/cms-msg-deploy-6987c5cb8d   (combined from similar events): Created pod: cms-msg-deploy-6987c5cb8d-r4hdd


#在10.169.248.131和10.169.248.132的服务器检查，提示磁盘到达了百分之85，超过了kubelet配置磁盘触发驱逐pod剩余的空间量
[root@VM_248_131_centos ~]# grep '85%' /var/log/messages
Nov 27 08:22:28 VM_248_131_centos kubelet: I1127 08:22:28.134844     935 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 5028521574 bytes down to the low threshold (80%).
[root@VM_248_131_centos ~]# grep '85%' /var/log/messages
Nov 27 08:22:28 VM_248_131_centos kubelet: I1127 08:22:28.134844     935 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 5028521574 bytes down to the low threshold (80%).
[root@VM_248_132_centos ~]# grep '85%' /var/log/messages
Nov 27 09:41:20 VM_248_132_centos kubelet: I1127 09:41:20.609047     940 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 4976412262 bytes down to the low threshold (80%).
Nov 27 15:26:21 VM_248_132_centos kubelet: I1127 15:26:21.135458     940 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 4357973606 bytes down to the low threshold (80%).

查看kubelet的pod驱逐策略
[root@VM_248_131_centos ~]# cat /var/lib/kubelet/config.yaml
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
[root@VM_248_132_centos ~]# cat /var/lib/kubelet/config.yaml
evictionHard:
  imagefs.available: 15%#磁盘空间小于百分之15就会发生pod驱逐
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%

#查看132的磁盘情况
[root@VM_248_132_centos ~]# df -h
Filesystem       Size  Used Avail Use% Mounted on
devtmpfs          32G     0   32G   0% /dev
tmpfs             32G   24K   32G   1% /dev/shm
tmpfs             32G  2.8M   32G   1% /run
tmpfs             32G     0   32G   0% /sys/fs/cgroup
/dev/vda1         50G   32G   16G  67% /
/dev/vdb          99G   43G   52G  46% /var/lib/docker
10.169.248.85:/  200G   22G  179G  11% /paas
tmpfs            6.3G     0  6.3G   0% /run/user/0
#查看131的磁盘情况
[root@VM_248_131_centos ~]# df -h
Filesystem       Size  Used Avail Use% Mounted on
devtmpfs          32G     0   32G   0% /dev
tmpfs             32G   24K   32G   1% /dev/shm
tmpfs             32G  2.6M   32G   1% /run
tmpfs             32G     0   32G   0% /sys/fs/cgroup
/dev/vda1         50G   15G   33G  30% /
/dev/vdb          99G   71G   24G  76% /var/lib/docker
10.169.248.85:/  200G   22G  179G  11% /paas
tmpfs            6.3G     0  6.3G   0% /run/user/0

解决方法扩容/var/lib/docker分区的磁盘空间，调整msg的pod产生的日志清理脚本

 

标签：node,DiskPressure,6987c5cb8d,deploy,54m,msg,pod,cms
From： https://www.cnblogs.com/yeyouqing/p/17859968.html

相关文章

  • yarn : 无法加载文件 C:\Program Files\nodejs\yarn.ps1,因为在此系统上禁止运行
    问题分析：这个错误提示说明在电脑系统上禁止运行PowerShell 脚本，因此导致无法加载Yarn的安装脚本。这是由于系统的执行策略（ExecutionPolicies）设置所导致的。解决方法：1.以管理员身份运行PowerShell。2.在窗口中执行 set-ExecutionPolicyRemoteSigned。3.执行完成后，......
  • Nodepad++格式化XML和JSON字符串
    （一）格式化XML1、安装XMLTools在notepad++中点击菜单栏【插件】-【插件管理】。在插件管理界面选择【XMLTools】，点击【安装】。2、打开XML文件在notepad++中点击菜单栏【文件】-【打开】。3、选择格式化XML全选XML字符串，点击【插件】-【XMLTools】-【Prettyprint(XMLonl......
  • npm下载node-sass包安装失败，需要下python2？
    问题这个问题真的老问题了，今天在跑一个老项目的时候又遇到了。每次遇到都会感慨，这个包真的是是非多啊，解决方案也很简单，不用下python环境，单纯是版本问题查了下网上有挺好的一篇博客：http://www.inspinia.net/a/388314.html?action=onClick里面的报错跟我的一模一样：解决方案......
  • node版本管理神器nvm安装使用教程（Windows11版本）
    安装前提在安装nvm之前，我们需要先将已有的node.js全部卸载！下载安装包进行安装第一步，下载安装包，nvm下载地址可下载以下文件：nvm-noinstall.zip：绿色免安装版，但使用时需要进行配置。nvm-setup.zip：安装版，推荐使用nvm-setup.exe：无需解压，直接安装，墙裂推荐！！！如下图： 第二步，进行......
  • node版本管理，勇士nvm下载安装使用
    node不同版本管理比较麻烦，推荐一个工具nvm进行版本管理 下载地址:nvm-windows  ......
  • node连接redis基础用法
     //*************redis基础用法**********************///1.导入redis，最好是下载3版本constredis=require('redis')//2.创建客户端constcli=redis.createClient({host:'localhost',port:6379})/***3.设置数据库*参1：k值*参2：v值*参3：'ex'|&#......
  • Node.js安装及环境配置
    一.安装Node.js步骤1、下载对应你系统的Node.js版本:https://nodejs.org/zh-cn/2、选安装目录进行安装3、环境配置4、测试二、前期准备1、Node.js简介简单的说Node.js就是运行在服务端的JavaScript。Node.js是一个基于ChromeV8引擎的JavaScript运行环境。Node.js使用......
  • <Index onVnodeUnmounted=fn<onVnodeUnmounted> ref=Ref< undefined > key=xxxx
    改了一下之前的程序，点开以后就出现了warn，初始加载页面是显示的然后切换到别的页面，就不显示了。 网上查了一下，有说是引用组件要驼峰，后来查了一下，不是这个原因。 想了一下是引入了一个对话框，然后就出现这种问题。究其原因：<template></template>下只能有一个root如果有多个......
  • bytenode nodejs 字节码编译工具
    bytenode是一个nodejs字节码编译工具，可以用来实现nodejs代码的一些加密简单使用安装目前推荐安装为cli全局模式，我使用npmscript不成功 sudonpminstall-gbytenode简单代码app.js定义一个模块 module.exports={name:"dalong",age:333}编译bytenode--compileapp.j......
  • nexe nodejs 应用打包工具
    nexe是类似pkg的一个nodejs应用打包工具支持的特性自包含可以用来运行多nodejsrumtime不依赖node以及npm构建幂等启动以及部署方便跨平台参考使用安装npminexe-g简单代码app.js console.log("demoapp")构建nexeapp.js说明nexe使用还是比较简单的，灵活性也很不错，值得试......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99