首页 > 数据库 >服务器上mysqld,java的进程Out of Memory,被kernel kill 掉了

服务器上mysqld,java的进程Out of Memory,被kernel kill 掉了

时间:2024-11-07 09:11:46浏览次数:1  
标签:kernel 26 java 22 centos 23 VM kill

/var/log/messages 里面日志如下

Aug 10 19:47:16 VM-0-7-centos kernel: 8936 total pagecache pages
Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages in swap cache
Aug 10 19:47:16 VM-0-7-centos kernel: Swap cache stats: add 0, delete 0, find 0/0
Aug 10 19:47:16 VM-0-7-centos kernel: Free swap  = 0kB
Aug 10 19:47:16 VM-0-7-centos kernel: Total swap = 0kB
Aug 10 19:47:16 VM-0-7-centos kernel: 2097016 pages RAM
Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages HighMem/MovableOnly
Aug 10 19:47:16 VM-0-7-centos kernel: 94855 pages reserved
Aug 10 19:47:16 VM-0-7-centos kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Aug 10 19:47:16 VM-0-7-centos kernel: [13357]    27 13357  1244419   814184    2246        0             0 mysqld
Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13357 (mysqld) score 407 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13357 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3256736kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 0 PID: 18863 Comm: VM Periodic Tas Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
Aug 10 19:47:16 VM-0-7-centos kernel: Call Trace:

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13388 (mysqld) score 407 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13388 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3257420kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 1 PID: 26485 Comm: logback-3 Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014

Aug 10 19:47:16 VM-0-7-centos kernel: [18851]  1000 18851  1442473   397376    1068        0             0 java
Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18851 (java) score 199 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18851 (java), UID 1000, total-vm:5769892kB, anon-rss:1589504kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18855 (VM Thread) score 201 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18855 (VM Thread), UID 1000, total-vm:5769892kB, anon-rss:1612704kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdum

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18863 (VM Periodic Tas) score 201 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service: main process exited, code=killed, status=9/KILL
Aug 10 19:47:16 VM-0-7-centos systemd: Unit mysqld.service entered failed state.
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service failed.
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service holdoff time over, scheduling restart.
Aug 10 19:47:16 VM-0-7-centos systemd: Stopped MySQL Server.
Aug 10 19:47:16 VM-0-7-centos systemd: Starting MySQL Server...
Aug 10 19:47:26 VM-0-7-centos systemd: Started MySQL Server.

 可以看到一开始是: VM Periodic Tas invoked oom-killer ,而VM Periodic Tag可能是JVM线程的东东。 Killed process 13357 (mysqld) 是指把分数最不理想的mysqld给kill了

后面则是 logback-3 invoked oom-killer ,killed process 13388 (mysqld)。再后面则是node invoked oom-killer. 自从ssr之后,对后台的压力增加了。

 

不过看监控在发生oom-killer时系统内存占用大概是6.8G左右,还有空闲。但还是发生了。而且看日志之前已经发生多次。

看网上的解释:The “OOM Killer” or “Out of Memory Killer” is a process that the Linux kernel employs when the system is critically low on memory.

 

看网上有人说建立swap分区会好一些,那我只好建了一个400M的swap分区文件。

========================================

已上并没有什么卵用,下面分析一下oom kill的log

Aug 23 22:26:50 VM-0-7-centos kernel: VM Thread invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
...........

Aug 23 22:26:51 VM-0-7-centos kernel: Mem-Info:
Aug 23 22:26:51 VM-0-7-centos kernel: active_anon:1550473 inactive_anon:321719 isolated_anon:0#012 active_file:8324 inactive_file:8839 isolated_file:32#012 unevictable:0 dirty:6 writeback:5 unstable:0#012 slab_reclaimable:29739 slab_unreclaimable:18215#012 mapped:655 shmem:157 pagetables:10440 bounce:0#012 free:27277 free_pcp:690 free_cma:0
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 DMA free:15892kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 23 22:26:51 VM-0-7-centos kernel: lowmem_reserve[]: 0 2830 7805 7805
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 DMA32 free:48256kB min:24468kB low:30584kB high:36700kB active_anon:2125528kB inactive_anon:555044kB active_file:20808kB inactive_file:22456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129192kB managed:2898760kB mlocked:0kB dirty:12kB writeback:8kB mapped:1824kB shmem:244kB slab_reclaimable:62172kB slab_unreclaimable:28556kB kernel_stack:2928kB pagetables:15376kB unstable:0kB bounce:0kB free_pcp:1308kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Aug 23 22:26:51 VM-0-7-centos kernel: lowmem_reserve[]: 0 0 4974 4974
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 Normal free:45948kB min:42976kB low:53720kB high:64464kB active_anon:4076364kB inactive_anon:731832kB active_file:12232kB inactive_file:13024kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:5242880kB managed:5093976kB mlocked:0kB dirty:12kB writeback:12kB mapped:796kB shmem:384kB slab_reclaimable:56784kB slab_unreclaimable:44288kB kernel_stack:9600kB pagetables:26384kB unstable:0kB bounce:0kB free_pcp:912kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:39131 all_unreclaimable? yes
Aug 23 22:26:51 VM-0-7-centos kernel: lowmem_reserve[]: 0 0 0 0
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 DMA32: 1784*4kB (UEM) 938*8kB (UEM) 773*16kB (UEM) 412*32kB (UEM) 107*64kB (UE) 5*128kB (UEM) 0*256kB 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 49216kB
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 Normal: 1003*4kB (UEM) 1872*8kB (UEM) 589*16kB (UEM) 386*32kB (UEM) 52*64kB (UEM) 11*128kB (EM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46012kB
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 23 22:26:51 VM-0-7-centos kernel: 26283 total pagecache pages
Aug 23 22:26:51 VM-0-7-centos kernel: 9173 pages in swap cache
Aug 23 22:26:51 VM-0-7-centos kernel: Swap cache stats: add 216402, delete 207233, find 812854/834105
Aug 23 22:26:51 VM-0-7-centos kernel: Free swap = 0kB
Aug 23 22:26:51 VM-0-7-centos kernel: Total swap = 409596kB
Aug 23 22:26:51 VM-0-7-centos kernel: 2097016 pages RAM
Aug 23 22:26:51 VM-0-7-centos kernel: 0 pages HighMem/MovableOnly
Aug 23 22:26:51 VM-0-7-centos kernel: 94855 pages reserved

Aug 23 22:26:51 VM-0-7-centos kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Aug 23 22:26:51 VM-0-7-centos kernel: [ 661] 999 661 180376 811 67 1501 0 polkitd
Aug 23 22:26:51 VM-0-7-centos kernel: [ 664] 998 664 2145 7 10 29 0 lsmd
Aug 23 22:26:51 VM-0-7-centos kernel: [ 666] 81 666 15390 297 32 122 -900 dbus-daemon
Aug 23 22:26:51 VM-0-7-centos kernel: [ 670] 38 670 12345 42 28 140 0 ntpd
Aug 23 22:26:51 VM-0-7-centos kernel: [ 671] 0 671 1097 1 8 33 0 acpid
Aug 23 22:26:51 VM-0-7-centos kernel: [ 674] 0 674 7103 290 19 309 0 systemd-logind
Aug 23 22:26:51 VM-0-7-centos kernel: [ 940] 0 940 25726 1 49 512 0 dhclient
Aug 23 22:26:51 VM-0-7-centos kernel: [ 1144] 0 1144 6477 7 18 46 0 atd
Aug 23 22:26:51 VM-0-7-centos kernel: [ 1171] 0 1171 27552 2 9 31 0 agetty
Aug 23 22:26:51 VM-0-7-centos kernel: [ 1172] 0 1172 27552 2 10 31 0 agetty
Aug 23 22:26:51 VM-0-7-centos kernel: [32380] 0 32380 11351 2 24 120 -1000 systemd-udevd
Aug 23 22:26:51 VM-0-7-centos kernel: [12665] 0 12665 146507 278 105 2986 0 tuned
Aug 23 22:26:51 VM-0-7-centos kernel: [19250] 0 19250 10728 2 26 67 0 lvmetad
Aug 23 22:26:51 VM-0-7-centos kernel: [19427] 0 19427 166116 622 175 89 0 rsyslogd
Aug 23 22:26:51 VM-0-7-centos kernel: [32536] 0 32536 1203030 37707 225 19159 0 java
Aug 23 22:26:51 VM-0-7-centos kernel: [ 9045] 0 9045 28234 67 58 195 -1000 sshd
Aug 23 22:26:51 VM-0-7-centos kernel: [ 6066] 0 6066 42016 381 22 199 0 redis-server
Aug 23 22:26:51 VM-0-7-centos kernel: [26425] 1000 26425 1013843 97449 486 58688 0 java
Aug 23 22:26:51 VM-0-7-centos kernel: [ 9542] 0 9542 51863 7632 43 1193 0 scrapyd
Aug 23 22:26:51 VM-0-7-centos kernel: [24774] 0 24774 46243 0 42 152 0 svnserve
Aug 23 22:26:51 VM-0-7-centos kernel: [ 5928] 0 5928 24373 96 20 28 0 sgagent
Aug 23 22:26:51 VM-0-7-centos kernel: [ 3104] 0 3104 11989 431 29 53 0 nginx
Aug 23 22:26:51 VM-0-7-centos kernel: [22261] 1000 22261 37951 2040 28 3645 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [24111] 1000 24111 38482 2796 30 2930 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [24112] 1000 24112 38482 2992 30 2751 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [24113] 1000 24113 38482 2991 30 2765 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [24114] 1000 24114 38482 3017 30 2747 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [24381] 1000 24381 616987 20964 139 527 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [21067] 0 21067 267754 597 42 211 0 YDLive
Aug 23 22:26:51 VM-0-7-centos kernel: [15268] 1000 15268 228656 8219 150 0 0 PM2 v5.3.0: God
Aug 23 22:26:51 VM-0-7-centos kernel: [ 5371] 1000 5371 179277 1643 22 0 0 mgtty
Aug 23 22:26:51 VM-0-7-centos kernel: [ 2652] 1000 2652 71523 2247 25 0 0 python
Aug 23 22:26:51 VM-0-7-centos kernel: [32094] 0 32094 287474 36843 164 10611 0 YDService
Aug 23 22:26:51 VM-0-7-centos kernel: [21810] 0 21810 312052 1141 47 444 0 sh
Aug 23 22:26:51 VM-0-7-centos kernel: [ 9387] 1000 9387 227773 10275 140 0 0 node /home/hjda
Aug 23 22:26:51 VM-0-7-centos kernel: [20365] 1000 20365 247573 22232 407 0 0 node /home/hjda
Aug 23 22:26:51 VM-0-7-centos kernel: [19566] 0 19566 33442 100 70 0 0 systemd-journal
Aug 23 22:26:51 VM-0-7-centos kernel: [ 3013] 0 3013 12914 1020 31 48 0 nginx
Aug 23 22:26:51 VM-0-7-centos kernel: [ 3014] 0 3014 12996 1044 31 48 0 nginx
Aug 23 22:26:51 VM-0-7-centos kernel: [ 3015] 0 3015 13066 1147 31 48 0 nginx
Aug 23 22:26:51 VM-0-7-centos kernel: [ 3016] 0 3016 13386 1463 32 48 0 nginx
Aug 23 22:26:51 VM-0-7-centos kernel: [ 8424] 27 8424 1795335 1019225 2442 0 0 mysqld
Aug 23 22:26:51 VM-0-7-centos kernel: [17805] 0 17805 31597 171 20 0 0 crond
Aug 23 22:26:51 VM-0-7-centos kernel: [ 911] 1000 911 240453 23601 335 0 0 node /home/hjda
Aug 23 22:26:51 VM-0-7-centos kernel: [ 4180] 1000 4180 1360179 202736 698 0 0 java
Aug 23 22:26:51 VM-0-7-centos kernel: [ 4371] 1000 4371 1024871 120478 410 0 0 java
Aug 23 22:26:51 VM-0-7-centos kernel: [21245] 0 21245 38922 1676 29 0 0 barad_agent
Aug 23 22:26:51 VM-0-7-centos kernel: [21261] 0 21261 41225 1863 35 0 0 barad_agent
Aug 23 22:26:51 VM-0-7-centos kernel: [21262] 0 21262 172850 3575 58 0 0 barad_agent
Aug 23 22:26:51 VM-0-7-centos kernel: [17196] 0 17196 39203 341 80 0 0 sshd

Aug 23 22:26:51 VM-0-7-centos kernel: [ 6579]  1000  6579 11058700   216642    3017        0             0 node

Aug 23 22:26:51 VM-0-7-centos kernel: Out of memory: Kill process 8424 (mysqld) score 485 or sacrifice child
Aug 23 22:26:51 VM-0-7-centos kernel: Killed process 8424 (mysqld), UID 27, total-vm:7181340kB, anon-rss:4076900kB, file-rss:0kB, shmem-rss:0kB
Aug 23 22:26:51 VM-0-7-centos systemd: mysqld.service: main process exited, code=killed, status=9/KILL

复制代码

https://community.wandisco.com/portal/s/article/Guide-to-Out-of-Memory-OOM-events-and-decoding-their-logging 这篇文章纤细介绍了log的看法

首先看 下面这行

Aug 23 22:26:50 VM-0-7-centos kernel: VM Thread invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
gfp_mask: The gfp_mask is used to tell the page allocator which pages can be allocated, whether the allocator can wait for more memory to be freed, etc. All the gfp flags are defined in include/linux/gfp.h:
/* Plain integer GFP bitmasks. Do not use this directly. */
#define ___GFP_DMA      0x01u
#define ___GFP_HIGHMEM      0x02u
#define ___GFP_DMA32        0x04u
#define ___GFP_MOVABLE      0x08u
#define ___GFP_WAIT     0x10u
#define ___GFP_HIGH     0x20u
#define ___GFP_IO       0x40u
#define ___GFP_FS       0x80u
#define ___GFP_COLD     0x100u
#define ___GFP_NOWARN       0x200u
#define ___GFP_REPEAT       0x400u
#define ___GFP_NOFAIL       0x800u
#define ___GFP_NORETRY      0x1000u
#define ___GFP_MEMALLOC     0x2000u
#define ___GFP_COMP     0x4000u
#define ___GFP_ZERO     0x8000u
#define ___GFP_NOMEMALLOC   0x10000u
#define ___GFP_HARDWALL     0x20000u
#define ___GFP_THISNODE     0x40000u
#define ___GFP_RECLAIMABLE  0x80000u
#define ___GFP_KMEMCG       0x100000u
#define ___GFP_NOTRACK      0x200000u
#define ___GFP_NO_KSWAPD    0x400000u
#define ___GFP_OTHER_NODE   0x800000u
#define ___GFP_WRITE        0x1000000u
#define ___GFP_CMA      0x2000000u

/*
 * GFP bitmasks..
 *
 * Zone modifiers (see linux/mmzone.h - low three bits)
 *
 * Do not put any conditional on these. If necessary modify the definitions
 * without the underscores and use them consistently. The definitions here may
 * be used in bit comparisons.
 */
#define __GFP_DMA   ((__force gfp_t)___GFP_DMA)
#define __GFP_HIGHMEM   ((__force gfp_t)___GFP_HIGHMEM)
#define __GFP_DMA32 ((__force gfp_t)___GFP_DMA32)
#define __GFP_MOVABLE   ((__force gfp_t)___GFP_MOVABLE)  /* Page is movable */
#define __GFP_CMA   ((__force gfp_t)___GFP_CMA)
#define GFP_ZONEMASK    (__GFP_DMA|__GFP_HIGHMEM|__GFP_DMA32|__GFP_MOVABLE| \
            __GFP_CMA)

/*
 * Action modifiers - doesn't change the zoning
 *
 * __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
 * _might_ fail.  This depends upon the particular VM implementation.
 *
 * __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
 * cannot handle allocation failures.  This modifier is deprecated and no new
 * users should be added.
 *
 * __GFP_NORETRY: The VM implementation must not retry indefinitely.
 *
 * __GFP_MOVABLE: Flag that this page will be movable by the page migration
 * mechanism or reclaimed
 */
#define __GFP_WAIT  ((__force gfp_t)___GFP_WAIT)    /* Can wait and reschedule? */
#define __GFP_HIGH  ((__force gfp_t)___GFP_HIGH)    /* Should access emergency pools? */
#define __GFP_IO    ((__force gfp_t)___GFP_IO)  /* Can start physical IO? */
#define __GFP_FS    ((__force gfp_t)___GFP_FS)  /* Can call down to low-level FS? */
#define __GFP_COLD  ((__force gfp_t)___GFP_COLD)    /* Cache-cold page required */
#define __GFP_NOWARN    ((__force gfp_t)___GFP_NOWARN)  /* Suppress page allocation failure warning */
#define __GFP_REPEAT    ((__force gfp_t)___GFP_REPEAT)  /* See above */
#define __GFP_NOFAIL    ((__force gfp_t)___GFP_NOFAIL)  /* See above */
#define __GFP_NORETRY   ((__force gfp_t)___GFP_NORETRY) /* See above */
#define __GFP_MEMALLOC  ((__force gfp_t)___GFP_MEMALLOC)/* Allow access to emergency reserves */
#define __GFP_COMP  ((__force gfp_t)___GFP_COMP)    /* Add compound page metadata */
#define __GFP_ZERO  ((__force gfp_t)___GFP_ZERO)    /* Return zeroed page on success */
#define __GFP_NOMEMALLOC ((__force gfp_t)___GFP_NOMEMALLOC) /* Don't use emergency reserves.
                             * This takes precedence over the
                             * __GFP_MEMALLOC flag if both are
                             * set
                             */
order: The 'order' of a page allocation is it's logarithm to the base 2, and the size of the allocation is 2order, an integral power-of-2 number of pages. 
这里order=0,表示分配 2 的0次方个page, 一个page=4k。至于 gfp_mask=0x201da 最后一个16进制是a,对应的二进制是1010,取low three bit = 010 Zone modifiers #define __GFP_HIGHMEM   0x02u 02对应的是010,所以应该是High Memory区(目前是这样理解的)
接下来看下面这个log
Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 Normal free:45948kB min:42976kB low:53720kB high:64464kB active_anon:4076364kB inactive_anon:731832kB active_file:12232kB inactive_file:13024kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:5242880kB managed:5093976kB mlocked:0kB dirty:12kB writeback:12kB mapped:796kB shmem:384kB slab_reclaimable:56784kB slab_unreclaimable:44288kB kernel_stack:9600kB pagetables:26384kB unstable:0kB bounce:0kB free_pcp:912kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:39131 all_unreclaimable? yes

Normal代表High Memory,这里 free:45948kB min:42976kB low:53720kB, 这里min<free < low, 根据https://community.wandisco.com/portal/s/article/Guide-to-Out-of-Memory-OOM-events-and-decoding-their-logging 这位老哥的介绍,当free < min时内存就无法分配了,但我这里不是这种情况,那是因为内存碎片太多了吗?看下面这个log

Aug 23 22:26:51 VM-0-7-centos kernel: Node 0 Normal: 1003*4kB (UEM) 1872*8kB (UEM) 589*16kB (UEM) 386*32kB (UEM) 52*64kB (UEM) 11*128kB (EM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46012kB

区区4k的内存,还是充足的,然后我又 搜索了一下,看到下面这里阿里云的链接:出现OOM Killer的原因与解决方案 (alibabacloud.com)

系统全局内存的使用率过高: 

如下日志记录的出现OOM Killer场景示例中,limit of host表示实例的全局内存出现了不足。在日志记录的数据中,空闲内存(free)已经低于了内存最低水位线(low)。
[六 9月 11 12:24:42 2021] test invoked oom-killer: gfp_mask=0x62****(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0,
[六 9月 11 12:24:42 2021] Task in /user.slice killed as a result of limit of host
[六 9月 11 12:24:42 2021] Node 0 DMA32 free:155160kB min:152412kB low:190512kB high:228612kB
[六 9月 11 12:24:42 2021] Node 0 Normal free:46592kB min:46712kB low:58388kB high:70064kB
原因:由于实例的空闲内存低于内存最低水位线,无法通过内存回收机制解决内存不足的问题,因此触发了OOM Killer。

内存节点(Node)的内存不足:

如下日志记录的出现OOM Killer场景示例中,部分日志记录说明:
limit of host表示内存节点的内存出现了不足。
实例存在Node 0和Node 1两个内存节点。
内存节点Node 1的空闲内存(free)低于内存最低水位线(low)。
实例的空闲内存还有大量剩余(free:4111496)。
[Sat Sep 11 09:46:24 2021] main invoked oom-killer: gfp_mask=0x62****(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
[Sat Sep 11 09:46:24 2021] main cpuset=mm_cpuset mems_allowed=1
[Sat Sep 11 09:46:24 2021] Task in / killed as a result of limit of host
[Sat Sep 11 09:46:24 2021] Mem-Info:
[Sat Sep 11 09:46:24 2021] active_anon:172 inactive_anon:4518735 isolated_anon:
    free:4111496 free_pcp:1 free_cma:0
[Sat Sep 11 09:46:24 2021] Node 1 Normal free:43636kB min:45148kB low:441424kB high:837700kB
[Sat Sep 11 09:46:24 2021] Node 1 Normal: 856*4kB (UME) 375*8kB (UME) 183*16kB (UME) 184*32kB (UME) 87*64kB (ME) 45*128kB (UME) 16*256kB (UME) 5*512kB (UE) 14*1024kB (UME) 0     *2048kB 0*4096kB = 47560kB
[Sat Sep 11 09:46:24 2021] Node 0 hugepages_total=360 hugepages_free=360 hugepages_surp=0 hugepages_size=1048576kB
[Sat Sep 11 09:46:24 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sat Sep 11 09:46:24 2021] Node 1 hugepages_total=360 hugepages_free=360 hugepages_surp=0 hugepages_size=1048576kB
[Sat Sep 11 09:46:25 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
原因:在NUMA存储模式下,操作系统可能存在多个内存节点(可运行cat /proc/buddyinfo命令查看相关资源信息)。如果通过cpuset.mems参数指定cgroup只能使用特定内存节点的内存,则可能导致实例在具备充足的空闲内存的情况下,仍出现OOM Killer的情况。

上面说的是当 free < low时,无法分配内存,看来阿里云的说法符合我的情况,可能是两种解释内核版本不同?

接下来解释几个名词:

total-vm: the size of the virtual memoy,Part of it is really mapped into the RAM itself (allocated and used). This is "RSS".

anon-rss:Part of the RSS is allocated in real memory blocks (other than mapped into a file or device). This is anonymous memory ("anon-rss")

file-rss:RSS memory blocks that are mapped into devices and files ("file-rss").

So, if you open a huge file in vim, the file-rss would be high, on the other side, if you malloc() a lot of memory and really use it, your anon-rss would be high also.On the other side, if you allocate a lot of space (with malloc()), but nevers use it, the total-vm would be higher, but no real memory would be used (due to the memory overcommit), so, the rss values would be low.

-------- https://stackoverflow.com/questions/18845857/what-does-anon-rss-and-total-vm-mean#

=================说了这么多,解决方法呢?

目前尝试了调整mysq的 innodb_buffer_pool_size = 2304M: 4G的75%,我的机器8G,因为我的机器人还跑了java和python,node服务。目前测试下来除了编译打包vue的ssr服务,node一下子占了好多内存导致oom发生外,还有chrome崩溃同时存在好多个chrome服务外还没有在正常运行中发生过

 

标签:kernel,26,java,22,centos,23,VM,kill
From: https://www.cnblogs.com/shujuyr/p/18531462

相关文章

  • 基于springboot框架在线生鲜商城推荐系统 java实现个性化生鲜/农产品购物商城推荐网站
    基于springboot框架在线生鲜商城推荐系统java实现个性化生鲜/农产品购物商城推荐网站爬虫、数据分析、排行榜基于协同过滤算法推荐、基于流行度热点推荐、平均加权混合推荐机器学习、大数据、深度学习OnlineShopRecommendEx一、项目简介1、开发工具和使用技术IDEA,jdk......
  • Java 8新特性深度解析:提升效率与功能
    Java8是Java语言发展史上的一个里程碑,它在2014年3月18日正式发布。Java8的发布引入了一系列创新的功能和改进,极大地提升了Java语言的表达能力、灵活性和性能。Java8的里程碑意义Java8标志着Java语言向函数式编程范式迈出了重要的一步。它不仅提高了开发人员的编码效率......
  • javascript 替代try catch方案详细完整案例和优缺点
    1.OptionalChaining(可选链)案例:constuser={name:"Kimi",details:{age:30}};constage=user.details?.age;//如果user或details是null/undefined,返回undefined而不是抛出错误优点:预防运行时错误,特别是在访问可能为null或undefined的对象......
  • java的类加载机制详解
    Java的类加载机制是一个复杂但非常重要的过程,因为它决定了程序如何找到并加载类文件。下面我会逐步详细讲解整个机制。1.什么是类加载机制?简单来说,类加载机制就是Java虚拟机(JVM)在运行时将.class文件中的字节码加载到内存中并进行解析的过程。这个机制让JVM可以动态加......
  • Java 中的 try-with-resources 详解
    在Java7之前,处理资源关闭通常使用try-catch-finally块。虽然这种方式可以确保资源被正确关闭,但代码显得冗长且容易出错。Java7引入了try-with-resources语法,使得资源管理变得更加简洁和安全。本文将详细介绍try-with-resources的使用方法和优势。1try-catch-fi......
  • JavasScript 的对象事件的处理程序
    1、鼠标事件常用的鼠标事件有MouseDown、MouseUp、MouseMove、MouseOver、MouseOut、Click、Blur及Focus等事件。mousedown:按下鼠标键时触发 mouseup:抬起鼠标键时触发 click:单击鼠标时触发 dblclick:在同一个元素上双击鼠标时触发 mouseenter:鼠标进入一个节点时触发,进......
  • 数组的介绍--Java
    1、数组是什么    数组就是一个容器,里面存放的是一组同种类型的数据。    Example:    1,3,5,7,8,10,12    int[]arr={1,3,5,7,8,10,12};    //  该数组存放的都是整型数据    李白,后羿,诸葛亮,刘邦,庄周    ......
  • javascript函数
    1.1初识函数1.1.1函数的定义1.函数用于封装一段特定功能的代码作用:提高代码的复用性,降低维护的难度(你将实现一个功能多段重复的代码变为一段代码,降低了维护的难度,你将这段代码用一个函数封装,要使用这个功能的时候就调用函数,即可提高代码的复用性)1.1.2函数的定义与使用1.函......
  • Java初学者指南
    Java是什么?Java是一种高级、面向对象的编程语言,设计目的是平台无关性,允许开发者“一次编写,到处运行”(WORA)。这意味着Java代码在一个平台上编译后,可以在任何支持Java的平台上运行,而无需重新编译。Java的历史由詹姆斯·高斯林和SunMicrosystems于1991年发明。......
  • 2025最新-计算机毕业设计java基于Springboot的智慧教学平台的系统设计与实现
    一、项目介绍  基于SpringBoot的智慧教学平台系统设计与实现是一个复杂但充满挑战的项目,旨在通过现代软件开发技术优化教育流程,提升教学质量和管理效率。以下是对该系统的详细介绍:智慧教学平台是一款集成了多种功能的综合性教育管理系统,通过运用SpringBoot框架、MySQ......