查看主机是正常重启还是意外重启:
last -x|head|tac
正常reboot重启显示如下:
root pts/0 10.5.3.207 Tue Nov 22 11:14 - down (00:04) shutdown system down 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:18 (00:00) reboot system boot 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:39 (00:20) runlevel (to lvl 5) 3.10.0-1062.el7. Tue Nov 22 11:18 - 11:23 (00:04)
正常按电源开关重启显示如下:
root pts/0 10.5.3.207 Tue Nov 22 11:29 - crash (00:02) reboot system boot 3.10.0-1062.el7. Tue Nov 22 11:31 - 11:39 (00:08) runlevel (to lvl 5) 3.10.0-1062.el7. Tue Nov 22 11:31 - 11:39 (00:08)
意外重启多数显示如下(非人为或断电,系统自己重启):
reboot system boot 3.10.0-1160.49.1 Tue Nov 22 02:27 - 13:52 (11:25) runlevel (to lvl 5) 3.10.0-1160.49.1 Tue Nov 22 02:27 - 13:52 (11:24)
那么问题就来了,到底是什么原因引起的重启?
答:80% 是硬件有问题,如电源,内存条,CPU等。
排查原因:
1. 查看message 在重启时间点上下有什么特别的信息
cat /var/log/message
在重启时间点往前翻,发现有多条报错如下:
Nov 21 17:43:08 host142 kernel: EDAC MC0: 3 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#0 or CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 page:0x2b980fe offset:0x0 grain:32 syndrome:0x0 - OVERFLOW area:DRAM err_code:0008:00c0 socket:1 ha:0 channel_mask:1 rank:255)
到这步基本确认是内存有问题了,是Channel#0_DIMM#0: 也就是A1有问题,参照下面的对应表
2. 再进一步查看一下硬件信息
dmesg |grep -Ei 'error|Fail'
如果得到以下信息可能是内存或cpu有问题:
[root@mysql /]# dmesg |grep -Ei 'error|Fail' [ 1.750394] ERST: Error Record Serialization Table (ERST) support is initialized. [ 2.893211] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 4163.052786] mce: [Hardware Error]: Machine check events logged [ 4163.052813] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR [ 4163.352856] EDAC MC0: 2 CE memory scrubbing error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#0 or CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 page:0x2b980fa offset:0x0 grain:32 syndrome:0x0 - OVERFLOW area:DRAM err_code:0008:00c0 socket:1 ha:0 channel_mask:1 rank:255)
3. 使用工具检查一下内存是否有报错:
yum install -y libsysfs edac-utils
#查看内存是否有报错:
edac-util -v
内存条对应关系如下(DELL R620):
其中 mc0 表示 表示内存控制器0, CPU_Src_ID#0表示源CPU0 , Channel#0 表示通道0
DIMM#0 标示内存槽0,Corrected Errors 代表已经纠错的次数,根据前面列出的CPU通
道和内存槽对应关系即可给edac-utils 返回的信息进行编号。
12条内存的对应关系:
mc0: csrow0: CPU#0Channel#0_DIMM#0: A1
mc0: csrow0: CPU#0Channel#1_DIMM#0: A2
mc0: csrow0: CPU#0Channel#2_DIMM#0: A3
mc0: csrow1: CPU#0Channel#0_DIMM#1: A4
mc0: csrow1: CPU#0Channel#1_DIMM#1: A5
mc0: csrow1: CPU#0Channel#2_DIMM#1: A6
mc1: csrow0: CPU#1Channel#0_DIMM#0: B1
mc1: csrow0: CPU#1Channel#1_DIMM#0: B2
mc1: csrow0: CPU#1Channel#2_DIMM#0: B3
mc1: csrow1: CPU#1Channel#0_DIMM#1: B4
mc1: csrow1: CPU#1Channel#1_DIMM#1: B5
mc1: csrow1: CPU#1Channel#2_DIMM#1: B6