这两天实验室的服务器总是崩溃，重启已经不能解决问题了，由于是跑深度学习的服务器，而且还是承接国家级项目的运行服务器，可以说是实验室的主要生产力了，给出报错的日志：

Oct 16 09:42:33 rootroot kernel: [ 7498.287883] perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
Oct 16 09:54:59 rootroot kernel: [ 8243.792856] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b1
Oct 16 09:54:59 rootroot kernel: [ 8243.793082] IP: _nv031733rm+0x79/0x940 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.793093] PGD 0 P4D 0
Oct 16 09:54:59 rootroot kernel: [ 8243.793100] Oops: 0000 [#1] SMP NOPTI
Oct 16 09:54:59 rootroot kernel: [ 8243.793280] CPU: 56 PID: 3976 Comm: gpustat Tainted: P OE 4.15.0-194-generic #205-Ubuntu
Oct 16 09:54:59 rootroot kernel: [ 8243.793527] RSP: 0018:ffffaf34a78078a8 EFLAGS: 00010202
Oct 16 09:54:59 rootroot kernel: [ 8243.793539] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
Oct 16 09:54:59 rootroot kernel: [ 8243.793553] RDX: ffff9b6c54600008 RSI: ffff9b6c53d98008 RDI: ffff9b6c69a34008
Oct 16 09:54:59 rootroot kernel: [ 8243.793593] R13: 000000000000000f R14: ffff9b6c53d98008 R15: 0000000000000001
Oct 16 09:54:59 rootroot kernel: [ 8243.793607] FS: 00007f782335e0c0(0000) GS:ffff9b2cffd00000(0000) knlGS:0000000000000000
Oct 16 07:37:46 rootroot kernel: [ 10.034623] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:d8/0000:d8:00.0/0000:d9:00.1/sound/card3/input2
Oct 16 07:37:46 rootroot kernel: [ 10.035662] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:d8/0000:d8:00.0/0000:d9:00.1/sound/card3/input3
Oct 16 07:37:46 rootroot kernel: [ 10.036394] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:d8/0000:d8:00.0/0000:d9:00.1/sound/card3/input4
Oct 16 07:37:46 rootroot kernel: [ 10.323371] bnxt_en 0000:18:00.0 eno1np0: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
Oct 16 07:37:46 rootroot kernel: [ 10.323374] bnxt_en 0000:18:00.0 eno1np0: EEE is not active
Oct 16 07:37:46 rootroot kernel: [ 10.323376] bnxt_en 0000:18:00.0 eno1np0: FEC autoneg off encodings: None
Oct 16 07:37:46 rootroot kernel: [ 11.444137] new mount options do not match the existing superblock, will be ignored
Oct 16 09:42:33 rootroot kernel: [ 7498.287883] perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
Oct 16 09:54:59 rootroot kernel: [ 8243.792856] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b1
Oct 16 09:54:59 rootroot kernel: [ 8243.793082] IP: _nv031733rm+0x79/0x940 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.793093] PGD 0 P4D 0
Oct 16 09:54:59 rootroot kernel: [ 8243.793100] Oops: 0000 [#1] SMP NOPTI
Oct 16 09:54:59 rootroot kernel: [ 8243.793280] CPU: 56 PID: 3976 Comm: gpustat Tainted: P OE 4.15.0-194-generic #205-Ubuntu
Oct 16 09:54:59 rootroot kernel: [ 8243.793527] RSP: 0018:ffffaf34a78078a8 EFLAGS: 00010202
Oct 16 09:54:59 rootroot kernel: [ 8243.793539] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
Oct 16 09:54:59 rootroot kernel: [ 8243.793553] RDX: ffff9b6c54600008 RSI: ffff9b6c53d98008 RDI: ffff9b6c69a34008
Oct 16 09:54:59 rootroot kernel: [ 8243.793593] R13: 000000000000000f R14: ffff9b6c53d98008 R15: 0000000000000001
Oct 16 09:54:59 rootroot kernel: [ 8243.793607] FS: 00007f782335e0c0(0000) GS:ffff9b2cffd00000(0000) knlGS:0000000000000000
Oct 16 09:54:59 rootroot kernel: [ 8243.793622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 09:54:59 rootroot kernel: [ 8243.793633] CR2: 00000000000000b1 CR3: 0000003f65f82003 CR4: 00000000007606e0
Oct 16 09:54:59 rootroot kernel: [ 8243.793646] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 09:54:59 rootroot kernel: [ 8243.793660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 09:54:59 rootroot kernel: [ 8243.793674] PKRU: 55555554
Oct 16 09:54:59 rootroot kernel: [ 8243.793681] Call Trace:
Oct 16 09:54:59 rootroot kernel: [ 8243.793857] ? _nv031847rm+0x82/0x270 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794036] ? _nv031880rm+0x17/0x30 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794217] ? _nv022845rm+0xc0/0x1b0 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794397] ? _nv022850rm+0x11b/0x230 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794576] ? _nv022850rm+0x211/0x230 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794755] ? _nv022852rm+0x310/0x310 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794855] ? _nv023526rm+0x32d/0x470 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.794954] ? _nv023526rm+0x304/0x470 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795056] ? _nv000719rm+0x32a/0x680 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795183] ? _nv000712rm+0x178a/0x2350 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795311] ? rm_init_adapter+0xc5/0xe0 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795397] ? nv_open_device+0x3e7/0x870 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795484] ? nvidia_open+0x310/0x510 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795573] ? nvidia_frontend_open+0x58/0xa0 [nvidia]
Oct 16 09:54:59 rootroot kernel: [ 8243.795587] ? chrdev_open+0xc4/0x1b0
Oct 16 09:54:59 rootroot kernel: [ 8243.795600] ? do_dentry_open+0x21d/0x370
Oct 16 09:54:59 rootroot kernel: [ 8243.796036] ? __inode_permission+0x5b/0x160
Oct 16 09:54:59 rootroot kernel: [ 8243.796446] ? cdev_put.part.2+0x20/0x20
Oct 16 09:54:59 rootroot kernel: [ 8243.796852] ? vfs_open+0x4f/0x80
Oct 16 09:54:59 rootroot kernel: [ 8243.797246] ? path_openat+0x6bf/0x18b0
Oct 16 09:54:59 rootroot kernel: [ 8243.797626] ? filename_lookup+0xf2/0x190
Oct 16 09:54:59 rootroot kernel: [ 8243.798005] ? __check_object_size+0xc8/0x1b0
Oct 16 09:54:59 rootroot kernel: [ 8243.798385] ? do_filp_open+0x9b/0x110
Oct 16 09:54:59 rootroot kernel: [ 8243.798753] ? __check_object_size+0xc8/0x1b0
Oct 16 09:54:59 rootroot kernel: [ 8243.799111] ? __alloc_fd+0x46/0x170
Oct 16 09:54:59 rootroot kernel: [ 8243.799453] ? do_sys_open+0x1ba/0x2c0
Oct 16 09:54:59 rootroot kernel: [ 8243.799782] ? do_sys_open+0x1ba/0x2c0
Oct 16 09:54:59 rootroot kernel: [ 8243.800094] ? SyS_openat+0x14/0x20
Oct 16 09:54:59 rootroot kernel: [ 8243.800394] ? do_syscall_64+0x73/0x130
Oct 16 09:54:59 rootroot kernel: [ 8243.800681] ? entry_SYSCALL_64_after_hwframe+0x41/0xa6
Oct 16 09:54:59 rootroot kernel: [ 8243.800958] Code: a7 07 00 00 41 bf 01 00 00 00 4c 8d 65 48 31 db 44 89 7d 10 66 0f 1f 44 00 00 41 f6 c5 01 0f 84 90 00 00 00 49 8b 86 30 1a 00 00 <80> b8 b1 00 00 00 00 74 12 b8 01 00 00 00 89 d9 d3 e0 41 85 86

====================================================

这个报错日志显示的是内核问题，是SMP构架下CPU同步超时后死锁，但是导致这个问题的进程又是NVIDIA的nv_queue，从这些信息上也是找不到如何解决的方法，也看了下网上相关的post:

465.24.02 page fault

于是大胆的猜测是NVIDIA的驱动问题，不过这服务器平时运行的好好的怎么会突然驱动有问题呢，这样一想又有些自相矛盾，不过再一想是不是有可能是有的用户跑了一些不兼容的cuda代码呢，因为以我的经验来说，我是知道一些TensorFlow1.x的代码运行在RTX2090或者是RTX3090显卡是会造成死机的，这样是不是可以推定到pytorch上呢，这么一想或许还真可能是显卡驱动的问题。于是我给出的解决方法就是升级系统（upgrade-release），然后再upgrade一下显卡的驱动，那么这样行不行呢，不清楚，先这样运行一段时间看看，如果不报错，不再死机就证明好用，如果这样的话本文就不再更新了。

还是希望这个法子可行，本文也不要再update了，毕竟兼职负责实验室服务器管理和当实验室的免费网管也不是啥好事情，劳心劳力还没回报。

====================================================