nvidia-smi Failed to initialize NVML: Driver/library version mismatch

nvidia-smi Failed to initialize NVML: Driver/library version mismatch

原因：NVIDIA 内核驱动版本与系统驱动不一致，

# sudo rmmod nvidia
rmmod: ERROR: Module nvidia is in use by: nvidia_modeset nvidia_uvm
首先要知道现在kernel mod 的依赖情况，首先我们从错误信息中知道，nvidia_modeset nvidia_uvm 这两个 mod 依赖于 nvidia, 所以要先卸载他们

# lsmod | grep nvidia
nvidia_uvm            769582  0
nvidia_drm             43547  2
nvidia_modeset       1053327  1 nvidia_drm
nvidia              15764359  2 nvidia_modeset,nvidia_uvm
drm_kms_helper        186531  1 nvidia_drm
drm                   456166  5 drm_kms_helper,nvidia_drm
ipmi_msghandler        56728  4 ipmi_ssif,ipmi_devintf,nvidia,ipmi_si

sudo lsof -n -w /dev/nvidia*
这些进程有个了解，如果一会卸载失败，记得关闭相关进程。

sudo lsof /dev/nvidia*
confirm you successfully unload those kmods

sudo rmmod nvidia_drm
rmmod: ERROR: Module nvidia_drm is in use

sudo rmmod nvidia_modeset
rmmod: ERROR: Module nvidia_modeset is in use by: nvidia_drm

sudo rmmod nvidia_uvm




lsmod | grep nvidia

you should get nothing, then confirm you can load the correct driver

nvidia-smi