OpenStack显卡穿透
-
注意事项
-
服务器主板CPU虚拟化打开
-
如果只有一张显卡物理机不要安装显卡驱动
1. 确认是否支持iommu
cat /proc/cmdline | grep iommu
- 针对Inter CPU(注意自己CPU型号)。如果没有输出,说明不支持iommu,需要手动开启。
vim /etc/default/grub
GRUB_CMDLINE_LINUX="intel_iommu=on crashkernel=auto rhgb quiet"
- 针对AMD CPU(注意自己CPU型号)。如果没有输出,说明不支持iommu,需要手动开启。
vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 amd_iommu=on"
2. 更新grub,并重启设备
- 更新grub(注意自己安装系统用的方式,这种是系统盘)
grub-mkconfig -o /boot/grub/grub.cfg
- 如果是efi安装的系统,那么使用如下更新grub
grub-mkconfig -o /boot/grub/grub.cfg
- 重启系统
reboot
- 重启以后如下确认是否正常
dmesg | grep -i iommu
- 检查当前显卡设备信息
- 注意,将得到的输出结果保存下来
lspci -nn | grep NVID
# 输出结果,第一个是显卡器,第二个是显卡音频
83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)
83:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
3. vfio
- 配置vfio
vim /etc/modprobe.d/blacklist.conf
blacklist nouveau
options nouveau modeset=0
blacklist xhci_hcd
blacklist nvidia
blacklist snd_hda_intel
blacklist nvidiafb
- 配置系统加载模块
vim /etc/modules-load.d/openstack-gpu.conf
vfio_pci
pci_stub
vfio
vfio_iommu_type1
kvm
kvm_intel
- 配置vfio加载的设备
- 注意下面的ids设备码对应自己显卡
vim /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c81,10de:0fb9
- 重启
reboot
4. 挂载VFIO驱动
echo 10de 1c81 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 10de 0fb9 > /sys/bus/pci/drivers/vfio-pci/new_id
- 检查VIFO驱动是否挂载成功
lspci -nnk -d 10de:1c81
lspci -nnk -d 10de:0fb9
5. OpenStack穿透配置
5.1 控制节点配置
- 配置nova文件
vim /etc/nova/nova.conf
[scheduler]
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter
- 重启nova-scheduler服务
service nova-scheduler restart
- 配置nova-api
- 注意显卡编码对应上,有多少个写多少个
vim /etc/nova/nova.conf
[pci]
alias = { "name": "nvidia1050vga", "product_id": "1c81", "vendor_id": "10de", "device_type": "type-PCI" }
alias = { "name": "nvidia1050aud", "product_id": "0fb9", "vendor_id": "10de", "device_type": "type-PCI" }
- 重启相关服务
service nova-api restart
5.2 计算节点
- 配置nova文件
- 注意显卡编码对应上,有多少个写多少个
vim /etc/nova/nova.conf
[pci]
alias = { "name": "nvidia1050vga", "product_id": "1c81", "vendor_id": "10de", "device_type": "type-PCI" }
alias = { "name": "nvidia1050aud", "product_id": "0fb9", "vendor_id": "10de", "device_type": "type-PCI" }
passthrough_whitelist = [{ "vendor_id": "10de", "product_id": "1c81" },{ "vendor_id": "10de", "product_id": "0fb9" }]
- 重启nova-compute
service nova-compute restart
- 安装ovmf,支持windows uefi启动
apt install ovmf -y
6. OpenStack调用显卡
- 创建及设置flavor,将显卡透传给虚拟机
openstack flavor create C2-4G-100G --ram 4096 --disk 100 --vcpus 2 --public
openstack flavor set C2-4G-100G --property "pci_passthrough:alias"="nvidia1050vga:1,nvidia1050aud:1"
- 因为NVIDIA显卡的驱动会检测是否跑在虚拟机里,如果在虚拟机里驱动就会出错,所以我们需要对显卡驱动隐藏hypervisor id
openstack image set [IMG-UUID] --property img_hide_hypervisor_id=true
- 之后利用相应的云主机类型创建相关实例即可
- 查看pci设备
root@ubuntu:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
# 这是我们的显卡驱动
00:05.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050] (rev a1)
00:06.0 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:08.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
- 安装完显卡驱动的效果
root@ubuntu:~# nvidia-smi
Fri Jan 20 13:07:31 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:00:05.0 Off | N/A |
| 21% 18C P8 N/A / 80W | 6MiB / 2048MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
标签:00,10de,nova,穿透,pci,显卡,OpenStack,id
From: https://www.cnblogs.com/wsxier/p/17063362.html