openstack PCI透傳(GPU)

一介布衣·GZ發表於2022-04-25
GPU

描述

kolla-ansible部署openstack的GPU透傳方法

一、gpu物理伺服器配置

在gpu伺服器上主啟用IOMMU
確認核心⽀支援iommu
$ cat /proc/cmdline | grep iommu 
輸出以下資訊：BOOT_IMAGE=/vmlinuz-3.10.0-1062.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on
如果沒有輸出, 則需要修改kernel啟動引數,接著下面操作。

對於Intel cpu：GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
對於AMD cpu：GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt iommu=1"
下面以intel cpu 為修改例子

$ vim /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rhgb quiet intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"
重新生成grub
  • 系統採用EFI啟動生成方式：grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
  • 系統採用非EFI啟動生成方式： grub2-mkconfig -o /boot/grub2/grub.cfg
將下列內容加入到blacklist中以避免被gpu伺服器佔用
$ vim  /etc/modprobe.d/blacklist.conf
blacklist snd_hda_intel
blacklist amd76x_edac
blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
blacklist nvidia
重新生成initramfs
$ mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
$ dracut -v /boot/initramfs-$(uname -r).img $(uname -r)
重啟系統
$ reboot

二、配置openstack

確認pci裝置驅動資訊

$ lspci -vnn | grep -i nvi
0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2204] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device [10de:1454]
0c:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:1454]

其中[10de:2204]的10de為NVIDIA pci裝置的vendor id, 2204為product id, 0c:00.0為裝置地址

增加 PciPassthroughFilter 和 available_filters = nova.scheduler.filters.all_filters
$ vim /etc/kolla/config/nova.conf 
*******
[filter_scheduler]
enabled_filters = RetryFilter,AvailabilityZoneFilter,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,NUMATopologyFilter,PciPassthroughFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,DifferentHostFilter,SameHostFilter
available_filters = nova.scheduler.filters.all_filters
新增pci裝置顯示卡資訊
$ vim /etc/kolla/config/nova.conf 
*******
[pci] 
alias = { "vendor_id":"10de", "product_id":"2204", "device_type":"type-PCI","name":"Nvidia3090a" }
passthrough_whitelist = { "vendor_id": "10de", "product_id": "2204", "address": "0000:0c:00.0" }
reconfigure nova元件
$ kolla-ansible -i /etc/ansible/hosts -t nova deploy
配置flavor並用這個flavor建立虛擬機器測試
$ openstack flavor set GPU-16C32G-e --property pci_passthrough:alias='Nvidia3090a:1'
參考文獻

自動化kolla-ansible部署openstack+GPU透傳方法

相關文章