How to isolate (reserve) GPUs with identical hardware IDs via IOMMU / dracut / kernel parameters (in VoidLinux)
0
votes
1
answer
684
views
I'm trying to set up pci/gpu passthrough on void.
Specifically, using the
vfio-pci
driver to "reserve" one of the GPUs.
I have been following the procedure outlined in [this void-docs PR](https://github.com/void-linux/void-docs/pull/217) (and the arch wiki where relevant)
I have an RX 580 and an RX 570, and my groups allow for isolating the former:
IOMMU Group 0 - 11:
dummy host bridges and such
IOMMU Group 12:
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
03:00.1 SATA controller : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
03:00.2 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
20:00.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:01.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:04.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
22:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
25:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 13:
26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
26:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 14 - 19:
dummy functions and audio devices
So, lumped in with a bunch of CPU-things, the RX570:
IOMMU group 12: 25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]
and completely separate, in its own group, the RX580:
IOMMU group 13: 26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]
Now as you can see these two share hardware IDs, and therefore I can't just specify a kernel param to push vfio-pci
driver onto one of them, it would catch both of them. (as it would be described in the guide in the above pull-request)
I tried separating them [following the guidelines on the arch-wiki](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs) , added the "Passthrough selected GPU" script to my dracut config:
/usr/local/bin/vfio-pci-override.sh
#!/bin/sh
DEVS="0000:26:00.0 0000:26:00.1"
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
for DEV in $DEVS; do
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
fi
modprobe -i vfio-pci
/etc/dracut.conf.d/10-vfio.conf
hostonly=" yes "
hostonly_cmdline=" amd_iommu=on iommu=pt "
force_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "
install_items+=" /usr/local/bin/vfio-pci-override.sh "
which should have an effect, if I'm not mistaken, if I run xbps-reconfigure --force linuxX.Y
but for some reason, the amdgpu
driver still catches both GPUs.
I have the following kernel parameters set in the /etc/default/grub
file:
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 amd_iommu=on iommu=pt rd.driver.pre=vfio_pci"
Any help is greatly appreciated, and will be forwarded to the aforementioned PR to extend on the guide.
(Though I know the void handbook isn't meant to describe things like this in detail, but maybe it'll be useful for the creation of an independent guide on this)
Asked by vn-ck
(1 rep)
Nov 9, 2021, 12:42 AM
Last activity: Mar 23, 2022, 05:01 AM
Last activity: Mar 23, 2022, 05:01 AM