Sample Header Ad - 728x90

How to isolate (reserve) GPUs with identical hardware IDs via IOMMU / dracut / kernel parameters (in VoidLinux)

0 votes
1 answer
684 views
I'm trying to set up pci/gpu passthrough on void. Specifically, using the vfio-pci driver to "reserve" one of the GPUs. I have been following the procedure outlined in [this void-docs PR](https://github.com/void-linux/void-docs/pull/217) (and the arch wiki where relevant) I have an RX 580 and an RX 570, and my groups allow for isolating the former:
IOMMU Group 0 - 11:
    dummy host bridges and such
IOMMU Group 12:
	03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
	03:00.1 SATA controller : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	03:00.2 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	20:00.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:01.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:04.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	22:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	25:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 13:
	26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	26:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 14 - 19:
	dummy functions and audio devices
So, lumped in with a bunch of CPU-things, the RX570:
IOMMU group 12: 25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]
and completely separate, in its own group, the RX580:
IOMMU group 13: 26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]
Now as you can see these two share hardware IDs, and therefore I can't just specify a kernel param to push vfio-pci driver onto one of them, it would catch both of them. (as it would be described in the guide in the above pull-request) I tried separating them [following the guidelines on the arch-wiki](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs) , added the "Passthrough selected GPU" script to my dracut config: /usr/local/bin/vfio-pci-override.sh
#!/bin/sh

DEVS="0000:26:00.0 0000:26:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

modprobe -i vfio-pci
/etc/dracut.conf.d/10-vfio.conf
hostonly=" yes "
hostonly_cmdline=" amd_iommu=on iommu=pt "
force_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "
install_items+=" /usr/local/bin/vfio-pci-override.sh "
which should have an effect, if I'm not mistaken, if I run xbps-reconfigure --force linuxX.Y but for some reason, the amdgpu driver still catches both GPUs. I have the following kernel parameters set in the /etc/default/grub file:
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 amd_iommu=on iommu=pt rd.driver.pre=vfio_pci"
Any help is greatly appreciated, and will be forwarded to the aforementioned PR to extend on the guide. (Though I know the void handbook isn't meant to describe things like this in detail, but maybe it'll be useful for the creation of an independent guide on this)
Asked by vn-ck (1 rep)
Nov 9, 2021, 12:42 AM
Last activity: Mar 23, 2022, 05:01 AM