So I've been trying to set up GPU passthrough on Arch, but I've run into a wall. I followed these two guides, mostly the first one though: GPU Passthrough Revisited , Non-root GPU passthrough . I followed each step exactly, and everything worked. Then, when I got to the part where I had to start QEMU, it didn't work. I successfully bound my GPU to vfio, but now I'm stuck. Here's the output of lspci -nnk for my GPU: 01:00.0 3D controller : NVIDIA Corporation GM107M [GeForce GTX 950M] [10de:139a] (rev a2) Subsystem: ASUSTeK Computer Inc. Device [1043:178d] Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia_drm, nvidia But then when I try to start qemu, I get this output: Invalid MIT-MAGIC-COOKIE-1 keyUnable to init server: Could not connect: Connection refused QEMU 2.9.0 monitor - type 'help' for more information (qemu) gtk initialization failed The MIT-MAGIC-COOKIE can be fixed by setting the WINDOW variable, but I still can't start up QEMU. Any idea what I'm doing wrong?

I'm having a laptop with i5-10210U, using it as a server-station. I am trying to passthrough the integrated Graphics Card to a VM using QEMU with this command, modified by fallowing this tutorial : virt-install --name windows11 --ram=8192 --vcpus=8 --machine q35 \ --device vfio-pci,host=00:02.0 --cpu host --hvm --disk path=w11vml,size=80 \ -object input-linux,id=kbd,evdev=/dev/input/by-path/platform-i8042-serio-0-event-kbd --cdrom W11.iso --graphics vnc,port=5901,listen=0.0.0.0,passwd='123456' response: virt-install: error: unrecognized arguments: --device vfio-pci,host=00:02.0 [alongside with -object input-linux] My problem is that I don't know how to translate the qemu-system-x86_64 arguments to virt-install after running the detaching script disabling the GPU on host, which would be: sudo qemu-system-x86_64 -machine q35 -m 2G -accel kvm -cpu host \ -device vfio-pci,host=00:02.0 -nographic -vga none \ -object input-linux,id=kbd,evdev=/dev/input/by-path/platform-i8042-serio-0-event-kbd \ -cdrom Fedora-Workstation-Live-x86_64-29-1.2.iso Additional info: VT-d is enabled in BIOS, OS: AlmaLinux9 I know the args could be very wrong, especially with graphics vnc args one, as it needs to be forced to fallback to the integrated graphics, but any corrections would be very helpfull.. I'm new to this "business" :D

I would like to passthru a PCI device to qemu for FreeBSD (14.2) without using virt-manager and vfio (because FreeBSD does not support it),but only the "raw" parameters. This is the device that I want to assign to qemu : marietto# lspci 05:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8192EE PCIe Wireless Network Adapter According with this post : https://unix.stackexchange.com/questions/96606/qemu-arm-how-to-passthrough-a-pci-card I've added the parameter "device pci-assign,host=05:00.0",like this : /usr/local/bin/qemu-system-x86_64 -machine pc-q35-9.1 -cpu max -m size=4292608k \ -vga std \ -drive file=/mnt/zroot2/zroot2/bhyve/img/Android/Android-qemu.img,format=raw \ -smp 4,sockets=4,cores=1,threads=1 -no-user-config -nodefaults \ -rtc base=utc,driftfix=slew \ -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=true,addr=0x2 \ -device pcie-pci-bridge,id=pci.2,bus=pci.1,addr=0x0 \ -device pcie-root-port,port=17,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=18,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=19,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x3 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=true,addr=0x1d \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \ -device ich9-ahci,id=sata \ -netdev tap,id=hostnet0,ifname=tap13,script=no,downscript=no \ -device e1000,netdev=hostnet0,mac=52:54:00:a3:e1:52 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \ > -device pci-assign,host=05:00.0 \ -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial,index=0 \ -drive if=pflash,format=raw,readonly=on,file=/usr/local/share/edk2-qemu/QEMU_UEFI_CODE-x86_64.fd \ qemu-system-x86_64: -device pci-assign,host=05:00.0: 'pci-assign' is > not a valid device model name Probably pci-assign is not a valid parameter anymore for the version of qemu that I'm using ? this one : marietto# qemu-system-x86_64 --version QEMU emulator version 9.1.0 Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers I have to say that if I boot the vm using bhyve instead of qemu,using these parameters,it is able to connect to internet,so the PCI-e device is recognized : bhyve-lin -S -c sockets=2,cores=1,threads=1 -m 4G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/zroot-133/bhyve/img/Android/Android-qemu.img,bootindex=1 \ > -s 8:0,passthru,5/0/0 \ -s 11,hda,play=/dev/dsp,rec=/dev/dsp \ -s 13,virtio-net,tap13 \ -s 29,fbuf,tcp=0.0.0.0:5913,w=1600,h=950,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd,/usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd \ vm0:13 < /dev/null & sleep 5 && vncviewer 0:13 && echo vncviewer 0:13 & I think that's only a matter of finding the correct syntax. Please,help me, thanks.

I have an AMD iGPU (integrated in an AMD Ryzen 9 9950x CPU) that I want to use as the video output for a Windows VM in proxmox (I have no other GPU installed). I discovered that it doesn't have the Function Level Reset capability, so from my understanding I can't unbind it from the host OS and rebind it to the VM. I tried using vendor-reset, but it is not compatible with my iGPU. Then, I tried to prevent the iGPU from ever being initialised by the host OS, by blacklisting my GPU drivers (amdgpu and radeon) by creating a config file in

/etc/modprobe.d/

that looks like this:

blacklist amdgpu
blacklist radeon
blacklist vgaarb
blacklist efifb
blacklist vesafb
install amdgpu /bin/false
install radeon /bin/false
install vgaarb /bin/false

I tried to specify every possible restriction on any frame buffer or GPU driver, as well as giving full control of the GPU to the vfio driver. I hoped this would prevent any part of the OS from ever touching the iGPU, but vgaarb still sets it as the boot VGA device, only to deactivate it afterwards, as displayed by dmesg:

[    0.472763] pci 0000:72:00.0: vgaarb: setting as boot VGA device
[    0.472763] pci 0000:72:00.0: vgaarb: bridge control possible
[    0.472763] pci 0000:72:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    0.490408] pci 0000:72:00.1: D0 power state depends on 0000:72:00.0
[    0.491001] pci 0000:72:00.0: Adding to iommu group 35
[    2.052946] vfio-pci 0000:72:00.0: vgaarb: deactivate vga console

My full kernel command line (as displayed by dmesg) is:

BOOT_IMAGE=/vmlinuz-6.8.12-5-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt nofb pcie_aspm=off nomodeset video=vesafb:off video=efifb:off video=vesa:off video=simplefb:off disable_vga=1 vga=none vgaarb=off video=off vfio-pci.ids=1002:13c0,1002:1640 vfio-pci.disable_vga=1 initcall_blacklist=sysfb_init,amd_iommu_v2_init modprobe.blacklist=amdgpu,radeon,snd_hda_intel,r8169 module_blacklist=amdgpu,radeon vgaarb.multi_gpu_switch=0 video.ignore_edd=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 rd.driver.blacklist=amdgpu rd.driver.blacklist=radeon rd.driver.pre=vfio-pci pcie_acs_override=downstream,multifunction radeon.modeset=0 amdgpu.modeset=0 amdgpu.aspm=0

I can confirm that in the end the GPU uses vfio as its driver, but it already has been initialised by the host OS and starting the VM returns the error

is already attached

The output of

-nnk

:

72:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:13c0] (rev c1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

The VM's configuration file:

bios: ovmf
boot: order=ide2;net0
cores: 8
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:72:00,pcie=1,x-vga=1
ide2: local:iso/windows-11.iso,media=cdrom,size=5695402K
machine: pc-q35-9.0
memory: 16384
meta: creation-qemu=9.0.2,ctime=1733853637
name: windows11
net0: virtio=BC:24:11:2B:1A:A2,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: local-zfs:vm-100-disk-1,aio=native,iothread=1,size=128G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=e73054d1-f03b-4b84-8cdc-9115d85676fa
sockets: 1
tpmstate0: local-zfs:vm-100-disk-2,size=4M,version=v2.0
vmgenid: e16f7ba2-2469-4495-ab4f-7480c7f89041
args: -device vfio-pci,host=72:00.0,multifunction=on,x-vga=on,rombar=0 -device vfio-pci,host=72:00.1

What else could I try? Thank you in advance!

I am trying to get my Windows VM to access my USB pen drive, but I am having problems passing this USB over to the VM. On my Linux host machine, I ran lsusb where my device was identified as Bus 001 Device 018 : ID 8564:1000 Transcend Information, Inc. Jetflash Looking at a few guides, I tried to pass this USB to kvm using qemu-system-x86_64 -m 3G --enable-kvm "Windows 10.qcow2" -usb -device usb-host,hostbus=1,hostaddr=18 No matter what I try, I get the error saying "usb-host" is not a valid device. I tried to change that to usb-storage, since that is the class of device I see for it when doing lsusb -t, but I don't know what to pass for Device ID..If I leave it blank, it says property ".drive" is missing, as expected. I tried doing a ls /sys/bus/usb/devices which printed out some information like 1-0:10 1-1 1-1.1 ........usb1 usb2 But no matter which id I try and pass along in the command qemu-system-x86_64 -m 3G --enable-kvm "Windows 10.qcow2" -usb -device usb-storage,drive=, it says no such property with given value exists. So, 1. Should I be concerned that usb-host is not a valid device? 2. What do I need to do to get usb-storage with device id to work? 3. Does qemu-kvm support usb 2, 3 and streaming camera? I'm using QEMU emulator version 4.1.0

When I do lspci on my **manjaro linux host** I can see that my Nvidia GPU is a part of [1025:1409] subsystem (as is everything else on my machine). The GPU is in its own IOMMU group and has vfio-pci driver. When I pass through my GPU into **windows 11 guest** in qemu the device instance path in windows device manager is PCI\VEN_10DE&DEV_1D52&SUBSYS_00000000&REV_A1\4&12829B10&0&0014. So subsystem is lost. Because of that I have problems with the drivers. First the driver installer couldn't find compatible hardware. Then I replaced all occurences of 1025:1409 sybsystem related to my device (1D52) to 0000:0000 subsystem in nvacig.inf (found no occurences in other *.inf files). Now driver installation starts but fails after some time. How do I fix it? It feels like if the subsystem was right then it would work.

I've been following the Official Arch Linux Guide (https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF) and everything has been working exactly how it should be, but once I start the VM the Monitor connected to my GPU gets grey (Before that It was showing the MSI Logo, so it detected something was happening) but nothing more happens. I have QEMU open on the second Monitor and I can see the output on the preview but not on the actual Display. Any help would be appreciated, thanks. GPU: NVIDIA RTX 3070 Ti CPU: i9-12900k (Using the iGPU for Manjaro)

I am newbie here and I can only find the blog or readme from github. Is there any official documents? Emm,someone wrote the "blacklist" on "/etc/modules-load.d/modules.conf" while someone wrote the "blacklist" on "/etc/modules-load.d/blacklist.conf". And someone wrote the "softdep" instead of "blacklist" For example, someone wrote echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf echo "blacklist snd_hda_intel" >> /etc/modprobe.d/blacklist.conf and someone wrote echo "softdep nouveau pre: vfio-pci" >> /etc/modprobe.d/nvidia.conf echo "softdep nvidia pre: vfio-pci" >> /etc/modprobe.d/nvidia.conf echo "softdep nvidia* pre: vfio-pci" >> /etc/modprobe.d/nvidia.conf I really don't know the difference,any suggestion would be appreciated.

I'm trying to set up pci/gpu passthrough on void. Specifically, using the vfio-pci driver to "reserve" one of the GPUs. I have been following the procedure outlined in [this void-docs PR](https://github.com/void-linux/void-docs/pull/217) (and the arch wiki where relevant) I have an RX 580 and an RX 570, and my groups allow for isolating the former:

IOMMU Group 0 - 11:
    dummy host bridges and such
IOMMU Group 12:
	03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
	03:00.1 SATA controller : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	03:00.2 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	20:00.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:01.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:04.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	22:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	25:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 13:
	26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	26:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 14 - 19:
	dummy functions and audio devices

So, lumped in with a bunch of CPU-things, the RX570:

IOMMU group 12: 25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]

and completely separate, in its own group, the RX580:

IOMMU group 13: 26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]

Now as you can see these two share hardware IDs, and therefore I can't just specify a kernel param to push vfio-pci driver onto one of them, it would catch both of them. (as it would be described in the guide in the above pull-request) I tried separating them [following the guidelines on the arch-wiki](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs) , added the "Passthrough selected GPU" script to my dracut config: /usr/local/bin/vfio-pci-override.sh

#!/bin/sh

DEVS="0000:26:00.0 0000:26:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

modprobe -i vfio-pci

/etc/dracut.conf.d/10-vfio.conf

hostonly=" yes "
hostonly_cmdline=" amd_iommu=on iommu=pt "
force_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "
install_items+=" /usr/local/bin/vfio-pci-override.sh "

which should have an effect, if I'm not mistaken, if I run xbps-reconfigure --force linuxX.Y but for some reason, the amdgpu driver still catches both GPUs. I have the following kernel parameters set in the /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 amd_iommu=on iommu=pt rd.driver.pre=vfio_pci"

Any help is greatly appreciated, and will be forwarded to the aforementioned PR to extend on the guide. (Though I know the void handbook isn't meant to describe things like this in detail, but maybe it'll be useful for the creation of an independent guide on this)

I bought AMD RyZEN 5 2400G, Gigabyte B450M D3SH and 16GB RAM. Some elder GPU AMD Radeon HD 6XXX I took for passing through. But I was unpleasantly surprised with the IOMMU grouping. I put the GPU in either first (16/16) or third (16/8) PCIe slot, and found that it was in both cases included in the iommu group zero. But other members of this iommu group cannot be passed though to VM neither excluded from host system. There were just four iommu groups (0 - 3), while in the tutorials and my other PCs I see twelve or more groups, and some PCIe slots are quite isolated in a single group. Questions: Does anybody knows: - If I should buy some other hardware? - If there are some kernel options to get more iommu groups (split)? - If do I have to use legacy BIOS instead of uEFI? SVM enabled, IOMMU enabled. OS: Debian Linux testitng, kernel 5.10, kernel cmd options iommu=on amd_iommu=on iommu group zero: 01:00.0 USB [0c03]: [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01) 01:00.1 SATA : [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01) 01:00.2 PCIbridge : [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01) 02:00.0 PCIbridge : [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) 04:00.0 Ethernet : Realtek Co. Ltd. RTL8111/8168/8411 PCIe Gigabit Eth [10ec:8168] (rev 0c) 05:00.0 VGA : [AMD/ATI] Turks PRO [Radeon HD 6570/7570/8550] [1002:6759] 05:00.1 Audio dev : [AMD/ATI] Turks HDMI Audio [Radeon HD 6500/6600/6700M Series] [1002:aa90] 06:00.0 NVMe : Intel NVMe Optane Memory [8086:2522] group 1 ... 08:00.0 VGA : [AMD/ATI] Raven Ridge [Radeon Vega (Mobile) Series] [1002:15dd] (rev c6) 08:00.1 Audio dev : [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de] ... As you can see, the APU (GPU integrated on the CPU chip) is a member of the group 1.

Short version: Using systemctl start libvirtd (or libvirtd.service, seems to have the same effect?) crashes the system, without warning, if one of the pools has /dev allocated. Long version: I'm trying install a guest OS with the help of passthrough, on Archlinux, with virt-manager. I'm following [this guide](https://dividebyzer0.gitlab.io/GPUpassthrough.html) . In the [*Configuring Virtual Machine*](https://dividebyzer0.gitlab.io/GPUpassthrough.html#configuring-virtual-machine) section, the first step is to add storage, and select the /dev. Once I do that, the system will reboot instantly, without any warning, after ~1 min. I have checked with systemctl status libvirtd at regular intervals, and the lowest time I got to see was 55 sec; I never got to see 1 min. The setting is saved in /etc/libvirt/storage/pool*.xml (pool-1.xml, or something like that). I deleted it in the meantime and now there are no more crashes. There are [one](https://old.reddit.com/r/archlinux/comments/o8wy6w/libvirt_reboots_system_after_1_minute/) and [two](https://old.reddit.com/r/archlinux/comments/mbdjot/system_automatically_reboots_1_min_after_booting/) threads on reddit, but none has any workaround. Is this a known behaviour, or is it a quirk? How do I get around it and finish installing the guest OS?

I have built a system with a Geforce GTX 960 and a Quadro M4000 graphics card, that I usually pass through to a virtual machine. The GTX 960 card is only used by the host. Normally, the Quadro card would not be available by the host, because the kernel driver vfio-pci prevents it from being used. However, when I don't use it in the virtual machine, then I would like to have it accessible from the host machine, e.g. to do some computation. But, there is this very strange behaviour in power consumption and fan speed... How can I reduce the power consumption and fan speed without needing to have nvidia-setttings open all the time? From my notes: ## Reuse a Passed-through-ready Device on the Host Supposed a secondary graphics card, that has been prepared for passing it through to a guest, should be used on the host instead. The device would normally not be usable on the host, since the wrong driver is loaded. Here, the Quadro M4000 has the vfio-pci driver in use, but instead the nvidia driver should be used.

sudo lspci -nnk | egrep -A3 "VGA|Display|3D"
  # 0b:00.0 VGA compatible controller : NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
  # Subsystem: Gigabyte Technology Co., Ltd Device [1458:36ac]
  # Kernel driver in use: nvidia
  # Kernel modules: nouveau, nvidia_drm, nvidia
  # --
  # 0c:00.0 VGA compatible controller : NVIDIA Corporation GM204GL [Quadro M4000] [10de:13f1] (rev a1)
  # Subsystem: Hewlett-Packard Company Device [103c:1153]
  # Kernel driver in use: vfio-pci
  # Kernel modules: nouveau, nvidia_drm, nvidia

Unload the vfio-pci driver and check the device status again. No kernel driver should be in use, hence line Kernel driver in use: ... is gone.

sudo modprobe -r vfio-pci
sudo lspci -nnk | egrep -A3 "VGA|Display|3D"
  # 0b:00.0 VGA compatible controller : NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
  # Subsystem: Gigabyte Technology Co., Ltd Device [1458:36ac]
  # Kernel driver in use: nvidia
  # Kernel modules: nouveau, nvidia_drm, nvidia
  # --
  # 0c:00.0 VGA compatible controller : NVIDIA Corporation GM204GL [Quadro M4000] [10de:13f1] (rev a1)
  # Subsystem: Hewlett-Packard Company Device [103c:1153]
  # Kernel modules: nouveau, nvidia_drm, nvidia
  # 0c:00.1 Audio device : NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

Also check the output of the nvidia driver tool nvidia-smi. It should list only one graphics card (the not-passed-through GTX 960).

sudo nvidia-smi 
  # Tue Sep 28 18:19:36 2021       
  # +-----------------------------------------------------------------------------+
  # | NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
  # |-------------------------------+----------------------+----------------------+
  # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  # |                               |                      |               MIG M. |
  # |===============================+======================+======================|
  # |   0  NVIDIA GeForce ...  Off  | 00000000:0B:00.0  On |                  N/A |
  # |  0%   51C    P8    19W / 160W |    477MiB /  4040MiB |      0%      Default |
  # |                               |                      |                  N/A |
  # +-------------------------------+----------------------+----------------------+
  # ...

Remove all associated PCI devices from the system. In this case, those are 0c:00.0 and 0c:00.1. Then check that those are actually gone.

echo 1 | sudo tee /sys/bus/pci/devices/0000\:0c\:00.0/remove
echo 1 | sudo tee /sys/bus/pci/devices/0000\:0c\:00.1/remove
sudo ls /sys/bus/pci/devices/ | grep 0c:00.
  # nothing...

Then let it rescan for PCI devices and check if the devices are there again and enabled. Also check which kernel driver is in use and what nvidia-smi is telling.

echo 1 | sudo tee /sys/bus/pci/rescan
sudo ls /sys/bus/pci/devices/ | grep 0c:00.
sudo cat /sys/bus/pci/devices/0000\:0c\:00.?/enable
  # 1
  # 1
sudo lspci -nnk | egrep -A3 "VGA|Display|3D"
  # 0b:00.0 VGA compatible controller : NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
  # Subsystem: Gigabyte Technology Co., Ltd Device [1458:36ac]
  # Kernel driver in use: nvidia
  # Kernel modules: nouveau, nvidia_drm, nvidia
  # --
  # 0c:00.0 VGA compatible controller : NVIDIA Corporation GM204GL [Quadro M4000] [10de:13f1] (rev a1)
  # Subsystem: Hewlett-Packard Company Device [103c:1153]
  # Kernel driver in use: nvidia      # <-- here!
  # Kernel modules: nouveau, nvidia_drm, nvidia
sudo nvidia-smi 
  # Tue Sep 28 18:26:16 2021       
  # +-----------------------------------------------------------------------------+
  # | NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
  # |-------------------------------+----------------------+----------------------+
  # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  # |                               |                      |               MIG M. |
  # |===============================+======================+======================|
  # |   0  NVIDIA GeForce ...  Off  | 00000000:0B:00.0  On |                  N/A |
  # |  0%   47C    P8    19W / 160W |    479MiB /  4040MiB |      0%      Default |
  # |                               |                      |                  N/A |
  # +-------------------------------+----------------------+----------------------+
  # |   1  Quadro M4000        Off  | 00000000:0C:00.0 Off |                  N/A |
  # | 45%   37C    P0    42W / 120W |      0MiB /  8127MiB |      2%      Default |
  # |                               |                      |                  N/A |
  # +-------------------------------+----------------------+----------------------+
  # ...

Funny enough, the Quadro M4000 consumes about 42 Watts under absolutely no load. I guess this is due to a driver problem... **However**, if the graphical nvidia-settings program is loaded, the power demand **drops** to about **12 Watts**.

# Terminal A
watch -d -n 1 sudo nvidia-smi
# Terminal B
nvidia-settings

Watch nvidia-smi and listen to the fan noise when the magic happens...

watch -d -n 1 sudo nvidia-smi
  # ...
  # +-------------------------------+----------------------+----------------------+
  # |   1  Quadro M4000        Off  | 00000000:0C:00.0 Off |                  N/A |
  # | 46%   38C    P0    10W / 120W |      0MiB /  8127MiB |      0%      Default |
  # |                               |                      |                  N/A |
  # +-------------------------------+----------------------+----------------------+
  # ...

Best of all -- nvidia-settings does not even list my Quadro card...

I have tried countless guides and always ended up with the same result, so it's time I ask those who know what's going on. I have all virtualisation IOMMU etc enabled in my BIOS. I am running the VM without issues when not passing through the GPU, no errors, great performance etc. When I do try to passthrough the GPU (both GPU and same IOMMU group devices) and press start the VM, nothing happens. Literally nothing, as if I did not press start the VM. When I try to remove the PCI passthrough, virt-manager crashes. If I forcibly close it then start it again, I can't connect to the server (qemu://system) until I fully restart my PC (logout doesn't help). What am I missing? I have a second GPU of course, which is what I've connected my monitor to. I have Nvidia drivers installed. Both GPUs are recognised and functional (tested by switching the monitor). The GPUs are 3060 and 560. Using Debian 10 and everything is updated to the latest version.

I'm using Debian 10.8 with Linux Kernel 4.19.0-14. I'm trying to use IOMMU to passthrough an entire USB controller (PCI Express card), but I'm having problems. The kernel is loading the ohci_pci and ehci_pci drivers and I can't seem to prevent it or unbind them. I've tried to remove them (echo 1 > /sys/bus/pci/devices/...) but it freezes the system. (the kernel shows a kernel BUG with a stacktrace, etc) Therefore, I'm trying to stop the kernel from binding the ohci and ehci drivers so that VFIO can use them without unbinding them first. I've tried several things: * Adding pci-stub.ids vfio-pci.ids to the GRUB kernel command line. * Aliasing the PCI devices to dummy drivers (/etc/modprobe.d/vfio-pci.conf) * Blacklisting ohci_pci and ehci_pci (/etc/modprobe.d/blacklist.conf) * Unbind the drivers using /sys/bus/pci/devices/.../unbind (but this freezes system) I think the kernel is using built-in ohci/ehci drivers (instead of modules) so perhaps it's loading these drivers before pci-stub or vfio can prevent it?

I inserted a mPCIe card - a USB controller - into one of my computers mPCIe slots. Attached USB devices are shown by lsusb and everything is working fine. Now I'm running KVM/qemu and I want to pass-through not just single USB devices but the USB controller as PCIe device to a virtual machine - as I already did with on-board NICs (also PCIe). So I figured the USB controller is announcing itself as a PCI connected device when calling lspci - but it isn't:

# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Mullins IOMMU
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Host Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Kabini/Mullins PSP-Platform Security Processor
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 11)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 39)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 42)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.7 SD Host controller: Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 5
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

The computer is a PC engines APU2 embedded computer, AMD based (https://www.pcengines.ch/apu2.htm) - running Debian Linux. lsusb shows the connected USB devices being connected to the USB controller in question. Question: How can I find out which PCI(e) ID/device my USB controller is connected to in order to know which one to pass-through so it pops up in my virtual machine as native PCI(e) device? I figured potentially by the USB bus ID of what lsusb spits out - however that's as far as I got.

I'm trying to passthrough my nvidia GTX 1060 from ubuntu 20.10 host to xen with ubuntu 20.10 guest os. Below you can see how is configured everything (default vga display adapter / nvidia 1060 driver and the monitor). The problem that I have is that even if the 1060 is detected inside the vm,the monitor attached to it is not detected. root@zioma-HVM-domU:/home/zioma# lspci -k 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) Subsystem: Red Hat, Inc. Qemu virtual machine 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] Subsystem: Red Hat, Inc. Qemu virtual machine 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] Subsystem: Red Hat, Inc. Qemu virtual machine Kernel driver in use: ata_piix Kernel modules: pata_acpi 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) Subsystem: Red Hat, Inc. QEMU Virtual Machine Kernel driver in use: uhci_hcd 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) Subsystem: Red Hat, Inc. Qemu virtual machine Kernel modules: i2c_piix4 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) Subsystem: XenSource, Inc. Xen Platform Device Kernel driver in use: xen-platform-pci 00:04.0 VGA compatible controller: Device 1234:1111 (rev 02) Subsystem: Red Hat, Inc. Device 1100 Kernel driver in use: bochs-drm Kernel modules: bochs_drm 00:06.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) Subsystem: ZOTAC International (MCO) Ltd. GP106 [GeForce GTX 1060 3GB] Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 00:07.0 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) Subsystem: ZOTAC International (MCO) Ltd. GP106 High Definition Audio Controller Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel root@zioma-HVM-domU:/home/zioma# ls -l /sys/class/drm/ totale 0 lrwxrwxrwx 1 root root 0 nov 1 23:14 card0 -> ../../devices/pci0000:00/0000:00:04.0/drm/card0 lrwxrwxrwx 1 root root 0 nov 1 23:14 card0-Virtual-1 -> ../../devices/pci0000:00/0000:00:04.0/drm/card0/card0-Virtual-1 lrwxrwxrwx 1 root root 0 nov 1 23:14 ttm -> ../../devices/virtual/drm/ttm -r--r--r-- 1 root root 4096 nov 1 23:14 version root@zioma-HVM-domU:/home/zioma# xrandr Screen 0: minimum 320 x 200, current 1024 x 768, maximum 8192 x 8192 Virtual-1 connected primary 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm 1024x768 60.00*+ 1920x1080 60.00 1600x1200 60.00 1680x1050 59.95 59.88 1400x1050 59.98 59.95 1600x900 60.00 1280x1024 60.02 1440x900 59.89 59.90 1280x960 60.00 1366x768 59.79 60.00 1360x768 60.02 1280x800 59.81 59.91 1280x768 59.87 59.99 1280x720 60.00 800x600 60.32 56.25 848x480 60.00 640x480 59.94 I can't turn off the screen 0 because it is the only screen that I have,the one attached to the xen default graphic adapter. And I can't turn on the external monitor attached with the vga cable because the screen is not detected inside the vm. So,I've thought,what happens if I blacklist nouveau and bochs-drm ? I read here how to blacklist bochs_drm : https://unix.stackexchange.com/questions/346090/disable-framebuffer-in-qemu-guests and here how to blacklist the nouveau driver : https://linuxconfig.org/how-to-disable-blacklist-nouveau-nvidia-driver-on-ubuntu-20-04-focal-fossa-linux my goal is to set the 1060 with the nvidia driver as the first and only graphic adapter inside the vm. This is what I did : nano /etc/modprobe.d/blacklist-nouveau.conf blacklist nouveau options nouveau modeset=0 nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash systemd.show_status=1 bochs_drm.fbdev=off" update-initramfs -u update-grub and then I rebooted and checked again with lspci -k,but the nouveau driver and the bochs_drm module haven't been blacklisted. nothing is changed from before. what's my mistake ?

I am trying to passthrough a USB card to a guest, but I can't seem to make the VFIO driver stick, instead it seems that the host loads and takes over the card. I have a GPU successfully passed through, so I am sure that vfio/IOMMU are working. card: Sonnet Allegro USB 3.1, Two-Port USB-C 10Gb PCIe Card (USB3C-2PM-E) motherboard: ASRock x399 Taichi CPU: Threadripper 1950x I tried following the same procedure that I used to passthrough a GPU, which I mostly learned here : First, get the ID of the card: ~ lspci | grep USB 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43ba (rev 02) 0a:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller 0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller 45:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller Next. get the vendor, product ID, and an alias, then modify system files accordingly in hopes that it will prevent the card from being grabbed by the host: ~ lspci -nn | grep 0a:00 0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242] cat /sys/bus/pci/devices/0000:0a:00.0/modalias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 ~ sudo nano /etc/modprobe.d/local.conf #paste at end: alias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 vfio-pci options vfio-pci ids=1b21:1242 then: ~ sudo update-initramfs -u and reboot and check if the card is using the vfio driver: ~ lspci -kn | grep -A 2 0a 0a:00.0 0c03: 1b21:1242 Subsystem: 16b8:7230 Kernel driver in use: xhci_hcd I also tried making an override script as per the same blog mentioned above: sudo nano /sbin/vfio-pci-override-usb.sh #!/bin/sh DEVS="0000:0a:00.0" for DEV in $DEVS; do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override done modprobe -i vfio-pci ##### sudo chmod 755 /sbin/vfio-pci-override-usb.sh sudo nano /etc/modprobe.d/local.conf ###commented out previous entry and added: install vfio-pci /sbin/vfio-pci-override-usb.sh sudo update-initramfs -u reboot but still no good: ~ lspci -kn | grep -A 2 0a:00 0a:00.0 0c03: 1b21:1242 Subsystem: 16b8:7230 Kernel driver in use: xhci_hcd also: When I look for the driver, in lspci (to see what else is using it), it brings up a bunch of devices but not 0a:00. ~ lspci -kn | grep -A 2 xhci Kernel driver in use: xhci_hcd 01:00.1 0106: 1022:43b6 (rev 02) Subsystem: 1b21:1062 -- Kernel driver in use: xhci_hcd 0b:00.0 1300: 1022:145a Subsystem: 1022:145a -- Kernel driver in use: xhci_hcd 0c:00.0 1300: 1022:1455 Subsystem: 1022:1455 -- Kernel driver in use: xhci_hcd 46:00.0 1300: 1022:1455 Subsystem: 1022:1455 Next I tried Tried blacklisting the driver: sudo nano /etc/default/grub # original: GRUB_CMDLINE_LINUX_DEFAULT=”amd_iommu=on” # to: GRUB_CMDLINE_LINUX_DEFAULT=”modprobe.blacklist=xhci_hcd amd_iommu=on” ##then: sudo update-grub ##Uncomment in local.conf: # options vfio-pci ids=1b21:1242 sudo update-initramfs -u still no: lspci -kn | grep -A 2 0a:00 0a:00.0 0c03: 1b21:1242 Subsystem: 16b8:7230 Kernel driver in use: xhci_hcd What else might I try? Am I correct that the kernel driver in use for that device must be vfio-pci?

On a Supermicro A2SDi-8C+-HLN4F Atom C3758 system I see that VT-d support is enabled in the BIOS but I can't find any IOMMU groups under Linux (Fedora 31):

$ ls /sys/kernel/iommu_groups
$

The kernel logs seem to indicate that VT-d is detected:

# journalctl -k | grep 'DMAR\|iommu' | sed 's/^.*kernel: //'
ACPI: DMAR 0x000000007E1629C0 000070 (v01 INTEL  BDW      00000001 INTL 00000001)
DMAR: Host address width 39
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
DMAR: RMRR base: 0x0000007f0fa000 end: 0x0000007f10afff
DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
DMAR-IR: HPET id 0 under DRHD base 0xfed90000
DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
DMAR-IR: Enabled IRQ remapping in x2apic mode
iommu: Default domain type: Translated

Do I need to configure something such that I also get IOMMU groups? Background: I'm looking into this because I can't bind an ethernet device to vfio-pci for dpdk:

# /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci
# /usr/share/dpdk/usertools/dpdk-devbind.py --status        

Network devices using kernel driver
===================================
0000:05:00.0 'Ethernet Connection X553 1GbE 15e4' if=eno1 drv=ixgbe unused=vfio-pci *Active*
0000:05:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno2 drv=ixgbe unused=vfio-pci 
0000:07:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno4 drv=ixgbe unused=vfio-pci 

Other Network devices
=====================
0000:07:00.0 'Ethernet Connection X553 1GbE 15e4' unused=ixgbe,vfio-pci
# dmesg | tail -n 3
[  +0.169963] ixgbe 0000:07:00.0: complete
[  +0.000561] vfio-pci: probe of 0000:07:00.0 failed with error -22
[  +0.000040] vfio-pci: probe of 0000:07:00.0 failed with error -22

I believe we are running into a possible bug with the GTX 1080 (driver) and PCI Passthrough. My host is an Ubuntu 14.04 system. My guest is an Ubuntu 14.04/16.04 system (both do the same thing). I can see device inside the guest VM: $ lspci -vnn | grep VGA 00:05.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1) I was able to successfully install the driver (370.2, latest driver). It installs, but is not recognized by nvidia-smi: $ nvidia-smi Unable to determine the device handle for GPU 0000:00:05.0: Unknown Error Looking in dmesg I see the following error message [29.535583] nvidia 0000:00:05.0: irq 45 for MSI/MSI-X [29.577727] NVRM: RmInitAdapter failed! (0x23:0x56:458) [29.577807] NVRM: rm_init_adapter failed for device bearing minor number 0 I can switch out the GTX 1080 for a different card (M4000, do passthrough and install drivers on guest) and it works. I am going to try tomorrow with another Geforce card. Another person on the NVidia forums had the SAME exact issue as me (but no answer). Is there anyway to debug this further?

I followed this quick guide to enable PCI passthrough using KVM on Fedora, the main steps are: - Enable IOMMU and Virtualization on the motherboard - Prevent my device from being grabbed by a module during statup So far so good. dmesg| grep -i iommu: [ 0.000000] DMAR: IOMMU enabled ... [ 1.165288] vboxpci: IOMMU found (even the vbox module notices it!) lspci -nnk: 00:02.0 Display controller : Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8534] Kernel driver in use: vfio-pci Kernel modules: i915 I did notice that vfio-pci is not present as an available module for my iGPU, but it still says it's being used, so I think this is good as well. I then proceeded to attach the iGPU to my VBox VM with the following: VBoxManage modifyvm "Windows 10" --pciattach 00:02.0 It didn't complain so I guess it's fine. I finally booted up the virtual machine but it still shows up the VirtualBox Graphics Adapter instead of the integrated graphics on the device manager (and there are no signs of the iGPU anywhere as well): Dmesg even says that an IOMMU domain is being created when I fire up the VM (not sure what that means, though, but I guess it means it's kinda working): [ 80.632320] vboxpci: created IOMMU domain ffff8816e45e9fe8 [ 159.674621] vboxpci: freeing IOMMU domain ffff8816e45e9fe8 So what did I do wrong? Some clarification: - I do have a discrete graphics card (NVidia GTX 1050) but I use it on the host, so I don't want to passthrough it - I enabled the iGPU on the motherboard so it doesn't get disabled when it detects a discrete graphics card - I just need to passthrough the iGPU for some light gaming that's only possible on windows with HW acceleration support, as software acceleration won't work (games refuse to even start) Since almost all guides out there are for passing through a secondary graphics card, they tell you to blacklist the nouveau module, but since I am doing it the other way around, should I also blacklist the i915 module?