VFIO single GPU passthrough - AMD-Vi: Completion-Wait loop timed out
1
vote
0
answers
38
views
I am trying to pass gpu to vm with libvirt following guides:
- https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Setting_up_an_OVMF-based_guest_virtual_machine
- https://github.com/joeknock90/Single-GPU-Passthrough
After domain is started the vfio driver is bound to gpu, but the screen is black:
@ubuntu:~$ lspci -nnks 03:00.0
03:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Picasso/Raven 2 [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15d8] (rev c2)
Subsystem: Lenovo ThinkPad E595 [17aa:5124]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu[ 329.365209] Console: switching to colour dummy device 80x25
[ 331.414644] VFIO - User Level meta-driver version: 0.3
[ 331.518909] amdgpu 0000:03:00.0: amdgpu: amdgpu: finishing device.
[ 331.537364] [drm] psp gfx command UNLOAD_TA(0x2) failed and response status is (0x117)
[ 331.574817] [drm] amdgpu: ttm finalized
[ 331.575932] ------------[ cut here ]------------
[ 331.575939] sysfs group 'power' not found for kobject 'amdgpu_bl1'
[ 331.575961] WARNING: CPU: 4 PID: 86 at fs/sysfs/group.c:282 sysfs_remove_group+0x85/0x90
[ 331.575979] Modules linked in: vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd ccm xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables bridge stp llc binfmt_misc intel_rapl_msr intel_rapl_common snd_sof_amd_acp63 amdgpu snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof edac_mce_amd snd_sof_utils kvm_amd snd_ctl_led snd_soc_core snd_hda_codec_conexant nls_iso8859_1 snd_hda_codec_generic snd_hda_codec_hdmi snd_compress ac97_bus amdxcp kvm snd_pcm_dmaengine drm_exec rtw88_8822be snd_pci_ps rtw88_8822b snd_hda_intel gpu_sched uvcvideo rtw88_pci btusb drm_buddy snd_rpl_pci_acp6x btrtl snd_intel_dspcfg rtw88_core irqbypass snd_intel_sdw_acpi drm_suballoc_helper btintel videobuf2_vmalloc snd_acp_pci uvc btbcm drm_ttm_helper videobuf2_memops btmtk snd_hda_codec ttm videobuf2_v4l2 rapl snd_acp_legacy_common bluetooth drm_display_helper think_lmi
[ 331.576257] videodev snd_pci_acp6x snd_pci_acp5x wmi_bmof mac80211 firmware_attributes_class snd_hda_core snd_rn_pci_acp3x videobuf2_common cec snd_hwdep snd_acp_config ecdh_generic rc_core mc snd_soc_acpi snd_pcm ecc k10temp i2c_piix4 i2c_algo_bit libarc4 snd_timer ccp snd_pci_acp3x i2c_scmi input_leds joydev cfg80211 serio_raw mac_hid sch_fq_codel dm_multipath efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel thinkpad_acpi nvme sha256_ssse3 sha1_ssse3 ahci nvram psmouse libahci snd ucsi_acpi nvme_core xhci_pci typec_ucsi xhci_pci_renesas soundcore nvme_auth typec video ledtrig_audio platform_profile wmi aesni_intel crypto_simd cryptd
[ 331.576437] CPU: 4 PID: 86 Comm: kworker/4:1 Not tainted 6.8.0-71-generic #71-Ubuntu
[ 331.576442] Hardware name: LENOVO 20NE000JPB/20NE000JPB, BIOS R11ET36W (1.16 ) 03/30/2020
[ 331.576446] Workqueue: events drm_connector_free_work_fn
[ 331.576453] RIP: 0010:sysfs_remove_group+0x85/0x90
[ 331.576458] Code: c0 31 d2 31 f6 31 ff e9 59 79 c8 00 48 89 df e8 11 9b ff ff eb c1 49 8b 55 00 49 8b 34 24 48 c7 c7 28 57 09 bb e8 4b 35 b3 ff 0b eb cb 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90
[ 331.576462] RSP: 0018:ffffa8b78041fd08 EFLAGS: 00010246
[ 331.576467] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 331.576470] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 331.576473] RBP: ffffa8b78041fd20 R08: 0000000000000000 R09: 0000000000000000
[ 331.576476] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffbaafe2a0
[ 331.576478] R13: ffff9c5c4b961490 R14: ffff9c5c6c380268 R15: ffff9c5c48044c00
[ 331.576481] FS: 0000000000000000(0000) GS:ffff9c5cf8800000(0000) knlGS:0000000000000000
[ 331.576485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 331.576488] CR2: 00007fe81402d838 CR3: 0000000123732000 CR4: 00000000003506f0
[ 331.576491] Call Trace:
[ 331.576494]
[ 331.576500] ? show_regs+0x6d/0x80
[ 331.576507] ? __warn+0x89/0x160
[ 331.576513] ? sysfs_remove_group+0x85/0x90
[ 331.576518] ? report_bug+0x17e/0x1b0
[ 331.576525] ? handle_bug+0x6e/0xb0
[ 331.576532] ? exc_invalid_op+0x18/0x80
[ 331.576538] ? asm_exc_invalid_op+0x1b/0x20
[ 331.576547] ? sysfs_remove_group+0x85/0x90
[ 331.576552] ? sysfs_remove_group+0x85/0x90
[ 331.576557] dpm_sysfs_remove+0x60/0x70
[ 331.576563] device_del+0xa2/0x3e0
[ 331.576568] ? srso_return_thunk+0x5/0x5f
[ 331.576574] ? __radix_tree_delete+0x9e/0x150
[ 331.576581] device_unregister+0x17/0x60
[ 331.576586] backlight_device_unregister.part.0+0x9d/0xb0
[ 331.576595] backlight_device_unregister+0x13/0x30
[ 331.576601] amdgpu_dm_connector_destroy+0xeb/0x140 [amdgpu]
[ 331.577205] ? drm_mode_object_unregister+0x6a/0xa0
[ 331.577213] drm_connector_free_work_fn+0x77/0xa0
[ 331.577220] process_one_work+0x184/0x3a0
[ 331.577227] worker_thread+0x306/0x440
[ 331.577232] ? srso_return_thunk+0x5/0x5f
[ 331.577238] ? _raw_spin_lock_irqsave+0xe/0x20
[ 331.577244] ? __pfx_worker_thread+0x10/0x10
[ 331.577248] kthread+0xf2/0x120
[ 331.577255] ? __pfx_kthread+0x10/0x10
[ 331.577260] ret_from_fork+0x47/0x70
[ 331.577266] ? __pfx_kthread+0x10/0x10
[ 331.577271] ret_from_fork_asm+0x1b/0x30
[ 331.577282]
[ 331.577284] ---[ end trace 0000000000000000 ]---
[ 331.577838] vfio-pci 0000:03:00.0: vgaarb: deactivate vga console
[ 331.577846] vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 333.700319] kauditd_printk_skb: 100 callbacks suppressed
[ 333.700328] audit: type=1400 audit(1754269730.212:112): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1422 comm="apparmor_parser"
[ 333.700338] audit: type=1400 audit(1754269730.213:113): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1422 comm="apparmor_parser"
[ 333.799726] audit: type=1400 audit(1754269730.312:114): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1425 comm="apparmor_parser"
[ 333.810460] audit: type=1400 audit(1754269730.323:115): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1425 comm="apparmor_parser"
[ 333.921948] audit: type=1400 audit(1754269730.435:116): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1429 comm="apparmor_parser"
[ 333.933480] audit: type=1400 audit(1754269730.446:117): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1429 comm="apparmor_parser"
[ 334.041433] audit: type=1400 audit(1754269730.554:118): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1433 comm="apparmor_parser"
[ 334.041829] audit: type=1400 audit(1754269730.554:119): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1433 comm="apparmor_parser"
[ 334.060569] virbr0: port 1(vnet0) entered blocking state
[ 334.060586] virbr0: port 1(vnet0) entered disabled state
[ 334.060612] vnet0: entered allmulticast mode
[ 334.060789] vnet0: entered promiscuous mode
[ 334.061279] virbr0: port 1(vnet0) entered blocking state
[ 334.061291] virbr0: port 1(vnet0) entered listening state
[ 334.154377] audit: type=1400 audit(1754269730.667:120): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1441 comm="apparmor_parser"
[ 334.160364] audit: type=1400 audit(1754269730.673:121): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1441 comm="apparmor_parser"
[ 334.455362] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable
[ 335.521105] AMD-Vi: Completion-Wait loop timed out
[ 335.679874] AMD-Vi: Completion-Wait loop timed out
[ 336.119893] virbr0: port 1(vnet0) entered learning state
[ 336.522658] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7bf0]
[ 336.804874] AMD-Vi: Completion-Wait loop timed out
[ 337.015900] AMD-Vi: Completion-Wait loop timed out
[ 337.335904] AMD-Vi: Completion-Wait loop timed out
[ 337.524658] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c20]
[ 337.399944] AMD-Vi: Completion-Wait loop timed out
[ 337.654878] AMD-Vi: Completion-Wait loop timed out
[ 337.834757] AMD-Vi: Completion-Wait loop timed out
[ 338.168892] virbr0: port 1(vnet0) entered forwarding state
[ 338.168905] virbr0: topology change detected, propagating
[ 338.168889] AMD-Vi: Completion-Wait loop timed out
[ 338.363888] AMD-Vi: Completion-Wait loop timed out
[ 338.526667] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c50]
[ 338.551950] AMD-Vi: Completion-Wait loop timed out
[ 338.950868] AMD-Vi: Completion-Wait loop timed out
[ 339.127921] AMD-Vi: Completion-Wait loop timed out
[ 339.137211] AMD-Vi: Completion-Wait loop timed out
[ 339.275191] AMD-Vi: Completion-Wait loop timed out
[ 339.528679] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c80]
[ 339.447945] AMD-Vi: Completion-Wait loop timed out
[ 339.590874] AMD-Vi: Completion-Wait loop timed out
[ 339.831958] AMD-Vi: Completion-Wait loop timed out
[ 339.959934] AMD-Vi: Completion-Wait loop timed out
[ 340.215929] AMD-Vi: Completion-Wait loop timed out
[ 340.530700] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7cb0]
[ 340.599897] AMD-Vi: Completion-Wait loop timed out
[ 340.727929] AMD-Vi: Completion-Wait loop timed out
[ 341.033943] AMD-Vi: Completion-Wait loop timed out
[ 341.119231] AMD-Vi: Completion-Wait loop timed out
[ 341.257066] AMD-Vi: Completion-Wait loop timed out
[ 341.532721] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7ce0]
[ 342.534749] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d10]
[ 343.536781] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d40]
[ 344.538818] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d70]
[ 345.540857] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7da0]
[ 346.542901] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7dd0]
[ 347.544950] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e20]
[ 348.547000] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e50]
[ 349.549052] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e80]sudo dmesg | grep iommu
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro iommu=pt amd_iommu=on
[ 0.027226] Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro iommu=pt amd_iommu=on
[ 0.360022] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.402401] pci 0000:00:01.0: Adding to iommu group 0
[ 0.402426] pci 0000:00:01.1: Adding to iommu group 1
[ 0.402458] pci 0000:00:01.6: Adding to iommu group 2
[ 0.402500] pci 0000:00:08.0: Adding to iommu group 3
[ 0.402525] pci 0000:00:08.1: Adding to iommu group 4
[ 0.402548] pci 0000:00:08.2: Adding to iommu group 3
[ 0.402589] pci 0000:00:14.0: Adding to iommu group 5
[ 0.402611] pci 0000:00:14.3: Adding to iommu group 5
[ 0.402703] pci 0000:00:18.0: Adding to iommu group 6
[ 0.402725] pci 0000:00:18.1: Adding to iommu group 6
[ 0.402748] pci 0000:00:18.2: Adding to iommu group 6
[ 0.402774] pci 0000:00:18.3: Adding to iommu group 6
[ 0.402797] pci 0000:00:18.4: Adding to iommu group 6
[ 0.402820] pci 0000:00:18.5: Adding to iommu group 6
[ 0.402843] pci 0000:00:18.6: Adding to iommu group 6
[ 0.402867] pci 0000:00:18.7: Adding to iommu group 6
[ 0.402903] pci 0000:01:00.0: Adding to iommu group 7
[ 0.402929] pci 0000:02:00.0: Adding to iommu group 8
[ 0.402963] pci 0000:03:00.0: Adding to iommu group 9
[ 0.403045] pci 0000:03:00.1: Adding to iommu group 10
[ 0.403079] pci 0000:03:00.2: Adding to iommu group 10
[ 0.403113] pci 0000:03:00.3: Adding to iommu group 10
[ 0.403148] pci 0000:03:00.4: Adding to iommu group 10
[ 0.403182] pci 0000:03:00.5: Adding to iommu group 10
[ 0.403217] pci 0000:03:00.6: Adding to iommu group 10
[ 0.403230] pci 0000:04:00.0: Adding to iommu group 3
[ 0.404218] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).hashibane@ubuntu:~$ ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices
0000:03:00.0GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt amd_iommu=on"xubuntu
400eec47-fd10-4b05-abd3-8ee6552e8a44
4194304
4194304
2
/machine
hvm
destroy
restart
destroy
/usr/bin/qemu-system-x86_64
/dev/urandom
libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44
libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44
+64055:+994
+64055:+994
Asked by Hashibane
(11 rep)
Aug 4, 2025, 01:56 AM