I am trying to pass gpu to vm with libvirt following guides: - https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Setting_up_an_OVMF-based_guest_virtual_machine - https://github.com/joeknock90/Single-GPU-Passthrough After domain is started the vfio driver is bound to gpu, but the screen is black:

@ubuntu:~$ lspci -nnks 03:00.0
03:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Picasso/Raven 2 [Radeon Vega Series / Radeon Vega Mobile Series] [1002:15d8] (rev c2)
        Subsystem: Lenovo ThinkPad E595 [17aa:5124]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

dmesg gives

[  329.365209] Console: switching to colour dummy device 80x25
[  331.414644] VFIO - User Level meta-driver version: 0.3
[  331.518909] amdgpu 0000:03:00.0: amdgpu: amdgpu: finishing device.
[  331.537364] [drm] psp gfx command UNLOAD_TA(0x2) failed and response status is (0x117)
[  331.574817] [drm] amdgpu: ttm finalized
[  331.575932] ------------[ cut here ]------------
[  331.575939] sysfs group 'power' not found for kobject 'amdgpu_bl1'
[  331.575961] WARNING: CPU: 4 PID: 86 at fs/sysfs/group.c:282 sysfs_remove_group+0x85/0x90
[  331.575979] Modules linked in: vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd ccm xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables bridge stp llc binfmt_misc intel_rapl_msr intel_rapl_common snd_sof_amd_acp63 amdgpu snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof edac_mce_amd snd_sof_utils kvm_amd snd_ctl_led snd_soc_core snd_hda_codec_conexant nls_iso8859_1 snd_hda_codec_generic snd_hda_codec_hdmi snd_compress ac97_bus amdxcp kvm snd_pcm_dmaengine drm_exec rtw88_8822be snd_pci_ps rtw88_8822b snd_hda_intel gpu_sched uvcvideo rtw88_pci btusb drm_buddy snd_rpl_pci_acp6x btrtl snd_intel_dspcfg rtw88_core irqbypass snd_intel_sdw_acpi drm_suballoc_helper btintel videobuf2_vmalloc snd_acp_pci uvc btbcm drm_ttm_helper videobuf2_memops btmtk snd_hda_codec ttm videobuf2_v4l2 rapl snd_acp_legacy_common bluetooth drm_display_helper think_lmi
[  331.576257]  videodev snd_pci_acp6x snd_pci_acp5x wmi_bmof mac80211 firmware_attributes_class snd_hda_core snd_rn_pci_acp3x videobuf2_common cec snd_hwdep snd_acp_config ecdh_generic rc_core mc snd_soc_acpi snd_pcm ecc k10temp i2c_piix4 i2c_algo_bit libarc4 snd_timer ccp snd_pci_acp3x i2c_scmi input_leds joydev cfg80211 serio_raw mac_hid sch_fq_codel dm_multipath efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel thinkpad_acpi nvme sha256_ssse3 sha1_ssse3 ahci nvram psmouse libahci snd ucsi_acpi nvme_core xhci_pci typec_ucsi xhci_pci_renesas soundcore nvme_auth typec video ledtrig_audio platform_profile wmi aesni_intel crypto_simd cryptd
[  331.576437] CPU: 4 PID: 86 Comm: kworker/4:1 Not tainted 6.8.0-71-generic #71-Ubuntu
[  331.576442] Hardware name: LENOVO 20NE000JPB/20NE000JPB, BIOS R11ET36W (1.16 ) 03/30/2020
[  331.576446] Workqueue: events drm_connector_free_work_fn
[  331.576453] RIP: 0010:sysfs_remove_group+0x85/0x90
[  331.576458] Code: c0 31 d2 31 f6 31 ff e9 59 79 c8 00 48 89 df e8 11 9b ff ff eb c1 49 8b 55 00 49 8b 34 24 48 c7 c7 28 57 09 bb e8 4b 35 b3 ff  0b eb cb 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90
[  331.576462] RSP: 0018:ffffa8b78041fd08 EFLAGS: 00010246
[  331.576467] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[  331.576470] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  331.576473] RBP: ffffa8b78041fd20 R08: 0000000000000000 R09: 0000000000000000
[  331.576476] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffbaafe2a0
[  331.576478] R13: ffff9c5c4b961490 R14: ffff9c5c6c380268 R15: ffff9c5c48044c00
[  331.576481] FS:  0000000000000000(0000) GS:ffff9c5cf8800000(0000) knlGS:0000000000000000
[  331.576485] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  331.576488] CR2: 00007fe81402d838 CR3: 0000000123732000 CR4: 00000000003506f0
[  331.576491] Call Trace:
[  331.576494]  
[  331.576500]  ? show_regs+0x6d/0x80
[  331.576507]  ? __warn+0x89/0x160
[  331.576513]  ? sysfs_remove_group+0x85/0x90
[  331.576518]  ? report_bug+0x17e/0x1b0
[  331.576525]  ? handle_bug+0x6e/0xb0
[  331.576532]  ? exc_invalid_op+0x18/0x80
[  331.576538]  ? asm_exc_invalid_op+0x1b/0x20
[  331.576547]  ? sysfs_remove_group+0x85/0x90
[  331.576552]  ? sysfs_remove_group+0x85/0x90
[  331.576557]  dpm_sysfs_remove+0x60/0x70
[  331.576563]  device_del+0xa2/0x3e0
[  331.576568]  ? srso_return_thunk+0x5/0x5f
[  331.576574]  ? __radix_tree_delete+0x9e/0x150
[  331.576581]  device_unregister+0x17/0x60
[  331.576586]  backlight_device_unregister.part.0+0x9d/0xb0
[  331.576595]  backlight_device_unregister+0x13/0x30
[  331.576601]  amdgpu_dm_connector_destroy+0xeb/0x140 [amdgpu]
[  331.577205]  ? drm_mode_object_unregister+0x6a/0xa0
[  331.577213]  drm_connector_free_work_fn+0x77/0xa0
[  331.577220]  process_one_work+0x184/0x3a0
[  331.577227]  worker_thread+0x306/0x440
[  331.577232]  ? srso_return_thunk+0x5/0x5f
[  331.577238]  ? _raw_spin_lock_irqsave+0xe/0x20
[  331.577244]  ? __pfx_worker_thread+0x10/0x10
[  331.577248]  kthread+0xf2/0x120
[  331.577255]  ? __pfx_kthread+0x10/0x10
[  331.577260]  ret_from_fork+0x47/0x70
[  331.577266]  ? __pfx_kthread+0x10/0x10
[  331.577271]  ret_from_fork_asm+0x1b/0x30
[  331.577282]  
[  331.577284] ---[ end trace 0000000000000000 ]---
[  331.577838] vfio-pci 0000:03:00.0: vgaarb: deactivate vga console
[  331.577846] vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  333.700319] kauditd_printk_skb: 100 callbacks suppressed
[  333.700328] audit: type=1400 audit(1754269730.212:112): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1422 comm="apparmor_parser"
[  333.700338] audit: type=1400 audit(1754269730.213:113): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1422 comm="apparmor_parser"
[  333.799726] audit: type=1400 audit(1754269730.312:114): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1425 comm="apparmor_parser"
[  333.810460] audit: type=1400 audit(1754269730.323:115): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1425 comm="apparmor_parser"
[  333.921948] audit: type=1400 audit(1754269730.435:116): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1429 comm="apparmor_parser"
[  333.933480] audit: type=1400 audit(1754269730.446:117): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1429 comm="apparmor_parser"
[  334.041433] audit: type=1400 audit(1754269730.554:118): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1433 comm="apparmor_parser"
[  334.041829] audit: type=1400 audit(1754269730.554:119): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1433 comm="apparmor_parser"
[  334.060569] virbr0: port 1(vnet0) entered blocking state
[  334.060586] virbr0: port 1(vnet0) entered disabled state
[  334.060612] vnet0: entered allmulticast mode
[  334.060789] vnet0: entered promiscuous mode
[  334.061279] virbr0: port 1(vnet0) entered blocking state
[  334.061291] virbr0: port 1(vnet0) entered listening state
[  334.154377] audit: type=1400 audit(1754269730.667:120): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44" pid=1441 comm="apparmor_parser"
[  334.160364] audit: type=1400 audit(1754269730.673:121): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44//passt" pid=1441 comm="apparmor_parser"
[  334.455362] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable
[  335.521105] AMD-Vi: Completion-Wait loop timed out
[  335.679874] AMD-Vi: Completion-Wait loop timed out
[  336.119893] virbr0: port 1(vnet0) entered learning state
[  336.522658] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7bf0]
[  336.804874] AMD-Vi: Completion-Wait loop timed out
[  337.015900] AMD-Vi: Completion-Wait loop timed out
[  337.335904] AMD-Vi: Completion-Wait loop timed out
[  337.524658] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c20]
[  337.399944] AMD-Vi: Completion-Wait loop timed out
[  337.654878] AMD-Vi: Completion-Wait loop timed out
[  337.834757] AMD-Vi: Completion-Wait loop timed out
[  338.168892] virbr0: port 1(vnet0) entered forwarding state
[  338.168905] virbr0: topology change detected, propagating
[  338.168889] AMD-Vi: Completion-Wait loop timed out
[  338.363888] AMD-Vi: Completion-Wait loop timed out
[  338.526667] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c50]
[  338.551950] AMD-Vi: Completion-Wait loop timed out
[  338.950868] AMD-Vi: Completion-Wait loop timed out
[  339.127921] AMD-Vi: Completion-Wait loop timed out
[  339.137211] AMD-Vi: Completion-Wait loop timed out
[  339.275191] AMD-Vi: Completion-Wait loop timed out
[  339.528679] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7c80]
[  339.447945] AMD-Vi: Completion-Wait loop timed out
[  339.590874] AMD-Vi: Completion-Wait loop timed out
[  339.831958] AMD-Vi: Completion-Wait loop timed out
[  339.959934] AMD-Vi: Completion-Wait loop timed out
[  340.215929] AMD-Vi: Completion-Wait loop timed out
[  340.530700] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7cb0]
[  340.599897] AMD-Vi: Completion-Wait loop timed out
[  340.727929] AMD-Vi: Completion-Wait loop timed out
[  341.033943] AMD-Vi: Completion-Wait loop timed out
[  341.119231] AMD-Vi: Completion-Wait loop timed out
[  341.257066] AMD-Vi: Completion-Wait loop timed out
[  341.532721] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7ce0]
[  342.534749] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d10]
[  343.536781] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d40]
[  344.538818] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7d70]
[  345.540857] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7da0]
[  346.542901] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7dd0]
[  347.544950] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e20]
[  348.547000] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e50]
[  349.549052] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002b7e80]

Iommu was successfully loaded.

sudo dmesg | grep iommu
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro iommu=pt amd_iommu=on
[    0.027226] Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro iommu=pt amd_iommu=on
[    0.360022] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.402401] pci 0000:00:01.0: Adding to iommu group 0
[    0.402426] pci 0000:00:01.1: Adding to iommu group 1
[    0.402458] pci 0000:00:01.6: Adding to iommu group 2
[    0.402500] pci 0000:00:08.0: Adding to iommu group 3
[    0.402525] pci 0000:00:08.1: Adding to iommu group 4
[    0.402548] pci 0000:00:08.2: Adding to iommu group 3
[    0.402589] pci 0000:00:14.0: Adding to iommu group 5
[    0.402611] pci 0000:00:14.3: Adding to iommu group 5
[    0.402703] pci 0000:00:18.0: Adding to iommu group 6
[    0.402725] pci 0000:00:18.1: Adding to iommu group 6
[    0.402748] pci 0000:00:18.2: Adding to iommu group 6
[    0.402774] pci 0000:00:18.3: Adding to iommu group 6
[    0.402797] pci 0000:00:18.4: Adding to iommu group 6
[    0.402820] pci 0000:00:18.5: Adding to iommu group 6
[    0.402843] pci 0000:00:18.6: Adding to iommu group 6
[    0.402867] pci 0000:00:18.7: Adding to iommu group 6
[    0.402903] pci 0000:01:00.0: Adding to iommu group 7
[    0.402929] pci 0000:02:00.0: Adding to iommu group 8
[    0.402963] pci 0000:03:00.0: Adding to iommu group 9
[    0.403045] pci 0000:03:00.1: Adding to iommu group 10
[    0.403079] pci 0000:03:00.2: Adding to iommu group 10
[    0.403113] pci 0000:03:00.3: Adding to iommu group 10
[    0.403148] pci 0000:03:00.4: Adding to iommu group 10
[    0.403182] pci 0000:03:00.5: Adding to iommu group 10
[    0.403217] pci 0000:03:00.6: Adding to iommu group 10
[    0.403230] pci 0000:04:00.0: Adding to iommu group 3
[    0.404218] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

The GPU has its own iommu group (9).

hashibane@ubuntu:~$ ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices
0000:03:00.0

I also modified grub

GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt amd_iommu=on"

Domain config

xubuntu
  400eec47-fd10-4b05-abd3-8ee6552e8a44
  
    
      
    
  
  4194304
  4194304
  2
  
    /machine
  
  
    hvm
    
  
  
    
    
    
      
    
    
      
    
    
  
  
  
    
    
    
  
  destroy
  restart
  destroy
  
    
    
  
  
    /usr/bin/qemu-system-x86_64
    
      
      
      
      
      
      
    
    
      
      
      
      
      
      
    
    
      
      
      
      
      
    
    
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
      
      
    
    
      
      
    
    
      
      
    
    
      
      
    
    
      
      
      
      
      
      
    
    
      
      
        
      
      
    
    
      
      
      
    
    
      
      
      
      
    
    
      
    
    
      
    
    
      
      
    
    
    
      
      
      
    
    
      
      
        
      
      
      
    
    
      
    
    
      
      
    
    
      /dev/urandom
      
      
    
  
  
    libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44
    libvirt-400eec47-fd10-4b05-abd3-8ee6552e8a44
  
  
    +64055:+994
    +64055:+994

I am running ubuntu server - Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-71-generic x86_64) on a thinkpad e495 AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (integrated graphics). I am stuck as on whats the exact issue here. I have also searched through theese but haven't found much success: - https://lenovopress.lenovo.com/lp1470.pdf - https://clayfreeman.github.io/gpu-passthrough/

Motherboard info: sudo dmidecode -t baseboard | grep -B 1 Product Manufacturer: Colorful Technology And Development Co.,LTD Product Name: BATTLE-AX B450M-HD CPU info: sudo dmidecode -t processor | grep Version Version: AMD Athlon 3000G with Radeon Vega Graphics OS info: uname -a Linux debian 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64 GNU/Linux I have enabled SVM in the bios setting,and set GRUB_CMDLINE_LINUX according to: https://unix.stackexchange.com/questions/263901/kfd-error-getting-iommu-info grep 'GRUB_CMDLINE_LINUX' /etc/default/grub GRUB_CMDLINE_LINUX="quiet amd_iommu=on iommu=pt" Reboot,check booting info on iommu: sudo dmesg | grep "error" [ 4.056168] kfd kfd: error getting iommu info. is the iommu enabled? [ 4.056221] kfd kfd: device 1002:15d8 NOT added due to errors Maybe we can't enable it on AMD CPU because it is not a bug? sudo dmesg | grep -e DMAR -e IOMMU [ 1.191400] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug. I have enabled the SVM mode in my motherboard's BIOS,and sudo update-grub,the iommu issue remains after reboot.

I am trying to figure out how to disable bounce buffers used in IOMMU when the hardware IOMMU is used. To give more context, when IOMMU_DEFAULT_DMA_STRICT is set in the kernel it enables strict IOTLB invalidations on page unmap. Also, it uses an "additional layer of bounce-buffering". Reference: config IOMMU_DEFAULT_DMA_STRICT bool "Translated - Strict" help Trusted devices use translation to restrict their access to only DMA-mapped pages, with strict TLB invalidation on unmap. Equivalent to passing "iommu.passthrough=0 iommu.strict=1" on the command line. Untrusted devices always use this mode, with an additional layer of bounce-buffering such that they cannot gain access to any unrelated data within a mapped page. According to the description, this feature enables both strict IOTLB invalidations and bounce-buffer for untrusted PCI devices. My current understanding is this config option still uses hardware IOMMU and bounce-buffers together (please correct me if I am wrong). I want a way to enable/disable only the bounce buffers in IOMMU to find the performance overhead involved. In other words, I want to find the overhead of this "additional layer". Please let me know if there is a way to enable/disable only the SW bounce buffers when hardware IOMMU is used. **What I tried so far:** I noticed there are kernel command line options such as iommu=soft and swiotlb={force | noforce}. iommu=soft seems to be a replacement for the hardware IOMMU using software bounce-buffering (reference https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt) which is not what I want. I want to enable/disable bounce-buffering when it's used as an "additional layer" for the hardware IOMMU. swiotlb=force seems to be what I want as it forces all the IO operations through SW bounce buffers. However, it does not specify whether hardware IOMMU is still used or not. It would be great if someone can confirm this. If this is the case, to enable hardware IOMMU without SW bounce buffers I will use below kernel cmdline parameters: intel_iommu=on iommu=force To enable SW bounce buffers with HW IOMMU: intel_iommu=on iommu=force swiotlb=force

In kernel boot options (https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt) I noticed the following option: iommu=[][,noagp][,off][,force][,noforce] [,memaper[=]][,merge][,fullflush][,nomerge] [,noaperture][,calgary] General iommu options: off Don't initialize and use any kind of IOMMU. noforce Don't force hardware IOMMU usage when it is not needed. (default). force Force the use of the hardware IOMMU even when it is not actually needed (e.g. because < 3 GB memory). soft Use software bounce buffering (SWIOTLB) (default for Intel machines). This can be used to prevent the usage of an available hardware IOMMU. It mentions that iommu=soft is the default option for Intel machines and also mentions that it can be used to prevent the use of a hardware IOMMU. So if this option is the default for Intel machines, do Intel machines not use hardware IOMMU or do they use hardware IOMMU + software bounce buffering by default? Also, is there a way to see which IOMMU configuration is being used in the kernel after bootup?

I have a working Qemu configuration with GPU-pass-through. Now I would like to pass also a SSD to the quest. I have read that the pass-through-devices should be in the same group. The following script: ~~~bash #!/usr/bin/env bash shopt -s nullglob for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:" for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})" done done ~~~ Shows that my GPU is in group 8, but the SSD is in group 9. ~~~none IOMMU Group 8: 01:00.0 VGA compatible controller : NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1) 01:00.1 Audio device : NVIDIA Corporation Device [10de:10fa] (rev a1) IOMMU Group 9: 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller [1022:43ee] 02:00.1 SATA controller : Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller [1022:43eb] 02:00.2 PCI bridge : Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port [1022:43e9] 03:00.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:04.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:08.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 03:09.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] Device [1022:43ea] 05:00.0 Non-Volatile memory controller : Samsung Electronics Co Ltd NVMe SSD Controller 980 [144d:a809] 07:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) ~~~ How to move the SSD into group 8? I used already driverctl set-override 0000:05:00.0 vfio-pci to change the driver from nvme to vfio-pci, but this does not change the IOMMU group.

I am trying to see how these two set of commands differ if I just have one nvme device. Block device

ls -l /sys/block/nvme0n1
lrwxrwxrwx 1 root root 0 Apr 28 11:59 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.1/0000:09:00.0/nvme/nvme0/nvme0n1

sudo sh -c "echo 1 > /sys/block/nvme0n1/device/device/remove"
# LINK DISABLE/DOWN
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x40:0x40
# LINK ENABLE
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x00:0x40 
sudo sh -c "echo 1 > /sys/bus/pci/rescan"

If I run sudo nvme list after the above, the device is present.

sudo modprobe -r nvme
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x40:0x40
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x00:0x40 
sudo modprobe nvme

If I run sudo nvme list after the above, the device is missing. However, if I remove the device from the PCI bus and rescan, it is able to come back up. There are no issues with either remove method when running without the link down commands. Does link down and up require mmu space to be reset? According to dmesg echo 1 > .. /remove; echo 1 > .. ./rescan commands do reset mmu and BAR

[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: Removing from iommu group 23
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: [1d82:0401] type 00 class 0x010802
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x10: [mem 0xea1e0000-0xea1e3fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x18: [mem 0xea1e4000-0xea1e7fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x20: [mem 0xd0000000-0xdfffffff 64bit pref]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: supports D1
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: PME# supported from D0 D1 D3hot
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1d8: [mem 0xea1e8000-0xea1ebfff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR0 space: [mem 0xea1e8000-0xea1fffff 64bit] (contains BAR0 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1e0: [mem 0xea180000-0xea18ffff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR2 space: [mem 0xea180000-0xea1dffff 64bit] (contains BAR2 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1ec: [mem 0xea000000-0xea03ffff]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR5 space: [mem 0xea000000-0xea17ffff] (contains BAR5 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: Adding to iommu group 23
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 4: assigned [mem 0xd0000000-0xdfffffff 64bit pref]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 12: assigned [mem 0xea000000-0xea17ffff]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 9: assigned [mem 0xea180000-0xea1dffff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 0: assigned [mem 0xea1e0000-0xea1e3fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 2: assigned [mem 0xea1e4000-0xea1e7fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 7: assigned [mem 0xea1e8000-0xea1fffff 64bit]
[Fri Apr 28 12:11:04 2023] nvme nvme0: pci function 0000:09:00.0
[Fri Apr 28 12:11:04 2023] nvme nvme0: 15/0/0 default/read/poll queues

but not modprobe

[Fri Apr 28 12:23:31 2023] nvme nvme0: pci function 0000:09:00.0
[Fri Apr 28 12:23:31 2023] nvme 0000:09:00.0: enabling device (0000 -> 0002)
[Fri Apr 28 12:23:31 2023] nvme nvme0: 15/0/0 default/read/poll queues
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 264 op 0x0:(READ) flags 0x80700 phys_seg 30 prio class 0
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 520 op 0x0:(READ) flags 0x80700 phys_seg 39 prio class 0
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 16777215096 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 0

I am trying to understand how IO devices are mapped into the 'regular' memory address space on modern x86 machines running Linux. Some details which I am trying to make sense of are: 1. cat /proc/iomem prints out a list of io memory mapped regions (printing the **physical** addresses) which are non-contiguous 2. These regions can be requested by kernel modules dynamically at runtime, and allocated via the function request_mem_region which is defined in 3. x86 machines use mov for both memory-access and IO (that is mapped into memory) So now, supposing kernel module code is running, we will likely encounter an instruction like mov [value] [virtual address] where the virtual address could be referring to either an mmio region or 'normal' data values that exist in memory. The process to separate mmio traffic from 'normal' memory ought to have 2 key steps: 1. determine **if** the address is mmio. The page table has a flag for whether it is memory-mapped, so I assume the mmu determines this while doing page table translation. 2. Determine the 'IO address' of the newly produced physical address (that the mmu gave as output from the page table translation) and pass this to whatever chip interfaces with the IO (Northbridge, root complex, etc) **Question 1**: is my understanding of step-1 above correct? **Question 2**: How is step-2 carried out? (by what **entity** and how is the map stored?) The ranges that need to be checked are listed in /proc/iomem, and the data which it draws from I guess is a map that looks like: map[mmio_address] = io_address_object Keeping in mind that all of this is happening **within** the context of a single mov instruction from the perspective of the cpu, I can't see how this translation could happen via anything other than hardware external to the cpu.

On my computer, I have 2 discrete GPUs. I've been using VFIO to pass the second GPU to a Windows VM to work with some programs. Now I want to pass my more powerful first GPU to the Windows VM in order to play some games.(I can create a second Windows VM if necessary, this is not a concern). I've checked with a [script](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid) if the first GPU has its own IOMMU group, and it has. The problem is, Gnome DE and gdm3 starts without an issue when I boot normally or pass the second GPU, but it doesn't when I pass the first GPU. The monitors do turn on/off depending on the passed GPU. I've tried to restart gdm3, kill gnome-shell, reboot, but it doesn't seem to have any effect getting to the DE. GDM3 started somehow, but it just got the password and returned to the user selection menu. The output of the script: (excluding non-necessary stuff, just the 2 GPUs)

IOMMU Group 12:
	00:15.0 PCI bridge : Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
	05:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
	05:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
IOMMU Group 14:
	01:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] [1002:67ef] (rev cf)
	01:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]

journalctl -u gdm when it does start (No passthrough):

Feb 11 17:29:53 Alienus-PC systemd: Starting GNOME Display Manager...
Feb 11 17:29:54 Alienus-PC systemd: Started GNOME Display Manager.
Feb 11 17:29:57 Alienus-PC gdm-autologin]: gkr-pam: no password is available for user
Feb 11 17:30:00 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user alienus by (uid=0)

journalctl -u gdm when it does not start (First GPU passthough):

Feb 11 17:25:58 Alienus-PC systemd: Starting GNOME Display Manager...
Feb 11 17:25:58 Alienus-PC systemd: Started GNOME Display Manager.
Feb 11 17:25:58 Alienus-PC gdm-autologin]: gkr-pam: no password is available for user
Feb 11 17:25:58 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user alienus by (uid=0)
Feb 11 17:25:58 Alienus-PC gdm-autologin]: gkr-pam: couldn't unlock the login keyring.
Feb 11 17:25:59 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session closed for user alienus
Feb 11 17:25:59 Alienus-PC gdm3: GdmDisplay: Session never registered, failing
Feb 11 17:25:59 Alienus-PC gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session opened for user gdm by (uid=0)
Feb 11 17:25:59 Alienus-PC gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session closed for user gdm
Feb 11 17:25:59 Alienus-PC gdm3: Child process -2688 was already dead.

journalctl -u gdm with debug enabled (/etc/gdm3/custom.conf, First GPU passthrough): https://paste.ubuntu.com/p/cSsDpBynyM/ (the output is about 52k characters, I can't post it here) Specs of the system: - Ubuntu 20.04.5 - Kernel 5.15.0-60-generic - [Gigabyte GA-970-D3 Motherboard](https://www.gigabyte.com/Motherboard/GA-970A-D3-rev-10-11#ov) - CPU [AMD FX 6100](https://www.amd.com/en/products/cpu/fx-6100) - First GPU [AMD RX 460 4G](https://www.amd.com/en/products/graphics/radeon-rx-460) - Second GPU [AMD Radeon 5450](https://www.amd.com/en/support/graphics/amd-radeon-hd/ati-radeon-hd-5000-series/ati-radeon-hd-5450)

I have a centos 7.2 linux installed. I want to check if IOMMU is enabled in the system or not. How can i figure it out? Is there any /proc or /sys entry i can check to make sure is it enabled or not?

Can a Linux swap partition be too big? I'm pretty certain the answer is, "no" but I haven't found any resources on-point, so thought I'd ask. In contrast, the main Windows swap file, pagefile.sys, can be too large. A commonly cited cap is 3x installed RAM, else the system may have trouble functioning. The distinction seems to lie in the fact that Linux virtual memory is highly configurable with kernel parameters, not to mention compile options, whereas Windows virtual memory is barely so. Windows virtual memory management consequently seems to rely on algorithms that are immutable or seem to rely on swap file size and how it is configured. Linux has its own virtual memory management algorithms, of course, but the question is whether and how they are affected by the size of the specified swap partition or file. This comes up because I have a system with 16GB physical RAM configured with a series of 64GB partitions to facilitate a multi-boot capability. For convenience / laziness, I've simply designated one of these 64GB partitions as swap, *i.e.*, 4x physical RAM in contrast to Windows' 3x cap (the latter being relevant only as a frame of reference because this is a Linux-only system). I'm debugging some issues around memory management and VMware Workstation and have come to wonder what, if any, effect the swap partition's size has on compaction, swappiness, page faults, and performance generally. Many thanks for any constructive input.

First of all I'm asking for an old PowerMac G5 Quad which ultimately is able to boot with - 2x1GB DDR2 - 2x4GB DDR2 - 2x4GB DDR2 - 2x4GB DDR2 Problem is it utilizes only the first 2 gigs. The other 3 pairs are identified as Unknown or something like that (in Apple Diagnostics), Anyway I'm asking about if it's possible with Linux kernel boot arg (or some other command) to force a specific amount of physical memory to be used or not. I tried the obvious mem=26G but ultimately free -h still only shows 1.9G. I saw in /dev/iommu that the maximum address is 0x7FFFFFFF (~2.1 G). So is it possible to somehow hack this - to show all my ram? *I'm running Gentoo Linux as a side note - and currently don't have access to the machine but I'm still wondering*

Is it worth installing VirtualBox with a Linux based distributed OS like Kali or Qubes if your hardware does not support iommu? Is there any benefit in regards to security if isolation is limited?

I'm trying to set up pci/gpu passthrough on void. Specifically, using the vfio-pci driver to "reserve" one of the GPUs. I have been following the procedure outlined in [this void-docs PR](https://github.com/void-linux/void-docs/pull/217) (and the arch wiki where relevant) I have an RX 580 and an RX 570, and my groups allow for isolating the former:

IOMMU Group 0 - 11:
    dummy host bridges and such
IOMMU Group 12:
	03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
	03:00.1 SATA controller : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	03:00.2 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	20:00.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:01.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:04.0 PCI bridge : Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	22:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	25:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 13:
	26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
	26:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 14 - 19:
	dummy functions and audio devices

So, lumped in with a bunch of CPU-things, the RX570:

IOMMU group 12: 25:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]

and completely separate, in its own group, the RX580:

IOMMU group 13: 26:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df]

Now as you can see these two share hardware IDs, and therefore I can't just specify a kernel param to push vfio-pci driver onto one of them, it would catch both of them. (as it would be described in the guide in the above pull-request) I tried separating them [following the guidelines on the arch-wiki](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs) , added the "Passthrough selected GPU" script to my dracut config: /usr/local/bin/vfio-pci-override.sh

#!/bin/sh

DEVS="0000:26:00.0 0000:26:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

modprobe -i vfio-pci

/etc/dracut.conf.d/10-vfio.conf

hostonly=" yes "
hostonly_cmdline=" amd_iommu=on iommu=pt "
force_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd "
install_items+=" /usr/local/bin/vfio-pci-override.sh "

which should have an effect, if I'm not mistaken, if I run xbps-reconfigure --force linuxX.Y but for some reason, the amdgpu driver still catches both GPUs. I have the following kernel parameters set in the /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 amd_iommu=on iommu=pt rd.driver.pre=vfio_pci"

Any help is greatly appreciated, and will be forwarded to the aforementioned PR to extend on the guide. (Though I know the void handbook isn't meant to describe things like this in detail, but maybe it'll be useful for the creation of an independent guide on this)

I bought AMD RyZEN 5 2400G, Gigabyte B450M D3SH and 16GB RAM. Some elder GPU AMD Radeon HD 6XXX I took for passing through. But I was unpleasantly surprised with the IOMMU grouping. I put the GPU in either first (16/16) or third (16/8) PCIe slot, and found that it was in both cases included in the iommu group zero. But other members of this iommu group cannot be passed though to VM neither excluded from host system. There were just four iommu groups (0 - 3), while in the tutorials and my other PCs I see twelve or more groups, and some PCIe slots are quite isolated in a single group. Questions: Does anybody knows: - If I should buy some other hardware? - If there are some kernel options to get more iommu groups (split)? - If do I have to use legacy BIOS instead of uEFI? SVM enabled, IOMMU enabled. OS: Debian Linux testitng, kernel 5.10, kernel cmd options iommu=on amd_iommu=on iommu group zero: 01:00.0 USB [0c03]: [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01) 01:00.1 SATA : [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01) 01:00.2 PCIbridge : [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01) 02:00.0 PCIbridge : [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) 04:00.0 Ethernet : Realtek Co. Ltd. RTL8111/8168/8411 PCIe Gigabit Eth [10ec:8168] (rev 0c) 05:00.0 VGA : [AMD/ATI] Turks PRO [Radeon HD 6570/7570/8550] [1002:6759] 05:00.1 Audio dev : [AMD/ATI] Turks HDMI Audio [Radeon HD 6500/6600/6700M Series] [1002:aa90] 06:00.0 NVMe : Intel NVMe Optane Memory [8086:2522] group 1 ... 08:00.0 VGA : [AMD/ATI] Raven Ridge [Radeon Vega (Mobile) Series] [1002:15dd] (rev c6) 08:00.1 Audio dev : [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de] ... As you can see, the APU (GPU integrated on the CPU chip) is a member of the group 1.

I have to follow a hardening procedure for a linux os, in this procedure it tels us that the kernel option must have iommu=force. What does it do ? What could be the difference without specifying anything about iommu? Is it compatible with the pci pass-through where I have set intel_iommu=on?

I have been trying to dual boot windows and Linux. I am facing similar problems for both Ubuntu and Mint. The attached image shows the errors that i am getting

I'm using Debian 10.8 with Linux Kernel 4.19.0-14. I'm trying to use IOMMU to passthrough an entire USB controller (PCI Express card), but I'm having problems. The kernel is loading the ohci_pci and ehci_pci drivers and I can't seem to prevent it or unbind them. I've tried to remove them (echo 1 > /sys/bus/pci/devices/...) but it freezes the system. (the kernel shows a kernel BUG with a stacktrace, etc) Therefore, I'm trying to stop the kernel from binding the ohci and ehci drivers so that VFIO can use them without unbinding them first. I've tried several things: * Adding pci-stub.ids vfio-pci.ids to the GRUB kernel command line. * Aliasing the PCI devices to dummy drivers (/etc/modprobe.d/vfio-pci.conf) * Blacklisting ohci_pci and ehci_pci (/etc/modprobe.d/blacklist.conf) * Unbind the drivers using /sys/bus/pci/devices/.../unbind (but this freezes system) I think the kernel is using built-in ohci/ehci drivers (instead of modules) so perhaps it's loading these drivers before pci-stub or vfio can prevent it?

I am using an amd laptop. - CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8) @ 2.100GHz - GPU: AMD ATI 03:00.0 Picasso While starting void I get this message on top AMD-Vi: Unable to read/write to IOMMU perf counter I tried some other solutions on stackoverflow/reddit where you add something to grub config and then it gets fixed but for my specific graphics It isn't working Please Help.
I have tried adding GRUB_CMDLINE_LINUX="iommu=soft" and also tried adding iommu=pt to the end of the GRUB_CMDLINE_LINUX_DEFAULT= line in /etc/default/grub;

I've got new laptop at work (Lenovo A485) and there are few issues with it. It prints AMD-Vi: IOAPIC not in IVRS table and kernel panic after that. So far I've figured few ways to get the system up and running. * noapic - terrible performance and high temperature, so not really a good way to do it * amd_iommu=off - not ideal either * ivrs_ioapic=00:14.0 ivrs_ioapic=00:00.2 - this seems to work fine * iommu=soft My questions are about iommu=soft. I'm not sure what exactly it does. What are the implications of this mode? What is preferable, overriding the IVRS table or iommu=soft?

Reserving all the PCIe slot for RX580 prevents my PC to boot, due to such errors AMD-VI IOTLB_INV_TIMEOUT, AER: Corrected error received or kernel panic. Adding the linux parameters with iommu=soft and pci=noaer solves the boot issue. On Lubuntu and Ubuntu 20.04 I see these logs from the kernel drm:

00:58:47 lubu kernel: [drm] PCIE GART of 256M enabled (table at 0x000000F400000000).
00:58:47 lubu kernel: [drm] UVD and UVD ENC initialized successfully.
00:58:47 lubu kernel: [drm] VCE initialized successfully.
00:58:47 lubu kernel: [drm] Cannot find any crtc or sizes

Additionally, on Ubuntu 20.04, the gdm3 floods my journalctl with such messages:

14:02:36 ub20 /usr/lib/gdm3/gdm-x-session: (II) AMDGPU(0): EDID vendor "GSM", prod id 19311
14:02:36 ub20 /usr/lib/gdm3/gdm-x-session: (II) AMDGPU(0): DDCModeFromDetailedTiming: 720x480 Warning: We only handle separate sync.
14:02:36 ub20 /usr/lib/gdm3/gdm-x-session: (II) AMDGPU(0): Using EDID range info for horizontal sync
14:02:36 ub20 /usr/lib/gdm3/gdm-x-session: (II) AMDGPU(0): Using EDID range info for vertical refresh
14:02:36 ub20 /usr/lib/gdm3/gdm-x-session: (II) AMDGPU(0): Printing DDC gathered Modelines:

However, if I use the GPU extensively, I experience a random system freeze, in both distributions. If I check the journalctl mostly, I see such error logs:

00:59:06 lubu kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=175, emitted seq=177
00:59:06 lubu kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
00:59:06 lubu kernel: amdgpu 0000:07:00.0: amdgpu: GPU reset begin!
30 00:59:06 lubu kernel: amdgpu: [powerplay] 
                              last message was failed ret is 65535

or

kernel: [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out!
kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=4226, emitted seq=4228

Some Linux user reported that iommu=pt solves such a problem, see https://bbs.archlinux.org/viewtopic.php?id=250297 . I've been confused since several months with this issue, I would like to learn and understand deeply, what is going on here. Thus, I've read the Linux kernel documentation reagarding the iIOMMU tuning in https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html for the possible parameter and the meaning of the parameter in https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt . Since I'm not an IOMMU expert and it is difficult to me to understand the terms, such as GART, re/unmapping, bounch buffering (SWIOTLB) and the interaction between amdgpu and the Linux system itself. I can read: iommu=soft means that Linux uses software bounce buffering (SWIOTLB), and the default value noforce prevents the OS boot on my PC. noforce means that the hardware IOMMU is not forced to use. Then, there is another parameter, called amd_iommu which is of course significant for my hardware setup. This option has three possible values to be assigned: fullflush, off, force_isolation. Unfortunately, I don't know what is the default value of this option. My short question would be: what is the best combination of the parameters iommu and amd_iommu for my hardware to have a fully utilization of my RX580s on Ryzen5? My extra questions: 1. is amd_iommu complements the iommu? or does amd_iommu refer to the AMD Ryzen CPU or the AMD GPU hardware? If there is also intel_iommu, I think amd_iommu refers to the Ryzen chipset. 2. if iommu is not forced to use hardware IOMMU, what is the impact of amd_iommu = off? I will be very grateful to read more deep explanation and a direct link to the source code. Thanks!