Gnome hangs with VFIO gpu passthrough

On my computer, I have 2 discrete GPUs. I've been using VFIO to pass the second GPU to a Windows VM to work with some programs. Now I want to pass my more powerful first GPU to the Windows VM in order to play some games.(I can create a second Windows VM if necessary, this is not a concern). I've checked with a [script](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid) if the first GPU has its own IOMMU group, and it has. The problem is, Gnome DE and gdm3 starts without an issue when I boot normally or pass the second GPU, but it doesn't when I pass the first GPU. The monitors do turn on/off depending on the passed GPU. I've tried to restart gdm3, kill gnome-shell, reboot, but it doesn't seem to have any effect getting to the DE. GDM3 started somehow, but it just got the password and returned to the user selection menu. The output of the script: (excluding non-necessary stuff, just the 2 GPUs)

IOMMU Group 12:
	00:15.0 PCI bridge : Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
	05:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
	05:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
IOMMU Group 14:
	01:00.0 VGA compatible controller : Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] [1002:67ef] (rev cf)
	01:00.1 Audio device : Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]

journalctl -u gdm when it does start (No passthrough):

Feb 11 17:29:53 Alienus-PC systemd: Starting GNOME Display Manager...
Feb 11 17:29:54 Alienus-PC systemd: Started GNOME Display Manager.
Feb 11 17:29:57 Alienus-PC gdm-autologin]: gkr-pam: no password is available for user
Feb 11 17:30:00 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user alienus by (uid=0)

journalctl -u gdm when it does not start (First GPU passthough):

Feb 11 17:25:58 Alienus-PC systemd: Starting GNOME Display Manager...
Feb 11 17:25:58 Alienus-PC systemd: Started GNOME Display Manager.
Feb 11 17:25:58 Alienus-PC gdm-autologin]: gkr-pam: no password is available for user
Feb 11 17:25:58 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user alienus by (uid=0)
Feb 11 17:25:58 Alienus-PC gdm-autologin]: gkr-pam: couldn't unlock the login keyring.
Feb 11 17:25:59 Alienus-PC gdm-autologin]: pam_unix(gdm-autologin:session): session closed for user alienus
Feb 11 17:25:59 Alienus-PC gdm3: GdmDisplay: Session never registered, failing
Feb 11 17:25:59 Alienus-PC gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session opened for user gdm by (uid=0)
Feb 11 17:25:59 Alienus-PC gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session closed for user gdm
Feb 11 17:25:59 Alienus-PC gdm3: Child process -2688 was already dead.

journalctl -u gdm with debug enabled (/etc/gdm3/custom.conf, First GPU passthrough): https://paste.ubuntu.com/p/cSsDpBynyM/ (the output is about 52k characters, I can't post it here) Specs of the system: - Ubuntu 20.04.5 - Kernel 5.15.0-60-generic - [Gigabyte GA-970-D3 Motherboard](https://www.gigabyte.com/Motherboard/GA-970A-D3-rev-10-11#ov) - CPU [AMD FX 6100](https://www.amd.com/en/products/cpu/fx-6100) - First GPU [AMD RX 460 4G](https://www.amd.com/en/products/graphics/radeon-rx-460) - Second GPU [AMD Radeon 5450](https://www.amd.com/en/support/graphics/amd-radeon-hd/ati-radeon-hd-5000-series/ati-radeon-hd-5450)