Can I use GPU through the M.2 interface (support PCIe) together with GPU installed in normal PCIe x16 slot in Linux?

I have a Nvidia GPU workstation for multiple users. I use KVM to create 4 VMs and use GPU in VMs in passthrough mode. However, I find when I pass through all 4 GPUs to these VMs. The host system (Ubuntu server LTS 20.04.02) always freeze. So I guess I have to keep at least one GPU on the host (The CPU is TR3960x, does not have integrated GPU). There is no pcie x16 slot left on the motherboard, so I just install a low-end AMD gpu on a M.2 slot with a M.2 to pcie x16 adapter. But after this, I cannot boot into the host system, with error

[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)]

I also tried to boot from a USB drive, but got

grub error: cannot allocate kernel buffer, you need to load kernel first.