How can I prevent USB drivers (EHCI) from loading on specific USB controller (PCIe card)?

I'm using Debian 10.8 with Linux Kernel 4.19.0-14. I'm trying to use IOMMU to passthrough an entire USB controller (PCI Express card), but I'm having problems. The kernel is loading the ohci_pci and ehci_pci drivers and I can't seem to prevent it or unbind them. I've tried to remove them (echo 1 > /sys/bus/pci/devices/...) but it freezes the system. (the kernel shows a kernel BUG with a stacktrace, etc) Therefore, I'm trying to stop the kernel from binding the ohci and ehci drivers so that VFIO can use them without unbinding them first. I've tried several things: * Adding pci-stub.ids vfio-pci.ids to the GRUB kernel command line. * Aliasing the PCI devices to dummy drivers (/etc/modprobe.d/vfio-pci.conf) * Blacklisting ohci_pci and ehci_pci (/etc/modprobe.d/blacklist.conf) * Unbind the drivers using /sys/bus/pci/devices/.../unbind (but this freezes system) I think the kernel is using built-in ohci/ehci drivers (instead of modules) so perhaps it's loading these drivers before pci-stub or vfio can prevent it?