Unloading KMS driver / Replacing NVIDIA Linux drivers without rebooting

In the past you could trivially replace NVIDIA proprietary drivers on the fly after switching to the text console, killing X.org and unloading (rmmod) the appropriate NVIDIA modules, and installing new drivers. However nowadays NVIDIA recommends to run the driver with KMS support, options nvidia-drm modeset=1 and that makes it impossible to unload kernel modules ("The device is busy"). The Linux kernel allows to unbind the graphical driver from the text console by running this command:

echo 0 > /sys/class/vtconsole/vtcon1/bind

However this command results in all the text consoles being completely dead. They just freeze, the system keeps running. Looks like after this command, one needs to run another command to make the kernel re-enable built-in drivers which drove text consoles prior to NVIDIA drivers, only the Internet has no information on that. This question is **not** limited to NVIDIA drivers. Would be nice to know how to unload an open source KMS driver, be it Intel, AMD or Nouveau. Here's what I see on boot:

Console: colour dummy device 80x25
printk: console [tty0] enabled
fbcon: Deferring console take-over
fbcon: Taking over console
Console: switching to colour frame buffer device 128x48

Here's what I have in my .config:

CONFIG_SYSFB=y
CONFIG_SYSFB_SIMPLEFB=y
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_FBDEV_OVERALLOC=200
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB=y
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_SIMPLE=y

Here's what I have in /sys:

$ find /sys -iname '*fb*'
/sys/class/graphics/fb0
/sys/class/graphics/fbcon
/sys/devices/platform/simple-framebuffer.0/graphics/fb0
/sys/devices/virtual/graphics/fbcon
/sys/module/drm_kms_helper/parameters/drm_fbdev_overalloc
/sys/module/drm_kms_helper/parameters/fbdev_emulation
/sys/module/fb
/sys/module/fb/parameters/lockless_register_fb