Manual removal of Nvidia driver fails due to init and busybox

I'm using PorteuX (Slackware based) and in the shutdown/reboot process I want to unmount everything, which works fine unless I boot with Nvidia drivers. In this case, just before reboot -f step if I run lsof /dev/nvidia* it says both init and busybox are using nvidia-modeset process, which prevents me from being able to kill any Nvidia process. I also tried to boot into init 3 and kill all Nvidia processes, but it says the device is busy, although is not clear why. Is there a way to force the Nvidia driver to unload without rebooting? I don't mind losing video, as this will happen in the last step before shutdown.