Disable KVM Kernel Module while its still in use

In order to run a virtual machine, VirtualBox tells me to disable the KVM kernel module. I googled around and found out that KVM is included in the qemu-*-packages, but none of them is/was installed on my system (Debian Wheezy). After that I simply tried to unload the module, but this doesn't want to work because it is in use: # modprobe -r kvm_intel FATAL: Module kvm_intel is in use. # modprobe -r kvm FATAL: Module kvm is in use. I don't know what program is using it and I have really no idea what that could be. I also tried with the -f option but I get the same result. My question is: How can I remove or disable the KVM Kernel module? ***Update:*** Here is the output of the lsmod command: # lsmod | grep kvm kvm_intel 138825 3 kvm 404853 1 kvm_intel