Can't use modprobe -r in ExecStop systemd service

I'm trying to make a service to load/unload a kernel module (wifi module to be specific) this is my service file:

[Unit]
Description=Wifi control
After=network.target

[Service]
Type=simple
RemainAfterExit=no

ExecStart=/sbin/modprobe brcmfmac
ExecStop=/sbin/modprobe -r brcmfmac

I have added blacklist brcmfmac to /etc/modprobe.d/brcmfmac.conf enabling/starting/disabling works perfectly But when I run systemctl stop myservice the ExecStop command will fail saying that "Module brcmfmac is in use." But when I run this command manually it works perfectly fine, I have tried a bunch of different options, running it through su or bash -c but nothing changes the outcome. I can just make it a oneshot service and instead of systemctl stop, run modprobe -r. But I like knowing why the behaviour is different here. Does anyone know why this is happening?