GPG agent doesn't remove my SSH key from the keyring

I have a really troubling problem. I can't get gpg-agent to remove my SSH key from its keyring and it even persists there after many reboots. $ ssh-add -D SSH_AGENT_FAILURE Failed to remove all identities. Even when I tell it to remove the identity: $ ssh-add -d /path/to/private/key Identity removed: /path/to/private/key I then look $ ssh-add -l 4096 1b:cb:52:a6:e5:13:e6:78:14:12:92:8f:34:8f:92:88 /path/to/private/key and it's still there. Where is this being cached? It seems to be writing to disk for some reason, which is a scary thing for a SSH agent to do. I'm running the following to start gpg-agent: gpg-agent --enable-ssh-support --daemon Everything else works fine, but it's caching this file somewhere and I need to delete it.