manage many ssh keys via ssh-agent

I use keepassxc (my password manager) to manage my ssh keys. This means the keys are stored within the database (not stored on disk in a traditional way). When I unlock my password database, all keys are added to the agent. However, with a growing list of ssh keys, I get the problem that ssh tries all the keys in the agent, which results in Too many authentication failures if the remote host allows less authentication tries than i have ssh keys (and the particulary ssh key is far back in the list). Is there a way to tell ssh: "for this host, use the key with fingerprint xyz, fetch it from the agent"? Alternatively pubkey xyz, but I can't use the IdentityFile option (and IdentitiesOnly yes), since I don't want to store my private keys on disk.