Use specific SSH keys for specific servers only

When using SSH key authentication, all available keys are tried against the server. Is there a possibility to set some SSH keys in a way, so that they are only disclosed to a given remote server list, for privacy? I've been looking but wasn't able to find any information or documentation regarding this at all. Thanks in advance