Is there a tool that can perform direct RSA decryption with a Yubikey?

The use case I'm looking for is that I walk up to a *headless* server and "unlock" it using a hardware key, where scripts on the server recognize that I've plugged it in and automatically use it without a pin or password or additional factors. The most primitive way of implementing this would be to have a USB thumb drive with unencrypted raw AES keys on it which the scripts on the server find and use to decrypt things. The downside is that the USB stick could be copied and there is no way to revoke it if it were lost. It also runs into trouble with possible filesystem corruption of the USB stick itself if the drive were removed when the scripts were still using it. It seems like a smartcard or Yubikey would be the obvious solution to these problems, but it also seems like most people describing Yubikey solutions pair it with gpg as a second factor of auth. I don't want "extra" auth factors, I want the key to be one of multiple possible decryption methods. I don't want to have to configure gpg on each host or have "identities" or expiration dates or trust chains or any of that. The other popular option is to integrate it with LUKS, but I was hoping for a more non-root userland option. I just want to take an encrypted AES key and directly ask the Yubikey to decrypt it with an RSA private key that lives in hardware (without entering a pin or password, but a short touch or long touch on the device is ok). Is there any existing tool that can accomplish this? Scripting language libraries are fine too.