GPG fails to decrypt file with Yubikey private key: No secret key

I'm trying to decrypt a file using GPG. The private key is stored on my Yubikey, but I get the following message from GPG:

shell> gpg --output test-temp --decrypt git-token.gpg 
gpg: encrypted with 255-bit ECDH key, ID 38033A6C1F5941E8, created 2022-04-22
      "User Name "
gpg: decryption failed: No secret key

However when checking the GPG card it can see the private key just fine:

General key info..: 
pub  ed25519/F5BA3C4BA7D63D15 2022-04-22 User Name 
sec>  ed25519/F5BA3C4BA7D63D15  created: 2022-04-22  expires: never     
                                card-no: [REDACTED]
ssb#  ed25519/A31508BC36769673  created: 2022-04-22  expires: never     
ssb#  cv25519/38033A6C1F5941E8  created: 2022-04-22  expires: never

As you see the key ID's match. The private/public keypair is also properly present in my keychain:

shell> gpg -k
pub   ed25519 2022-04-22 [SC]
      0D9E4996BF56ED20DC3162BEF5BA3C4BA7D63D15
uid           [ultimate] User Name 
sub   ed25519 2022-04-22 [A]
sub   cv25519 2022-04-22 [E]

shell> gpg -K
/home/user/.gnupg/pubring.kbx
-------------------------------
sec>  ed25519 2022-04-22 [SC]
      0D9E4996BF56ED20DC3162BEF5BA3C4BA7D63D15
      Card serial no. = [REDACTED]
uid           [ultimate] User Name 
ssb#  ed25519 2022-04-22 [A]
ssb#  cv25519 2022-04-22 [E]

The file was encrypted using

--encrypt --armor --output git-token.gpg --recipient user@username.com git-token

What gives? My setup works fine with signing (in Git for example).