Yubikey PIV not working without OTP

I just started using a Yubikey to do SSH logins by following this guide. It works fine - but only as long as the Yubikey is able to do OTP. After a few accidental touches on the key spewing a random string into places I don't want it I figured I could just disable OTP as I'm only using PIV. Cert/key is there and working:

$ ykman --version
YubiKey Manager (ykman) version: 5.1.0
$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)
$ ykman piv info
PIV version:              5.4.3
PIN tries remaining:      3/3
Management key algorithm: TDES
CHUID: No data available
CCC:   No data available
Slot 9A (AUTHENTICATION):
  Algorithm:   ECCP256
  Subject DN:  CN=SSH key
  Issuer DN:   CN=SSH key
  Serial:      14009452700000000000
  Fingerprint: 1fa375971a89c6f82f3b73218f717cb1d031fbd61c94965qqqqqqqqqqqqqqqqq
  Not before:  2023-03-10T10:02:12
  Not after:   2024-03-09T10:02:12

I can disable OTP:

$ ykman config usb --list
OTP
FIDO U2F
FIDO2
OATH
PIV
OpenPGP

$ ykman config usb --disable OTP
USB configuration changes:
  Disable OTP
  The YubiKey will reboot
Proceed? [y/N]: y

$ ykman config usb --list
FIDO U2F
FIDO2
OATH
PIV
OpenPGP

But after doing that, SSH no longer works, and I'm prompted for a password instead of the expected key-based login I can enable OTP again:

$ ykman config usb --enable OTP
USB configuration changes:
  Enable OTP
  The YubiKey will reboot
Proceed? [y/N]: y

Which restores SSH functionality. Why must OTP be enabled for PIV to work?