SSH Publickey+TOTP and Password+TOTP simultaneously

I am trying to setup an SSH server where some users prefer to use an SSH key but others prefer LDAP provided passwords. I found out that I can set in sshd_config for publickey + PAM or PAM only authentication.

AuthenticationMethods publickey,keyboard-interactive:pam keyboard-interactive:pam

The PAM part works as expected. User enters their LDAP password and they are asked a verification code. However publickey authentication works so that user uses key + then asked to enter LDAP password + then enter verification code. (we do not want key users to also be asked for password) Is it possible to support key users so they do not need to enter their password while 2FA verification code is still asked. Meanwhile people who does not provide a key, still can use passwords for authentication and also asked for 2FA verification code? Note: We want to have a single SSH server process running in the system. We prefer not to use a solution like running multiple SSH servers on different ports. Thanks!