Restrict ssh access to single host per user

I am trying to restrict SSH for a user to a specific host: I would like

to be able to SSH to any host, allowing only pubkey authentication, and only if the connection comes from 192.168.1.17. I've tried the following configs in

/etc/ssh/sshd_config.d/sshd_config.conf

:

Match Group ansible Address !192.168.10.17
    DenyUsers                       ansible
    PubkeyAuthentication            no
    KbdInteractiveAuthentication    no
    PasswordAuthentication          no
    KerberosAuthentication          no
    GSSAPIAuthentication            no
    AllowAgentForwarding            no
    AllowTcpForwarding              no
    X11Forwarding                   no


Match User ansible Address 192.168.10.17
    AuthorizedKeysFile              ~/.ssh/authorized_keys
    PubkeyAuthentication            yes
    KbdInteractiveAuthentication    no
    PasswordAuthentication          no
    KerberosAuthentication          no
    GSSAPIAuthentication            no
    AllowAgentForwarding            no
    AllowTcpForwarding              no
    X11Forwarding                   no

This doesn't work and I'm out of ideas. I did see question "Restrict users to specific ssh" but that didn't really help much. Any ideas are appreciated, thanks!