How to match IPv6 addresses as a host pattern in ssh_config

My VPS listens to SSH on port 222. On my clients I created a ssh_config such that the non-default port needs not to be explicitly provided on the command line every time. (Some daemons which internally use SSH and must connect to my server do not even allow to specify a different port). Matching of the DNS name and IPv4 address works as expected, but IPv6 is not matched. How do I specify IPv6 addresses in ssh_config? My configuration looks like

Host my-server.my-domain.tld x.y.w.z [xxxx:yyyy::zzzz]
    HostName my-server.my-domain.tld
    Port 222

Currently, when some process tries to connect to my server, I see an error that the server refused the connection on port 22 via IPv6 and then the process falla back to IPv4 and successfully connect to the server on port 222. I have already tried to specify the IPv6 adress without the square brackets, but then SSH complains about an syntactically invalid DNS or IPv4 address. I also have tried to move the IPv6 address to its own Host-block without success.