Funtoo Keychain keeps asking for password when logging in using SSH

I have followed the instructions here https://www.funtoo.org/Keychain and added eval keychain --eval --agents ssh --inherit any id_rsa to - I have tried both but the result was the same - .zshrc and .zprofile. Now the first terminal that I open requires me to enter the password for the SSH key (as expected) but after that, I never need to enter the password, independently of how many terminals I open. However, if I log in to this machine using SSH I need to enter the SSH key password every time I connect. Obviously there is some difference how the shell is setup if I open a new terminal session sitting in front of the machine or open an SSH session. **How do I make SSH logins "inherit" the Funtoo Keychain setup/environment so I don't need to enter the password for the SSH key every time I log in to this machine using SSH?**