How do I set up KDE, KWallet and ssh so I'm not asked to enter my ssh passphrase?

I have ssh keys set up and I can see my key in KWalletManager under ksshaskpass. How do I get it so that I am not asked for my ssh passphrase eg. every time I do something in Git. (It says: "Enter passphrase for key ...") What I would like is that my KDE login is the only time I am asked for a password (as in Cinnamon and Gnome). I guess that this should result in some kind of interaction between KWallet and ssh so that my passphrase is auto-entered into ssh. I've tried searching for information but could not find a step by step solution for Kubuntu. What do I need to do?