git with ssh keeps asking for passphrase everytime

I'm using a git bare repository for managing dotfiles, and some other repositories for custom DWM build and all. I use SSH for pushing and pulling from git, but it keeps asking me my SSH passphrase every time try to push or pull. i have already tried

eval $(ssh-agent)

then

ssh-add ~/.ssh/id_rsa

and its added successfully, but no luck I am missing something?