Repeated "from agent: agent refused operation", even after having added private key

I am trying to unlock the gnome keyring on my server machine so that I can push git commits via ssh to github. The ssh key for this is encrypted. What I tried is this:

# See https://unix.stackexchange.com/a/676655 
function unlock-gnome-keyring () {
    read -rsp "Password: " pass
    export $(echo -n "$pass" | gnome-keyring-daemon --replace --unlock)
    unset pass
}

unlock-gnome-keyring
# Enter password...
# Prints
# discover_other_daemon: 0** Message: 20:29:52.540: Replacing daemon, using directory: /run/user/1000/keyring

ssh-add /home/username/.ssh/id_ed25519
# Enter password for keyfile...
# Prints:
# Identity added: /home/username/.ssh/id_ed25519 (comment)

That looks OK so far and I can then push to the repo. However, after some time this does not work anymore. Similarly, when I close the shell and reopen it, I am asked again for the passphrase. This may be expected if the shell that sharted the keyring daemon gets closed? I don't quite know. Either way, I try to unlock the keyring again, as above and get the same output. Then,

$ git pull
sign_and_send_pubkey: signing failed for ED25519 "/home/username/.ssh/id_ed25519" from agent: agent refused operation
git@...: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

This can only be fixed by re-adding the key to the keyring. How can this be fixed?