How to configure separate ports and keys for restic usage?

I want to use restic for remote backups, but seeing as SFTP and SSH will both require access to the system, is there some way to configure the SSH access to the server in such a way that the directories accessible by SSH and SFTP are protected from each other? By that I mean connecting to the server using the SSH key for normal user logons (which grant access to a terminal) will not grant access to the directories used for the restic backup, and vice versa, ie connection with SFTP will not grant access to the terminal like the regular logon. The idea is that if one of the keys get compromised the other will not be affected. The only way I can think of is if the HTTP based remote server option is used, which can probably create directories that cannot be directly accessed from the regular user login.