Backup Pictures folder using unison

I've been very successfully using unison to back up my personal data to a remote machine. The problem is that I cannot seem to be using it to back up my pictures as I always get the error

Error in scanning directory:
Operation not permitted [opendir(/Users/xxxx/Pictures/Photos Library.photoslibrary)]

Clearly system integrity protection is preventing unison from accessing the photo library, either on the local machine or on the remote machine (both are Macs). I have given full disk access to pretty much everything I could imagine: Terminal.app and unison on the local machine, sshd and unison on the remote machine, but without any success. I'm now running out of things to try... Any idea?