Operation not permitted when accessing Downloads folder

When I try to ls the ~/Downloads folder on my mac I get a ls: Downloads: Operation not permitted error. My user does have rwx permissions on the folder. I can look at the folder just fine in finder. I use tmux in iterm2. The issue does not exist when I use terminal.app. It does work if I start tmux in terminal.app, regardless of which terminal emulator I use to control tmux. iTerm.app already has Downloads folder permissions in Security & Privacy settings. What is the reason for this?