How to permanently swap esc and caps lock in xfce / xubuntu?

I followed some post a while ago, and it said to make an ~/.xmodmap file like so: ! Swap caps lock and escape remove Lock = Caps_Lock keysym Escape = Caps_Lock keysym Caps_Lock = Escape add Lock = Caps_Lock I did, and it now I can swap esc and caps lock by doing xmodmap ~/.xmodmap. I wanted this to be permanent, so I added that command to my ~/.bash_profile, but it doesn't seem to work; I still have to manually xmodmap. As an EEng dropout, I am tempted to just rewire my keyboard }:-) but I know there must be a better way. How can I make this work, permanently?