Delete all files without user permissions

I have a prompt that asks me to delete all the files in a directory that the owner (u) can't r, w, nor x, in one command. I tried this command:

find data -type f ! -perm -u=rwx -exec rm -f {} \;

... but I think it removes too many files.