Find hex value in files/folder

Is there a way to search for a hex value in files / folders? I've tried the following: grep -P "[\x01\x02\x03]" but that doesn't work (there is no error, but it doesn't search for these hex values).