rsync to USB flash drive always transferring all data

When rsyncing a directory to a freshly plugged-in external USB flash drive, via rsync -av /source/ /dest/ all files get transferred (i.e. rewritten) despite no changes in the files. Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents. Including the -u update option and explicitly adding the -t option did not change anything. The mount point remains the same (i.e. /media/user/, the drive is automouted by xfce, the /dev/sdxy obviously changes) The hard drive source is ext4, while the USB is vfat with utf8 character encoding. What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.