List processes accessing device after `umount --lazy`

I want to remove my external HDD as safely as possible. I want to use umount --lazy: > Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.) Then after a short delay I plan to kill any processes with open files on the device where the filesystem is still quasi-mounted. * I can't use lsof for an accurate list of the open files as the filesystem has become invisible to new processes. * If I use lsof before umount -l, there is a race contition of a new file being opened in between the two invocations. Is there any way of finding out which processes are accessing a DEVICE rather than a filesystem?