When is it safe to disconnect my external HDD?

In Windows, there's an option to "safely remove hardware". Ignoring the fact that it almost always fails, and that most of the time I simply have to plug the device out, I get the initiative behind it (not completely though). So, when using Debian, when is it "safe" (i.e. not damaging to any party) to remove my external HDD? After it has been unmounted (Xfce seems to use the term "ejected" but I guess that's the same thing)? Or simply when there are no programs using the disk?