Xfce panel buttons don't minimize windows when focus follows mouse is set

I like to have the focus follow my mouse, but I also like to be able to both minimize and restore windows by clicking the window button in the panel. If "click to focus" is set, then I get that behaviour. If I change to "focus follows mouse", then the behaviour of the window buttons is a bit unpredictable. Sometimes if I click to focus one window, then click another, then the second window minimizes while staying in the background. Is this by design, or potentially a bug? I'm using Xubuntu 14.04.2 and haven't modified Xfce in any way.