Ignore history when using zsh

I set the HISTORY_IGNORE variable in ~/.zshrc and it correctly prevents those commands from being written to ~/.zsh_history. But what I really want is for those entries not to show when pressing up and down. (Same as HISTIGNORE when using bash). How do I accomplish that?