In Bash Vi mode, I get thrown into Insert mode automatically after one key press

I am using the bash shell with the XFCE4 terminal. I have configured the shell so that it is in Vi mode. I have done this by adding the following line in my ~/.bashrc file: set -o vi When in a shell in Normal Mode, the first character I press gets inputed in Normal Mode but then I get thrown in Insert Mode. ie: I am in Normal Mode. If I press b, I will go back one word and get dropped in Insert Mode. If I press bb, I will have got back one word, enter Insert Mode and then insert the b character. Another example, if I press dd, the first d will be caught by Normal Mode, I will then go to Insert Mode where the second d will be inserted. If I manually enter set -o vi in the shell (whether it is in my .bashrc or not) I will not get this behaviour and instead get the normal behaviour. I know the change in my .bashrc is getting read as I am not in Emacs mode. I am using bash 4.2.45(1)-release. Here is the full content of my .bashrc: PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting PATH=$PATH:/home/bleakcabal/.rvm/gems/ruby-2.0.0-p247/bin set -o vi