Echoing "!" inside a string does some weird things

If I type in this:

echo "Hello, World!"

I don't know the name of it, but it prompts me for the next line. You know the PS2 thing. Or if you type echo \ and press Enter. Why? Well I know that ! is a **Special Variable** that you can use to reference your history. But as soon as I use this:

echo "Hello, World"!

I get my desired output. What is happening and why can't you use ! inside ""? --- Thanks for your help :)