Change when phone beeps for notifications

My phone makes a lot of noise when it's not in use. I think this is because apps running in the background have experienced an event, like Clash of the Clans. How can I change this? I'd really like to find out what apps just made my phone beep. I'm running Cyanogenmod 12.1 with android 5.1.1 (though soon to upgrade to android 6).