Laggy dialer app

I have an Xperia P with latest update (Build No. 6.2.A.1.100, Android 4.1.2). It's phone app becomes horribly slow as the size of call log increases. Also, whenever there's an incoming call, it usually takes around 4 to 5 secs to respond after the call is accepted. There is a great time lag during dialing too. I tried clearing app data from settings but of no use! The problem goes away when I clear the call log. Also, the scroll lag disappears while operating in landscape mode. Does anyone know of any workaround for this? Currently, I'm using Ex-Dialer (the best dialer app I've ever seen from performance perspective) as a replacement for stock dialer (I'm not rooted and stock dialer is still installed in my phone) but I really don't want to use it or any third party app. One of the reason is that the Ex-Dialer contact list doesn't show Facebook thumbnail images for contacts. Please help!